In stark contrast to only five years ago, much of the prominent new research on scholarship programme outcomes adopts a longitudinal model. Some examples have been posted within this blog series: see, for instance, essays by Anna Seifried and Cate Lawrence or Mirka Martel. A recent request for proposals from the MasterCard Foundation – architects of the Scholars Program – reinforces the point:
The Scholars Program Longitudinal Cohort Study will be a central component of the Program's learning partnership. It will seek to track post-Program Scholar outcomes and to understand the impact of Scholars on their families, communities, organizations, and societies.
(RFP 3, 4th March 2017)
The increased profile of longitudinal research within scholarship evaluation is a testament both to the difficulties presented by previous approaches and to the will of policymakers and administering agencies to grapple with those complexities and strengthen the evidence-base for grant making. Whether conducted within administering agencies or outsourced to expert consultants, contemporaneous research tracking scholarship alumni during and post-scholarship is fast becoming the 'gold standard' for assessing individual-level impacts.
Like others in this field, the Commonwealth Scholarship Commission in the UK (CSC) has now invested considerable time and energy into building an effective longitudinal study of mid-range scholarship outcomes (professional development, organisational change, etc.). Unlike most others in the field, however, the CSC Longitudinal Research Framework is designed to continue perpetually as a component of basic scholarship administration. We introduce the logic and methodology of the framework in our article.
Evaluating Commonwealth Scholarships
The CSC administers the UK Government's contribution to the Commonwealth Scholarship and Fellowship Plan: a programme offering grants for academic study and professional mobility to citizens of Commonwealth countries. Since its inception in 1959, the CSC has funded over 26,000 grants: a combination of full academic scholarships (course fees, subsistence, travel, etc.) at UK universities and short-term placements at UK organisations for visiting professionals. As of 2017, funding for the CSC is primarily from the UK Department for International Development (DfID).
The CSC has an independent policymaking board appointed by parliament, including a steering group for monitoring and evaluation of Commonwealth Scholarships. Many of the analyses completed by the evaluation programme are available on the CSC website.
Until recently, the CSC evaluation programme has conducted cross-sectional tracer surveys to investigate broad issues in career trajectory and social impact. These tracer surveys have been primarily about accountability and setting the groundwork for detailed research. They have had some significant benefits. Firstly, tentative, but interesting conclusions have been drawn about impact within specific regions or fields, geographical mobility, and maintenance of networks between the UK and returned Commonwealth Scholars (see, for instance, our most recent report). Secondly, a precedent has been set for systematically reaching out to alumni to understand their post-scholarship careers. And finally, the surveys helped to highlight major successes and compelling stories of real-life social and economic impacts from Commonwealth Scholars, some of which were turned into individual case studies.
New thinking, new approaches
Nonetheless, this model of research also had several deficits. Many of the respondents to tracer surveys dated back to the early years of the programme and thus several decades had elapsed between their original scholarship and more recent survey response. For these alumni, the 'contribution story' linking their scholarship-funded studies to their subsequent achievements could be difficult to tell. For a similar reason, the surveys were not timely tools for policymaking. The experiences of alumni who held scholarships during the very different global landscape of the 1970s or 1980s are arguably less directly relevant to policymaking in the contemporary context of Commonwealth Scholarships.
Organisational thinking within the CSC about the role of outcomes research has thus developed alongside the groundwork set by the tracer study activities. Increasingly, complex questions about both the most effective investments and the comparability of outcomes across types of scholarship are being considered, with the expectation – rather than the ambition – that outcomes data is available to aid decision making. This shift is, of course, not absolute. Identifying major successes as an accountability mechanism remains important, yet research activity within the CSC increasingly shares this goal with the goals of organisational learning and the evaluation of performance.
To underpin the shift, two changes – one conceptual, one practical – have been made:
- Greater attention to empirical investigation of the normative model of scholarship outcomes – that is, the ideal and expected model of how funding translates into specific outcomes; distinct from identifying exceptional outcomes of major successes which are, by definition, not the norm
- A move from a 'tracing' to a 'tracking' approach: a structured and systematic 'intervention'-style research model that is better able to collect data that can be linked to expected outcomes within the current grant-making cycle
These are principles of the CSC Longitudinal Research Framework, launched in 2016.
Longitudinal research framework
A research exercise can reasonably be described as 'longitudinal' if it collects repeated data points over a protracted period of time. The CSC framework is longitudinal because it collects contemporaneous baseline (pre-scholarship) and follow-up (post-scholarship) data from Commonwealth Scholars. The approach is an intervention and follow-up study design. It is not typical in international scholarship research, but is common within academic research in psychology, education, and many other empirical subjects.
Web surveys are the fundamental technology of the framework, sent to alumni at standardised points within their Commonwealth Scholarship journey. A useful distinction here is between the surveys and the framework. The longitudinal framework is a research approach to tracking scholarship grants; the surveys are a method for collecting the data.
We track academic scholarship programmes alumni for 10 years post-scholarship, by sending invitations to participate in survey data collection every two years (making five post-scholarship surveys per grant). Whereas, alumni of short fellowship programmes are only tracked for 2.5 years and we invite them to participate in three post-fellowship surveys. A comparison group of unsuccessful finalists for a Commonwealth Scholarship are invited to participate in the counterfactual survey, also using a ten year follow-up period and two-year survey intervals.
The longitudinal framework is largely prospective – for tracking future Commonwealth Scholars – but to generate an initial corpus of data we have also applied this follow-up structure retrospectively. For example, in 2016 we surveyed scholarship alumni who completed their CSC funding in 2006 as if they were receiving their +10 year follow-up survey.
Harmonisation and comparability
The standardisation of data collection points within the tracking framework is crucial. Although there will always be some small variation – for example, because of administrative differences between Masters programmes – the +2 year surveys will always be approximately two years after each Scholar has completed their Commonwealth funding. Benchmarking the follow-up 'timetable' in this manner has several benefits:
- It helps to overcome inconsistent follow-up periods encountered in our (and many other agencies') earlier tracing surveys: some respondents to surveys had finished only 1-2 years prior to the survey, whereas others had finished 40-50 years prior! In the longitudinal framework, the length of time between completing funding and participating in the evaluation programme is consistent.
- It helps to ensure that analysis is consistently looking at outcomes of Commonwealth Scholarship funding, rather than of academic degrees generally. A minority of Scholars will take longer to finish their degree than the period for which CSC funds them (for example, taking 5 years to finish a PhD when the CSC fully funds only 3 years) and the evaluation should relate to the funded period specifically.
In addition to harmonising the follow-up period, the longitudinal framework harmonises the measures of scholarship outcomes. The CSC has common programme outcomes that span all individual scholarship and fellowship schemes (see below), although certain outcomes may be more evident in some schemes than in others.
CSC Common Scheme Outcomes:
- Implementation of new skills and content knowledge, and skills and technology transfer in the workplace
- Professional development
- Leadership and the capacity to influence and disseminate knowledge
- Improved teaching quality, capacity and outputs
- Improved research quality, capacity and outputs
- Improved networks, partnerships and international links
Surveys within the longitudinal framework seek to investigate how far these outcomes are being achieved by including the same topics and using the same questions, allowing comparison across data collected with the alumni of different grants (e.g. masters, PhD). This approach is not designed to homogenise the aims of different grants, but to harmonise the ways in which progress toward those aims are measured, allowing better comparison.
Using the data
Complex analysis is possible within the longitudinal framework because it provides multiple levels of comparison. Firstly, individual measures baseline to follow-up, and specific data points in the follow-up period (+2 years; +4 years, etc.), can be examined for a specific alumnus. Secondly, the same comparison – baseline to follow-up – can be examined for entire cohorts: for example, all who started a PhD in 2014; all who finished a taught masters in 2016. Thirdly, the same comparison can be examined between cohorts: for example, all who start a PhD in 2014 vs all who started a PhD in 2015. Fourthly, for academic scholarships, comparison can be made between successful Scholars and the counterfactual group on any or all of the comparisons noted above (individuals, matched individuals, between and within cohorts).
What might be interesting topics for detailed analysis? Clearly the development of individual career trajectories over the follow-up period is the crux of understanding all of the common programme outcomes. We might reasonably expect these trajectories to differ between employment sectors, countries, and along other lines, as well as over time, yielding a rich variety of data to aid reflection on current scholarship programming. One compelling finding from an initial reading of results is how quickly (often extensive) catalytic impacts on communities and organisations can accrue. To cite one of the many examples from alumni only two years post-scholarship:
I have solved a protein structure upon my returning to home country. This is the first protein structure solved in my institution. I have initiated the software[s] installation in the institution and this will benefit the next to continue the effort. Outcome: We do not need to depend on attaching students to the overseas institutions for X-ray protein crystallisation and protein structure solution. We can do it ourselves with the knowledge that we have learned.
Thoughts for the future
The CSC longitudinal research framework is a structured, rigorous approach to assessing outcomes. A closer link between post-scholarship research and the grant cycle should better support more authoritative statements about realising the scholarship programme's aims. In turn, this provides the 'bare bones' for building a convincing empirical case about Commonwealth Scholarships' contribution to social, economic, and civic development. The approach need not be restricted to one agency, such as CSC. Multiple agencies can collaborate to implement the framework in the hope of establishing common indicators and comparable datasets, and with the aim of improving the 'global' richness of analysis available to all.
On the other hand, this is not a framework for explicitly assessing impacts: the real lived experiences, beneficial or otherwise, of downstream effects from the scholarship 'intervention'. Impact data for scholarship programmes is notoriously difficult to gather, not least because the second-order beneficiaries – those who benefit from actions by the beneficiaries of scholarships – are often beyond the reach of agencies to survey or interview without huge resource investment. Nevertheless, understanding impact is underpinned by understanding outcomes. Additionally, focusing at an early stage on 'impact' tends to encourage championing of exceptional cases (prime ministers, Nobel laureates, major community leaders etc.) that are hardly representative of either the broader programme impacts or the process by which scholarships can and do lead to impacts. An effective and rigorous 'contribution story' for such a programme requires more than only good stories about virtuous alumni: it needs a compelling explanation of the pathway to change.
Our approach is not without deficiencies. At its foundation lies self-report web surveys, a research method with many limitations, all too well understood in the international scholarship community. Yet web surveys are not the only part of the framework: rather, they are the hub to which many spokes – detailed qualitative field studies, bibliometric research, social network analysis, etc. – can be affixed. As some of the most impressive studies in this field are showing us, much can still be achieved with web surveys, if intelligently combined with other research methods. Another limitation is that most of the indicators we currently use to analyse topics such as 'leadership' or 'professional development' are 'act-based' – they examine whether someone is undertaking an activity and roughly how often – but are not cumulative. They cannot tell us, for example, what the cumulative total of academic production is within a defined period, they can only observe whether it is changing over that period. Clearly this approach has significant limitations for cost-benefit analysis: another reason why additional 'spokes' around the survey 'hub' are very important.
Two potentially helpful resources from our framework are available on request: 1) the surveys, 2) a 'dictionary' of consistent question / data labels, to make comparison across datasets easier. Please contact us if you would like a copy of either resource: communications@acu.ac.uk
Image by the Commonwealth Scholarship Commission in the UK
The authors
The Measuring Success blog series draws from the ACU's experience in scholarship design, administration, and analysis, and our connections in the sector, to explore the outcomes of international scholarship schemes for higher education.