Methodologies for evaluating short fellowship programmes

Measuring success?

The ACU manages several short fellowship programmes which offer academics the opportunity to spend a period of time at universities or organisations in the UK and undertake professional development tailored to their specific professional needs. The fellowships can be seen as a collection of individual training paths, and their evaluation can therefore present some distinct challenges. Here, we take a look at two specific fellowship schemes and discuss the methodologies at play for assessing impact.

CSC Fellowships: As part of the UK’s contribution to the Commonwealth Scholarship and Fellowship Plan (CSFP), the Commonwealth Scholarship Commission in the UK (CSC) offers fellowship programmes for academics from developing Commonwealth countries to develop knowledge and expertise in their specialist areas. The programme, funded by the Department for International Development (DFID) of the United Kingdom, include Professional Fellowships for mid-career professionals to spend time at a UK host institution, Academic Fellowships for early career academic to conduct research at a UK university and Medical Fellowships to provide doctors and dentists the opportunity to update their clinical skills at UK hospitals.

CIRCLE Visiting Fellowships: CIRCLE (Climate Impacts Research Capacity and Leadership Enhancement) visiting fellowships commenced at the beginning of 2015. The programme, also funded by DFID, is set to run over three years, supporting 100 CIRCLE Visiting Fellows (CVFs) in total with a target of 60 post-doctoral and 40 post-masters Fellowships and a 50/50 gender allocation. The programme is relatively small in scale and timeframe compared to some of the larger scholarship programmes, however it is ambitious in its objectives and is striving for a sustainable legacy of improved institutional research output by building the capacity of institutions involved in the programme to provide systematic research support through an Institutional Strengthening Programme (ISP).

Evaluation challenges

Firstly fellowships are designed to provide a development opportunity to individual fellows and therefore the evaluation of these programmes needs to examine the extent to which they deliver on the knowledge, skills and learning for which they are developed, in addition to the longer term outcomes and impact of the individuals. This can be particularly challenging when the knowledge and skills developed are being examined across a wide spectrum of disciplines and specialities.

The dual objectives of these programmes to strengthen both the individual capacity of the fellows as well as their institutions also present an evaluation challenge. While understanding the impact on individuals and tracking longer term careers is important, it is balanced with the need to understand the immediate transfer of knowledge and implementation of skills to the organisations where fellows are employed. Long-term evaluations aimed at understanding the impact on organisations also have to account for individuals who move on in their careers.

Targeted approaches

For the CIRCLE programme the relatively small number of fellows and the short timeframe of the fellowships allows for a more in-depth, hands on approach to programme management and monitoring. The contact with CVFs and their institutions is more frequent than may be the case with larger programmes. CVFs, their supervisors and their institutions regularly report on progress, both in terms of the fellowship research as well as the CVFs’ engagement in disseminating the outcomes of their research. Additional resources such as a research support fund and a conference and training fund means that the programme managers are also able to closely track the activities undertaken by the fellows. Furthermore, the CVFs have been very active in discussing the progress of their fellowships on an online forum, which led to the development of a blog to showcase their achievements and reflections externally. These various forms of tracking ongoing progress give a rich picture of the activity of the CVFs and their engagement with the programme.

In contrast, the ISP component of the programme is, by nature, a more gradual and complex process, often involving multiple individuals and departments, and measures of outcome and impact are therefore less tangible and quantifiable than individual interventions. A number of training events have been run and as the first year draws to a close, institutions are setting up their ISP teams and preparing to assess their institutional capacity, based on the understanding they have developed over the past year. A gap analysis will inform a workshop in December on the next steps the institutions will take in strengthening their capacity.

In addition to the ongoing monitoring of the programme, surveys on application serve as a baseline to evaluate the progress of the CVFs and their institutions. A counterfactual survey has also been conducted among early career researchers who have not participated in the CIRCLE programme. These initial surveys have already thrown up some interesting insights, suggesting that the CVFs selected represent researchers already showing more activity and engagement before the start of the programme than others at their level. This will need to be taken into consideration when conducting the final evaluation of the first cohort of CVFs as well as comparing them to future ones.

In contrast to the CIRCLE programme, the CSC’s Fellowships are long established and on-going. Their evaluation was initiated in 2008 with a survey sent to all fellowship alumni since the inception of the CSFP scheme (1959) to gather data on their career trajectories and fellowship outcomes. These Fellows have subsequently been sent an additional evaluation survey between 2011 and 2015. However, recognising the importance of following up with fellowship alumni soon after the completion of their UK visit, the CSC has implemented a longitudinal approach for Fellows commencing their Fellowships in 2012 and onwards. At the same time the CSC conducted a review of its Professional Fellowships scheme which highlighted the need for data collection from other stakeholders, including the organisations which host Fellows in the UK and employers in their home countries. 

Fellows now receive two evaluation surveys within the first two years after the completion of their awards. New methods have been developed to capture feedback from the UK hosts and home country employers for the Professional Fellowships scheme, with survey data collected from these groups and focus group discussions with Fellows and hosts conducted to understand experiences of both parties during the fellowship. For the Academic and Medical Fellowship programmes, baseline surveys have been developed in 2015 to understand the context within which Fellows are working prior to their fellowship and host and employer surveys will be extend to these schemes from this year. Qualitative data through interviews will also complement the surveys by providing a richer context for greater understanding in, for example, issues Fellows may face when they return to work such as barriers to application of the knowledge and skills that they gained in the UK.

The blend of methodologies employed in the evaluation of fellowships demonstrates the rich, in-depth data that can be obtained for these short programmes, as evidence beyond the experience of the fellows can be gathered by working with other key programme participants. However the range of methods also reflects some of the complexities in measuring the impact of fellowships and by capturing the perspectives of multiple stakeholders it is possible to gain a holistic understanding of the learning, institutional improvement and longer term outcomes of Fellowships programmes.


Dr Caroline Cage is CIRCLE Programme Officer at the ACU, and Shireen Quraishi is CSC Programme Officer (Evaluation) at the ACU.


Measuring success?’ – This blog series draws from the ACU’s experience in scholarship design, administration, and analysis, and our connections in the sector, to explore the outcomes of international scholarship schemes for higher education. New posts are published every three to four weeks, authored by experts from all around the world. Find out more

If you are interested in contributing to the series, please email info@acu.ac.uk

Last modified on 05/02/2016
Tags: early careers, data, impact, 'Measuring success?' blog series

Comments

Dear Dr. Cage, I would like to express my interest in the short fellowship program for 2 or 3 months. I am doing my PhD study in the Technical University of Madrid in Spain. This is my second year of a 3-year PhD program. I have a scholarship that cover my PhD tuition and living expenses including the time I spend as a visiting scholar outside of Spain. So, the financial aid is not necessary for me during my fellowship period. My area of study is improving the reverse logistics for fashion industry to promote environmental and economic sustainability which is one of the research lines under Supply chain Management. Would you please provide me some information about how to apply and the possibility to apply to this program? Best Regards, Mohammad Abdullatif Bukhari Universidad Politecnica de Madrid
by Mohammad Abdullatif Bukhari
on 29/02/2016 17:26

Reply

*
* Your email will not be published
*
*
CAPTCHA Image   Reload Image
It is a well-known fact that Illuminati consist of Multi Millionaires, Billionaires who have major influence regarding most global affairs, including the planning of a New World Order. Many world leaders, Presidents, Prime Ministers, royalty and senior executives of major Fortune 500 companies are members of Illuminati. join a secret cabal of mysterious forces and become rich with boundless measures of wealth in your company or any given business, the great Illuminati can make everything possible just contact : liliaparker2000@gmail.com or WhatsApp +2347019637728 EL IAI LEXION Thaddeus Iam Vice-President of Citizen Outreach THE ILLUMINATI ORGANIZATION Do not hesitate to contact us by WhatsApp. Whatsapp: +2347019637728 Email : liliaparker2000@gmail.com BEWARE OF SCAMMERS, THERE IS NO SUCH THING AS REGISTRATION FEE AND YOU MUST BE ABOVE THE AGE OF 18YRS. THANKS,,,,
by mr kings
on 21/11/2017 23:08

Reply

*
* Your email will not be published
*
*
CAPTCHA Image   Reload Image
Dear Caroline, Thank you for this insightful write up. I have earlier shared some thoughts on assessing impacts of the CIRCLE programme (2014-2018) with Jay and Ben. I will be delighted to work with you on actualizing the research proposal. Thank you. Best regards, Olawale Olayide (CVF 2015)
by Olawale Olayide
on 03/03/2016 19:22

Reply

*
* Your email will not be published
*
*
CAPTCHA Image   Reload Image
Lake Doctoral Dissertation Fellowship Lake Institute on Faith & Giving at the Indiana University Lilly Family School of Philanthropy is pleased to offer a one year “Lake Doctoral Dissertation Fellowship”. The winners will get a one-year doctoral dissertation fellowship of $22,000 for the academic year 2017-2018. The submission of fellowship application will close on January 31, 2017. http://usascholarships.com/lake-doctoral-dissertation-fellowship/
by ama
on 25/11/2016 08:56

Reply

*
* Your email will not be published
*
*
CAPTCHA Image   Reload Image

Leave a comment

*
* Your email will not be published
*
*
CAPTCHA Image   Reload Image