One would expect and wish for a direct link between evaluation studies and policymaking. In an ideal world, policy making is influenced by the outcomes of evaluation studies, and evaluation studies are designed to improve and inform policy decisions. In practice, this is not always the case. In many instances policies are determined by politics and interests rather than insights gathered from evidence, and evaluation studies are seldom commissioned to support a political decision. Evaluations of scholarship programmes are no exception to this.
The long history of the Netherlands Fellowship Programmes (NFP), and the many evaluation studies that have been carried out, give an interesting insight into the unreliable relationship between evaluations and policymaking. The NFP functions by giving mid-career professionals fellowships for study and training in the Netherlands. Through their studies, fellows are expected to improve the services and products of their employing organisations. At regular intervals programme evaluations and tracer studies have been conducted to assess the effectiveness and impact of both the programme and individual fellowships. Over the years, the main objective of the NFP has hardly changed, remaining committed to the sustainable strengthening of (human) capacity of organisations in developing countries.
Changing development policies
On the other hand, Dutch development cooperation policies have undergone major changes in 2000 and 2011. In 2000 the traditional moral and do-good aims were exchanged for demand-driven development cooperation objectives, with project ownership remaining in the South. In 2011, the focus on social development was exchanged for one on economic development as a driver for wellbeing and welfare in developing countries. Aid was exchanged for trade in order to stimulate economic development in developing countries and, at the same time, benefit our own Dutch interests.
Despite the changes to development policies in 2000, the evaluation and tracer studies of the NFP continued to focus on the impact of the scholarships on individuals and their employing organisations, as had been the case in the years before. These evaluations resulted in adaptations to the programme, which aimed to increase its effectiveness and efficiency and still exist to the present day:
- Tailor Made Training - short-term skills training to groups of professionals from an organisation
- Refresher Courses - short courses for alumni to update their knowledge in their field of expertise
Another modality, the Multi-Year Agreement, which consisted of a package of scholarships in support of an organisation's Human Resources Development plan was tried out, but not continued.
Transitioning to a demand-led approach
The changes in government policies have not altered the objectives and programme elements of the NFP, but they have drastically affected the distribution of scholarships. Until 2000, the available scholarships were divided between a number of specialised education institutions in the Netherlands, using a quota system. These institutions offered courses in the English language for mid-career professionals of developing countries, which focused on practice-oriented subjects thought to be relevant for developing countries.
After 2000 this was changed to a system of scholarships allocations, based on demand for certain courses from interested NFP candidates in developing countries. Prospective scholars could choose from a wider range of international courses, offered by Dutch higher education institutes that could demonstrate they had the infrastructure and facilities to host foreign students. This broadened the options for NFP candidates to choose from among a wide variety of courses offered in the English language, but had serious consequences for the specialised Dutch institutions traditionally hosting NFP scholars; they lost their monopoly on NFP fellowships and entered an uncertain future. This drastic change had no relation whatsoever to the conclusions or recommendations of the previous evaluation of the NFP. Rather, it was linked to a broader interdepartmental review of the effectiveness of all Dutch support to higher education which was conducted in 1999-2000 and was politically and ideologically motivated. The Dutch government wanted to replace the existing portfolio of support instruments with a small number of more flexible programmes that would serve capacity demands in developing countries and make use of a broad spectrum of Dutch education and training organisations.
From development cooperation to soft power
Since 2000, several external evaluations and tracer studies have proven the success and value of NFP in achieving its original objectives, despite the shift to a more demand-driven orientation. However, in the last five years new changes in politics and policies have affected the expectations and objectives of the NFP. The NFP is no longer seen as a development cooperation instrument, but increasingly as an instrument to pursue Dutch economic and diplomatic interests.
This has led to changes in the purpose and design of the evaluations and tracer studies of the fellowship programme. Nowadays NFP evaluations have to focus more on the contributions scholarships make to economic relations, economic development, economic diplomacy and the promotion of peace and stability on fragile environments. NFP alumni are seen as global ambassadors for the Netherlands and in fragile states as instruments to support peace and stability in the region. The NFP has also introduced a number of activities to catalyse the economic diplomacy aspect of the fellowships through stimulating and sponsoring various alumni activities and forging links between alumni, the Dutch embassies and Dutch private companies in NFP countries.
The louder call for accountability of how tax payers' money is spent on development cooperation has further influenced the evaluation studies. Tracer studies are valuable in getting an impression of the effectiveness of scholarships and individual examples of impact, but fail to systematically assess the return on investments involved in scholarships. Other research methods need to be used to that end, with a strong economic orientation, the construction of baseline data and gathering of counterfactual evidence. EP-Nuffic has tried to conduct simple training needs assessments in a small number of NFP countries, focussing on organisations which play a role in the Dutch bilateral aid programmes and would be preferred partners to nominate staff for NFP fellowships. These assessments could then serve as the baseline for an impact assessment in the future. Unfortunately, the exercise proved to be time consuming and, more importantly, not appreciated by the Dutch embassies who, for diplomatic reasons, are hesitant to openly prefer one organisation over another.
Final thoughts: links between evaluation and policymaking remain weak
If the history of the NFP is illustrative for other scholarship programmes it shows that where in the past the main purpose of the evaluations and tracer studies was to demonstrate and improve the effectiveness of programmes, they are increasingly being used to defend political choices and to account for how tax money is being spent. The learning aspect of evaluations has lessened in importance and has been replaced by a focus on accountability and impact. It is also shows that the link between evaluations and policymaking is to a great extent determined by ad-hoc political choices and is not as causal and logical as evaluators and development practitioners would like it to be.