Scotland Rural Development Programme 2007-2013: ex-post evaluation

1. Introduction, Context and Study Method

This report presents the findings of the ex-post evaluation of the Scotland Rural Development Programme (SRDP) 2007-2013.

The study was commissioned by the Scottish Government in December 2015 and was undertaken by EKOS Limited in collaboration with the Rural Development Company, P&L Cook and Partners, and Prof. Bill Slee.

1.1 Context and Study Objectives

The European Commission requires Member States to evaluate the assistance from the European Agricultural Fund for Rural Development (EAFRD) for the programming period 2007-2013.

The programming period incorporated a number of features, such as a new EC Common Monitoring and Evaluation Framework (CMEF) which included the intervention logic for each Programme Measure in relevant Measure Fiches, and an Ongoing Evaluation approach.

The ex-post evaluation was guided by the guidelines for 'Ex Post Evaluations of Rural Development Programmes' ^[1] .

As described in the 'EC Guidelines for Ex Post Evaluations':

Ex Post Evaluation is a summative evaluation of a Rural Development Programme after it has been completed. It is conducted at a point where it is possible to assess impacts and the added value of the programme funding. ….Demonstration of the policy achievements, thus legitimising funding for rural development measures, is important at European, national and regional levels, especially when budgets are tight. The Ex Post Evaluation also provides the opportunity to see whether the policy was designed and implemented appropriately to address the most relevant needs in the programme area.

The key objectives of the ex-post evaluation were to assess a number of factors, including the:

• relevance in terms of addressing the most important needs in the programme area;
• effectiveness and achievements towards policy objectives;
• efficiency in terms of receiving best value for money;
• results in terms of programme achievements within the group of direct programme beneficiaries;
• socio-economic impacts in terms of programme contributions to the change observed in the programme area; and
• success and failure factors and lessons learned for the future policy design.

Although the evaluation guidelines include an assessment at Measure level, the overall impact of the SRDP is assessed at Axis and Programme level.

In combination with the ex-post evaluation guidance, the CMEF system provided the basis for the ex-post research design and assessment criteria.

1.2 Study Method

The study used a mixed-method approach drawing on secondary research as well as undertaking primary research. The following key stages were accomplished:

Stage 1 - Inception: The study commenced with an Inception meeting with the client at which the study scope and detailed method were agreed. A short Inception Report was produced and approved by the client.

Stage 2 - Desk-based Research: A number of comprehensive desk-based reviews of available monitoring information, research studies, mid-term evaluation (MTE), and ex-ante workshop findings have been undertaken. Key findings were presented in the first Interim Report in March 2016.

Stage 3 - Primary Research: The primary research phase incorporated a comprehensive programme of activities. It was designed to consult with a range of stakeholders and beneficiary groups throughout Scotland as well as a range of Scottish Government officials, Scheme Managers and the Scottish National Rural Network (SNRN).

The primary research included the following:

• 2 workshops with SRDP Scheme Managers and LEADER Managers;
• 12 Scheme Managers and strategic stakeholders - 1-to-1 consultations;
• Postal survey of Measure 321 ^[2] beneficiaries;
• 8 case studies;
• 5 thematic focus groups across Scotland (MAPP method); and
• A top-up survey of beneficiaries who could not attend the focus groups.

At the end of Stage 3, a second Interim Report was produced outlining the key headline findings of the fieldwork.

As the primary research was conducted over the summer period, it was agreed to conduct the five thematic focus groups by the end of August delaying the original timeframe of the evaluation by two months.

Stage 4 - Analysis and Reporting: Following the completion of the primary research, all desk-based and primary research findings were analysed and the Topic Guides for each Axis completed. Further analysis was conducted at the programme level and a detailed impact assessment was undertaken whereby findings from previous annual surveys were utilised. Stage 4 assessed the socio-economic impact of the programme, and the effectiveness and efficiency with which the SRDP was implemented.

Following the completion and submission of the draft report, we will hold a learning workshop with the client to reflect on the findings and draft recommendations following which we will amend the report and produce the Final Report to the exact specifications of the client.

1.3 Study Issues

A small number of study issues were experienced, mainly influencing the depth to which the ex-post evaluation could analyse data and beneficiary experience:

• the ex-post evaluation was commissioned on a limited budget restricting most beneficiary research to utilising findings from previous evaluation surveys (with the exception of one postal survey focusing on Measure 321, and five workshops with a limited number of beneficiaries participating).
• SRDP monitoring data were made available to the study team on an aggregate basis. Without being able to interrogate and analysis the raw monitoring data sets, the evaluation team was unable to analyse the data in detail (type of beneficiary, size of projects, type of projects).

The above issues placed limitations on the extent to which a number of the Evaluation Questions and Common Evaluation Questions posed by the 'Ex Post Evaluation Guidelines' could be answered.

The relatively poor quality of the performance indicator targets and/or missing baselines further impinged on the delivery of a decisive ex-post assessment as the extent to which the SRDP was effective in achieving its targets was difficult to assess.

1.4 Overall Structure of the Report

Chapter 1: Introduction, Context and Study Method;

Chapter 2: Description of Programme, Measures and Budget;

Chapter 3: Answers to the Evaluation Questions; and

Chapter 4: Conclusions.

There are six Appendices that accompany the report.