Scotland Rural Development Programme 2007-2013: ex-post evaluation

4. Conclusions and Recommendations

4.1 Conclusions

4.1.1 Relevance of Strategy

The SRDP intervention logic was considered by most stakeholders to be appropriate to the needs identified in rural Scotland. The logic was robust, especially at programme level. Most Measures were well chosen to address some of the key weaknesses identified at the outset and during the Programme period.

The consistency of the SRDP with national rural policy priorities was confirmed by the annual reports of the Programme. The 2008 Spending Review and a depreciation of Sterling against the Euro of around 25% following the financial downturn in 2008 resulted in a significant reduction of originally allocated public sector resources to the Programme by nearly half (48%), inevitably impacting on programme outcomes and creating less change than originally anticipated. The European Economic Recovery Plan later added funding to a selected range of Measures to address some of the perceived needs of the rural economy following the economic crisis. As all available funding was taken up, the demand for the SRDP was confirmed.

The significant emphasis of the Programme on environmental interventions was a direct response to the long-term decline in farmland biodiversity and the condition of many designated sites and was in close strategic alignment with Scottish policy direction thereby addressing important current needs.

Regarding those interventions targeted at improving the rural economy, the SRDP was largely seen as a vitally important support mechanism for rural businesses in difficult and uncertain times when confidence levels were low. High global prices for food significantly buffered the farm sector from the worst effects of the economic crisis, but for many non-agricultural rural businesses issues of sustainability were of more immediate and higher priority than business expansion or growth over the programme period.

The target setting of the Programme was weak which limited the extent to which an assessment could be made regarding its ability to achieve its outputs effectively and efficiently. In a number of cases, the SRDP seemed to have changed its original approach towards supporting more people but in a more light-touch manner (although in other areas of the Programme the opposite approach was evident).

This was particularly the case under Axis 3 with a large increase in the number of beneficiaries and activities targeted, although in practice the outturn fell somewhere between the original and revised targets.

The economic downturn in 2008 and associated uncertainties influenced Programme up-take negatively, including availability of domestic funding (which was drastically reduced); and strategic emphasis. Having said this, agriculture showed some resilience to the crises until more recent events.

The SRDP had a significant emphasis on supporting agri-environmental investment to improve biodiversity etc. in line with overall Government objectives, although the largest Axis 2 Measure (212, LFASS) seemed to be more significant in terms of income support than biodiversity enhancement.

The inclusion of rural businesses as well as farm and forestry holdings was relevant and appropriate with a focus on supporting diversification and growth and fundamentally sustainability of beneficiaries.

4.1.2 Implementation Approach

The original delivery ambition was strategically designed to complement the intervention logic in aiming to link together existing schemes in an integrated manner, in particular the Land Managers Contracts approach with the Axes and Measures of the SRDP, especially through the RP and LMO schemes. This could have been transformational but needed to be much better targeted and more actively facilitated to create more scope for incremental actions in response to specific Programme objectives.

The management and implementation were a major topic of concern throughout all stages of the Programme and subject to a first stage review, ongoing evaluation, and MTE. In many respects concerns over delivery obscured the strategic objectives and focus on results. It is questionable whether the sought-after strategic learning which the integrated delivery approach intended was actually achieved. The ambition for an integrated approach was laudable but in practice integration was limited with the concentration of resources in four Measures and the experienced difficulties with the online application process. In addition, the large but unrelated direct income transfer of LFASS did little to enhance integration.

A complex IT based system made the application process difficult for many farmers and other small beneficiary businesses and organisations who either relied on using consultants to apply or felt excluded from the Programme.

The RPAC approach was seen as highly beneficial in joining up agency interventions, sharing approaches and creating effective cross-agency working. However, RPACs were felt to be less effective in targeting and allocating resources against priorities and related governance/appraisal/empowerment issues. Appropriate means of renewing such collaborative efforts between key agencies should be considered.

Small farms and crofts were disenfranchised by the scoring system in the 2007-2013 SRDP. Where they are making discernible gains at reasonable cost, their actions should be supported by a simplified scheme that avoids over-elaboration.

In terms of LEADER many lessons have been learned during the Programme period regarding shortcomings in implementation particularly regarding the availability of guidance for LAGs, clarity of eligibility criteria and project expenditure. Audit issues created some considerable upset in some LAGs.

4.1.3 Effectiveness and Achievements

The SRDP contributed substantially to sustaining the rural economy and supported particularly the safeguarding of jobs in farm and forestry holdings. In addition, effects of rising farm product prices for some of the period and timber price over a longer period on incomes in the farm and forestry sector generated supporting factors in terms of positive policy effects. It was felt that the SRDP investments played a part in supporting and complementing other initiatives particularly in the areas of climate change, renewable energy, and water quality.

There were thought to have been significant agri-environmental gains which can be attributed to SRDP interventions, though more robust methods of evaluation would have enabled greater certainty of judgement.

In some areas, the SRDP reached a larger number of beneficiaries than anticipated (and the Programme was often perceived by stakeholders as being effective in achieving high participation rates). However, its success in performance varied between Measures and a number of Axis 1 and Axis 3 Measures failed to reach their targeted number of beneficiaries.

Overall, the effectiveness in achieving the Programme's aims is difficult to assess due to weak target setting and insufficiently robust monitoring being in place (particularly regarding environmental achievements). Therefore, the assessment of effectiveness relied primarily on the perceptions of wide range of stakeholders consulted throughout the Programme period.

Whilst it was cost-effective to have continued with some of the established, well-known and easy to administer schemes such as LFASS in Axis 2, it was widely believed that the SRDP could have been more effective, particularly in reaching smaller businesses, farm and family holdings in Axis 1 and Axis 3 if application and reporting procedures of some of the SRDP Schemes had been easier. .

SRDP support related well to improving business sustainability and delivered significant benefits and there is consensus that on the whole, the SRDP was effective in achieving its objectives (however, due to the weaknesses of target setting and the monitoring systems in place this is less reflected in a quantitative manner).

Primary research found that the SRDP was successful in supporting the Food & Drink sector which was also a sector that received substantial political support.

SDS linked well with a number of SRDP Measures. It secured good industry buy-in and delivered some very successful skills development/knowledge transfer initiatives often involving third parties, e.g. a renewables development initiative and a Crofting Training Programme delivered by the Scottish Crofting Federation as well as the successful Monitor Farm Programme (with a relatively small budget).

Studies commissioned by SRDP found that RP has contributed positively to improving assessed condition on designated sites compared to condition on unsupported sites. The benefits included a shift from substandard condition to good condition on some supported sites.

In terms of biodiversity, RP options were targeted, both at the holding and landscape scale, and where options were tailored to deliver the desired outcomes, then they worked well (however, robust quantitative measurement of effects was in most cases lacking).

LMO was seen to have performed satisfactorily for a broad and shallow scheme both in terms of allocation of funds and in terms of spatial disbursement of these funds across a wide geographic reach. The attempt at regionalisation through the RPACs was not universally liked or successful but was thought to reflect a positive policy principle and enhanced awareness of the integrated approach. While LEADER funding was small participating communities and local partnerships generally developed well, and a lot of effective community-based initiatives were implemented. There are many good examples where the integrated approach to rural development worked very well.

4.1.4 Efficiency of the Programme

Although generally regarded by respondents as successful in achieving desired results and outcomes, the limited reliability of the monitoring data, mainly due to poor target setting and weak baselines, prevented a more detailed assessment of the extent to which the SRDP was able to achieve the expected Programme results.

Primary research findings indicate that the aims and objectives of the SRDP, particularly with regard to introducing innovative approaches and helping the agricultural and forestry industries to restructure and modernise, were achieved successfully by increasing businesses' capacities and productivity.

Apart from LFASS the bureaucracy of the SRDP was often perceived as a key obstacle and challenge for most applicants/beneficiaries, raising questions over its efficiency of delivery.

The complexity of the application forms (often needing consultants) almost certainly resulted in certain categories of potential applicant being excluded e.g. crofters, small farmers - and for some of the intended learning effects of participation to miss their target audience (i.e. the farm and forestry holding/rural businesses rather than the consultants).

The intended regional targeting by RPACs was largely lost as those who selected the RP scheme did not want to close options and preferred to draw down funding for their regions, and this significantly diminished the effectiveness of geographical targeting.

One of the key lessons learned regarding LEADER was that the management of LEADER required more resources than were available to make it more efficient, particularly for LAGs operating in a small LEADER area. Overall, however, the efficiencies of all LAGs suffered from a heavy policy compliance burden which in turn placed demands on staff capacity.

4.1.5 Results Achieved

The importance of the SRDP in sustaining and safeguarding jobs was emphasised by respondents throughout the ex-post evaluation primary research.

Generally speaking, at the time of the ex-post evaluation, the performance of Axis 1 schemes and interventions was felt to have been broadly successful in introducing innovative approaches and in restructuring/modernising the farming and forestry sectors, thereby helping to increase capacities and productivity and improve the quality of life in rural communities.

Similarly, Axis 3 schemes and interventions contributed through sustaining, safeguarding and to a lesser extent creating employment. The support was relevant to beneficiary needs and the results of the support were substantial but were not clearly evidenced by the CMEF indicators as they were unable to capture the range of results achieved. Over and above performance of LEADER was not added to the monitoring data, thereby remaining under-reported.

The availability of support for small rural businesses and communities, regardless of its source was thought important in providing confidence to sustain rural businesses.

Surveys of beneficiaries throughout the Programme period showed that the majority of respondents reported positive effects on their business efficiency, output, quality and competitiveness. Much of the SRDP investment was considered effective in terms of additionality.

Monitoring data for results indicators are generally far below target. Some of this could potentially be explained by challenges in reporting on results indicators via monitoring processes. Particular challenges were associated with a widespread collection of the GVA result indicator due the complexities of the CMEF formula to be applied. Having said this, the MA commissioned a number of annual surveys to alleviate these difficulties and to be able to have some insight into achievements.

4.1.6 Impacts Created

The SRDP has created or safeguarded between 30,400 to 33,400 jobs and between £1.03bn and £1.12bn of GVA. The wider primary research suggests that the majority of jobs were safeguarded rather than created.

On average, the cost per job ranged between £41,000 and £43,300 creating a return on investment of between £2.30 and £2.40 for every £1 spent by the Programme.

Some agri-environment options were well used and almost certainly contributed to species recovery (e.g. Corn bunting). A significant number of new hedgerows were established. However, the Farmland Bird Index, the key impact indicator, declined slightly over the period, with some component species, especially upland waders, faring very badly.

New afforestation will reduce GHG emissions, but other Axis 2 Measures, especially LFASS may well increase emissions. LFASS was a major contributor to protecting jobs in remote areas but its environmental benefits are more questionable. NPIs are likely to have enhanced water quality in priority catchments. Soil quality remains compromised by falling soil carbon levels and erosion risk over significant areas.

4.1.7 Monitoring Systems

Agri-environmental monitoring was rather limited and was not proportionate to the scale of the investment. It was made more complex and challenging by the large number of options. Following commissioned research, new systems were put in place whereby cost-effectiveness of the new system was favoured.

A number of studies were commissioned to fill gaps in knowledge and system capacities to deal with monitoring of outputs and results.

While there was little time to address failings during the Programme period, the evaluation team and stakeholders feel that lessons have been learnt and that the new SRDP is currently benefiting from a new system and better guidance.

Regarding targets, one of the main issues was the basis and realism of the targets, which was acknowledged but never fully resolved. The issue was more about the determination of targets than the performance itself.

There were a number of issues surrounding eligibility of LEADER expenditure, which were addressed during the Programme period. These lessons have informed the new SRDP and better guidance, budgets, and support are now in place for the new Programme.

4.1.8 Finance Review

The overall budget for the SRDP reduced from an anticipated public spending value of €2.133 billion when approved in 2008 to €1.425 billion, a reduction of 33%. Following the Scottish Government Spending Review in response of the economic downturn in 2008, there were reductions across Axes 1 to 4, with Axis 2 being reduced by 26% (€ 381m); Axis 3 reducing most by 52% (€127m reduction); but Axis 5 increasing in value by 61% (€2m increase).

However, EAFRD spending was achieved as originally planned. The percentage of EAFRD drawdown actually increased from the original budget, due to changes in the intervention rate.

The actual spend intervention rate was 48%, slightly above the final budget figure of 47% (the more dramatic budget changes occurred in 'smaller' Axes and Measures).

The spending cuts varied between Programme Measures resulting in a considerable degree of concentration of SRDP resources on four Measures (jointly representing 72% of the total SRDP spent), thereby setting clear strategic priorities:

• Modernisation of agricultural holdings (Measure 121) - 11% of spent;
• Payments to farmers in areas with handicaps (Measure 212) - 37% of spent;
• Agri-environment payments (Measure 214) - 14% of spent; and
• First forest afforestation of agricultural land (Measure 221) - 11% of spent.

At Programme level, the finances were managed very closely to final budget, with total public spending being 99% of budget, EAFRD spending being 99.9% of budget and Scottish Government spending being 98% of target.

The final budget and actual spend of Axis 3 represented 8% of the total Programme, thus missing the required minimum allocation of 10%.

4.2 Recommendations

In line with the above conclusions, the following recommendations can be made:

1. Consideration should be given to improving the target setting against performance indicators, this will require a thorough understanding of unit costs in line with the strategic approach taken by the Programme. Adjustments to targets need to be consistent with budget changes occurring over the Programme period.

2. The logic justifying changes in strategic emphasis or approach should be clearly set out in relevant Programme documentation, e.g. changing from 'intensive training delivery to a small number of beneficiaries' to a 'light touch training delivery for many'.

3. Consideration should be given to continuing with an integrated approach to rural development where the different interventions complement one another. However, this cannot be achieved through the application process alone but requires dedicated building of know-how and capacities within the delivery chain to foster more in-depth learning and the overall understanding of the approach and outcomes sought.

4. The Managing Authority should seek to design application mechanisms and processes which are less complex thus reducing the transactional costs and reducing obstacles to accessing funding e.g. for smaller businesses.

5. Consideration should be given to providing more in-depth induction support to LEADER project managers as well as project leaders on eligibility and administrative compliance requirements. Care should be taken that LEADER groups have sufficient resources available to them to enable them to adhere to the relevant Programme requirements.

6. Adequate performance monitoring processes are needed in order to assess the progress and effectiveness of the Programme to inform strategic Programme decision making. A number of areas of improvement should be addressed, for example:
   a. creating relevant Programme specific performance indicators which address the Programme aims and objectives;
   b. establishing robust and measurable baselines at Programme level against which progress can be measured;
   c. creating user-friendly monitoring and reporting methods through which relevant data can be gathered and disseminated;
   d. the specification, creation and maintenance of up-to-date monitoring data sets; and
   e. Establishing and implementing reporting systems through which physical performance is regularly analysed and reported to the PMC.

7. Ensure that all agri-environmental schemes are equipped with relevant measurable performance indicators and robust and practical monitoring procedures.

8. Monitoring and reporting should in future take account of the fact that 'safeguarding jobs' is as important as 'creating jobs' in rural development.

9. Consideration should be given to re-establishing RPACs to help join-up regional interventions and integrate Programme delivery, in doing so it will be essential to furnish RPACs with the appropriate level of devolved decision-making and resources to enable informed, detailed appraisal capacities.

10. Ensure that performance data of all RDP implementation Schemes, including LEADER is gathered and reported by the monitoring system to present the Programme's achievements comprehensively.