Rural affairs, food and environment research programme 2016 to 2022: evaluation report

1. Executive Summary

1.1. Introduction

The Rural Affairs, Food and Environment Research Strategy for 2016-21^[1] ("the Strategy") provided the vision and strategy for investment by the Scottish Government in environmental and agricultural research over a five-year period.

The vision (as extracted from the Strategy (Scottish Government, 2015, p3)) states:

We will have delivered research that is relevant, respected and responsive

to Scotland's communities, its people and to the rural economy in 2021 by:

• Having a Strategic Research Programme, which has interdisciplinarity at its core and has a single clear identity that is recognised nationally and internationally for its excellent science.
• Demonstrating increased levels of collaboration with researchers from other institutions through leverage of our investment in research.
• Having evidence of increasing innovation activity associated with the Programme with a range of non-commercial and commercial funders.
• Ensuring we demonstrate the impact of our research undertaken on the communities, businesses, public sector and the economy of Scotland in a clear and measurable way.
• Making the significant data holdings we support through our funding more visible and accessible.
• Creating Centres of Expertise at points of significant demand in the system for the translation of scientific understanding from across Scotland into solutions to critical questions that will emerge over time.

Due to the Covid-19 pandemic, the Programme was extended for one year to cover 2021/22 and to enable ongoing projects to finish. Main Research Providers (MRPs) delivered the majority of the research. These included the James Hutton Institute (incorporating Biomathematics and Statistics Scotland - BioSS), Scotland's Rural College (SRUC), Moredun Research Institute, and the Rowett Institute. Dedicated Centres of Expertise (CoEs) including ClimateXChange (Centre of expertise connecting climate change research and policy (CXC)), CREW (Centre of Expertise for Waters), EPIC (Centre of Expertise on Animal Disease Outbreaks) and the Plant Health Centre (launched 2017/18 therefore during the Programme) provided specialist expertise. Additional Research Providers (ARPs) such as universities and Higher Education Institutes also undertook some research.

Following completion of the six-year Programme, there was a need for evaluation in line with HM Treasury's Magenta Book. Risk & Policy Analysts (RPA) was contracted by the Scottish Government to undertake the Programme evaluation. The evaluation had the following specific objectives:

1. Gather and present the Programme inputs, outputs, delivery and outcomes;

2. Understand and quantify the impact to Scotland from the research undertaken – including, community benefit, net zero contribution, policy contribution and scientific benefit, where appropriate, use should be made of guidance from the Magenta Book;

3. Understand and quantify what value Scotland, and the Scottish Government, gained from the research undertaken within the Programme. This includes a robust estimate of the economic impact of the Programme, again referring to Magenta Book methodology where and when appropriate;

4. Provide appraisal on the Programme delivery and performance of both the MRPs and the Scottish Government. This includes detail on the advantages and disadvantages to the delivery model. Consideration should be given to delivery vehicles, the procurement arrangements and the research delivery framework; and

5. Provide a comparative assessment against the final research outputs and the vision and principles set out in the Rural Affairs, Food and Environment Research Strategy for 2016 – 2021.

1.2. Approach

The evaluation began with the co-creation of a Theory of Change (ToC), which showed the components of the Programme from inputs through to the vision. The study team used the ToC, alongside the evaluation objectives, to develop an evaluation framework which included questions, indicators and data sources.

Evidence for the evaluation included Programme documentation supplied by the Scottish Government (e.g. annual reports) and information from the internet (e.g. the SEFARI^[2] website). The study team reviewed the Programme documentation as part of a desktop assessment. This involved collating and analysing basic Programme statistics such as staff inputs, funding and outputs (e.g. policy outputs).

Programme documentation was supplemented with engagement data. The study team held 30 online semi-structured interviews with stakeholders from the Scottish Government, MRPs, CoEs and ARPs. A sampling strategy was used to ensure engagement included stakeholders from different organisations, roles, etc. Tailored interview questions were developed for the different stakeholder types.

The study team analysed all the evidence against the evaluation questions. Programme data were used to estimate the economic impacts of the Programme. Interview data was used to explore the reasons for any patterns or trends, to help identify impacts, and to determine the extent to which any impacts could be attributed to the Programme. Interview evidence also fed into recommendations.

1.3. Findings

Objective 1: programme inputs, outputs, delivery and outcomes

The Programme allocated £279 million across the Strategic Resaerch programme (SRP), Underpinning Capacity, CoEs, knowledge sharing through SEFARI gateway and on supporting innovation. The programme supported 354 staff in 2016/17, declining to 267 in the extension year (2021/22). Work was undertaken on all three of the Programme's themes of natural assets; productive and sustainable land management and rural economies; and food, health and wellbeing. Outputs included 1,244 policy outputs, 2,674 peer-reviewed publications and 607 publications for trade, which captured topics such as sheep scab and agri-policy for pollinators.

Objective 2: impact to Scotland

The Programme has resulted in many different impacts for Scotland. The net zero contribution of the Programme is strongly linked to work by CXC, which was involved in producing a Scottish Transport and Air Pollution model. This has implications for transport policy in terms of identifying the best options to decrease emissions and improve air quality. In relation to community benefits, EPIC's knowledge exchange work on disease preparedness is expected to benefit the farming community. The contribution of the Programme to policy has been seen in work on rural depopulation, which fed into the creation of the National Island Plan by the Scottish Government's Island Policy team. Considering scientific benefits, research has led to a new method for monitoring and conserving genetic diversity.

Objective 3: value gained by Scotland and the Scottish Government

The benefits of the Programme have been estimated as £470 million to £680 million (£2022) based on monetising eight different types of impact. This figure includes the economic benefits of gross value added from jobs and spin-outs, income generated from intellectual property and the reduced impacts of animal diseases (specifically sheep scab). It also captures the environmental benefits of reduced greenhouse gas emissions (from changes in diet resulting from a food swap tool) and social benefits associated with jobs provided by the Programme, new skills and the return on public investment in research. The total estimated benefits are greater than the costs of £279 million, and still, the actual value of the Programme is likely to be even higher. Some benefits will take time to be realised and are not yet apparent.

Objective 4: appraisal of Programme delivery

Programme delivery was generally seen as positive. Interviewees from MRPs, CoEs and ARPs were keen to highlight the Programme's encouragement of co-working between MRPs instead of making them compete for funding, and the way the Programme allows for more strategic research. The Programme was also seen to encourage direct communication with policy makers. The performance of the Scottish Government was broadly praised, especially in relation to accessibility of staff during the Covid-19 pandemic. Strong personal relationships were recognised as key. However, it was sometimes difficult to build relationships across institutions if they did not already exist. Other issues raised by MRPs related to the use of Excel spreadsheets for reporting. These were seen to take a lot of time and effort, diverting resources away from research. Comments were also made in relation to the length of the funding cycle, and how annual cycles sometimes led to staff looking for other jobs due to lack of funding certainty.

Objective 5: comparative assessment against the Strategy

The Programme performed favourably against the principles from the vision, although comments from interviewees suggested further progress could be made. Interviewees were positive about the Programme's reputation and the SEFARI Gateway. The involvement of MRPs with others outside the Programme in research and innovation was apparent, with £85 million of funding coming from research councils, the EU and Defra, and £70 million of industry input, showing commercial interest in innovations developed. Creation of the Plant Health Centre CoE in 2018 enabled research on topics such the impacts of removing the molluscicide metaldehyde, with the results feeding directly into Scottish Government decision making.

Interdisciplinary working was apparent based on outputs, although there was a desire from interviewees for more opportunities to work together. Underpinning capacity was used, but interviewees felt it could have been promoted more. Some interviewees felt the individual institutes were more visible than the Programme.

Benchmarking

The Programme was seen as having a long-term view when compared with other funders. For example, UK Research and Innovation (UKRI) was felt to be more reactive. Whilst the overall UKRI programme is bigger in terms of amount of funding, and has a different scope, it was commented that expertise within the SRP was on a par. However, it was also suggested that UKRI work might have a higher standard reputation. This could link to the issue of identity, perhaps reflecting the lower visibility of the Programme itself.

1.4. Recommendations

The evaluation identified several recommendations for future Programmes. Key recommendations included:

• Length of funding cycle: consider the possibility of longer-term funding cycles within the Programme to provide sufficient resources to ensure continuity of research projects. The loss of key personnel impacted the quality of science conducted, and meant strong personal relationships that had been established during prior Programmes were lost;
• Importance of relationships: embed strong relationship building practices as part of project planning, and encourage this throughout. Numerous interviewees identified a key reliance upon effective relationships across institutions and work package teams, with difficulties being encountered where relationships were not productive. Pre-established relationships were seen as vital in delivering quality science;
• Project reporting: review project reporting to see if it could be made more flexible as well as undertaken in a different format to Excel; and
• Promotion of the Programme: build on the work already undertaken by SEFARI to further promote the Programme itself, to build the Programme's identity. Interviewees felt that audiences were sometimes more aware of the individual institutions than the overall Programme, and more could be done to ensure the Programme's strategic approach to research was appreciated.