Rural affairs, food and environment research programme 2016 to 2022: evaluation highlights

Highlights

What is this Evaluation about?

This report sets out the approach and findings of the evaluation of the Rural Affairs, Food and Environment Research Programme 2016-22. The Scottish Government commissioned the evaluation for two main reasons:

1. Accountability: To determine the impact and value of the Programme, and assess the Programme against the Rural Affairs, Food and Environment Research Strategy for 2016-21; and

2. Lesson learning: to appraise Programme delivery including advantages and disadvantages of the delivery model.

The Programme was originally intended to last five years but was extended for one year due to the Covid-19 pandemic. The evaluation covers all six years.

What did we do?

We developed a Theory of Change (ToC) (an evaluation tool) to identify how we expected the Programme to deliver the vision from the Strategy. We used the ToC to produce an evaluation framework. This framework included a set of evaluation questions, which defined the scope. We gathered evidence from Programme documents such as annual reports from Main Research Providers (MRPs) and Centres of Expertise (CoEs). We complemented this with information gathered through 30 interviews with representatives of the Scottish Government, MRPs, CoEs and Additional Research Providers (ARPs). We analysed the evidence against the evaluation questions.

What did we find out?

The Programme aimed to deliver research that is relevant, respected and responsive to Scotland's communities, its people and the rural economy. Interviewees were positive about the Programme overall, commenting that it "focused on achieving practical outcomes", included "quite a lot of impactful science" and there had been "a growing number of stakeholders getting involved".

The Programme's vision included a set of principles. The following bullets summarise the performance of the Programme against these principles.

• Interdisciplinary research: 82 policy outputs covered two Programme themes, whilst 21 covered all three main themes. Interviewee feedback suggested the Programme did have a multi-disciplinary team, but there was less communication across themes than within a thematic area.
• Programme identity: interviewees had mixed views on identity. Four explicitly stated that the Programme had a good reputation whilst one noted that the Programme "had its own identity". However, others implied that more could be done to publicise the overall Programme since "audiences could see individual pieces of science, but not the [Programme]". It was commented that the Programme "isn't appreciated as much as it should be either within or out with Scotland" and that the government could "better promote the fact that they've created this mechanism to undertake strategic research".
• Collaborative working: MRPs undertook 2,329 collaborative projects with other organisations. Interview evidence noted that the Programme encouraged working together, and that collaborations between researchers resulted in those in other roles (including directors) also working together. Collaborations enabled studies to have a global reach, for example, SRUC was involved in an international study to estimate greenhouse gas emissions from aquaculture. They also allowed research to go beyond the laboratory and connect with industry, for instance, on integrated pest management.
• Innovation activity: little information is available on activities directly supported by £1.0 million of innovation funding. However, on top of the Programme's £279 million total investment, funding from industry brought in over £70 million, demonstrating commercialisation of research and innovations. Examples included the James Hutton Institute obtaining royalty income from soft fruit cultivars. Their breeding programme covers both raspberry and blackcurrant and allows exchange with researchers around the world. This helps ensure fruits are bred with the traits needed to meet the demands of the soft fruit industry and their environments.
• Demonstrating the impact of research: interviewees were positive about the SEFARI Gateway (the Programme's knowledge exchange and impact hub), noting it did "an excellent job to disseminate a digestible summary of the type of research done". The Gateway was seen as a place to share breakthroughs with one interviewee explaining that "it helped bring materials together" to tell a story and inform policy. There is also evidence of industry contacts seeing outputs on SEFARI which subsequently motivated them to get in touch with researchers.
• Visibility and accessibility of data holdings: collections such as the National Soil Archive form part of the Underpinning Capacity of the Programme. Interview evidence suggests that underpinning capacity was not promoted as strongly as it could have been. However, data does indicate some increased usage during the Programme (e.g. the James Hutton Institute's live insect collection service).
• Creation of CoEs at points of significant demand: the Plant Health Centre was established in 2018. Within its first 18 months, five of the centre's projects directly informed policy, showing a demand for its expertise. It commissioned a study on assessing critical biosecurity risks to Scotland, which covered non-specialist and online horticultural sales. This research resulted in best practice options to help this important but understudied and difficult to reach sector to reduce the risks to plant health.

Benchmarking

Interviewees provided some comments on how the Programme compared with other such funds. In terms of impact, the Programme was seen as providing more of a big picture and having a longer-term view than research funded by UK Research and Innovation (UKRI). Considering processes, the Programme was viewed as being very flexible, with academics experiencing a "lighter" funding application process than the processes used under UKRI. Indeed, other funders were viewed as being more prescriptive about the way that research should be carried out. However, Programme reporting was seen as more burdensome than for similar funds with comparable budgets.

Economic impact

There are various ways we can present the potential impacts the funded research Programme had on Scotland. The first being the economic impact. The economic impact of the Programme has been estimated as £470 million to £680 million (present value, £2022). This includes benefits attributed to:

• £100 million of Gross Value Added (GVA) from jobs, including those directly employed, as well as indirect jobs and induced jobs;
• £890,000 of gross value added generated from spin-outs. This assumes three spin-outs are formed and each lasts three years;
• £4.6 million of income from intellectual property relating to licensing. This covers work by Moredun and Hutton, for example, for raspberries;
• £9.3 million from reduced impacts of animal diseases. This assumes developments in the 2016-22 Programme enable the continued reduction of the impacts of sheep scab, following the previous programme;
• £13 million from avoided carbon emissions. This assumes carbon emissions are saved through dietary changes resulting from the Food Swap tool developed by Moredun and BioSS;
• £24 million of social benefits from those employed by the Programme having a secure job;
• £18 million of benefits to the exchequer from individuals having gained a qualification (based on number of PhDs); and
• £290 million to £510 million benefits from the return on public research, assuming a 20% to 35% annual return on Scottish Government funding.

The monetised impacts for the 2016-22 Programme are greater than the Scottish Government's investment of £279 million. This shows the benefits of the Programme are greater than the costs. There are likely to be additional benefits that are not captured here. These may result from specific projects. For example, CXC's contribution to the Scottish Government's property flood resilience plan could lead to benefits for those in flood risk areas, if implementation of the plan leads to reduced likelihood or impacts from flooding. Additional impacts and benefits are therefore expected over time.

The total benefits (range of £470 million to £680 million) are of a similar order of magnitude to the estimated benefits from the previous programme. The 2011-16 programme covered five years and invested £246 million. It resulted in an estimated total of £620.7 million in benefits (covering operational impacts and wider economic benefits)^[1]. However, the figures are not necessarily comparable, because different approaches were used. Furthermore, there are other, more qualitative ways we can discuss research impact. The following text provides some examples of the 2016-22 Programme benefits.

Community Benefits

There are various examples of research providing community benefits. The Rowett Institute and BioSS developed a FoodSwap tool to enable more sustainable yet nutritious and affordable food choices. This tool featured on the SEFARI blog and, as a result of SEFARI Gateway Responsive Opportunity Funding, is now available on the internet as a demo model ("Greatest Grocery List"^[2]). The researchers intend to further test and refine the model and have already contacted the NHS and groups at Scotland Food & Drink. This free to use tool can help individuals have a healthier diet.

Centre of Expertise for Waters (CREW) were involved in developing a method to track the presence of Covid-19 in wastewater^[3]. This led to the formation of a wastewater monitoring network for Covid‑19. By 2021, the network covered around 70% of the Scottish population. It helped provide information to the Scottish Government for the public health response during the pandemic.

Scientific Benefits

In total, 38 research scientific/analytical tools and methods were developed. Examples include the Moredun Research Institute producing a novel diagnostic test for Bovine Respiratory Disease Complex, which will assist with disease control, and a CREW project to develop a method to estimate erosion risk. The James Hutton Institute developed a new method to separate and analyse microplastics in environmental sediments. This has implications for knowledge development and policy on plastics in the environment since it should enable cheaper and more efficient analysis of samples.

Contribution to NetZero

Research carried out by ClimateXchange (CXC) fed into the Scottish Government's Draft Public Engagement Strategy for Climate Change^[4], the Heat in Buildings Strategy^[5] and the Hydrogen Action Plan^[6]. CXC research also supported the implementation of the Energy Efficient Scotland Programme. Work by CXC, SRUC, and the James Hutton Institute developed a framework methodology to assess greenhouse gas emissions intensity data on Scottish beef farms^[7]. Implementing the framework will help inform decision making related to delivery of the Scottish Government commitment to decrease the carbon footprint of food consumed.

Contribution to Policy

Research has informed policy making in many areas such as peatland restoration, flood prevention, greenhouse gases, coastal erosion and land management. Specific examples include research feeding into the creation of local coastal management plans, and outputs that help farm systems to become carbon neutral. Other examples include work led by Scotland's Rural College (SRUC) which informed the National Island Plan^[8]. SRUC also produced updates on EU-exit and food security, which were classed as useful intelligence by the Scottish Government. The Centre of Expertise on Animal Disease Outbreaks (EPIC) undertook several peer reviews which provided material to support decision making relating to the avian influenza outbreak.

What will we do next?

The evaluation also identified recommendations for improvements. These included considering longer-term funding to secure project resources and avoid loss of research staff, who may look for jobs elsewhere under yearly funding cycles. Strong relationship building practices should be embedded in project planning, since interviewees saw pre-established relationships as vital for delivering quality science. Project reporting should be revised to increase flexibility. The Programme should continue to be promoted through SEFARI so it gains its own identify and reputation beyond those of the institutes.