Rural affairs, food and environment research programme 2016 to 2022: evaluation report

5. Conclusions and Recommendations

5.1. Overview

The ambition of the Research Strategy was for the Programme to deliver "research that is relevant, respected and responsive to Scotland's communities, its people and to the rural economy in 2021". Whilst the original timetable was extended to 2022 due to the Covid-19 pandemic, this evaluation suggests that the Programme overall has achieved outputs, outcomes and impacts that are consistent with the vision. Section 5 provides conclusions for each of the evaluation's objectives, as well as some recommendations for future programmes.

5.2. Objective 1: programme inputs, outputs, delivery and outcomes

The 2016-22 Programme allocated funding across the SRP, Underpinning Capacity, CoEs, knowledge sharing through SEFARI gateway and to supporting innovation. It translated £279 million of funding into 1,244 policy outputs, 2,674 peer-reviewed publications and 160.5 full-time funded PhDs (with additional PhDs partially supported). All three Programme themes (natural assets; productive and sustainable land management and rural economies; and food, health and wellbeing) were represented in the outputs. One science advisor within RESAS praised outputs for being "really good and translatable, which is really good for policy", whilst the Programme was lauded for its "long term focus" which helped lead to "a large amount of influential papers". Interviewees noted that findings were disseminated to the wider research community through high profile academic journals whereas a range of approaches including institution websites, information events (such as workshops), social media and the SEFARI Gateway were used to inform the general public. Furthermore, they highlighted instances where research had been taken up by others, such as for example, research on barley has been picked up by the Scottish whisky industry. Importantly, some of the research findings have also fed into policy outcomes, with research on climate change informing the Scottish Government's climate change plan. Research resulted in 607 publications for trade, with a majority of these related to the Programme's sub-theme of livestock production, health, welfare & disease control. Examples included an article by Moredun on "Sheep scab diagnostics: what's new?" in the magazine Sheep Farmer, and a piece by SRUC on "New agri-policy must protect pollinators" in The Scottish Farmer.

Programme delivery supported 1,891 staff, ranging from 354 in 2016/17 to 267 in 2021/22. Scottish Government funding was complemented with £100 million of funding from non-industry sources including research councils, the EU (through various programmes) and Defra. MRPs also brought income through undertaking consultancy work for industry and through licensing of intellectual property. For example, Hutton managed 12 contracts relating to the institute's plant breeding, germination and propagation work, demonstrating how research directly feeds into the needs of industry.

5.3. Objective 2: impact to Scotland from the research undertaken

The Programme has had economic, environmental and social impacts for Scotland. Research with economic impacts includes, for example, work undertaken by EPIC, to create a computational tool that better assesses disease risks in populations. The tool can be used to improve responses to disease outbreaks in livestock. Environmental benefits are expected, for example, from CREW's guide for farmers and land managers to help them create sustainable drainage systems, to limit the impacts of diffuse pollution following fertiliser application or chemical spraying. Collaborative work between SEFARI organisations and others on the species associated with oak trees is informing woodland biodiversity conservation and so contributing to the management of natural assets. Social benefits are expected from SRUC's analysis of Scotland's first rural mental health survey, which fed into the Scottish Government's Mental Health, Isolation and Suicide Strategies, as well as influencing the formation of a new National Rural Mental Health Forum.

Net zero

The net zero contribution of the Programme is strongly linked to work undertaken by CXC, given the CoE's focus on climate. CXC work has fed into the Heat in Buildings Strategy; the Energy Strategy; Scotland's Energy Efficiency Programme; the Heat Network Delivery Plan; the Clean Air Zone policy; the 2018 Climate Change Plan; and Net Zero Nation, the public engagement strategy for climate change. CXC was also involved in producing a Scottish Transport and Air Pollution model. This can be used to analyse alternative transport scenarios and so determine which approaches best meet goals relating to reducing emissions and improving air quality. The CoE has also guided and evaluated the Scottish Government's substantial peatland restoration efforts, with potential benefits for CO₂ sequestration.

Community benefits

Community benefits are likely to result from research in areas such as flooding, energy and adaptation. For example, CXC helped to formulate and implement the Scottish Government's property flood resilience action plan, assisting those in flood risk areas. The farming community is expected to benefit from EPIC's work on disease preparedness and associated knowledge exchange events, as well as PHCs' research on potato cyst nematode control, which led to recommendations for industry and research. The education community benefited from the SEFARI Gateway providing online educational resources for teachers and home-schooling parents during the Covid-19 pandemic.

Policy contribution

Work funded under the Programme has had clear policy impacts. Examples provided during interviews showed how research has informed policy making in areas such as peatland restoration, flood prevention, greenhouse gases, and coastal erosion. Research also covered the monitoring of sewage to gauge the amount of viral Covid-19 matter and so determine the spread of Covid-19 in the community, with direct implications for policy. MRP stakeholders were often reticent to directly link their work to specific policy changes, but two interviewees did point to specific work undertaken on rural depopulation, which fed into the creation of the National Island Plan by the Scottish Government's Island Policy team.

Scientific benefits

Research has resulted in scientific benefits, with advances in methods and techniques covering all three of the Programme's themes. For example, work under the programme has led to the development of an innovative method for monitoring and conserving genetic diversity. Research on integrated pest management could help reduce pesticide use, whilst work on pre-natal food preference has potential health benefits.

5.4. Objective 3: the value that Scotland, and the Scottish Government have gained from the research undertaken

The impact of the Programme has been estimated at £470 million to £680 million (£2022) based on monetising eight different impacts covering:

• Gross value added generated from jobs (£100 million): this includes 1,891 directly employed by the Programme, as well as indirect jobs (in the supply chain) and induced jobs (jobs related to income spent by others);
• Gross value added generated from spin-outs (£890,000): capturing the potential benefits of an assumed three spin-outs generated from the Programme (annual reports show at least one is expected from Hutton);
• Income generated from intellectual property relating to licensing (£4.6 million): covering work by Moredun and Hutton (e.g. for raspberry varieties);
• Economic benefits from reduced impacts of animal diseases (£9.3 million): based on continuing work on sheep scab, building on the research carried out under the 2011-16 programme;
• Environmental benefits of avoided carbon emissions (£13 million): based on potential carbon savings from dietary changes that could result from a Food Swap tool developed by Moredun and BioSS. Other projects could result in additional savings;
• Social value of having a secure job (£24 million): the social benefits to the 1,891 who are directly employed by the Programme;
• Social benefits of having new skills (£18 million): this captures the benefits to the exchequer of individuals having gained a qualification and the increased tax receipts over time; and
• Social return on investment (£290 million to £510 million): public research may generate an annual rate of return on investment of between 20% and 50%. The lower and mid-rates (35%) are applied to avoid overestimating the benefits given that several benefits are estimated separately.

All benefits expected in the future have been discounted during the Treasury Green Book standard rate of 3.5%, whilst all monetary values have been uprated to £2022 using the GDP deflator from the Office for National Statistics.

The estimated benefits show that the value resulting from the Programme is greater than the Scottish Government's investment (£279 million), therefore demonstrating that the Programme is cost-beneficial. The full value of the Programme is likely to be even greater than the figure presented here, since the estimate does not capture all impacts. Some impacts will only become clear over time as research is taken up and implemented outside the Programme.

5.6. Objective 4: appraisal on the Programme delivery and performance of both the MRPs and the Scottish Government

Some elements of Programme delivery were praised by interviewees, whilst some parts were acknowledged as having potentially negative impacts. Interviewees from MRPs, CoEs and ARPs were keen to highlight several positives such as the Programme's encouragement of co-working between MRPs instead of making them compete for funding, and the way the long-term nature of the Programme allows for more strategic research. They also commented that the Programme was sufficiently broad to accommodate a wide area of research. Furthermore, they found that the Programme encouraged direct communication with policymakers.

However, it was suggested that there could have been further harmonisation between the Programme's themes, and that reporting requirements were too arduous, especially at the beginning of the Programme, with the reporting requirements felt to have improved over time, and into the 2022-27 (current) Programme, with the deployment of ResearchFish. The importance of personal relationships was noted by the majority of stakeholders but was also acknowledged to be a potential weakness or issue. It was commented that a lack of pre-existing e personal relationships made it difficult to work effectively participating institutes in some instances. The importance of such relationships became more pronounced during the Covid-19 pandemic, with the switch to remote working.

MRP, CoE and ARP stakeholders believed there was sufficient money to achieve their objectives, but most suggested that more money would have been beneficial to enabling better scientific research. It was noted by some interviewees from these stakeholder groups that they spent more time on administration than they would have liked, with this taking money away from scientific research. The performance of the Scottish Government was broadly praised, especially during the Covid-19 pandemic where the accessibility of communications with RESAS staff was seen as excellent. This is despite Scottish Government interviewees noting that there was a lack of staff within the team for a broad duration of the Programme. The delivery of the extension year, during a time of hitherto unheard-of uncertainty due to the pandemic, was universally praised by interviewed stakeholders, especially for helping to prevent staff lay-offs and ensuring the continuation of research.

5.5. Objective 5: comparative assessment against the Strategy

This section of the report presents the conclusions on the extent to which the Programme achieved what was set out, as per the principles in the vision.

Principle from vision: Demonstrating increased levels of collaboration with researchers from other institutions through leverage of our investment in research

Conclusions

Collaborative working was apparent: there were 30 collaborations between CoEs; 979 collaborations between CoEs and other types of organisation; and 2,329 between MRPs and other types of organisation.

Interviewees reported being encouraged to work together, with collaborations between scientists resulting in those in director roles also working together. Collaborations enabled research to leave the lab and connect with industry. However, some interviewees raised concerns about the Programme not being as cross institutional as previous ones, and that collaborations depended on individual relationships rather than the Programme itself.

Principle from vision: Having a Strategic Research Programme, which has interdisciplinarity at its core and has a single clear identity that is recognised nationally and internationally for its excellent science

Conclusions

Programme data show a limited level of interdisciplinary working: 691 policy outputs were linked to at least one of the Programme's three themes, of which 85% (588 outputs) were related to one theme, 12% (82) linked to two themes and 2% (21) covered all three themes.

Interviewee feedback reinforces the view that there was interdisciplinary working, but there was a desire for more opportunities to bring complementary areas of expertise together.

Interviewee views on programme identity were mixed. Four explicitly commented that the Programme had a good reputation, with another highlighting the positive role of SEFARI in bringing the different institutes together in one portal. Others thought that stakeholders had greater awareness of the individual institutions than the Programme, therefore evidencing a limited visibility of the Programme.

Principle from vision: Ensuring we demonstrate the impact of our research undertaken on the communities, businesses, public sector and the economy of Scotland in a clear and measurable way

Conclusions

Interviewees were particularly positive about the SEFARI Gateway, which allowed effective knowledge exchange. It was viewed as a place to share breakthroughs, and to bring different areas of research together to tell a story and inform policy.

Individual projects also undertook their own knowledge exchange and dissemination. For example, Moredun Institute undertook work to analyse the impact of medicated grit on the health of red grouse. This project led to dialogue between grouse moor owners, managers, and gamekeepers that resulted in many more Scottish Estates improving their practices and therefore creating a more sustainable sector.

Principle from vision: Making the significant data holdings we support through our funding more visible and accessible

Conclusions

Interviewees indicated that underpinning capacity was not promoted as strongly as it could have been. However, Programme documentation did provide evidence of data sets being used by stakeholders outside of the Programme. For example, Hutton responded to two requests for samples from the National Soils Archive in 2012/22, whilst Moredun's collection of pathogens and linked materials continued to be used throughout the Programme.

Principle from vision: Having evidence of increasing innovation activity associated with the Programme with a range of non-commercial and commercial funders

Conclusions

Innovation activity is assumed to capture research collaborations and the involvement of organisations outside the Programme in funding, contributing to, and/or taking up research. Industry investment brought in over £70 million as companies and others took up innovations developed under the Programme. An illustrative example of this is the James Hutton Institute obtaining royalty income from breeding soft fruit cultivars.

The Programme also attracted an additional £100 million from non-commercial funders including the EU, research councils and Defra, showing interest from those outside the Scottish Government in the research and development being advanced by the Programme.

Principle from vision: Creating Centres of Expertise at points of significant demand in the system for the translation of scientific understanding from across Scotland into solutions to critical questions that will emerge over time.

Conclusions

The Plant Health Centre was set up in 2018 to improve resilience to plant health threats in Scotland by connecting science to application to inform policy, planning, responses and solutions. Five PHC projects directly informed policy within the CoE's first 18 months, showing a demand for the research. PHC commissioned work included a project to assess critical biosecurity risks to Scotland from non-specialist and online horticultural sales. The research resulted in recommendations for this important but understudied and difficult to reach sector.

Considering the extent to which the Programme has helped deal with policy challenges, MRP interviewees tended to be of the opinion that the Programme had funded the right balance of topics across its three themes. However, it was also highlighted that initial priorities became less relevant as issues arose in emerging areas. This is perhaps a natural result of setting a five year strategy, although one interviewee did comment that research priorities would be adapted over time.

The Strategy additionally included three national priorities. There is evidence to suggest that research contributed to all three, yet to different extents:

• Valuing and protecting the natural environment: since 36% of scientific outputs produced related to the natural assets theme, this suggests around one third of research was relevant to this outcome;
• Reducing the local and global impact of consumption and production: the food, health and wellbeing theme was only responsible for 14% of scientific outputs produced, suggesting that whilst work on consumption did occur, it was less common that other topics. Research on production was, however, more common, with 50% of scientific outputs linked to the land management theme; and
• Being better educated, skilled and successful, and renowned for research: the Programme directly contributed towards education and skills through funding PhDs. Continual representation by researchers at both government and non-government advisory groups throughout the Programme demonstrated the reputation of researchers and their work.

5.7. Recommendations

Throughout interviews with stakeholders from the Scottish Government, MRPs, CoEs and ARPs, a number of comments were provided that the study team felt value in capturing for recommendations. These are provided in the subsections below. Key recommendations are those relating to reviewing the possibility for longer term funding, ensuring strong relationship building, changing the approach to project reporting and working on publicising the identity of the Programme itself (rather than individual projects or institutions).

5.7.1. Length of funding cycle

Interviewees noted that funding provided for short terms resulted in employment insecurities for researchers. This resulted in researchers looking for more secure employment elsewhere, and ultimately loss of key personnel.

Recommendation

• The Scottish Government should review the possibility for longer term funding (i.e. greater than one year) to ensure continuity for projects and for staff.

5.7.2. Relationships

Interviewees identified a key reliance upon effective relationships across institutions and work package teams. They identified difficulties where relationships were not productive, and/or where key staff left the research team (causing disruption). Related to this, interviewees highlighted benefits of a co-productive approach to research.

Recommendation

• The Scottish Government should encourage strong relationship building practices as part of project planning, and ensure applicants detail a risk mitigation process for staff losses during application process.
• MRPs/CoEs should ensure project teams provide adequate handover of project work when staff members change to enable replacement staff to fulfil the role effectively.
• The Scottish Government should emphasise the benefits of co-production and support collaboration in future research Programmes.

5.7.3. Reporting

Interviewees believed monthly reporting was too frequent for strategic research. Feedback suggested the use of Microsoft Excel spreadsheets required a substantial amount of time to fill in and focussed on granular level data where impacts may not have been achieved. Alternative methods as used in the 2022-27 Programme (including ResearchFish) were felt to be more effective.

Annual reporting and mid-point review effectively communicated summary impacts from research projects. The Scottish Government should avoid the use of monthly monitoring and reporting requirements, especially when focusing on granular levels.

Recommendation

• The Scottish Government should avoid the use of Microsoft Excel spreadsheets as a monitoring return for large scale strategic research.
• The Scottish Government should continue the use of six monthly and annual reporting as interviewees felt this was a suitable timeframe for progress to be monitored.

5.7.4. Programme identity

Some interviewees felt audiences saw individual science projects, or individual institutions rather than the Programme as a whole. Linked to this were interviewee comments suggesting that the research objectives of the 2016-21 Programme were not well defined. The SEFARI group of organisations should continue existing work on publicising the Programme through SEFARI.

Recommendation

• The Scottish Government should ensure Programme objectives, research questions, and subject areas are specific and clearly defined as part of establishing the identity of the Programme.

5.7.5. Application writing and assessment

Interviewees stated that strategic research is often long-term and complex, with applications requiring considerable time to plan and coordinate across research partners. Applications ran in parallel to reporting for the previous (2011-16) Programme. Page limits were seen as constraining and did not discriminate for projects of different size and scope.

Interviewees felt the application process would have benefitted from being synergised with other academic funds (e.g. UKRI). Interviewees felt they were more familiar with academic funding requirements and level of scrutiny.

Interviewees highlighted that applications were submitted through a specific HEI submission portal. Not all application writers had access to this portal.

Interviewees noted staff shortages within the Scottish Government led to additional burdens for science advisors and administrative team. Science advisors often took over work packages that they were not specialised in.

Interviewees also questioned how applications were assessed and whether the assessor had the specialist knowledge required for their application.

Recommendation

• The Scottish Government should consult stakeholders on timeframes for applications. If delays occur, consider providing extensions to ensure applicants have appropriate time to prepare tenders.
• The Scottish Government should consider revising page limits for funding bids whilst ensuring workloads for assessors remain manageable.
• The Scottish Government should review application process and if suitable, align processes with other commonly used HEI funding routes, and/or have an information day.
• The Scottish Government should emphasise the submission process (including the portal needed) in the tender terms of reference, and advise applicants to gain access from relevant HEI partners well in advance of submission.
• The Scottish Government should ensure administrative staff and scientific advisors are adequately resourced to fulfil the assessment of applications and cope with the number of projects within their thematic area.
• The Scottish Government should provide communication with all applicants regarding how their application was assessed, and if relevant, the credentials of the assessor.

5.7.6. Partner performance during projects

Interviewees operating at the research delivery level identified issues with partners not communicating or collaborating effectively to meet reporting deadlines.

Recommendation

• The Scottish Government should ensure Research Deliverable coordinators have more accountability and routes for escalating and resolving issues with partners.
• MRPs and CoEs should consider implementing single points of contact at institutions responsible for chasing of deliverables.

5.7.7. Ad-hoc communications

RD interviewees felt they benefitted from the loose grip approach from the RDF. Ad hoc communications with science advisors were viewed as beneficial.

Recommendation

• The Scottish Government, MRPs and CoEs should ensure open communication lines are maintained across the Programme.

5.7.8. Transparency of communications

One interviewee felt that communications may benefit from more transparency and documentation of discussions, commenting that conversations occurring behind closed doors would prevent efficient project operation.

Recommendation

• All stakeholders should ensure meetings (and their outcomes) are documented and communicated within delivery teams

5.7.9. Project funding

Interviewees noted that funding received was a real terms decrease compared to what was outlined in the original bid due to inflation. This made it more difficult to provide the budgeted quality over time

Recommendation

• The Scottish Government should provide flexibility for projects to adjust deliverables as a consequence of real terms decreases in funding

5.7.10. Extension Year

Interviewees noted that communication about whether extension year would or would not be deployed was muddled and at times quite unclear. Interviewees were cognisant of the extremely unusual situation.

Recommendation

• The Scottish Government should ensure that communications on key issues like extension year(s) are timely and as clear as possible to provide reassurance researchers, and allow them time to plan alternative courses, if necessary.