Environmental, natural, resources and agriculture research programme 2022–27: commissioning evaluation

Executive Summary

Introduction

The Scottish Government (SG) Rural and Environment Science and Analytical Services Division (RESAS) will invest almost £50 million a year over the next five years in scientific research in line with its Strategy for Environment, Natural Resources and Agriculture (ENRA) Research for 2022–2027. The Strategy was published in March 2021 and the research programme launched in April 2022. Its main goal is to support policymaking within the SG with robust and relevant research and evidence. A wider intention is to enhance the continuity of long-term research programmes and science facilities.

The focus of this evaluation was the Programme Commissioning Process (PCP), which is the journey from Strategy publication to the research programme launch and which has involved six main steps:

1. Creation of research questions and strategy themes;

2. Contract creation and contract types;

3. Invitation to tender;

4. Peer review process;

5. Principal Investigators appointment;

6. Project start up.

Evaluation aims

The overarching aim of this evaluation was to explore and understand what has worked well and what has not worked well, what improvements could be made, and overall perceptions of the Programme Commissioning Process.

More specifically, the evaluation focused on the successes, lessons learned, and perceptions of stakeholders involved in different stages of the following types of projects within the ENRA PCP: Strategic Research Programme (SRP), Underpinning National Capacity (UNC), and Centres of Expertise (CoE).

Methodology

The evaluation consisted of three key stages:

1. evaluation scoping which involved informative discussions with the SG team and a rapid evidence assessment of the PCP documentation;

2. fieldwork comprising of 35 in-depth interviews, of c. 60 minutes long, were completed with ENRA research programme applicants, peer reviewers and the SG internal topic and policy leads;

3. analysis and reporting which includes comprehensive qualitative analysis of conducted interviews and producing a draft and a final report.

The breakdown of completed interviews is as follows:

• 5 x MRP co-ordinators (SRP and UNC)
• 7 x PIs (SRP and UNC)
• 4 x CoE co-ordinators
• 3 x CoE PIs
• 6 x Peer reviewers
• 5 x SG internal topic leads
• 3 x SG policy lead
• 2 x Unsuccessful/replaced applicants

Key findings

Overall, stakeholders think the research questions and themes are comprehensive and address Scottish National Outcomes. The SG internal topic and policy leads appreciated the collaborative nature of the development of research questions and themes. It was acknowledged that sometimes policies can change too quickly to be addressed with long-term research funded under programmes such as the ENRA research programme.

The Invitation to Tender process had its benefits, as well as areas identified for improvement. For example, the guidance and the provided templates were exhaustive and clear which was helpful for proposal preparation. On the other hand, applicants mentioned challenges around timings and the amount of workload that proposal writing generated for them. However, they regard such detailed requests for proposals reasonable, considering value and length of research.

Peer review process resulted in thorough and valuable feedback for applicants. The guidance for peer reviewers was perceived as comprehensive and clear which was well received. Nevertheless, peer reviewers noted they would prefer to have more continuous communication with the SG, especially around the next steps following their work. Applicants experienced challenges over the amount of time allowed to respond to feedback to their proposals, particularly when they had to co-ordinate input of partner organisations. Most stakeholders acknowledged that the timings were likely more challenging due to the impact of the COVID-19 pandemic which caused delays in various areas of their work.

PI appointment is perceived as being a relatively smooth process. Stakeholders largely welcome the change of PIs responsibilities in the ENRA research programme compared to the previous research programme. A minority of applicants mentioned that the change generated notably higher workload for them.

The project start-up process is largely described as good, with notable improvements to the reporting system from the previous research programme. A few applicants noted that other research programmes, for example those that are UKRI-funded, have simpler finance processes which makes a project kick-off overall more effective.

Stakeholders generally found that the ENRA PCP is a well organised and standardised process, with high levels of clarity, with the exception of timings, which could be improved in future research programmes.

Conclusions

This process evaluation revealed that interviewed stakeholders overall have a positive outlook on the ENRA PCP.

Research themes and questions are perceived as comprehensive and well thought through.

The ITGF process involved clear and detailed guidance for ENRA research programme applicants which was well received.

The peer review process generated thorough and helpful feedback for applicants, and peer reviewers received an extensive information pack – for some it was perhaps too extensive.

The appointment process for PIs ran smoothly, and greater responsibilities for PIs within ENRA research programme are generally perceived as reasonable, with role details and guidance regarded as clear.

The project start-up stage is described as good by most stakeholders, with direct engagement between project teams and the SG advisers seen as one of the biggest advantages of the ENRA research programme, even in comparison to other programmes, for example those that are UKRI-funded.

Several areas for improvement are identified for the ENRA PCP. The overarching issue for stakeholders was time. Most of them wished for clearer defined timeframes, a more streamlined ITGF process, and greater time allowed particularly for applicants' response to feedback to their proposals. Some applicants suggested improvements around project finances and payment procedures to reduce the burden on project teams, including streamlining finances to avoid multiplied invoicing.

Recommendations

Upon completion of the ENRA PCP evaluation, we recommend that the SG considers the following in order to increase effectiveness of a future PCP, in summary:

• timely appointment of a project manager who will oversee a PCP planning and roll out, including setting out clear roles, responsibilities and timelines for each activity and stakeholder, with generous contingency time calculated in the timeframe;
• more time allowed for the peer review process and applicants' response to feedback;
• a more succinct information pack for peer reviewers with stronger engagement with them to ensure future collaborations;
• continue with the generally-successfully implemented ITGF and PI appointment process.