Environmental, natural, resources and agriculture research programme 2022–27: commissioning evaluation

3. Conclusions

3.1. About this report

The current report provides an overview of findings derived from a process evaluation of the ENRA PCP conducted from September to December 2022. The evaluation comprised 35 in-depth interviews with four different groups of stakeholders: ENRA research programme successful applicants, unsuccessful applicants or those who went through application clarification stages, peer reviewers, and the SG policy and internal topic leads.

The following concluding remarks are based on the analysis of the interviews completed.

3.2. What has worked well in the ENRA PCP

Overall, stakeholders have a positive outlook on the ENRA PCP. Starting with its first step, development of research themes and questions, participants largely think that these are comprehensive, and they aim to achieve Scottish national outcomes. ENRA research programme applicants find the research questions clear which they found helpful for their proposal preparation.

The Invitation to grant funding (ITGF) step and its evaluation involved reviewing and discussing guidance documents for applicants. ITGF documents are described as detailed and clear, and providing a proposal and project plan templates which present a constructive steer for applicants.

The peer review process, including guidance for peer reviewers is broadly perceived as a well-planned standardised process. Peer reviewers received detailed instructions from several different sources, for example ITGF documents, as well as separate email communications and documents published and continuously updated on the Objective Connect platform. Most of ENRA research programme applicants found feedback by peer reviewers thorough and valuable, which enabled them to strengthen their proposal, and even opened additional avenues of thinking about their research themes and research questions.

The PI appointment process is regarded positively by applicants and the SG policy and internal topic leads, with whom we discussed this aspect of the PCP. They mention that guidance for the PIs, including the outline of their responsibilities and expectations in terms of their relationship with the SG was exhaustive and clear. It is recognised that PIs' responsibilities are greater in the current ENRA research programme compared to the previous one. However, applicants broadly feel that this increase aligns to the PIs' role and works well for projects overall. Appointment of PIs was largely smooth within research organisations and it did not generate an excessive amount of work for them, which was well received.

Project start-up broadly received positive comments from successful applicants and the SG policy and internal topic leads. The process is seen as being clearly established. It is tied to reporting which is, notably, considered to have been improved in the current ENRA research programme compared to the previous one. Successful applicants praised use of enhanced reporting tools, including ResearchFish and simplified Excel spreadsheets, which made project monitoring more effective.

Stakeholders who compared the ENRA research programme to other research programme commissioning processes largely say that the ENRA PCP is a very standardised process which for them resulted in a similar experience when applying for funding and commissioning it. Some successful applicants commend direct engagement and close collaboration with the SG advisers, which is seen as one of the ENRA research programme's greatest advantages. This communication is important for project progress and is motivating for project teams because they receive continuous feedback and guidance on the impact of their work.

3.3. What can be improved within the ENRA PCP

In each of the PCP steps, there are some aspects stakeholders would like to see improved in future. For example, the SG policy and internal topic leads mostly commented on the research themes and questions developments. Some of them felt that this process could have been more efficient had the SG had more staff at that time. An additional suggestion is that there should be cross SG agreement on what the research gaps are, which areas can be feasibly addressed, and which groups would benefit from the funded research.

The main suggestion for improvement across several different areas, including the ITGF stage, is around timescales. Stakeholders – applicants and the SG policy and internal topic leads mentioned that timeframes presented a challenge for coordinating their work and achieving collaborations between partner organisations. It is considered an appointed project management resource could resolve some of these issues. From the SG perspective, some preparations for ITGF were a bit late or timed poorly, mainly due to the lack of staff, or new members of staff employed too late for effective ITGF planning and roll out. Applicants felt that the ITGF stage was time consuming. It would have helped if it was more focused and streamlined.

The peer review process is another PCP step in which stakeholders would have benefited from better timeframes, they say. Applicants contend that they did not have a sufficient amount of time to respond properly to feedback received, which caused stress across their and partner organisations' teams. Some SG stakeholders found it challenging to communicate effectively with peer reviewers and ensure seamless running of the process. Peer reviewers felt similarly with regard to communication with the SG. An overarching observation is that communication between all parties involved, including details of the peer review process and scheduled timeframes could be clearer in future.

The only negative comment in relation to the PI appointment process is that some PIs found the assigned responsibilities overwhelming. This was perhaps new to them considering change from the previous research programme. However, these remarks represent a minority of comments received to date.

In terms of project start-up, there were a couple of suggestions for improvements, mainly around financing and organising payments, particularly in cases where several research organisations are delivering a single project. In this context, a few applicants referred to UKRI funded research programmes as having simpler payment procedures. A couple of applicants also mentioned they would have found more engagement with the SG advisers helpful, specifically at the project start-up stage, when projects are being set up, and key points discussed and agreed.