Environmental, natural, resources and agriculture research programme 2022–27: commissioning evaluation

2. Key Findings

2.1. Overview

This section presents main findings of the process evaluation of the ENRA PCP. Conducted interviews broadly followed the aforementioned six steps of the PCP, and findings are discussed in that order, which is outlined below:

We completed 35 interviews, with the breakdown as follows:

• 5 x MRP co-ordinators (SRP and UNC)
• 7 x PIs (SRP and UNC)
• 4 x CoE co-ordinators
• 3 x CoE PIs
• 6 x Peer reviewers
• 5 x SG internal topic leads
• 3 x SG policy lead
• 2 x Unsuccessful/replaced applicants

2.2. Creation of research questions and themes

The process of developing the questions and themes started with engaging the SG policy team RESAS supports and asking for their priorities and what is needed over the programmes five year duration. These were collated by those who were to become topic leads. The policy team input was converted centrally into a coordinated group of topics and themes, or priority areas.

The interviewed stakeholders that had a hand in developing the research questions were asked whether they believe the questions to be sufficiently comprehensive. Responses were split amongst those that said yes generally (three participants, 38% of SG stakeholders), and those that noted they were not sure (two, 25% of SG stakeholders). In part, this was because the ones who were unsure were not closely involved in the development (having come in late, or being more administrative in their job role), and in part this was because some of the topic leads felt determining research questions is not their area of expertise but that it seems to have worked out well.

Some concerns (three participants, 38%) with how comprehensive the questions are given the rapid evolution of policy, which changes so quickly that sometimes questions do not remain as relevant as they might have been when they were set up. Additionally, the science advisory team were engaged with arranging the research needs alongside the policy team almost a year and a half prior to the development of the research questions, with no time to reassess research areas, which might have given rise to some gaps in the needs/research questions as well.

Overall, there seemed to be a consensus among internal topic leads that the research questions were broad enough that they remained relevant and comprehensive. In addition to the questions themselves being appropriate in the eyes of the topic leads, when asked what worked particularly well, two things stood out:

1. The development of the research questions involved a lot of collaboration, with meetings to discuss the best way to phrase a question.

2. In the previous research programme, as indicated by SG stakeholders, the research questions were not developed by the SG internal topic and policy leads. This change was welcomed by SG stakeholders, however there were no comments as to who developed the research questions.

Collaboration between the policy and the science and economic advisers was mentioned by two stakeholders (25% of SG stakeholders) as being beneficial to the process and that it worked well – though it was noted that policy team members do not have a lot of capacity to engage with the programme. One of them pointed out that they thought the team looked to the right people to be involved in the process, the ones best suited to discuss the topics.

There were some comments that highlighted aspects of the research question development process that could be improved in future. One suggestion (13% of SG stakeholders) is that there should be cross SG agreement on what the research gaps are, and an understanding of which can feasibly be filled with this fund, with a clear sight on the biggest impact on different groups if the research is funded. Additionally, one stakeholder sought more clarity, for example, on what other bodies and organisations are doing so that the research questions developed for the ENRA research programme can be best placed to utilise the funding the SG has available for the research.

One (13%) SG topic lead suggested that in future, they would prefer to have more reassurance and support on the actual research questions, as the creation of research questions was not a part of their daily duties, nor something they had a lot of experience with. In order to combat that lack of experience, the stakeholder worked closely with colleagues in policy to ensure the questions were phrased as well as they could be. However, they had concerns that to be as robust as possible, they would need further support in this area in future.

There were concerns about the time taken to develop the questions. As already mentioned, the size of the programme and project means that policy can change, but individual topic leads felt that the time they were given to complete the tasks was not adequate to ensure the consistency and thoroughness they would have liked. They lacked time to revisit colleagues' needs, one participant said, and to ensure that the overlaps were examined properly.

The consensus from the successful applicants is that the themes are good, and familiar seeing as they suggest the themes have not changed much over the years. On from those, the sub-themes and research questions were also well received in most areas. It is worth noting that the unsuccessful applicant we spoke to also thought the questions and themes were well thought out and are comprehensive enough.

However, one stakeholder (5% of successful applicants) suggested that the research questions are not comprehensive enough. That was due to their desire to know more details and with better clarity. Another said that there are one or two areas, potentially involving health or aquaculture in which the questions could have gone further – though the respondent said that they appreciated that aquaculture is not under RESAS's remit.

2.3. Invitation to Tender

Overall, it seemed that the invitation to tender process of the ENRA PCP was something that stakeholders thought worked well and served its purpose. However, in almost every respondent group there was the opinion that more time could have been given over to the tendering portion of the commissioning process. There were notable changes to the ITT process in this programme compared to previous rounds that for the most part have worked well.

For example, one (13%) internal topic lead believes that the high level of collaboration between stakeholders to discuss the ITT was beneficial, and another mentioned that the approach to setting out the research questions was good. Internal stakeholders were pleased that despite internal resource limitations, they sent an ITT out on time. One highlighted that this process focused more on data management, for example, which they indicated was a "sign of the times" (Successful bidder, Pye Tait, 2022). Additionally, splitting the programme by the new themes has been received well by stakeholders. However, there are some cases where stakeholders wished it was more similar to the previous research programme, especially regarding face-to-face meetings.

Interviewed topic leads that had a hand in the ITT were particularly critical of the planning and time given over to this segment of the process. Short timescales made the invitation to tender difficult to execute – though they all acknowledge that they did manage to execute it on time. Every topic lead that mentioned working with colleagues and expert advisers said that it was valuable to have their insight.

A challenge for the topic leads was that the internal human resources allocated to the commissioning process were not high enough. There were big gaps in scientific knowledge, which made the task harder than it could have been. But despite these challenges, the topic leads reported that in general it seemed to work.

Overall, more than half (55%) the successful applicants believe that the ITT instructions and/or requirements were clear. Several mentioned there were cases where clarification had to be sought, or that interpretation was required, or it was obvious that they had interpreted it differently due to viewing the world through a different lens (seven, 37%).

Of those who noted instructions were clear, two (11%) also mentioned that the pricing proposals' requirements were the most unclear or needed to be queried, and almost a quarter reported that there were errors in the instructions that were corrected eventually. In those financial sheets, one stakeholder (5% of all applicants) asked that the SG be more consistent in the templates and remove merged cells as this makes it difficult to use the spreadsheet.

Five (24%) applicants said that the instructions were unclear, with three (14%) suggesting the instructions had been rushed and should have been checked for internal consistency before being released. Some (four, 19%) reported the wording was confusing in places, and one (5%) noted that having to read the "verbiage" was a chore, though another balanced that by suggesting that the SG is generally good at using plain English compared to the EU-funded programmes instructions, which use a lot of EU-specific jargon.

"We struggled on the clarity of setting the boundaries for the resources to scale the ambition of the research effort to the resources available. For a set of research questions there was not a block amount of money available. In some sense working with a movable set of goalposts in terms of designing the research effort to the number of resources available. Only at later stage did we get firmer figures and still some reshuffles even towards the end- again I understand why this may be, but having a target resource amount to aim for to structure workplans would have been helpful."

(Successful applicant, Pye Tait, 2022)

Stakeholders thought the feedback they received was good, overall, but one (5% of all participants) mentioned that more detailed feedback would be useful – particularly, and even, when there was little to change as positive feedback is just as useful as constructive criticism. Furthermore, when receiving feedback and being asked to make changes, participants referenced too short turnaround times, despite it taking a long time for proposals to come back with comments.

"I think the times were a bit slow, we put together our proposal and it took a long time to come back, and they came back with request for changes with a small timeframe and the requirements were quite large and we needed to contact institutions that were not originally involved in our proposals. We probably had two to four weeks to make changes and you don't know people's availability, and plans, like holidays, family commitments, sick leave etc."

(Replaced proposal applicant, Pye Tait, 2022)

Overall, better communication from the SG team was desired by applicants, as there were comments that it was difficult to determine what decisions had been made across the whole programme. Helping the institutions and applicants understand what has been commissioned across the whole programme would be useful, especially now that the programme has been split into project-based work. One stakeholder (5% of all applicants) added that further clarity on how budgeting was going to be divided across themes would have been useful, as well.

There was at least one (5%) applicant who suggested the CoE tendering process was difficult, with "painful unfamiliar terrain" to navigate leading to an "all consuming" amount of work. Additionally, the changes to the system for CoEs meant that the changes were mostly of legal nature, leading to a shift in process that means it is harder for consortia to work together.

When asked what kind of improvements could be made to the invitation to tender process, four (19%) applicants said that they were not sure, and one (5%) noted that the process was possibly as good as it will get. However, others who commented referred to timelines involved, but worth pointing out is the specific example that with the process ending with submission in July, the start of the Scottish school summer holidays, there were a lot of times that key members of staff were no longer available. Shifting the time back a month to start in March and finish in June would have been better.

None of the applicants, successful or otherwise, mentioned they were not planning to apply for future SG programmes of research (other than one who noted they were no longer going to be working in this sector).

2.4. Peer Review Process

ENRA research programme applicants, the SG topic and policy leads, and peer reviewers were asked about their opinions on the overall peer review process. Specifically, they discussed what has worked well, what could be improved, how did it impact final outputs and if the peer review guidance provided to each type of stakeholder was adequate.

Over half of successful applicants (ten, 53%) noted they did not feel there was enough detail within the feedback specifically, with two (11%) mentioning that some projects received none at all. They also stated that there was no provided indication of time frames or when they should expect to receive the review report, which was problematic for planning of their other commitments. A couple (25%) topic/policy leads within the SG followed this up and stated that the process felt rushed and under-resourced, with too few reviewers involved.

Despite these comments, the majority (11, 58%) of the interviewed successful applicants commented that the overall peer review process was fair and reasonable. Approximately half (nine) noted that the feedback did help their application and project, although most (ten, 53% of successful applicants) stated that this was minimal in terms of clarifications of wording, wider context and mobilisation, insight into the questions and responses to the research themes. A handful (five, 26%) noted that whilst the feedback received was appropriate, they still felt there was more content, and detail, needed. Two (11%) commented that the peer review reports helped them improve their approach and develop stronger projects.

"Approximately 40-45% of projects did not receive any [substantial] feedback; they only received an email that said 'this is acceptable'…also the timing was a problem; they went out to review in August so many of the reviewers that weren't based in Scotland would have been on leave – this was not ideal."

(Successful applicant, Pye Tait, 2022)

With regard to what has worked particularly well within the peer review process, a handful of successful applicants (four, 21%) outlined that it helped refine their proposals to a higher standard, and it was a positive element to the programme commissioning process.

Five peer (83%) reviewers felt that the process was smooth, well organised, and concise. A couple (33%) highlighted that the initial template provided to them to undertake the review was very clear and was easy to fill in, although in some cases the word limit imposed was a little low.

One (17%) peer reviewer and one (13%) topic/policy lead noted that technology allowed for a smoother process as it enabled fully remote, online sharing of documents. However, a couple stakeholders (6% of all stakeholders) felt the timings receiving the documents were delayed or inconvenient.

Primary improvement for the peer review process that all stakeholder types noted is the need for better timeframes. Five (26%) of the successful applicants felt the turnaround time for the feedback was uncertain and the deadline confirmed late, such that is caused issues for change implementation.

Some (five, 14%) stakeholders across all groups also believe that there should be more transparency in the process. Both applicants and peer reviewers noted that there should be greater involvement and opportunity for PIs to talk with the reviewers and vice versa in order to collaboratively provide feedback. One (5%) successful applicant also wished for verbal feedback options where possible.

One (17%) peer reviewer additionally commented that they felt there should be more strict consistency across submissions in terms of the content and how the structures and formats are presented. They feel this would make the role of the reviewer easier. This is furthered by all peer reviewers noting that the work they undertake is particularly time consuming and does not pay a lot per project, with most stating they tend to go over the budgeted timeframe when completing a review.

"We didn't get a chance to talk to other parties – all it is, is being sent the materials, being asked to review and return by a certain date and then invoice. We didn't get any feedback on whether our comments were taken forward at all or had an impact…you can end up feeling a little neglected."

(Peer reviewer, Pye Tait, 2022)

In terms of guidance provided, SG topic/policy leads commented that there was no initial help given and it was very unclear as to what was expected. They feel that as a result the administrative burden and workload were unnecessarily high, and overall the process took too much time to manage. Two (25%) noted that some additional preplanning stages and preparation of up to six months prior to the peer review process could help alleviate this issue. One stakeholder (13%) suggested the introduction of a flow chart to help assess the peer review outcomes and highlight what scores should be awarded would be beneficial.

"From a topic lead perspective, a lot of the role was administrative. It was a massive task and should have had some dedicated central resources for that. A lot of people drafted many emails, but that could have been drafted centrally. Time required for this part of the process was underestimated. Lack of clarity on the role and I don't remember getting guidance."

(SG topic/policy lead, Pye Tait, 2022)

The peer reviewers noted they received a large quantity of guidance, including a large pack of project information and content, forms regarding impartiality and details on the scoring system including a comprehensive template. Two (33%) commented that whilst the guidance was good, there was a lot to go through and was seen as an overload of information. This took a large amount of time to read and as such one feels that there may be ways to tailor the information per peer reviewer.

"They gave comprehensive advice in the form of standard forms of questions and scoring – it clearly highlighted what they wanted, and I had no problem undertaking this. There was appropriate help to understand what was expected of us. You get the documents submitted by the research parties outlining what they want you to do and how much it will cost."

(Peer Reviewer, Pye Tait, 2022)

2.5. Principal Investigator appointment

Stakeholders were asked about their opinions regarding the PI appointment process, with particular focus on what they considered to have worked well and what improvements could be made in future.

The majority (12, 63%) of successful applicants noted that the process of appointing the PI was smooth with little issues or internal competition. They stated that it was common for the lead to either be self-appointed, internally chosen or a natural choice based on current and past roles on ITTs. It was a soft approach with a few indicating a formalised process. Six (60%) PIs confirmed this, and stated they were either the obvious choice from past experiences, or internal discussions were held and agreed upon.

A couple (20% of PIs) commented that the requirements imposed on the PIs regarding their relationship and contact with the SG were effective. They indicated that it is a good idea as the communication aids the understanding of what outcomes are desired from the projects.

Additionally, three (16%) successful applicants indicated that the responsibilities afforded to the PIs have worked especially well as part of the process. They noted the level of responsibilities PIs have in the current ENRA research programme ensures the project meets the needs of the government, is completed within good time, and develops the capabilities of the PI directly. Specifically, two (11%) commented that the responsibility of reporting through ResearchFish has worked particularly well. However, one (5%) feels that it is too much pressure to put the responsibility of the project on one sole PI, rather than a shared responsibility across project leads.

"I think the PI's taking responsibility for their projects and taking responsibility for reporting through the ResearchFish system [is what has worked well]. The ResearchFish system will allow them to highlight more of their research than maybe they cared to do in the in the past programmes."

(Successful applicant, Pye Tait, 2022)

Most successful applicants (13, 68%) believe that the system requires little improvement as they feel the process was overall smooth with no clear obstacles or issues. The only complaints of note were that two (11%) feel that it is difficult for one organisation to hold the primary PI when multiple partnerships are involved. They noted this adds pressure and leads to potential duplication as each partner organisation would have their own data management policies for each project. They think there should be more encouraged cooperation and shared liability to alleviate this. One (13%) topic/policy lead suggested there should be an additional admin role connected to the PI to facilitate the updating and management of documents.

2.6. Project start up

Successful applicants, mainly MRP co-ordinators, hold mixed opinions of the overall project start-up process. However, they mostly (ten, 53%) report that their project start-up was good. They highlighted that despite some changes, the 2022–2027 programme is largely the same as previous programmes.

Four (21%) successful applicants referred to project start-ups meeting whereby they met with the SG advisers. One (5%) explained that, whilst the project start-up process was straightforward within their organisation, this was not the case with the start-up ongoings with RESAS as they experienced legal issues relating to ownership of intellectual property.

In terms of their relationship with the SG, most stakeholders (13, 62% of all applicants) described it as good. Reasons for this include the SG's willingness to answer questions, responsiveness, openness, and patience. One (5% of all applicants) explained that before publishing guidelines, for example on reporting, the SG often asked for comments ahead of sending a final version, which they found beneficial. Others highlighted that whilst sometimes the SG may take a while to answer questions, they feel that the quality of guidance and clarifications provided are high, which they suggested is due to expertise of the staff at the SG. Another stakeholder added that they like the iterative process of developing the research with feedback during this process.

In comparison to the previous research programme, one (5%) applicant explained that collaboration with the SG is much better and suggested that it is perhaps due to an increase in the number of staff at the SG as well as them being more receptive. However, three (14%) would like to see more direct engagement during the project start-up stage. For example, one (5%) mentioned that there could have been better discussions around processes (especially governance mechanisms) ahead of the beginning of the programme. Another feels that there has not been much collaboration with the SG – however, they think that processes are well defined and welcome the use of the retrospective reporting.

Two (40%) MRP co-ordinators reported that improvements need to be made in relation to timings. One (20%) referred to issues in relation to the SG's requirement to hold the project start-up meeting either within the last month of the previous programme or the first month of the new programme, which they believed to be a "a little ambitious".

Further, timings were reported as being tight to work to and requiring a large amount of work to be done in a small amount of time. Some participants (nine, 26% of all stakeholders) commented that the timings for the tendering process could be improved, particularly because they had an impact on the start-up process. A minority (five, 14%) also mentioned delays, with one CoE stakeholder mentioning that they have not yet gone through the project-start up due to delays in the programme.

Staffing issues in combination with challenges around timings were raised by eight participants (23% of all stakeholders). They discussed staff availability and changes to contacts, including staff shortages within the SG, as well as struggles with staff turnover and delays in recruiting within their organisations. One (3%) suggested differences in sizes and number of permanent staff in the research centres should be recognised and considered when setting project timeframes.

Stakeholders noted that the project start-up involved receiving clarity about ResearchFish as it is a new reporting system. Some stakeholders' organisations held in-house meetings and staff attended open briefs run by ResearchFish. One (5% of all applicants) added that YouTube videos could help to provide a clarified demonstration of the information ResearchFish required. Within the context of reporting, another (5%) suggested that greater lead time is needed for talking with the SG about it, although they appreciate it was a learning experience for the SG too.

Multiple (six, 29%) applicants commented on the impact that the project start-up process has upon financing. A few (three, 14%) noted challenges around project finances, including complexity of paying out partner organisations which has multiplied invoicing and accounting procedures. One (5%) explained that because it is not possible to transfer money between years or between projects, budgeting individual projects was the biggest challenge, and that it would be beneficial if the SG could change this. No additional comments on project finances were received from other stakeholders.

"Trying to get the budgeting down to individual projects was horrendous… But having to actually be able to budget right down so that each individual project has a budget assigned to it so they can see where the money goes adds so many layers of restrictions to us that it's actually hard to deliver science. And it made the tender process really quite difficult."

(Successful applicant, Pye Tait, 2022)

2.7. Changes to the ENRA PCP compared to the previous Research Programme

Compared to the previous programme commissioning process, there were noteworthy changes participants commented on. Many of those changes were seen as positive, such as the reporting and monitoring aspect. SG stakeholders and applicants (seven, 24% of all stakeholders) suggested that the monitoring of the reporting was much easier in this programme because of the changes to the process, where the reports are now shorter, project-focused, and produced less frequently than in previous commissions. Applicants (four, 19%) noted that completing the reports for the ENRA research programme is much better than before, as the platform, ResearchFish, is a good system to use compared to the Excel spreadsheet used last time.

However, one (5%) applicant indicated that new reporting styles meant that more overhead was required, as the team had to provide more briefings to get the reports completed. Other applicants did not report this issue. Another (5%), however, said that it was too early to make a judgement on how useful the compartmentalised reporting would be. Nevertheless, there were suggestions that the reporting change is much more reasonable and is a more streamlined way of recording metrics and outputs as it was a much more manual process previously.

Another theme in the changes from previous programmes was the format and flexibility in the applications that an institution could submit. Tying into being able to write across themes, one (5%) applicant mentioned that it was good the flexibility was there to allow the experts in the MRPs to give the SG their best opinion on how a project should be approached. However, another (5%) suggested that RESAS could consider allowing proposals submitted in other formats than the written word, as, for example, video can be an effective tool for communicating ideas.

In terms of the interviewed applicants, there were impressions that the splitting of the programme into individual projects worked well and was an improvement over the previous research programme (four, 19%). This enabled the smaller organisations to have a piece of ownership over their research, rather than being 'deputised' by the larger MRPs, and it made for changes in the reporting that spread the workload out and gave the responsibility to the principal investigators, rather than the coordinators. This means that the reports are written by the person closest to the project work.

One applicant (5%) suggested splitting the programme into projects was difficult and caused problems, particularly with regard to finances, but from responses received it appears to be more of an administrative issue than a structural issue with the tendering process.

Another change between programmes that was mentioned multiple times was the format of meetings. Before, the programmes would usually require face to face meetings between the institutes and the SG, or between institutes. Many (eight, 23% of stakeholders) said that meetings are virtual in the current process. There were mixed opinions on whether this was positive (six applicants, 29%) or negative (three applicants, 14%), with some noting that the lesser impact on the climate and the ease with which a meeting could be joined was a benefit, there were others that stated that the lack of spontaneity in virtual meetings was a downside of the change. On the positive side, one (5%) applicant mentioned that it had enabled conversations with people wider afield, in the United States and Australia, that would not have occurred before. Digital accessibility is something that should be considered, however, when projects are relying on digital tools – can all stakeholders access the digital platforms necessary to perform the research?

In terms of the changes within the tendering process, one of the SRP and UNC applicants (5%) stated that if it had been open competition, it would have been much more difficult, but when CoE applicants were asked about their experience the general opinion appears to be positive. One (5%) CoE applicant mentioned that comparing between the grant process and the open competition process is difficult, but that "it does seem to be better", whilst another suggested that it will not be possible to know for some years if the process was weaker fundamentally. Another stakeholder stated that competition is a good thing, fundamentally, because without it there is little incentive to examine costs and the competition can encourage or trigger innovation in science.

Overall, the CoE stakeholders (seven, 33% of all applicants) seemed to suggest that open competition was fine for them. As with the invitation to grant funding, there were some concerns about the timelines involved with the process. One applicant (5%) thought that it was implemented in too short a timescale which put a lot of pressure on staff. Additionally, some (three, 14%) said the application was large and required a lot of work that had not been accounted for in the timeline given, which comes with a suspicion that there may have been potential applicants that did not apply because of how involved the process is.

Additionally, there were delays in the open competition process that meant there were people uncertain of work contracts for a long period of time, and if the process could be started earlier that would be of benefit.

2.8. ENRA Research Programme Commissioning Process compared to other Research Programmes

Stakeholders discussed best practices they noticed in other research programmes' commissioning processes or within their organisations, which could be implemented into the ENRA PCP. Good communication and engagement techniques were most commonly mentioned. One (5%) successful applicant noted that across other research councils they hold 'Town Hall Meetings' where those involved have the opportunity to meet and ask questions.

Other examples of best practice include having a clear statement and strategy of aims as well as constraints, adopting straightforward language without 'buzzword', and providing examples of good projects and proposals.

Some stakeholders (nine, 43% of all applicants) made comparisons with the EU-funded research programmes' processes. They largely think that the EU and the SG commissioning processes are similar such that they are standardised and clear. However, one (5%) applicant noted that the SG procedures are easier and run smoother.

Others commented (11, 52%) on UKRI research programme practices compared to the ENRA PCP. They commonly mentioned that UKRI's processes are more efficient, such that they involve online submission, more critical and detailed feedback, and simpler payment procedures. One (5% of successful applicants) stated that grants from UKRI are easier to get commissioned as a CoE. Another observed that the ENRA PCP has moved more towards a UKRI model.

Three (14%) applicants mentioned that the ENRA PCP was a better experience compared to UKRI process, mainly due to it having clearer requirements, being more flexible in its procedures, and having direct communication with the SG advisers. On the other hand, two (10%) noted that the ENRA PCP generated more work for them compared to both EU and UKRI funded programmes.

"Organisations such as UKRI or when we could usually apply to the EU, they are quite a lot more structured and ridged which is helpful in some ways but loses that element of the flexibility that you get with an ad hoc system."
(Successful applicant, Pye Tait, 2022)

Eight (38%) applicants reported they do not have sufficient knowledge of other processes to make a comparison. Of these, two are of a view that the processes are too different to be compared. Some peer reviewers acknowledged that their role in the process was limited, and one (17%) explained that they could not comment on improvements to the commissioning process as they were not involved in it. Similarly, a policy lead explained that they are not involved in other grant processes so cannot compare the process with other grant processes.

One peer reviewer (17%) provided an example of the Swedish Research Councils approach, whereby they explained that the system is quite good as it is not always reliant on external peer reviewers and reviewing is conducted by members of the panel who are not complete experts in the field. In comparison, the UKRI system relies on expert peer review as well as a couple of panel members, however this can create challenges for PIs as they are developing proposals for two audiences.

Generally, the programme commissioning process appears to have generated no more work than other comparable programmes have or would have. That is to say that respondents indicated that all programme commissioning processes tend to generate a lot of work, but that it "seems reasonable" for the funding level involved. Some people involved in the commissioning process suggested that they have less to do this time because their PIs were leading on reporting and on proposals, and for those working on multiple projects they said that as long as each milestone and deliverable are met it would be fine.

2.9. Additional comments

Stakeholders made several additional comments across the different stages of the PCP and in relation to their role within the PCP. A summary of the additional points is outlined below:

• The SG to have more frequent communication with peer reviewers, should they wish to continue collaborating with them;
• A suggestion to introduce an option of donating peer reviewers' fees to charities;
• Considering using commercial review platforms, such as Smart Simple, for improved streamlining;
• Improvements around project management, and having clearer roles and responsibilities within the SG policy teams; and,
• Considering having two versions of the tender template, depending on the size of the project.