Scottish Government Rural Affairs and the Environment Strategic Research Strategy 2011 - 2016

6. Assessing / Measuring the Impact of our Research Investment

A Performance Management Framework (PMF) has been developed to better and more systematically measure and assess the impacts of the SG's RAE research investment. Data and evidence collected under the Framework will enable Government and the Main Research Providers to:

• Ensure accountability for the research spend;
• Demonstrate the benefits being derived from that spend;
• Use performance data to enhance impact and value for money.

Based around the strategic objectives which the spend is aimed to achieve (see Section 3), the framework uses a logic model to link inputs to activities, outputs and short and longer-term outcomes (see Figure 2 below).

Measurement of the impact of investment in science is widely acknowledged as a challenge for a number of reasons, including the time lag between research and impact, the non-linearity and complexity of the links between activity and impact and associated challenges in establishing causal connections and the effects of external factors and events. The framework therefore uses multiple lines of evidence to monitor performance and impact. These include key indicators reported annually, monitoring trends over time, comparison between different elements of the research portfolio and, where appropriate, benchmarking with external organisations and funders.

Figure 2: Generic Logic Model for Research Activity with Example Indicators

Annual reporting is a major component of the framework. Reporting requirements were developed in consultation with scientists at the Main Research Providers. Key indicators were selected to reflect progress towards delivering the five strategic priorities for research portfolio:

• Supporting Policy and Practice;
• Supporting Innovation and the Economy;
• Collaboration and Multidisciplinary Working;
• Scientific Excellence;
• Scientific Resilience.

Comprehensive information on outputs and outcomes across these five strategic priorities is gathered using narrative examples under twelve categories and seventeen key performance indicators (see Table 1 below). Information is collated in an interactive "dashboard" that links the aims of the research with its outputs and outcomes.

Table One: Summary of Quantitative Performance Indicators⁴