Innovation data baseline: final report

Executive Summary

This report summarises the findings of a study into methods and approaches for measuring the impacts of investment in innovation support in Scotland.^[1] The study is part of a wider programme of work being progressed by the Scottish Government on behalf of the Enterprise and Skills Strategic Board (ESSB). Specifically the research focused on the activities of the Enterprise^[2] and Skills agencies (Scottish Enterprise (SE), Highlands and Islands Enterprise (HIE) and Scottish Funding Council (SFC)).

The objectives of this research were to:

• Identify the baseline data and methods required to improve the evaluation of innovation support interventions in terms of short and longer term outputs and impacts on the Scottish economy.
• Review the existing data and methods used by the agencies to monitor and evaluate innovation support activities.
• Make recommendations for new approaches to address gaps in current data collection and encourage consistent measurement and evaluation across all agencies.

Role of the Agencies and Innovation Expenditure

Each of the agencies has a different role in supporting innovation. The primary role of SFC relates to supporting research capability and activity in Higher Education (HE) and supporting and encouraging engagement between education/research institutions and external actors in the innovation system (e.g. firms, public sector agencies, etc.).

As the bodies responsible for promoting and supporting economic development, SE and HIE focus on developing innovative capacity and supporting innovation activity within the business base. While offering broadly similar packages of support for businesses, these are tailored to reflect the different nature of the economy across lowland Scotland and the Highlands and Islands.

In 2018/19 it was estimated that Scottish Government's total annual spend on innovation interventions was around £479m. Further detail on the breakdown of expenditure is available in Section 1 and Appendix A in the main report.

Of this total, £349m is delivered through the Enterprise and Skills agencies: SFC - £308m (88%), SE - £37m (11%) and HIE - £4m (1%). The Research Excellence Grant funding that SFC administers to Scottish universities is by far the largest single programme by value (£238m, 68%).

Why Support Innovation?

There is a strong and well-established evidence base demonstrating the role of innovation as a key driver for economic growth and productivity through the development of new products, processes, services and business models, which increase sales, and thus economic output, ultimately creating wealth and employment.^[3] The important role of innovation as an enabling tool is also reflected in the Scottish policy environment.^[4]

While many competing definitions exist, in order to place some practical parameters around the scope of the research, the Scottish Government's Enterprise and Skills Analytical Unit have developed a broad high-level definition of innovation:

"new ways of combining existing (and/or new) resources to better address existing (and/or new) needs".

Conceptual Framework and Theory of Change

The research drew on existing literature^[5] to develop a conceptual model that would simplify (as far as reasonable) the complexities of the innovation process and demonstrate the (direct/indirect) linkages across the system from knowledge creation to eventual exploitation (Figure 1). This model was agreed with the agencies and then used as a framework against which to assess current measurement practice.

Figure description:

This diagram presents a model of the innovation process with theory of change and activities. The 5 steps of the innovation process listed are; Knowledge Creation, Innovation Capacity, Knowledge Flows and Diffusion, Innovation Development and Application and Exploitation.

While the model depicts innovation as a linear process, it also recognises that real world innovation is in fact rarely linear. The outputs of one stage may not lead to the next, and iterative 'feedback loops' are possible across the model. It is also true to say that while much of the focus here has been on economic impacts, the model can and does also accommodate wider social and environmental impacts arising from innovation activities.

Finally, while the model does not directly address spillover effects, it may be possible and indeed useful to identify these by other means. Knowledge spillovers are often (unintended) outcomes from engaging in, or undertaking, innovation - new knowledge created has a number of potential applications by a number of different actors, outwith its intended use. However, at the individual intervention level it is extremely challenging to attribute, track and quantify the wider knowledge spillover impacts/benefits from one particular intervention to other firms, industry sectors, countries, society, etc. However, given the evidence to suggest the importance of these effects,^[6] this is an area worth further consideration.

Further detail on the component elements of the conceptual model, and the accompanying theory of change, is provided in the main report (Section 2).

Review of Current Practice

Investment in innovation was then mapped thematically across the innovation model and by type of intervention. Figure 2, below, provides an overview of the total annual expenditure (2018/19) in each category and provides examples of programmes delivered by the agencies.

The programmes highlighted in red were then examined in more detail to review current data collection and reporting practices, with greatest priority given to those with higher levels of expenditure.

These interventions together account for 87% (£303m) of the agencies' spend on innovation support in 2018/19.

Figure description:

This diagram maps the current landscape of innovation support, showcasing what support programs are in place for each stage of the innovation process. There is also a note of total spend in each stage of the innovation process; Knowledge Creation £244m, Innovation Capacity £3.6m, Knowledge Flows and Diffusion £35.4m, Innovation Development £30.5m and Application & Exploitation £0.6m.

Summary of Main Findings

Across the agencies there are no developed frameworks for the specific goal/purpose of measuring and monitoring the impact of their investments in innovation support.^[7] Instead, additional measures are typically included within existing (often long established) frameworks designed to measure wider agency targets/objectives. These frameworks are necessarily quite high level and often too broad to support a detailed and consistent account of the impacts of specific forms of innovation support.

As a result, a number of common issues/challenges were identified:

• attribution - there are challenges in attributing changes in performance (e.g. employment or sales) to specific interventions;
• timescales - long timeframes of many innovation interventions to deliver quantifiable impacts;
• inconsistencies in terminology and interpretation of indicators - many are similar, but expressed in slightly different terms; and
• there is limited attempt made to capture the extent of "successes" across the innovation system.

Specific Agency Practice

Scottish Funding Council

SFCs' role in the innovation system is different to that of SE and HIE, as its investments are directed to HE and FE institutions, and are often distributed as formula-based grants. The data sources used to track these core grants tend to be existing secondary sources (such as those provided by HESA) which provide more general measures of institutional performance rather than measures specific to SFC's particular investments. Attributing impacts to SFC's core grants is therefore challenging.

This is less the case with SFC's strategic funds, which are often used to support defined programmes (such as the Innovation Centres). Many of these are subject to more specific measurement and reporting frameworks, which allow clearer attribution.

Scottish Enterprise

Overall, driven by its strategy,^[8] SE has adopted a consistent approach across the breadth of innovation project and programme activity, and gathers a range of specific data.

This includes; R&D jobs, sales/exports from innovation activity, leveraged investments (private and collaborative R&D) and 'innovation' active companies. While not performing the same function as a measurement framework, SE also periodically commissions quantitative and qualitative evaluation studies to inform the evidence base that adds a rich source of intelligence.

Highlands and Islands Enterprise

In common with SE, HIE has a defined measurement system for capturing data to report against its published targets and assess the impacts of its services and activities.

This measurement system is aligned to the National Performance Framework and is driven by the priorities of the Scottish Economic Strategy and HIE's own strategy and operating plans.

HIE's framework has a set of general and more innovation specific metrics such as: R&D jobs, clients engaged in knowledge transfer, patents/licences approved and new products/processes developed, etc. In addition, HIE also has a series of tracking measures in the form of 'ladders' which track progress in five areas including innovation. The innovation ladder comprises five stages (chance, intent, active, embedded and evolving) - each of which assess the extent of commitment to, and action on, innovation within organisations. These measures allow tracking over time.

Current Practice Delivering Against Policy Goals

In support of the current practice review, a GAR (Green Amber Red) assessment was undertaken and can be viewed at Section 5 of the main report. The main finding is that the collection and reporting processes for gathering data on the inputs to innovation (which are largely financial) and the activities that are delivered/supported, are relatively robust across the cross-section of programmes reviewed.

In contrast, there is little in the way of data being gathered across the agencies that would support a consistent and accurate assessment on the outputs/outcomes and impacts being generated through the innovation spend.

Specifically, we would note that a key strategic objective for Government is for Scotland^[9] to be within the top quartile of OECD countries in terms of productivity (and equality, wellbeing, sustainability). Investing in innovation is one of the key mechanisms to help drive inclusive growth across all these areas. However, current approaches do not gather the data necessary to measure the contribution of Scottish Government innovation expenditure to this over-arching policy goal.

Conclusions and Recommendations

One of the main objectives of the study was to assess whether it was feasible to measure the totality of impact generated through Scotland's investments in innovation across the three agencies, which was estimated at £349m in 2018/19.

On the basis of current approaches and measurement practice undertaken by the agencies, such an assessment is not possible, for the following reasons:

• there are significant gaps where either data are nor collected at all, or the available data are not sufficiently robust or complete;
• measures/metrics are sometimes inconsistent in their definition across different agencies, even if similar in general meaning;
• the attributional links between investments and impacts are not always clear, and there is reliance in some places on benchmark measures of innovation performance (such as those in the HE BCI for universities or BERD for the wider economy) which cannot easily be attributed to specific support or investment;
• there is considerable potential for double counting of impacts where companies have received multiple forms of support from the same/or another agency/organisation;
• there is little sharing of data across the agencies;
• impact data (e.g. turnover and employment gains) are frequently based on forecast rather than actual data; and
• the long timescales between intervention and impact in innovation projects are such that impacts are not always captured by existing data collection methods as longer term tracking is limited.

Recommendations for Action

In providing recommendations about how to improve and address gaps in measurement systems, there is a need to be pragmatic, given the complexity and overlapping nature of the linkages across the innovation journey and conceptual model. In particular, there is a need for a pragmatic means of assessing, and therefore balancing, the costs and benefits of monitoring and evaluation.

In order to implement a measurement framework that can better evidence the impact of Scotland's expenditure on innovation, there will also be practical considerations such as:

• securing the "buy-in" and cooperation of the three agencies and wider stakeholders such as the universities;
• strategic commitment reflected in agency/partner's policy and allocation of resources to develop and populate a new framework; and
• timescales for implementation and 'payback' i.e. reflecting the longitudinal nature of the innovation journey it will be a number of years post-implementation before any framework starts to generate meaningful data and intelligence.

Taking all this into consideration, we have outlined an updated Framework of proposed measures across the innovation journey, presented at Section 6.5 of the main report.

At a high level, in addition to the commitment of resources there are implications for each of the agencies as follows:

Scottish Funding Council

SFC should seek to gather data from HEIs on:

• the use of Core Grant monies (specifically the Research Excellence Grant and University Innovation Fund) against defined categories of expenditure (these could relate to infrastructure, research activity and staff, for example);
• more comprehensive measures of research activity and resources including the number of active research projects and research active staff (taking account also of the Scottish Government work on research and KE measures and review of existing data sources such as the HE-BCI Survey);
• the number and location of businesses and other external organisations with which they engage (this could be captured in the HE-BCI and could be an issue to input into the HESA review process); and
• engage with and support the universities to collect data on the impacts of their innovation activities.

Enterprise Agencies

For the enterprise agencies there are three overarching principles that should be considered to support development and implementation of a new framework:

• there is a need to adopt a consistent set of measures between the agencies such that data collection and reporting can be more aligned;
• there should be greater commitment to longer term tracking of assisted organisations to assess impacts over time - this is particularly important for innovation support where timescales can be lengthy; and
• data collection should move from reliance on forecasts to actual impact data - which will require the longer term tracking noted above.

Other Stakeholders

Delivering and undertaking innovation activity is not the sole responsibility of the agencies and there is a role for wider actors and stakeholders. These wider stakeholders include: ESSB, Scottish Government, universities, research institutions, innovation centres and businesses. There are resource implications for these wider stakeholders, which also need to be considered in defining their role in any future framework.

Next Steps

The next steps in taking forward the new framework would be as follows:

• further engagement with the three agencies to define in more detail the specific (and consistent) set of measures within the framework and scope out the various data collection methods - this might also extend to wider partners and stakeholders;
• agree and develop a data governance model that identifies roles and responsibilities including responsibility for the co-ordination of the data collection and reporting processes;
• develop a common reporting format (template) for the agencies to input data on an annual (or otherwise agreed) basis;
• develop data collection processes (including arrangements for longer term tracking) and allocate necessary resources to support; and
• begin data collection to establish a baseline position for Year 1.

It is worth noting that some of these actions could be implemented with limited impact on resources. These include:

• developing a consistent set of measures and common reporting template across the agencies; and
• establishing agreed data collection processes and timeframes across the agencies.

Implementing new data collection processes may then require the allocation of resources and this is a decision to be made by the agencies and the ESSB.