Innovation data baseline: final report

4. Current Practice

4.1 Introduction

In agreement with the Scottish Government and the three agencies, we have considered the data collection methods across a sample of key projects/programmes/schemes. In identifying which interventions should be included, we sought to prioritise those with the greater levels of expenditure, while also seeking a balance across the agencies and across different types of support.

These interventions, as shown below, account for 87% (£303m) of the agencies' spend on innovation support, as defined.

Table 4.1: Interventions for Further Review

	Investment	Capacity Building	Infrastructure
Knowledge Creation	Research Excellence Grant (SFC, £237.8m)
Innovation Capacity		Workplace Innovation Funding (SE, £2.6m) Northern Innovation Hub (HIE, £1.3m)
Knowledge Flows	Innovation Vouchers (SFC/ HIE, £0.62m)	University Innovation Fund (SFC, £15.8m)	Innovation Centres (SFC/SE/HIE, £14.1m)
Innovation Development	R&D Grants (SE/ HIE, £17.8m) SMART (SE, £7m) By Design Grant (SE, £0.9m) Aquaculture Fund (HIE, £0.32m)	Innovation Project Support (SE, £4.8m)
Application & Exploitation		IP Audit (SE, £0.26m)

As it is the individual agencies that have responsibility for gathering and reporting performance data, the section that follows provides an account of the data collection methods employed by each of the agencies in relation to the programmes/interventions set out above.

As already noted, some of these interventions/projects could sit within other categories and/or overlap - this therefore represents a best fit framework.

4.2 Scottish Funding Council

Introduction

As noted above, SFC provides funding and support to universities and colleges in Scotland. Its primary function in the innovation system relate to supporting research capability and activity in higher education (HE) and supporting and encouraging engagement between HE, Further Education (FE) and external actors in the innovation system (e.g. firms, public sector agencies etc.) for the purposes of advancing innovation.

Research Excellence Grant

The Research Excellence Grant (REG) is a formula-based funding model designed to:

• support excellence wherever it is found in the research base;
• develop and enhance research in Scottish universities to ensure Scotland remains globally competitive and attractive to the best researchers;
• recognise and reward the effective translation of research 'impact' as measured by the Research Excellence Framework (REF),^[21] including effective business support by universities;
• uphold the principles of the dual support system, including contributions to the full economic costs from Research Council, charity, European and other research income to retain confidence in the Scottish university research base; and
• address SFC's responsibilities to support the personal, professional and career development of researchers.

The formula for the funding combines a score to reflect the quality of institutions' research with scores based on their success in securing research income from different sources. The quality score is informed by the REF (see below).

Institutions have considerable autonomy in relation to how the REG funding can be used, and may invest in infrastructure and staff that support research activity, as well as directly into research projects. At present there is no detailed account of how REG funding is used by institutions (Scotland-wide or across individual institutions), as these data are not collected or reported to SFC in this way.

SFC does have access to sources of data on HE income and expenditure both through the Transparent Approach to Costing (TRAC) data and HESA's HE Income and Expenditure datasets. Together these provide a rich account of HE finances, but do not directly identify how REG funding has been used.

The primary means through which SFC assesses the value of the REG is through the UK REF.

The first REF was run in 2014 and replaced the previous Research Assessment Exercise (RAE). It is a UK-wide system for assessing research in higher education institutions and is conducted jointly by all four UK higher education funding bodies.

Give the scale of the REF it is conducted every seven years with the next originally scheduled for 2021. This has subsequently been delayed as a result of the Covid-19 pandemic.

Institutions provide submissions to the REF based on individual academic discipline area (Units of Assessment) in which they provide information about research active staff and research activity and outputs. Panels of experts in individual academic subject areas then assess institutions' research submissions under three broad headings:

• research outputs - the quality of research publications and other forms of research output;
• impact - the benefits delivered to the economy, society, culture, public policy or services, health, the environment or quality of life; and
• research environment - the vitality and sustainability of the research environment, including the contribution to the wider discipline or research base.

The assessment of research outputs focuses on originality, significance and rigour and includes consideration of the academic outputs of research activity such as written outputs (journal articles, books, book chapters, conference presentations etc.), physical artefacts (e.g. devices, materials, products etc.), digital artefacts, exhibitions and performances and other.

This accounts for 60% of the total REF rating for the Unit of Assessment with an institution.

The assessment of impacts is probably of greatest interest to the current study as this is where the wider effects of research activity are considered.

Impact is defined in this context as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia (REF 2021). Impact also includes the reduction or prevention of harm, risk, cost or other negative effects, but does not include impact in terms of the advancement of knowledge or impacts on students or staff.

Institutions provide case study examples of research impacts, which are largely narrative (and qualitative). They may include more quantitative impact measures (such as creation of spin out companies and employment) and there are moves towards standardising these.

The impact case studies provide useful accounts of how academic research can deliver wider social benefits (and can be used to infer whether the Theory of Change is accurate/relevant). However, they only provide a narrow cross-section of the impacts that may arise from specific research activity. They also do not provide aggregate figures for, say, the economic impact of research activity in any given year. Therefore, while the REF does provide the funding bodies with a means of assessing the performance of UK higher education research, it does not measure the impacts of the totality of that research activity.

Thus, it is true to say that we do not have direct measures of how the REG is spent by the funded institutions, nor of the impacts of that expenditure. Based on this system, it is therefore not possible to determine the return on this investment other than in terms of research income generated from other sources and the outputs of the REF.

University Innovation Fund

The University Innovation Fund (UIF) is also a formula-based fund designed to support knowledge exchange (KE) and innovation activities in HE institutions. The purpose of the UIF is stated as follows:

"Scotland's universities are a critical part in our innovation system. The University Innovation Fund (UIF) is a contribution, in support of the broad range of knowledge based activity and services through which our universities collaborate with each other and the wider world to stimulate economic, cultural, and social benefits. It ensures that our universities are supported to evolve and deliver their services in line with Scotland's priorities, including the outcomes in the National Performance Framework, and Climate Emergency. It is allocated on the basis of commitment to continuous improvement, working in partnership where appropriate."^[22]

The grant comprises two elements:

• the Platform Grant: a flat level grant of £250K (to be adjusted for inflation) which is to be matched by the HEIs (with the exception of small specialist institutions) and invested into core infrastructure that supports innovation and KE; and
• the formulaic UIF grant: based on institution's performance in the 2015/16 KE Metrics (see below). This is currently under review, as discussed below.

The primary mechanism for determining allocations for the UIF (which accounts for c.60% of the total funding provided through the UIF) is based on the KE Metrics that SFC collects from institutions. These gather data on the total income for each institution across a number of activity categories, including: research grants and contracts (split by UK Government and public sector, industry and commercial), consultancy, licences, CPD, enterprise schemes, translational awards, venturing and outreach.

The KE Metrics do not collect any information on how the UIF monies are spent, but institutions are required to specify this within the Outcome Agreements provided to SFC each year.

There is also a specific UIF submission required of each institution to detail how they propose to use the findings and what the expected outcomes will be.

The UIF is currently being calculated on the basis of 2015/16 KE Metrics and work is underway to review the scheme and its funding method. Of specific relevance here, the proposal is to move away from the collection of the KE Metrics in favour of using the UK-wide Higher Education Business and Community Interaction Survey (HE-BCI)^[23] administered by Research England (institutions are required to complete this survey as a condition of UIF funding).

The HE-BCI is a more comprehensive dataset that collects information on institutions' income by activity (collaborative research, contract research, consultancy, CPD, intellectual property (IP), facilities and equipment, regeneration and development) and by partner type (SMEs, large businesses, public and third sector). It also provides data on the scale of activities including numbers of IP disclosures, patents and licences, numbers of spin-out and staff start-up companies, and the turnover and employment within these new firms, and the (qualitative) benefits from regeneration activities.

Finally, the survey also provides data on wider public engagement activities such as lectures, exhibitions and performances (numbers and attendances).

As such, the HE-BCI is certainly a useful source of data on universities' external engagement and KE activities and is one way in which their contribution to the innovation system can be assessed. However, it has some limitations.

First, the outcomes that it reported are not tied to specific investments. Thus, UIF investments could not be said to lead directly to these outcomes, as other sources of investment will also contribute (not least research funding). Secondly, the HE-BCI data does not track the impacts of these KE activities on participating organisations (e.g. firms) and simply presents a snap shot of activity (based on the individual firms that completed the survey and provided data - which may change in a given year). As this is a primary source of economic (and other) impacts arising from higher education's innovation activities, this is a major gap in the available data.

Innovation Centres Programme

SFC established the Innovation Centres (IC) Programme^[24] in 2012 and has since funded (in partnership with the enterprise agencies) eight ICs. The Programme aims to support transformational collaboration between universities and businesses, to enhance innovation and entrepreneurship, and to support economic growth, particularly, but not exclusively, in Scotland's priority sectors.

The Vision for Innovation Centres is as follows "Using the Scottish university infrastructure, human resources and research excellence as a platform for collaborations across the whole of Scotland, ICs will create sustainable and internationally ambitious open communities of university staff, research institutes, businesses and others to deliver economic growth and wider benefits for Scotland."^[25]

The Centres each have different structures and activities, but all have a focus on collaborative R&D between universities and industry and on the development of high-level skills in their specific fields (data science, biotechnology, sensors, digital health, precision medicine, construction, and aquaculture).

The Programme is now into its second phase, and each IC is required to provide regular (quarterly) reports to the funding partners. These reports each vary slightly in style and format, but all combine a narrative report on progress and activities with a more quantitative update using an agreed Monitoring and Evaluation Framework (MEF) for the Programme.

The MEF was revised in 2018 following recommendations made in the review of the first phase conducted by Professor Graeme Reid.

The revised MEF was developed using a logic model approach and specifies indicators in four categories:

• Inputs: quarterly income and expenditure (split by source);
• Activities: including measures of the number of new, continuing and completed collaborative innovation projects; business engagements; number and type of businesses engaged; HEIs involved in collaborative projects; events and attendances; and individuals involved in learning activities (mainly masters courses developed by the ICs in partnership with HEIs) -reported quarterly;
• Outputs: including measures of follow-on from collaborative projects (e.g. signposting, follow on projects and commercialisation); new products, processes and business models developed as a result of collaborative projects; research income for HEIs as a result of projects; and individuals gaining new qualifications. These are reported quarterly; and
• Outcomes/ Impacts: including jobs, turnover and GVA impacts arising from IC activities. These are reported annually.

The MEF provides a clear means of tracking the activities and impacts of the Programme against a logical structure. However, our understanding is that there are some inconsistencies in how the measures are being interpreted and reported by different Centres, and in the mechanisms used to collect the required data. For example, different accounting systems and practices used by the Centres direct the reporting of financial measures in different ways, and there are different methods used to collect impact data from supported firms, including some that use forecast data, and others that rely on assumptions about the attribution of wider sector growth to the activities of the Centre.

It should be noted here that the Programme was always intended to be a long term intervention and that the full impacts would take many years to materialise. As the revised MEF was introduced for Phase 2 of the Programme, the extent of prior tracking of companies supported in Phase 1 is not clear (although here was a previous MEF and also an external impact evaluation).

It is also not clear that the IC Programme MEF can track backwards to identify, for example, where research funding may have contributed in the past to the creation of new knowledge (or IP) that has subsequently informed the collaborative innovation projects supported by the Centres.

Innovation Vouchers

The Innovation Vouchers scheme is funded by SFC and administered by Interface. It provides small amounts of funding in the form of a voucher that an organisation can use to 'buy' academic time or input from a university to support an innovation project. The value of a Standard Innovation Voucher is between £1,000 and £5,000, which covers the academic project costs and is paid directly to the university or college.

Vouchers are awarded to support projects which are innovative (e.g. a novel/pioneering idea) and lead to either new products, services, processes, or an innovative workplace business process/practise that will benefit the company, the academic institution, and the Scottish economy.

As well as administering the scheme, Interface also provides ongoing monitoring data to track uptake and progress. Interface's monitoring systems are robust and are completed with considerable diligence.^[26] This includes a final report^[27] at the end of each project which seeks impact data as well as a follow up survey 6 months later to track impacts. This report details the outcomes from the project and any next steps that will be taken to progress the work undertaken.

It should be noted that one of the current weaknesses in the system is that the data from final reports is not yet automated into the CRM system, although this is an action in progress.

The following impact performance data is gathered (achieved at 6 months post project completion and forecasted over the next 12 months):

• number of new/improved products;
• number of new/improved processes;
• number of new/improved services;
• number of new/safeguarded jobs;
• increase in company turnover;
• increase in company profit;
• increase in productivity;
• increase in exports;
• company investment in further R&D - Value (£); and
• does the company have an ongoing relationship with HE/ FE.

While Interface gathers a range of relevant impact data, again, there are limitations to the data collection for this scheme. Most obviously, Interface does not have the resources to undertake longer term tracking of companies beyond six months post support. Also, the next step for beneficiary companies is often a follow-on voucher or support from a programme like Knowledge Transfer Partnerships (both common next step routes for companies after the initial voucher project).

As such, the impacts directly attributable to the Innovation Vouchers will be harder to distinguish form those arising from follow on support. This is not a challenge that is unique to the Innovation Vouchers scheme and is discussed further below.

HIE, SE and SFC also provide funding for follow-on vouchers to support follow up activities. In addition to the data collected by Interface for SFC, HIE also collects additional impact data on:

• the number of clients engaged in knowledge exchange activities;
• the number of clients engaged in R&D activities;
• the number of academic collaborations completed; and
• increase in turnover as a result of KE projects.

Discussion

Measuring the impacts of research investment is challenging for a number of reasons:

• as discussed in Section 2, research funding is driven by objectives relating as much (if not more) to the advancement of knowledge as to economic, social and environmental benefit through innovation. As such, a substantial (and currently unknowable) proportion of the REG will support research that is unlikely to feed into the innovation system;
• the processes through which academic knowledge eventually feed into innovation projects can be long and convoluted, and may involve multiple forms of additional investment and support;
• the principle that universities are autonomous bodies with control over their own finances means that they may use research funding as they see fit so long as it is contributing to their research performance (as assessed by the REF). As a result, SFC does not have detailed information on the activities supported by REG funding; and
• SFC operates within a UK-wide system for research funding, and the principle means of assessing research performance remains the REF. While the REF does take account of impacts (25% of overall ratings) it does not link impacts to investments, and is based only a very small sample of research activities. It cannot, therefore, provide a measure of return on investment for research funding.

Therefore, under the current arrangements, it is not possible to determine with any accuracy or consistency, the return on investment (economic or otherwise) from the Research Excellence Grant. To do so would require a means of tracking research outputs/impacts over potentially very long time horizons. Such a system would then be required to identify research outputs that are subsequently used in innovation processes.

This, however, is only a part of the picture. Knowledge Exchange is fundamentally people-based, and the value that higher education contributes to innovation projects is often through the tacit knowledge of academic staff. It is difficult to identify a means through which this may be tracked other than through identified support programmes (such as Knowledge Transfer Partnerships, for example).

SFC's support for innovation related activities also presents a mixed picture regarding measurement. The UIF is paid directly to universities and there are four main sources of evidence:

• University Outcome Agreements, which are largely narrative and set out what the institution intends to do rather than what has been achieved;
• UIF submissions, which are again an ex-ante assessment of what the funding will achieve and do not provide a means of assessing impacts;
• SFC's KE Metrics, which have not been collected for some time and are currently under review; and
• the HE-BCI, which is UK-wide and does provide useful data on universities' KE activities.

Of these, the HE-BCI is the most useful in assessing the returns to innovation support, but suffers from the following limitations:

• impact data is limited. It provides good data on HEI's activities and income to the institutions, but far less on wider impacts within beneficiary companies;
• recorded impacts are not directly attributable to any specific support or investment, but are more generally attributable to the activities of institutions however funded; and
• while it records the extent of interactions with businesses, the survey does not categorise businesses by their location. It is therefore not possible to identify how many Scottish (or even UK) businesses have been supported.

Thus, determining the returns from the UIF is also not possible under current arrangements.

The situation regarding Innovation Centres and the Innovation Vouchers scheme is somewhat different. Both programmes have defined measures, including impact measures, and systems in place for collecting relevant data. It is possible, at least in theory, to provide some assessment of the impacts arising from these programmes, albeit with some caveats regarding the quality and consistency of the impact data.

4.3 Scottish Enterprise

SE Measurement Framework

SE's strategy, Building Scotland's Future Today,^[28] is driven by the Scottish Government's economic agenda as set out in the Programme for Government, Economic Action Plan and Enterprise and Skills Strategic Board Strategic Plan. The strategy places considerable emphasis on international competitiveness and R&D and innovation as a key driver.

The organisation's progress against its strategic priorities is measured through a defined set of targets and performance measures based around five key priority outcomes:

1. planned new/protected jobs paying at least the real living wage;

2. planned R&D investment (businesses and sectors);

3. planned capital investment (businesses and sectors);

4. growth funding raised by supported businesses; and

5. planned international export sales.

These headline measures are supplemented by a number of supporting measures, many of which relate to, or can be aligned to innovation:

• SE Supporting Measures
• number of R&D jobs
• number of high value added jobs
• R&D investment leverage ratio (SE investment to non-SE)
• R&D investment by inward investors
• R&D investment attracted from Innovate UK, EU, etc.
• collaborative R&D funds from other sources
• R&D investment attracted to sector assets
• number of new innovation active companies
• planned sales from innovation;
• number of collaborations (businesses to business and business to academia)
• number of academic spin outs and start-ups
• planned international export sales from innovation
• number of new products/ services launched in international markets

SE's activities are expected to contribute to these measures, but the framework includes the flexibility to add project specific measures. It is worth noting that many of the impact measures are based on forecasts (e.g. 'planned sales from innovation') rather than actual measures.

This may reflect both the timescales from support to impact, which can be lengthy for innovation support, and the practical challenges in tracking companies over time. Forecast data (as discussed below) is collected at the application stage for the main innovation grant programmes, but actual impact data are often also based on estimated future impacts (which may be subject to unknown levels of optimism bias).

SE also conducts periodic evaluation of its programmes (quantitative and case study based), and these exercises tend to produce more robust estimates of impact, albeit within the evaluation period.

This approach therefore seeks to use historical data and feedback on past performance to test the Theory of Change (i.e. did they achieve the desired change and impact that they intended at the outset).

The main limitation with this approach is that, while the grant management system processes claims (and therefore collects some data) on a quarterly basis during delivery of the project, without regular tracking of performance beyond project end dates, SE is unable to accurately assess the ongoing impact of their interventions until such time as an evaluation is conducted. More specific issues are addressed below in relation to the selected support programmes.

It is also worth noting that the evaluations tend to be somewhat ad-hoc in their timing, and are subject to wide variation in method.

R&D Grants

The R&D Grants Programme is designed to address and alleviate commercial and technical risk for companies undertaking R&D projects, and is also a useful stimulus in attracting inward investment projects. The programme supports projects that will lead to innovative new products, processes, and services to improve company competitiveness and benefit the Scottish economy. Lasting between 6 and 36 months, projects must be based in Scotland and must be new, as the grant will not cover routine or periodic changes made to products, processes and services. The amount of funding offered is at the discretion of SE and can cover up to 50% of project costs. The value of grants awarded ranges widely from c £25K to £7m.^[29]

At the application stage, companies are required to provide details and milestones for the project for which they are seeking funding along with evidence of match funding - this is followed up via the grant management team. They are also asked about employment in the company (R&D and non-R&D jobs) and complete an economic impact template in which they provide forecasts of the likely employment and sales (turnover) impacts of the project.

During the life of the funded project, the grant management process collects regular data on project expenditure and progress against milestones and project objectives.

Beneficiary companies are also required to complete a Project Monitoring form on completion of the project. This provides an account of the project's success or otherwise and of the extent to which it achieved its objectives. It also seeks information on any job creation/ protection and turnover attributable to the project, any disposal of intellectual assets (e.g. licences) and a qualitative account of the impact of the project on the company's growth.

The collection of these data at the point of project completion means that in many cases the full impacts arising from the R&D activities will not yet be apparent. SE does undertake some limited follow up 18 months post completion (although the timing of this can vary) to gather further impact data, but this is with only a small sample of supported firms. Also, 18 months remains, in many cases, too early for impacts to be identified. For example, SE evaluation evidence identifies that across a range of R&D and innovation projects, the 'impact profile' shows that the majority of impacts are generated in years 8 to 10 post the start of the project (SE measures the impact of support over a 10 year period).^[30] This 'time-lag' in terms of generating impact is supported by wider case study evidence which identifies that "development of basic scientific research and its findings and implementation in commercialisable products may be very long".^[31]

The result is that the main impact reporting for the R&D Grants Programme is based on planned/ forecast impacts - mainly jobs and turnover. Data on R&D investments (e.g. investment leveraged by SE funding) are based on actual figures, and the project monitoring will also capture R&D jobs created directly through the projects.

Work on impacts is also undertaken through periodic evaluation and the collection of case studies of supported companies. It can, however, remain challenging, even in evaluation studies to assess longer term impacts, particularly if the original support was some time in the past. Changes in company personnel, lack of corporate memory and complications with attributing benefit to a specific intervention are all common issues.

SMART Grants (SMART: SCOTLAND)

Similar to the R&D Grants, the purpose of SMART: SCOTLAND is to alleviate commercial and technological risk for SMEs seeking to undertake early stage R&D and funding is available for feasibility studies and R&D projects that have a potential commercial application.

For feasibility studies lasting between 6 and 18 months, a maximum grant of £100,000 is available, and for development projects lasting between 6 and 36 months, a maximum grant of £600,000 is available.

Applicants must be an SME, university spin-out or start up with an operational base in Scotland, and projects should exhibit a significant advance in technological innovation and technical challenge. SMART grants are available to businesses that operate in both the SE and HIE regions (and also the South of Scotland under the new agency).

Data collection processes for SMART are similar to those for R&D Grants. The application form collects data on employment within the applicant firm and also on the employment and turnover gains expected as a result of the funded project. Again, a Project Completion Monitoring form is issued 18 months post project which collects information on some early commercialisation outcomes (mainly qualitative) and expected future impacts. However, SMART supports early stage R&D and the impacts attributable to these projects will in most cases be many years in the future, and may also require subsequent support and numerous rounds of investment (seed funding, Venture Capital, series investment) to realise. This makes attribution of eventual benefit to the original support more difficult and complex. The early stage nature of the project also means that some may not progress at all, although the knowledge gained may be subsequently diffused.

The extended timeframe for realising impacts therefore suggests a need for longer term tracking, but there is a lack of clarity about who is responsible for this and to what extent SE has sufficient resources to support such an approach (and also how proportionate is this to the scale of the investment). There are also issues with personnel change within the assisted firms, and lack of corporate memory, particularly for grants that are more than a few years old.

The result of all of these factors is that there is little consistent tracking of impacts beyond the project completion stage and for SMART (where grants are smaller scale) there are issues with the reliability of the project completion returns.

This means that reporting of impacts from SMART is again based mainly on forecast jobs and turnover growth. The exception here is the leverage of private sector investment (which itself may directly support employment).

By Design Grants

The By Design grant provides funding to companies to help design and develop new products and services. The fund was developed to stimulate Scottish companies' engagement with design innovation, and to contribute to SE's 'innovation active' figure (definition below). It is part of a cross-border collaborative project with Interreg and User Factor.^[32]

The grant can support up 70% of the total eligible project costs, up to a maximum grant of £5,000.

Prospective applicants are asked to complete a By Design Application. This includes baseline information on the company (sector, turnover employees), details of the project to be supported and forecast benefits as follows:

• new products/ services to be developed;
• improved products/ services;
• new markets to be entered;
• additional profits;
• increased competitiveness;
• additional sales;
• value of expected sales as a direct result of the project; and
• number of expected new jobs as a direct result of the project.

On completion of the project, the client is asked to complete a project closing report. The data capture is recorded as part of SE's published measures. The relevant measures are outlined below:

• planned R&D investment (reported as an estimate and entered against the project at the start following receipt of the client's signed contract agreeing to support);
• planned sales from innovation. This is estimated project turnover over 3 years - recorded as an estimate on the Business Support Application, and reported as a published measure on project close as an 'expected' figure; and
• planned international sales from this project over 3 years. This is recorded as an estimate on the Business Support Application, and reported as a published measure on project close as an 'expected' figure.

Innovation Project Support

Innovation Project Support grants are intended to support all forms of innovation project work aimed at creating, developing or preparing new products, processes or services for commercialisation. This grant is discretionary and is only available by referral from an SE Innovation advisor.

There are two levels of Innovation project support:

• Level 1 covers 70% of eligible project costs up to value of £5,000; and
• Level 2 is for follow-on support which covers 30% of eligible project costs up to a maximum £20,000. The grant may assist businesses that are undertaking innovation projects of £30,000 - £70,000. The innovation project must comprise activities related to the design and development of a new product or service, which the business will seek to commercialise.

Most applicant companies will have previously received one of the smaller innovation (Level 1) grants (up to £5k of support).

Applicants complete a Business Support Application Form, which captures basic company information (turnover and employment etc.) and forecast/ planned impacts as follows:

• total forecast increase in turnover as a result of the project (split by international and domestic turnover);
• total forecast increase in jobs - split by those paying at least the living wage and high value jobs (salary in excess of £40,275);
• total forecast jobs safeguarded (split as above);
• planned R&D investment; and
• planned capital investment.

On completion, beneficiary firms compete a project completion form and this gathers the same data as described above for the By Design Grants.

In addition to the above, SE also captures data on all 'new' companies engaging with innovation across a wide range of products and services in SE, including By Design and Innovation Project Support.

These are companies who are receiving a product or service the first time over a 3- year period. This figure is reported on a monthly basis as part of a management report.

Workplace Innovation Support

SE's Workplace Innovation Support provides around 900 companies each year with support, advice and best practice guidance on how to engage employees and improve productivity. The support comprises different elements including, one-to-one tailored advice and support training and development programmes, seminars and self-help guides to enable companies to implement changes in working practices to improve productivity.

SE collects a range of data to assess the performance and impacts of the Workplace Innovation Support. Impact measures are as follows:

• increased turnover;
• employee numbers;
• jobs safeguarded;
• absenteeism;
• staff retention;
• sales; and
• savings.

However, other, more specific measures (which may be more subjective in nature) are also collected via evaluation forms. These are:

• improved workforce skills;
• enhanced employee representative participation;
• improved performance, pay and reward;
• diverse and inclusive work practices developed;
• improved organisational design;
• enhanced workplace culture;
• improved workspace design;
• improved integration of technology, data and automation;
• enhanced flexibility in working practices;
• improved team performance;
• employee engagement level (including motivation); and
• leadership and management potential (including succession planning)

These measures are specific to the nature of the support and the focus on employee engagement and working practices. However, some may be more useful than others as measures of innovation support. Clearly employee productivity will contribute to business success but establishing what exactly constitutes "innovation in working practices" may be less straightforward and open to broad interpretation.

The data are collected via an evaluation form distributed to supported firms. The firms then identify which of the impacts and benefits above they have experienced as a result of the support received.

IP Audit

IP Audits are offered through both SE and HIE (although funded by the intellectual Property Office) and allows for an IP audit professional to come in and independently assess a business's IP assets (e.g. patents, Trademarks, copyright, designs, etc.) with a view to supporting the development of an IP management plan or strategy. The support often forms part of a wider package of Innovation support, including SMART Scotland, Innovation R&D or market entry assistance. The audit provides advice and guidance towards identifying and maximising the value of IP, for example, if IP protection is significant to the development and commercialisation of the product/service. The support is generally targeted at growth businesses and the costs are £3,000 per audit (with businesses expected to contribute £500).

On receipt of the IP audit report the business will also be offered dedicated follow up support from their nominated contact. The nominated contact will discuss the report's recommendations and work with them to identify next steps and priorities in respect of their IP.

The application asks for very basic details regarding business background and the scope and focus for the IP audit and the expected actions/benefits.

Follow up occurs post-completion and is focused on whether businesses plan on taking forward recommendations in the audit report - there does not appear to be any formal monitoring undertaken.

A short evaluation is completed every year (via an online survey of beneficiaries) and asks businesses to outline the main impacts and benefits.

Past evaluations have shown that businesses identified the following impacts:

• identified new business opportunities - for example through licensing or franchising;
• secured more business;
• filed/registered a trade mark;
• reviewed their business terms and conditions; and
• identified confidential information - trade secrets or other commercial information.

The data gathered can be useful in terms of identifying drivers of innovation, however, this is not collected in any systematic manner.

Partnership Projects

SE is a partner, along with SFC and HIE, in both the Innovation Centres Programme and the Innovation Vouchers scheme (like HIE, SE contributes to Advanced Vouchers).

As described above, the primary data collection mechanisms for the Innovation Centres programme is the MEF, and SE was involved in its development. In addition, SE has also been working with the Centres on the development of a Project Log.

The Project Log gathers some additional detail on new projects, those that are ongoing and those that are completed. It seeks information on the scope/nature of projects, costs and timescales, as well as forecasts (for new and ongoing projects) of the expected launch date for a new product or service and the anticipated impacts in terms of jobs created and safeguarded.

For competed projects, ICs have to provide updated information on the performance of the project against the initial estimates and actual job created/ safeguarded figures.

The log also breaks the employment impacts (forecast and actual) into high value and R&D jobs, and for completed projects also seeks information on carbon reduction impacts, although a specific measure for that has not yet been agreed.

The Project Log was prepared and planned for launch in March 2020, but has been delayed as a result of the Covid-19 pandemic.

SE is also a partner in the Bayes Centre, Edinburgh University's innovation hub for Data Science and Artificial Intelligence. The Bayes Centre is part of the Data-Driven Innovation (DDI) Programme within the Edinburgh and South East Scotland City-Region Deal and combines research, education, innovation and tech entrepreneurship within a multidisciplinary environment.

The Bayes Innovation Programme has a straightforward measurement framework based around six objectives:

• attracting corporate R&D teams to establish a presence in or near Bayes;
• securing new jobs by attracting these corporates;
• attracting additional high growth ventures
• supporting investment in high growth companies;
• generating revenue to support research, innovation and entrepreneurship; and
• delivering a business case outlining a larger follow-on project.

There are also two additional tracking measures supporting the third and fourth objectives of creating high growth ventures and high growth companies. These are:

• creating additional jobs over a 12 month period (linked to objective 3); and
• creating additional jobs (linked to objective 4).

Each of these measures is defined with targets, and the framework specifies a data collection process and the supporting evidence required to verify the data.

Summary

Overall, the SE monitoring is consistent across the breadth of innovation project and programme activity that they support and provides a number of relevant metrics to gather business specific innovation impacts. In this sense, it can be considered relatively comprehensive and provides a range of relevant quantified metrics to assess impact

That said, there are opportunities to improve the approach, and these are summarised below.

The SE systems rely on planned and forecast data to assess and measure impact, and there is little in the way of structured follow-up to track progress (limited number of follow-ups post completion at 18 months).

As noted, the agency undertakes (either internally or through external suppliers) evaluations and case study reviews of its innovation support, however, these are done on an ad hoc basis and the feedback gathered through beneficiary engagement may be subject to errors/inconsistencies due to timescales elapsed, or change in personnel within the supported company. Evaluation is therefore not a reliable substitution for ongoing monitoring.

Timescales - much of SE's own evidence points to a notable time-lag between innovation support and the eventual creation/realisation of impacts and benefits. For example, any sales or turnover impacts attributed to support may be predicated on the firm launching a new product or process which may take years to develop/launch and even longer to generate a positive financial return.

An evaluation of the large R&D programme identified that the majority of quantifiable impacts (increased turnover and jobs) occurred in later years^[33] - data gathered at the application and post completion stage is unlikely to reflect this time lag (and in our experience having evaluated numerous SE innovation projects/programmes, underestimates the extent of the time lag).

Attribution - this is a key issue and while not a unique challenge to SE, companies can access support multiple times and through multiple products. This makes it challenging to disaggregate impacts/benefits, and also to assess the total impact of the support. For example, a company seeking to bring a new product to market could access SMART funding for the feasibility element and various other subsequent grant awards e.g. R&D to support future product development and funding to attend trade shows to identify new markets/customers, etc.

The SE systems are designed to capture the impacts/benefits of specific/individual awards and do not take account of other support provision (including private investment) as firms move through the innovation journey. This can be further complicated with changes in staffing across supported companies and corporate memory - with companies unable to differentiate between the impacts/benefits of the individual awards (i.e. the data being collected may be subject to inconsistencies).

It is therefore unclear as to what extent impacts/benefits can be directly attributed to the support, although to some extent this is an evaluation question in measuring/attributing the gross and net effects. As noted, SE periodically commission evaluation studies either by programme or thematically.

4.4 Highlands and Islands Enterprise

HIE Measurement Framework

HIE's Strategy and Vision^[34] (2019 - 21) sets out the Agency's commitment to developing a vibrant rural economy but with a focus on community development supporting both people and place.

In common with SE, HIE has a defined measurement system for capturing data to report against its published targets and assess the impacts of its services and activities. This measurement system is aligned to the National Performance Framework and is driven by the priorities of the Scottish Economic Strategy as well as HIE's own strategy and operating plans.

The system is designed to capture both expected outputs and impacts (i.e. forecasts prior to support) and actual outputs and impacts. All of HIE's innovation-related interventions sit within this framework.

The framework is broadly hierarchical in structure with a range of primary measures sitting at the top. It is these measures against which HIE sets targets and these tend to be general rather than innovation-specific.

Nonetheless, impact measures such as jobs created and/ or sustained, increase in turnover and increase in export sales are all relevant impact measures for innovation activities.

Moving down the hierarchy there is a range of supporting measures, some of which are more innovation-specific, such as number of R&D jobs, number of high value jobs, innovation active companies and number of clients engaged in R&D activity.

Further, under the heading of 'business' the framework also identifies more specific measures including the following (all of which could be considered relevant to innovation support).

HIE Supporting Measures

• number of academic knowledge transfer projects complete
• number of clients engaged in knowledge transfer activities
• number of clients engaged in R&D activities
• number of clients implementing change management
• number of clients implementing improved working practices
• number of new licenses
• number of new markets entered;
• number of new patents
• number of new products/ processes developed;
• number of new spin-outs

Finally, HIE also has a series of tracking measures in the form of 'ladders' which track progress in five areas - international trade, innovation, digital maturity, business values and social enterprise capacity. The innovation ladder comprises five stages - chance, intent, active, embedded and evolving - each of which assess the extent of commitment to, and action on, innovation within organisations. These measures are primarily used with account managed companies and organisations with which HIE has an ongoing relationship thereby allowing this kind of tracking over time.

All of HIE's activities are measured via this framework although there is the option to include further project specific measures as required.

R&D Grants

Similar to SE, HIE also distributes R&D grants. As described above, applicant businesses are required to provide details of the R&D project and how the funding will be used along with plans for commercialisation and forecasts of the potential economic impacts. The measures used to assess progress and performance are contained within the HIE Measurement Framework and include:

• number of businesses engaged in R&D activity;
• business expenditure on research and development (BERD);
• number of R&D jobs created;
• increase in turnover.

For larger grants (>£100K) HIE collects actual impact data from the beneficiary companies and seeks some form of evidence to verify claims (similar to the processes used for Regional Selective Assistance grants). This is not usually required for smaller grants.

Northern Innovation Hub

The Northern Innovation Hub (NIH) is part of the Inverness City-Region City Deal programme and comprises a range of projects and interventions aimed at developing innovative capacity within the Inverness City-Region. Projects include support for business innovation, skills and training and networking across key sectors such as food and drink, life sciences, tourism and creative industries.

Performance measures are defined for each project element and reported on a quarterly basis through a dashboard which summarises spend and benefits against targets.

In relation to project benefits, many of the reported measures are activities or immediate outputs. These include measures such as:

• numbers of events held;
• numbers of companies participating and/ or attending events;
• numbers of businesses using collaborative space (and as tenants);
• individuals accessing training or placements; and
• businesses participating in training courses.

One project element (Impact30 - a business support programme) has identified impact measures in the form of jobs created and additional turnover, but these are forecasts at this stage. Actual data will be collected later in the project lifespan at agreed milestones.

As the NIH is part of the City Deal Programme, and is part funded by the UK and Scottish Governments, there will be a requirement to report against economic impact targets (through the Monitoring Assurance Framework). It is, however, still too early in the life of the project for this data collection to have taken place.

Innovation Vouchers

Data collection for the Innovation Vouchers programme is discussed above in the section relating to SFC interventions. However, in common with SE, HIE has contributed additional funding to supporting Advanced Vouchers. In addition to the data reported by Interface, HIE also undertakes its own data capture for this investment which tracks:

• number of clients engaged in knowledge transfer activities;
• number of clients engaged in R&D activities;
• number of academic collaborations completed; and
• increase in turnover.

These measures are all included within the HIE Measurement Framework. It should be noted that HIE will no longer be funding this programme forthwith.

Innovation Centres Programme

Along with SFC and SE, HIE is a partner in the Innovation Centres Programme with a particular involvement in the Scottish Aquaculture Innovation Centre (SAIC), reflecting the importance of the aquaculture industry to the HIE region.

There is little in the way of additional performance measurement beyond that which is described above as provided through the Innovation Centres' MEF reports.

Aquaculture Fund

The Aquaculture Fund is a small fund supporting R&D and training for aquaculture innovation projects. Again, performance reporting is aligned to the HIE Measurement Framework and tracks:

• number of R&D jobs created;
• number of new products/ services commercialised;
• increase in turnover;
• export sales;
• jobs created; and
• jobs created in fragile areas.

Other Areas of Activity

Again in common with SE, HIE will contribute to innovation related outcomes and impacts through more general support services, in particular through account management. As such, these outcomes may not always be attributed to innovation-related expenditure.

Thus, while HIE can report against broad innovation-related outcomes and impact such as new products and services created/ commercialised, R&D jobs and BERD, a proportion of these impacts may arise from general rather than innovation-related support.

HIE also raised the issue of wider infrastructure and its role in supporting innovation, with UHI Inverness Campus as an example. These investments are not captured within the current scope of innovation support yet may make a contribution to innovation performance through the development of company and R&D facilities, for example, digital infrastructure could be considered another example.

Summary

Like SE, HIE has a defined measurement framework against which projects and programmes report. Partnership projects and those with third party funding, such as those in the Inverness City Deal or the Innovation Centres Programme may have specific measurement frameworks agreed by the partners, and defined reporting protocols. These tend to be based on logic model structures and include useful and valid measures of innovation. Issues are most likely to arise in the measurement of impacts, particularly if projects are long term in nature, requiring tracking of beneficiary firms over time. Lengthy timescales are typical in innovation support, and there is a need to consider the resource requirements of longer term tracking.

As noted earlier, even where there are well-defined measurement frameworks (e.g. Innovation Centres) issues with the consistency of data collection and reporting practice remain.

This is partly due to differing interpretations of the measures, and partly due to the practical challenges of collecting reliable data for companies, particularly relating to historic support.

Comparing HIE's approach to that of SE, it seems that HIE's overarching measurement framework contains more in the way of detailed measures of innovation activity e.g. companies involved in knowledge transfer activities. In some places, the agencies are measuring similar things, but expressed in different ways. For example, SE measures the 'number of collaborations (businesses to business and business to academia)', and HIE measures 'number of clients engaged in knowledge transfer activities'. Given the cross-cutting nature of the enterprise agencies objectives and activities, this therefore suggests there is some scope for greater consistency and alignment in terms of collecting and reporting performance data (considered later in this paper).

Wider Comments

It is clear that none of the agencies have developed frameworks for the specific goal/purpose of measuring and monitoring the impact of their investments in innovation support.

Most have included additional measures within existing (often long established) frameworks designed to measure wider agency targets/objectives. These frameworks are necessarily quite high level and too broad to offer a detailed account of the impacts of innovation support.

As noted throughout, there are numerous challenges with regards to accurately and consistently reporting the benefits/impacts of innovation e.g. timescales to impact, attribution, failures, fluidity within the innovation journey, etc. While these challenges remain, a more specific framework for measuring innovation can at least partly address these. Interestingly, the Innovation Centres (which is a collaboration across the three agencies) demonstrates the closest alignment with the conceptual Framework and was designed with the sole purpose of tracking and reporting innovation activity. However, as noted, implementing this monitoring Framework has not come without its challenges.

This is considered further below at Section 5 which provides a high level mapping of the agencies against the conceptual framework.