Coronavirus (COVID-19) business support measures: evaluation

2. Evaluation Approach

This report considers the outputs and indicative outcomes of the COVID-19 business support measures available in Scotland up to summer 2021 to guide ongoing policy thinking on business support as the COVID-19 situation evolves. Outputs are considered in terms of numbers of businesses and jobs supported and indicative outcomes are considered in terms of the extent to which the measures introduced have helped businesses survive through the immediate crisis. It is an action-focused exercise that combines ongoing monitoring of uptake of support under each of the schemes with an assessment of what the schemes are achieving.

The initial evaluation report (completed in summer 2020) focused on the package of support covering the first lockdown in spring-early summer 2020. This report builds on the initial evaluation report and covers the extended package of support available up to summer 2021. Further evaluation beyond this point will be considered depending on the direction of the pandemic.

2.1 Data Sources

Qualitative and quantitative data for the evaluation is drawn from a range of sources as outlined below to identify and measure the outputs and short-term outcomes of the business support measures. It should be noted that due to the nature of some these statistical publications, data lags exist and some figures may refer to different years or periods. The statistics set out in this report are the most recent data available at time of writing:

• scheme management information on applications and approvals
• data on scheme uptake and the impact of the overall package of support from the Office for National Statistics (ONS) Business Insights and Conditions Survey (BICS) and the UK Government Department for Business, Energy & Industrial Strategy's (BEIS) Longitudinal Small Business Survey.
• business intelligence on the immediate impacts of support gathered from the business representative organisations and enterprise agency account managed businesses (individual businesses were not surveyed given the pressures they are under due to the pandemic and the availability of the data set out in the bullets above)
• OCEA's Strategic Framework Sector Viability modelling framework, which simulates the impact of COVID-19 on business viability and evaluates the impact of Government support across sectors.

At the time of writing the initial evaluation, in summer 2020, there were limited timely sources available for measuring the impact of the pandemic. This second evaluation report has the advantage of being undertaken over a year since COVID-19 was first identified in Scotland, which allows the analysis of, now published, secondary sources that measure the impact of the pandemic on Scotland's economy:

• existing macro and micro economic data on impact of COVID-19 on businesses and the economy
• existing published business surveys and enterprise agency business intelligence.

Scheme management information data

Information on applications and approvals (by number and value) is available from the management information generated from administration of each scheme. This second evaluation report has limited scope to cover schemes that are closed, however the management information data for certain schemes is still incomplete with some schemes still processing applications or appeals. For some schemes, the figures presented in this report may therefore be subject to change which will be made clear. In addition, data for Scotland is not yet available for some of the UK Government schemes. For other schemes, such as non-domestic rates (NDR) relief, the management information is still being processed. Further updates will be provided as more evidence becomes available.

The extent to which disaggregated management information is available differs significantly by each scheme. Where possible, breakdowns are provided by:

• sector: where possible, breakdowns are provided at UK Standard Industrial Classification (SIC) section level.
• geography: where possible, breakdowns are provided by local authority and by urban and rural classification using the SG's Rural and Environmental Science and Analytical Services (RESAS) Division's classification which classifies local authorities according to their level of rurality. The RESAS classification of local authorities as either urban or rural can be seen in Annex 3.
• firm size: where possible, breakdowns are provided by business size based on number of employees: micro (1-9 employees); small (10-49 employees); medium (50-249 employees); and large (250+ employees).
• equalities characteristics: equalities information is not available for most schemes, with the exception of the UK Government's Future Fund, Job Retention Scheme and Self-Employment Income Support Scheme.

Business Insights and Conditions Survey

The Business Insights and Conditions Survey (BICS) is a UK-wide fortnightly business survey, published by the ONS. It delivers real-time information to help assess the impact of issues and events affecting the economy and UK businesses in areas including financial performance, workforce, prices, trade and business resilience. The survey was previously named the Business Impact of Coronavirus Survey (BICS) but has since been renamed to reflect all aspects of the questionnaire but its purpose remains the same. Some of the Scotland level data is currently classed as experimental statistics.

The results for Scotland presented here have been weighted using the SG's Office of the Chief Economic Adviser's (OCEA) own weighting system to be more representative of the business population in Scotland. Note that in some cases, the weighted data for Scotland cannot be compared to ONS' published UK data.

The results in this report relate to different waves of the survey depending upon the latest available wave that questions were asked (as not all questions are covered in each wave). The data is for businesses with 10+ employees with a presence in Scotland. The coverage of micro businesses (1-9 employees) in BICS for Scotland is currently not robust. Breakdowns are provided for sector and business size (SMEs and large businesses), however breakdowns by geography are not available at this time due to small sample sizes. The results for Scotland, particularly the sectoral results, should be treated with caution given small sample sizes. Where possible, it should be corroborated with other sources to test for robustness.

Longitudinal Small Business Survey

The Longitudinal Small Business Survey is an annual UK-wide survey of small and medium-sized enterprises (SMEs) commissioned by BEIS. The 2020 survey, published in August 2021, asked questions on the uptake and impact of COVID-19 business support offered by both the Scottish and UK Governments. Fieldwork was undertaken during the pandemic over the period September 2020 to April 2021. High level data is provided for Scotland in the publication with a sample size of 505 for SME employers and 158 SMEs with no employees (sole traders). The results for Scotland should be treated with caution given small sample sizes.

High level results for Scotland as a whole will be referenced in this report, however disaggregated results for Scotland will be available with the subsequent publication of the Small Business Survey Scotland 2020. These disaggregated results, which will include breakdowns by sector, size and urban rural location, will be included in the full evaluation of COVID-19 business support schemes.

Business intelligence

OCEA invited representatives of business organisations to provide feedback on the business support measures introduced to mitigate the impact of the COVID-19 pandemic and health measures on businesses. Views were invited on:

• the extent to which the package of support offered by the Scottish and UK Governments assisted businesses to continue trading and maintain employment, and scheme(s) that had the biggest impact
• the extent to which SG support addressed gaps in the support offered by the UKG
• the extent to which the application process and information provided for the SG schemes was sufficient and easy to navigate
• any recommendations for potential similar support in the future.

Returns were received from the Federation of Small Businesses (FSB), the Scottish Chambers of Commerce (SCC), the Scottish Tourism Alliance (STA), and the Institute of Directors (IoD). Supplementary information was also provided by most of these organisations, and others who did not respond to the OCEA questionnaire, in the form of surveys they had undertaken of their own members and letters sent to Scottish Ministers on the package of business support provided. Intelligence gathered by Scottish Enterprise and Highlands and Islands Enterprise from their account managed firms provided further evidence.

OCEA Strategic Framework Sectoral Viability Model

To better understand the sectoral impacts, including the viability of businesses, OCEA has developed a framework for assessing how the COVID-19 pandemic and lockdown are affecting business profitability across different sectors of the economy. The framework simulates the impact of COVID-19 on sectoral profitability using survey data from Scottish Annual Business Statistics.

The framework applies estimates of impacts on output for each sector under different levels of restrictions to simulate the projected impacts of COVID-19 on business viability in terms of profitability and risk to employment. It evaluates the impact of government support such as business rates relief, the Job Retention Scheme and other grants. The framework also allows for monitoring of how profitability is impacted by the re-opening of the economy as we emerge from lockdown. This will inform assessment of the need for on-going support to sectors of the economy and the risk to businesses and jobs as the COVID-19 context changes.

There are some notable sectors missing from the results of this modelling presented here. The methodology used for this analysis cannot be applied to sectors reliant on atypical sources of revenue. For example, at section level, Education and Health & Social Work sectors are reliant on public sector funding rather than accruing turnover through sales of goods and services, and so business viability is not aligned to profitability as estimated using this method. Similarly, Arts, Entertainment & Recreation, while recognised as being among the sectors most significantly affected by the pandemic, is not included here – this sector contains sub-sectors reliant on atypical sources of revenue, as well as sub-sectors for which underlying base data is deemed to be unreliable.