National Partnership for Culture: independent report

5. Data and Evidence

The Measuring Change Group (MCG) was convened in 2020 as a standing sub-group of the National Partnership for Culture. The MCG's remit is to advise the NPC on appropriate measures, data and research that would best support decision-making related to the delivery of the Culture Strategy for Scotland.

The MCG provided an information paper based on their work programme and hosted an NPC workshop in September 2021. These underpin the following recommendations. A summary of the information paper and the workshop output report will be published alongside the NPC's recommendations.

The current data landscape is fragmented, in terms of both the cultural workforce and public engagement with culture, and in relation to readily available information on the overall level and distribution of public funding for culture. There is a clear opportunity for improvement in these areas.

Alongside the challenges in data and evidence availability, it is important to note that the Culture Strategy itself, which the MCG was founded to advise upon, was not written in a way that is easily measurable. The Culture Strategy sets out bold and ambitious aspirations and contains several actions, but these are generally not linked to specific outcomes, indicators or targets, which hampers efforts to monitor and evaluate its impact.

Considering the context set out above, we propose the following three overarching recommendations.

5.1 The four National Performance Framework indicators for Culture should be reviewed, to better encapsulate the spirit and priorities of the Culture Strategy.

The Scottish Government's outcome for Culture ('We are creative and our vibrant and diverse cultures are expressed and enjoyed widely') is currently measured by four indicators:

• Attendance at cultural events or places of culture (percentage of adults who have attended or visited a cultural event or place in the last 12 months).
• Participation in a cultural activity (percentage of adults who have participated in a cultural activity in the last 12 months).
• Growth in the cultural economy (the amount of income generated by businesses, measured by Approximate Gross Value Added [aGVA], of the Creative Industries Growth Sector [GBP Millions]).
• People working in arts and culture (the number of jobs in the Creative Industries Growth Sector [culture and arts]).

These are relatively narrow quantitative indicators, which are being used to measure a largely qualitative outcome. On the basis that the National Performance Framework is due to be reviewed in 2023, we propose that an updated set of indicators should have a mix of quantitative and qualitative measures, which seek to evidence tangible and meaningful change related to the National Outcome for Culture. The indicators should be focussed not only on the sector but should also include the perceptions of the public: what they value about culture, how culture brings meaning to their lives, and the cultural activities they find meaningful. The cultural values of children and young people should also be considered alongside those of adults. Sufficient time and consultation should be factored into the refresh process to allow these indicators to be altered accordingly.

Additionally, the Scottish Government's planned refresh of the Culture Strategy's work streams and Action Plan should aim to provide clarity about any intermediate outcomes and indicators that will be used in understanding its impact and explaining its contribution to the overarching National Outcome for Culture.

5.2. The primary generators and users of cultural data should be asked to standardise, manage, and share current, ongoing and future data sets so that cultural organisations and artists are not required to provide the same data repeatedly and a wide range of cultural organisations and bodies can easily access, use and contribute to them. This should include the Scottish Government providing updated guidance to the national public bodies about the key data they should be collecting from the culture sector in a standardised manner to aid aggregation and comparison.

To do this, the Scottish Government should create clear guidelines for the relevant public bodies (those who regularly undertake or commission data collection), which should set out: what key data should be collected and monitored; how it should be collected in order to ensure consistency and facilitate aggregation; for what purpose it is being asked; and how it should be shared. Consideration should also be given to how duplication and repetition of requests for artists and organisations to provide similar monitoring data to multiple agencies can be reduced or removed entirely.

Alongside this, there should be a joined-up approach to evaluation and research, to increase the efficiency and coherence of this work and data collection related to it. Evaluation should be proportionate to the scale of the work being evaluated, and the focus should be on less, but better evaluation that is oriented towards learning and the potential for work to be scaled up or learnt from in other areas of Scotland.

The need for improved data sharing relationships between Government, local authorities, national bodies, and organisations underpins this proposal, and in delivering on these recommendations, the Scottish Government should explore whether there is a requirement for: 1) additional investment within Government in data analysis and expertise dedicated to the culture sector; 2) key cultural public bodies to fully collaborate in ensuring they align and standardise their approaches to data monitoring and ensure they collectively adopt a strategic approach to evaluation and research that builds on rather than replicates existing work; and/or 3) the creation of a single, neutral third party, to look across the data landscape, akin to the role played by HESA in the Higher Education sector.

5.3 The Scottish Government should commit to undertaking a meaningful evaluation of the Culture Strategy for Scotland and the extent to which the Strategy has produced change.

The Culture Strategy did not set out a timeframe for the achievement of its aims and aspirations nor clear outcomes against which the impact or effectiveness of the Strategy can be assessed.

The Scottish Government should work with stakeholders and national public bodies to decide which outcomes and indicators are appropriate to these aims and aspirations. This should take into account the MCG's proposed long list of potential indicators, which will be included in the output report when published. Following this, the Scottish Government should produce a terms of reference for an independent evaluation capable, at an appropriate 'mid-term' point, of a) assessing the qualitative and quantitative dimensions of progress towards delivering the aims of the Strategy, and b) capturing lessons on the quality of its implementation. In designing and delivering this evaluation, qualitative data should be placed as equal in value and importance to quantitative data.

The Scottish Government should also require the taskforce, detailed in 2.1 above, to monitor and publicly report on progress towards delivery of the agreed outcomes on an annual basis, drawing on the shared monitoring data proposed in 5.2 above and any interim data generated as part of the agreed evaluation framework.