National Partnership for Culture: Workshop 4 – measuring change information paper

This is an abridged version of the information paper produced for the National Partnership for Culture, Culture and Measuring Change workshop on 10 September 2021.

If you would like the full version, please email culturestrategyandengagement@gov.scot

This paper offers a brief summary of the work of the Measuring Change Group, a standing sub-committee of the National Partnership for Culture. The original remit, objectives and background to the group are below. 

Background information

Scottish Government made the case for establishing a Measuring Change Group after analysis of the results of the public consultation for the Culture Strategy.

The Group consists of key analytical stakeholders, research experts and academics. They are responsible for providing advice on the development of an appropriate approach to monitoring and evaluating the Culture Strategy.

Group approach

Members met 10 times between September 2020 and September 2021, as a whole group or as part of two, smaller working groups.  The group was mindful that it was not their role to develop specific policies. Instead the discussion focussed on the type of data that would be valuable to be available to those making judgements about progress towards the aspirations in the Culture Strategy.

Members were familiar with existing datasets and their coverage and frequency. This meant the group were able to identify where the current state of knowledge was good, where it could be improved, and where new measures could be necessary.

Throughout these meetings the group have attempted to work in a pragmatic manner remaining aware of the resource limitations that will constrain any data generation and analysis. However they have not limited proposals to only those which are immediately available and/or obviously feasible.

The Culture Strategy sets out ten aims which sit under three high level aspirations.  There are few specific actions/targets/objectives set out underneath these aims. What the Culture Strategy sets out to achieve is neither easily or immediately quantifiable. As such, it was important for the group to avoid inadvertently setting specific objectives that would, in effect, become the delivery plan for the aims.

The group benefitted from a Theory of Change developed by the Scottish Government. This helped clarify what the Culture Strategy sought to change and what might usefully be measured.

The Theory of Change exercise identified inequality as the major challenge. The actions to address this needed:

• to strengthen culture (change within the sector)
• transform through culture (change within government)
• empower through culture (change for people)

In developing its initial recommendations, the group decided that it would  focus on highlighting where attention should be if we rather than proposing a very specific set of measurement indicators.

This is to facilitate informed judgements about the extent to which the aspirations and aims of the strategy are being delivered. The group have proposed a set of six critical questions that they believe should be reflected on annually, informed by a range of relevant quantitative and qualitative data.

Recommendations: critical questions and proxy indicators

The aims of the Culture Strategy are not, in and of themselves measurable. It would always be a matter of judgement as to the extent to which, for example, Scotland recognises each community’s own local cultures in generating a distinct sense of place, identity, and confidence. Rather than attempt to create a set of specific indicators under each aim the group considered the types of questions that, should the aims of the Culture Strategy be delivered, one would expect to be able to answer ‘yes’ to. These critical questions were gathered into six groupings: 

1. does Scotland's arts, heritage and creative industries have a diverse workforce? Is this workforce working under inclusive, accessible and Fair Work conditions? Are decisions made by diverse collectives of decision makers?
2. are we encouraging investment in the arts, heritage, and creative industries to help us to achieve our cross-policy goals and to secure our infrastructure for future generations?
3. does this, and the next generation, of cultural and creative workers have the appropriate skills and levels of education to support Scotland's world class cultural and creative offer? Is everyone with the talent and ambition, irrespective of their background or personal characteristics, supported in their education, learning, and skills development? What are the barriers to skills and educational development and what is the Scottish Government doing to remove them?
4. are our diverse cultural events, organisations and creative products recognised as high quality by our own citizens? Is this excellence visible to audiences and publics internationally, and do they view us as a culturally vibrant nation?
5. do people living in Scotland have equitable access to the spaces, places, and resources they require to pursue their cultural interests in a way that is meaningful to them?
6. do we have the physical assets needed to host, make, and communicate our cultural endeavours? Do we have a healthy base of voluntary and community groups undertaking and enjoying cultural activities?

It is proposed that an independent, representative group, such as the NPC or its equivalent, is charged with reflecting on the questions on an annual basis and coming to a judgement about how they should be answered. In coming to this judgement, the group should be able to draw on a range of evidence/data, some of which is also generated for other purposes and some of which would be generated specifically for this purpose.

The complexity of both the aims of the Culture Strategy and the critical questions proposed by the Measuring Change group mean that judgements can only be made by considering a range of indicators and would require a combination of quantitative and qualitative data. In short, this is because concepts such as ‘openness’ can never be fully or adequately judged through a single proxy indicator.

The group have focused their discussions to date on the type of quantitative data that it could be useful for them to access for each of these six sets of critical questions.

At present some of these data sets are available, some are partially available, and some would require new work to be undertaken. It is unlikely that all of them could be generated with dedicated resources. Therefore it is important to identify which of these data sets are priorities and to find ways to generate priority data if currently unavailable.

General principles

The Measuring Change group have also begun to outline a set of general principles that should inform the approach to any work undertaken in order to use data to make judgements about the extent to which progress is being made towards the aims and aspirations of the Culture Strategy:

• as much as possible data should be generated in such a way as to ensure that it can be disaggregated down to local (i.e., postcode) levels.
• as much as possible indicators should be directional and reviewed as such i.e., what is the direction of travel?
• quantitative data should be considered alongside targeted qualitative data.
• arbitrary targets should be avoided.
• the creation of any new data sets should be centrally coordinated and managed through all relevant agencies in a consistent manner to facilitate consistent comparison and aggregation.
• the generation of new data needs adequately resourced and centrally co-ordinated and not left to relevant agencies to try and coordinate locally.
• data sets should be made openly available to aid wider analysis by other bodies, organisations, and researchers.

Roundtable workshop

This workshop is intended to bring together both users and holders of data in what is, for culture, a particularly fractured landscape that reflects the different ways that culture is defined and engaged with. Please come prepared to discuss the following:

• reflecting on the six sets of critical questions that the Measuring Change working group have suggested need to be regularly reflected on to make judgements on progress towards delivering the aspirations set out in the cultural strategy reach consensus about a prioritised list of quantitative indicators that should be monitored in order to inform such judgements
• reach consensus on the quality and relevance of existing data sources in relation to those quantitative indicators describing the current state of how our sector is being monitored, mapped, and measured 
• reach consensus on how existing data sources can be improved and/or shared better to allow these indicators to be monitored – what is available, what are the gaps, is a bespoke national survey on the cultural sector the best way to fill them?
• identify key barriers and enablers: What are the barriers/enabling factors in gathering and/or sharing the data that is needed most? How much would that cost? Who pays? Who owns the data?
• reach consensus on action: What are the key actions needed to address the above questions? Which of them are for Scottish Ministers? Local authorities? The Convention of Scottish Local Authorities (COSLA)? Cultural organisations? Individuals?

Note produced by the Measuring Change Group