Culture strategy for Scotland consultation: analysis of responses - full report

Monitoring and evaluating the strategy

The consultation paper explains that the Scottish Government wants to establish a Measuring Change Group consisting of key analytical stakeholders, research experts and academics that will be responsible for providing advice on the development of an appropriate approach to monitoring and evaluation of the Culture Strategy. The intention to learn from what works well and to avoid reducing impact evaluation to simplistic target and output indicators is emphasised.

Question 15: What is your view of the proposed approach to monitoring and evaluating the strategy set out in section 5?

Question 16: If you have further comments on the proposed monitoring and evaluation approach, please provide them below.

Table 6: Question 15 – What is your view of the proposed approach to monitoring and evaluating the strategy set out in section 5?

* if figures do not sum to 100% this is due to rounding.

A majority of respondents – 62% of those answering the question – supported the proposed approach to monitoring and evaluating the draft strategy, while 8% did not support the approach and 30% said they did not know. The views of those who did not support the approach and who left a comment are noted at the end of the analysis below. First, the views of those who supported the approach, those who did not know and those who did not answer are considered together, since in practice there was much overlap.

Respondents sometimes suggested they were unable to judge the proposed approach without further detail. However, the intention to monitor was often welcomed or acknowledged to be necessary, with the proposed approach described as interesting, positive, and pragmatic. The need to improve on current monitoring methods was noted, including suggestions that questions on culture in the Scottish Household Survey should be reviewed. Although it was argued that the draft strategy contains few measurable outcomes, that the process will be difficult, subjective to an extent, and will require a lot of work to develop useful measures, it was also suggested that there is an opportunity to:

• refresh how impact is understood and measured.
• develop metrics which measure the pattern of provision and its change over time across Scotland.
• develop a comprehensive evidence base on the health of culture and cultural participation.

Measuring Change Group and stakeholder involvement

Several respondents welcomed inclusion of the Academic Roundtable^[3] as part of the Measuring Change Group or suggested that it will be important to know who the academic members will be. Others noted that they would be interested in knowing who group members are more generally or observed that membership should reflect the diversity of the sector.

Additional suggestions included:

• representation or a subgroup for British Sign Language users. Such representation was suggested to be in line with the British Sign Language National Plan 2017-2023.
• representation for the archives sector, for museums or from across government.
• representation on a rotational basis, so organisations being monitored can feel included.

A small number of respondents noted that they expected to be, or would welcome the opportunity to be, part of the Measuring Change Group as part of the Academic Roundtable³, while respondents from a range of respondent types suggested that they would also like to contribute to the process. It was suggested that there should be transparency around appointments, with terms and conditions for membership made available.

The importance of stakeholder involvement was noted by several respondents and the intent to work collaboratively with key partners was welcomed. In particular, it was suggested that communities and grassroots organisations should be involved.

Approach to monitoring and evaluation

Respondents who commented on statements made in the consultation paper with respect to the proposed approach, most frequently:

• welcomed the commitment to 'avoid reducing impact evaluation to simplistic target and output indicators', sometimes still expressing a view that monitoring will become a set of targets and a tick box exercise.
• welcomed the intention to focus on 'longer-term change rather than immediate impacts'. One respondent urged:

…that the commitment to long-term approaches articulated within this document, are not sacrificed in the desire to demonstrate impact in the short-term.

(Representative or umbrella group respondent)

Other intentions set out in the consultation paper and noted as positive included:

• 'avoiding a restrictive evaluation approach that might stifle the independence and creativity of cultural activity'.
• taking a learning-centred approach to monitoring.
• the commitment to work with key partners.
• alignment with the new National Outcome for culture, and indicators on the refreshed National Performance Framework. It was also suggested that alignment to United Nation's Sustainable Development goals could add further weight.

Respondents also made comments with respect to the types of data they thought should be collected, with the most frequent arguments in favour of a focus on greater use of:

• qualitative (or soft) data.
• longitudinal studies.
• case studies.

It was also seen as important to keep monitoring simple to avoid any additional burden or costs to either delivery bodies or smaller organisations. The need for investment in data collection was suggested, along with ensuring that any new approaches align with those already used in the sector. Other elements suggested to be desirable or important in the system for monitoring and evaluation included:

• guidance and support on collecting and recording data, or a simple tool that grassroots cultural groups can use to demonstrate their impacts.
• regular feedback on progress.

Respondents who did not support the proposed approach sometimes referred to their more general disagreement with the draft strategy or took a view that any monitoring approach would represent additional box ticking. Specific issues raised included that:

• monitoring and evaluation need to be built into the ambitions and actions or should be embedded in creative practice.
• the logic modelling referenced in the consultation paper can be challenging for charities and is not representative of success.