A Healthier Future: analysis of consultation responses

12. Monitoring change (Q13)

12.1 The consultation paper set out the commitment to ensuring that the development of Scotland’s approach to improving health is based on robust evidence. The paper confirmed that a monitoring and evaluation programme to measure the impact of the new proposals would be put in place, and that a biennial international conference to measure progress and share good practice would be hosted.

12.2 Respondents were asked for their views on whether further steps should be taken to monitor change.

Question 13: What further steps, if any, should be taken to monitor change?

12.3 A total of 207 respondents (110 organisations and 97 individuals) provided comments in response to Question 13. For organisational respondents, whether or not a comment was provided varied by organisational type. In particular, a smaller proportion of private sector and business organisations offered comments, as compared to public health and third sector organisations.

12.4 Respondents who commented affirmed the importance of robust monitoring and evaluation procedures to measure progress. They highlighted that this was a complex area and that it was important to develop a consistent approach with well-specified definitions and measures, where outcomes are used to drive further improvement.

12.5 However, respondents differed in their views in relation to the breadth of the data collection which should be undertaken, and – by implication – which outcomes were the most important to monitor. In addition, respondents discussed the strengths and weaknesses of existing measurement approaches (including surveys and routine administrative data) and their possible application to the current proposals. They identified sources of existing data as well as other monitoring and evaluation programmes and approaches that had already been developed, and which might provide a starting point. These aspects are discussed further below. A final section summarises a range of other relevant points made by respondents.

12.6 Note that respondents also used this question as a further opportunity to restate their views on which interventions were most likely to be effective, and which interventions they would wish to see implemented. This included comments about the targeting of any initiatives and in particular the importance of targeting (geographic) areas of high deprivation and the importance of focusing on ‘upstream interventions’. These comments have been discussed in full in relation to earlier questions and are not repeated here.

Data to be collected

12.7 Respondents offered a range of views on what data should be collected in relation to monitoring and evaluation. There were three main perspectives as follows:

• The main focus should be on monitoring weight, and other related measurements such as BMI and waist circumference.
• The main focus should be beyond weight, and should focus on issues relating to health and wellbeing more generally – or ‘quality of life’. This would involve measuring indicators relating to diet, fitness, activity (including uptake of exercise classes), mental health, and so on, as well as measuring impacts on health inequalities. This perspective was particularly common among respondents from public health and third sector organisations.
• Other respondents (including those in the food and drink industry, but also individuals and respondents from a range of other organisations) thought it was important to monitor impacts on health and wellbeing (including weight) but also on retail sales, nutritional content, and consumption. This group argued that since the Scottish Government’s proposals (on promotions, advertising and labelling) were aimed at changing consumption patterns and nutritional content, then any monitoring and evaluation programme should include measurement of these elements. They also argued that these types of measures might give an earlier indication about whether the proposals were having an effect (compared to impacts on weight and / or health and wellbeing which would take longer). These respondents suggested that information should be collected on sales, the provision of product information in food outlets, the nutritional content of foods and nutritional standards generally, and food consumption.

12.8 In all cases, respondents recognised that changing the diet, exercise and eating habits of the Scottish population was a long-term project which might take a generation or more to achieve. Therefore, there needed to be a long-term perspective in any monitoring and evaluation activities.

12.9 The topic of ‘surveillance’ was discussed extensively at the ScotPHN engagement events. Additional key points from these events were that:

• Monitoring and evaluation activities would be useful in addressing the current lack of consistency in interventions (by highlighting which approaches were most effective), and would also improve the arrangements for follow-up of outcomes (which were currently inadequate).
• Economic evaluation should also be undertaken to understand the return on investment in prevention, as well as assessing the cost benefits of treatment.
• Long-term population surveillance would be required, and it was not clear how this could be sustained with only short-term funding. Given the pressures on NHS Boards and local authorities, work in this area may not be prioritised. It was suggested that funding for interventions should be allocated on condition that monitoring / evaluation is undertaken.

Strengths and weaknesses of data collection approaches

12.10 Respondents from a wide range of organisations, but particularly those concerned with public health, asked for a better national surveillance system for monitoring weight. This would include measuring children’s weight at the end of primary school (as had been done previously), but would also involve (i) using GP visits and other health service contacts (e.g. in ante-natal clinics) to measure and record people’s weight, and (ii) encouraging self-monitoring of weight. It was suggested that ‘body fatness’ rather than ‘overweight’ or BMI was the most relevant measure.

12.11 While respondents were supportive of using the Scottish Health Survey ( SHeS) to collect data on weight on an annual basis, it was thought that the sample size would have to be increased considerably if SHeS was to be useful as a monitoring tool at local level. Public sector respondents suggested that disaggregation by local authority, NHS board, integrated joint board, health and social care partnership, community planning partnership, and even individual GP practice would be useful. It was also thought that the way physical activity data are collected within the SHeS should be reviewed.

Learning from other monitoring and evaluation activity

12.12 Some respondents, especially those with experience of the NHS, emphasised the importance of using routinely available administrative data, including GP-held patient data and data on diabetes from the Scottish Care Information – Diabetes Collaboration ( SCI- DC) initiative in relation to monitoring and evaluation. Third sector respondents also described a range of measurement tools which they had developed or were using (e.g. the Cycling Potential Tool, the Hands Up Scotland Survey) and suggested that investment in these tools would be worthwhile.

12.13 Participants in the ScotPHN engagement events suggested that a range of other existing population level data may also be used to support monitoring and evaluation efforts. These included (in addition to SHeS) UK Treasury data, and data collected through Healthy Working Lives and the Healthy Living Award schemes. Reference was also made to Kantar (consumer panel) data which (it was suggested) could be used to monitor the impact of promotions. In addition, it was thought there was an opportunity to use and extend current cohort studies (for example, the ‘Growing Up in Scotland’).

12.14 More broadly, respondents referred to a range of examples where comprehensive monitoring and evaluation programmes or approaches had already been developed, and which could contribute to developing a programme in this area. In particular:

• Monitoring and Evaluating Scotland’s Alcohol Strategy ( MESAS) – developed by NHS Health Scotland – was thought to provide a good example of a comprehensive approach to monitoring and evaluation of a complex series of interventions. Respondents suggested that a similar approach might be adopted in the current context.
• The National Observatory for Obesity ( NOO) set up by Public Health England ( PHE) was thought to provide an example of good practice in relation to monitoring obesity. The approach was thought to be comprehensive and to include key performance indicators across a wide range of topic areas.
• The Scottish Food Enforcement Liaison Committee Diet and Nutrition Working Group ( SFELC) are currently working on monitoring the activities of local authorities in relation to diet and nutrition. These could be built into any comprehensive approach which was developed.
• It was thought that revisiting the Healthy Weight Outcomes Framework would be worthwhile.

Other issues

12.15 The relevance of improvement methods and ‘small tests of change’, and of a regular international conference to discuss progress – as set out in proposals in the consultation paper – were not widely discussed. However, on the occasions where they were discussed, they did not attract support. Respondents did not think that tests of change were particularly relevant in such a complex field where change – if it happened – would take place over a long timeframe. The conference was not viewed as providing ‘value for money’.

12.16 A wide range of respondents emphasised the importance of focusing on children and schools in relation to this agenda. School meals, health and wellbeing in schools, and nutritional quality near schools were all mentioned repeatedly, as was the importance of measuring children’s weight as they left primary school. It was suggested that a ‘nutritional scoring risk rating scheme’ to evaluate nutritional quality near schools should be developed.