AN EVALUATION OF THE COMMONWEALTH GAMES 2014 LEGACY: PRE-GAMES REPORT

2. Aims, Key Questions and Methods

Introduction

2.1 The SG, GCC and partners are committed to assessing progress on legacy. The Games Legacy Evaluation Working Group (GLEWG) was established in January 2012 to design and deliver an evaluation of legacy from the Glasgow 2014 Commonwealth Games^[5]. GLEWG agreed that an evaluation would be of value to Parliament, government and Games partners, as well as to future bidders and hosts. It will also add to the international evidence base on legacy from major sporting events.

2.2 To help finalise the evaluation questions and design we conducted a literature review on whether, and under what circumstances, major international multi-sport events create legacy. We also examined the methods used to measure and evaluate legacy plans and programmes. This updates a previous review conducted in 2012. A synthesis of the literature is presented in the evidence review published alongside this report^[6]. Findings from this review appear throughout. These findings have also been used extensively by SG and partners in legacy planning.

2.3 Overall, the evidence on legacy from major sporting events varies in quality and rigour. However, it tells us that hosting major events can potentially create legacy, but it won't happen by chance. Successful legacies are well-planned, well-delivered and embedded in existing strategies, policies and programmes.

2.4 Early decisions made on the evaluation design were set out in our first report^[7]. GLEWG concluded that a 'counterfactual' or 'control' was not feasible. By 'counterfactual' we mean what would have happened in the absence of the Games. Many legacy programmes are embedded in existing programmes. This makes the generation of a 'counterfactual' position challenging.

2.5 This, in turn, makes it difficult to attribute legacy outcomes to specific Games-related programmes. Rather, this evaluation has been designed (where relevant) to assess the contribution that programmes are likely to have made to outcomes. 'Contribution analysis'^[8] can be used when it is not feasible to design an 'experiment', and where it is clear that multiple interventions might affect outcomes.

2.6 This decision on the counterfactual has implications for the whole evaluation, in particular for the economic assessment. It will not be possible to provide a single figure of economic impact. Rather, the economic assessment will provide a series of data that describes the economic contribution of the Glasgow 2014 Commonwealth Games.

2.7 Finally, this evaluation has been developed within constrained resources. A clear principle of proportionality underpins the design. We have made best use of existing data where possible and have added questions to existing surveys to help address the key questions. We have only supported new work where there is a clear gap that could not be filled in any other way.

Aims and key questions

2.8 At the outset the challenge posed for this evaluation was to assess the extent to which legacy outcomes have been achieved between 2008 and 2019. We are tracking indicators across all national themes and monitoring all national programmes. However, given both methodological challenges and the available resources, it was important to focus on some key questions.

2.9 After considering the evidence from major sporting events and consultation with the Games Legacy Executive Board (GLEB) the six key questions we agreed to ultimately address are:

1. What effect have the Games had on the economy, businesses, employment, training and volunteering? (theme - flourishing)

2. What effect have the Games had on the community in the area of Glasgow's East End most directly affected? (theme - sustainable)

3. What effect have the Games had on infrastructure, sports participation and physical activity? (theme - active)

4. What effect have the Games had on the cultural sector and cultural engagement?(theme - connected)

5. What effect have the Games had on civic pride and/or international reputation?(theme - flourishing and connected)

6. Is there a partnership legacy from the Games?

2.10 These questions vary in their relevance spatially. Some are relevant nationally, some at city level and some in Glasgow's East End where much of the new investment has been. Table 2.1 sets out in more detail the key questions, supplementary questions and the relevant spatial level.

2.11 A baseline of 2008 has been chosen as the 'before' year, with the years 2009-2019 the period when strategies and programmes are delivered.

2.12 There are also some questions we have posed, given some of the theoretical debate to date on hosting major sporting events. This includes questions about the 'demonstration' and 'festival' effects described in the literature on physical activity legacy (see Chapter 5). The evidence also suggests that partnership working can, in theory, expand and improve as a result of hosting a major sporting event. We plan to explore the impact of the Games on the nature and quality of partnership working in Scotland.

2.13 There may also be additional questions that Glasgow, as a host city, address directly, particularly on the impact of their legacy programmes. A Glasgow Evaluation Group (GEG) is addressing these.

2.14 There are many additional or alternative questions that could be asked. This evaluation cannot address them all. We hope research conducted by academia and the wider public sector in Scotland will add to the evidence base on legacy. To this end we have joint funded two PhD studentships with the Economic and Social Research Council (ESRC).

Methods

2.15 Our evaluation design comprises of seven key components. The first two components cover all the national themes and programmes. The remaining components are designed to address the evaluation questions set out above. We are undertaking, or supporting, the following:

1. Tracking over 50 outcome indicators from 2008-2019
2. Monitoring and evaluation of national legacy programmes in collaboration with partners
3. A new community longitudinal study (GoWell East) as part of the GoWell programme in Glasgow
4. An economic assessment
5. Secondary analysis of key national surveys
6. Online and social media monitoring
7. Primary research on partnership and collaboration

Tracking Outcome Indicators

2.16 A set of 52 outcome indicators are being tracked from a 2008 baseline to 2019 across all four national themes. These provide an important 'canvas' or 'backdrop' for the evaluation.

2.17 The selection of these indicators was based on four criteria, including: quality, regularity, disaggregation and relevance. The types of data used as indicators are statistical data, survey data and indices which provide a ranking.

2.18 We commissioned a 'Data Review in support of the Commonwealth Games Legacy evaluation' in 2013^[9]. The review concluded that the current indicator set is close to optimal if we are using already available data and statistics. However, some gaps were identified including volunteering among young people and national data on civic pride.

2.19 Data are at Scottish, Glasgow, and East End level, where relevant data are available. By 'East End' we are referring to the area in Glasgow where there has been substantial investment due to the Games. A study area was defined which accords with the GCC East End Local Development Strategy. This is also the 'GoWell East' study area that we refer to in this report (see paragraph 2.32).

2.20 Changes in outcome indicators would theoretically be expected at different points in time. While we present baseline data from 2008 we would not expect to see a change in some of these indicators due to the Games at this point.

2.21 These indicators are described in Annex 1. The data now appear on a website entitled 'Assessing Legacy 2014' which is updated twice per year www.scotland.gov.uk/AssessingLegacy2014. The data are also set out in Annex 2.

Monitoring and Evaluation of National Legacy Programmes

2.22 There are currently over 50 national legacy programmes underway in Scotland. Some are completely new, some were existing programmes that have been scaled up, or expanded. Over 80 legacy programmes are underway in Glasgow. GCC and their partners will be reporting on these projects separately.

2.23 Evaluating every one of the national legacy programmes would be disproportionate to the Games investment. However, we are keen to champion evaluation of the larger programmes, particularly where they will help directly address the 6 key research questions.

2.24 Almost all of the national programmes are now providing monitoring data on a six monthly basis. The next update of Assessing Legacy 2014 will be in summer 2014.

2.25 The SG and a number of partners have plans to evaluate their legacy programmes. These broadly fall into two categories. Firstly, a relatively small number of legacy programmes have commissioned external evaluations. These tend to be the larger programmes and include Community Sport Hubs, Culture and Festival 2014 and Game on Scotland, the education programme for the Games.

2.26 Secondly, and more commonly, delivery organisations are expecting to self-evaluate. To help champion, and advise on, self-evaluation we have provided written guidance^[10] to national legacy leads and to Local Authorities developing legacy plans. We also provide support through our knowledge hub where an information video is available.

Longitudinal Community Study in Glasgow's East End (GoWell East)

2.27 Our second question asks what effect the Games (and broader regeneration activity) has had on the lives of the community in Glasgow's East End where much of the new investment is directed. This is not something we can readily answer from existing data. We have supported a new study area on the GoWell programme and refer to it throughout this report as 'GoWell East'.

2.28 GoWell^[11] is Glasgow-based research programme that provides evidence on the impact of regeneration. The study generates evidence on how neighbourhood regeneration and housing investment affect individual health and wellbeing; the degree to which places are transformed through regeneration and housing investment; and the effect of regeneration policy on area-based health and inequalities.

2.29 GoWell East is a longitudinal community study with several components: three survey sweeps; a qualitative component; and an ecological analysis of health and deprivation indicators. As far as we know, this is the first longitudinal community study in a Games 'host population'.

2.30 GoWell East will allow us to identify changes among a cohort of the adult population living in the East End. These include changes in physical activity and wellbeing, perceptions and experiences of neighbourhood and housing, experience and outcomes of employment, learning and volunteering, engagement with legacy programmes, and support for, and engagement with, the Games themselves.

2.31 In paragraph 2.4, we discussed the difficulties of generating a 'counterfactual'. A benefit of building on the GoWell programme is the scope for comparability with other regeneration areas in Glasgow. This helps address the question of 'additionality' of the Games-related regeneration if the effects are positive.

2.32 The study area covers 6 communities in Glasgow's East End - Calton, Camlachie, Gallowgate, Parkhead (in part), Dalmarnock, and Bridgeton. The study area comprises 27 datazones, 21 of which are in the 15% most deprived in Scotland^[12]. The area includes the Emirates Stadium, the Velodrome and the Athletes' Village, with the International Hockey Centre and Tollcross International Swimming Centre on the fringes. The study area is below.

Figure 2.1: GoWell East study area

2.33 A total of 1,015 interviews have been achieved in Sweep 1. The headline findings from this sweep were published in May 2013^[13]. Sweep 2 will commence after the Games when the Games venues have reopened to the public in 2014, and sweep 3 is scheduled for 2016.

2.34 In 2014 the GoWell Team will also be developing the ecological component of the study and conducting qualitative research. The ecological work uses secondary data sources to examine changes in the relative position of the GoWell East study area across health, demographic, social, economic and environmental indicators.

Economic Assessment

2.35 The SG and GCC have established a Games Legacy Economic Evaluation (GLEE) group^[14] to deliver the economic assessment. The remit of this group is to produce a high quality, robust evaluation of the economic legacies from the Glasgow 2014 Commonwealth Games.

2.36 The economic assessment will provide a well-evidenced narrative of the economic contributions of the Games over delivery and legacy phases, at different spatial levels, across five topic areas. This includes the short term impacts of the Games activity directly associated with the delivery and hosting of the event, but also includes the potential longer-term legacies for business, the labour market, the tourism and events industry and the physical environment.

2.37 The economic assessment will have a number of components. Firstly, the SG Input Output (IO) Framework will be used to estimate GVA and employment impacts from the Games-related capital investment programme and the activities of Glasgow 2014 to prepare and host the Games.

2.38 Secondly, to examine the impact of visitors, a primary study has been commissioned. The project is a collaboration between SG, GCC, Glasgow Life and Glasgow City Marketing Bureau (GCMB). Fieldwork will take place during the Games and the survey will report in late 2014. When tourism national statistics are available in early 2015 it will then be possible to produce a Games-time visitor analysis with standard considerations made for leakage, deadweight, displacement and substitution effects.

2.39 Finally, we will draw on evaluations of legacy programmes, survey data and the GoWell East study to assess whether longer term legacy is achieved. Further primary research may also be required after the Games. A full report from the economic assessment will be available in 2015.

Secondary Analysis of Key Surveys

2.40 We have, and will, undertake further analysis of the underlying trends and patterns for some outcome indicators, where they help address the research questions specifically. Some of this analysis appears in this report.

2.41 The aforementioned Data Review recommended we make better use of longitudinal data sources. We have negotiated the inclusion of questions on engagement with the Games in the Scottish sample of Understanding Society (US) in 2014. This will allow an analysis of the relationship between Games engagement and taking part in sport. We have also proposed new questions on physical activity in the birth cohort study Growing Up in Scotland (GUS) to assess whether there is any evidence of a 'Commonwealth Games cohort' of children in Scotland whose physical activity is influenced through engagement with the Games.

Media monitoring

2.42 Online and social media monitoring and analysis has been developed. This will analyse and evaluate UK and overseas online media coverage and social media mentions of the Glasgow 2014 Commonwealth Games.

2.43 This will provide evidence for one of our key questions "Has there been a change in civic pride and/or international reputation?". Specifically, this work will help assess the contribution that the Games have upon the reputation of Scotland and Glasgow, both in the UK and internationally, across a range of domains including creativity, heritage, culture, arts, food and transport (among others).

2.44 We will focus on volume of mentions, topic of mentions and sentiment. English-speaking online media outlets and online social media platforms (including Twitter) will be monitored. The time period over which data will be analysed is pre-Games (January-July 2014) and post-Games (August-December 2014).

Partnership research

2.45 Finally, we plan to examine the longer-term impact of the Games experience on the nature and scale of partnership working across sectors in Scotland with a view to drawing out the transferable lessons for the public sector. This is likely to be qualitative work with partners, and will commence after the Games.

Conclusion

2.46 Table 2.1 provides a summary of the 6 key questions, supplementary questions and the methods we expect will address each of the research questions. We also set out what spatial level(s) the research question will be addressed.

2.47 It is too early to make summative statements about legacy from the Games. However, this report sets out progress to date and provides a prospective assessment of the plausibility of achieving legacy outcomes. It draws on:

• The evidence review published alongside this report
• Programme monitoring data
• Initial findings from GoWell East study
• Early data from indicator tracking (where appropriate)
• Economic modelling of capital investment

Table 2.1 - Analytical Plan Key / supplementary questions	Area
A=active; F=flourishing; C=connected; S=sustainable; O-overarching	Scotland	Glasgow	East End	Outcom Indicators	Secondary analysis	GoWell East	Monitoring and evaluation	Economic assessment	Online media monitoring	Other primary research
What effect have the Games had on the economy, businesses, employment training and volunteering? [ F ]
What has been the effect of Games investment on economic outcomes?
How do the outcomes of the Games investment compare with other investments?
Have legacy investments and programmes which aim to support businesses contributed to change? What can we learn for the future?
Have legacy investments and programmes which aim to support people into employment, training and volunteering contributed to change? What can we learn for the future?
What effect have the Games had on the lives of the community in the area of Glasgow's east end most directly affected? [ S , A , F ]
What change is observed over time in key legacy outcomes e.g. (physical activity, cultural engagement, volunteering)?
Have there been changes in the physical, social and retail environment over time?
How do changes in key outcomes in this area compare with other regeneration sites in Glasgow? Is there additionality?
What effect have the Games had on infrastructure, sports participation and physical activity? [ A ]
Are sustained changes in physical activity and sports participation evident among particular groups?
Is there evidence for a 'demonstration effect' among those already, or recently, physically active?
Is there evidence of a 'festival effect' among those who are currently sedentary?
Have legacy investments and programmes which aim to increase physical activity contributed to change? What can we learn for the future?
What effect have the Games had on the cultural sector or cultural engagement due to the Games? [ C , F ]
Are sustained changes in cultural engagement evident among particular groups?
Have legacy investments and programmes which aim to increase cultural engagement contributed to change? What can we learn for in the future?
What effect have the Games had on civic pride and / or international reputation? [ C , F ]
Has there been a change in civic pride among particular groups?
What aspects of international reputation have changed over time?
Is there a partnership legacy from the Games? [ O ] How have organisations worked together to maximise the Games' opportunities? Did partners capitalise on the impacts of the Games post-2014? What are the lessons for the future?