An Evaluation of the Commonwealth Games 2014 Legacy for Scotland Report 1: Questions, Methods and Baseline
This report sets out the broad approach to the Glasgow 2014 legacy evaluation,the research questions it will address, and the range of methods that will be deployed between now and 2019. It also sets out the priorities for the next 18 months and a forward timetable for publication of reports.The report also has a series of Annexes on the 4 national themes (flourishing, active, connected and sustainable).
2 Research Questions and Methods
Aim and Research Questions
2.1 The main aim of this evaluation is to assess the extent to which legacy outcomes have been achieved between 2008 and 2019. A baseline of 2008 seems sensible as the 'before' year, with the period 2009-2019 the period when strategies and programmes are delivered. In consultation with partners we have identified key questions that we consider crucial to address. The research questions are set out in Table 2.1.
2.2 The key questions generally seek to answer whether progress on outcomes and intermediate outcomes has been achieved, and whether legacy programmes are contributing to change. These questions vary in their relevance spatially - some are relevant nationally, some at city level and some in Glasgow's east end where much of the new investment has been made. Some questions sit very clearly within one of the national themes (active, flourishing, sustainable and connected); others are more cross-cutting.
2.3 There are also some questions we would like to pose, given some of the theoretical debate and research evidence to date on hosting major sporting events. We have touched on these in Chapter 1. They include questions about the 'demonstration' and 'festival' effects described in the literature on physical activity legacy and the potential for strengthening partnership and collaboration.
2.4 The questions in Table 2.1 are our priority to address at the time of publication. As the planning and implementation of legacy programmes and projects progresses there may be new and additional questions that the evaluation may seek to address. For example, last month Education Scotland launched plans for 'Game On Scotland', a schools programme for the Glasgow 2014 Games. We will explore how we address questions about whether the programme has stimulated learning and increased international links through schools. Any changes to the core research questions will be highlighted in future reports.
2.5 This chapter also sets out an analytical plan that shows the methods we will use to address these at Scotland, Glasgow and at the East End level (see Annex 5). In many cases, a combination of, or triangulation of, methods will be used to address the research question at hand.
2.6 There will also be additional questions that Glasgow, as a host city, may want to address directly; particularly on the benefits and impact of their programmes. A Glasgow Evaluation Group will work closely with GLEWG on the development of this Scotland-wide evaluation of legacy, and will also tackle questions that Glasgow specifically might want to address beyond this current work.
2.7 Of course, there are many additional or alternative questions that could then be asked about the design, delivery and outcomes of the Glasgow 2014 Games and associated legacy activity. This evaluation cannot address them all, and indeed we hope research conducted by academia and the wider public sector in Scotland will also add to the evidence base on legacy. We are working with the Glasgow Commonwealth Games Partnership Legacy Research
co-ordinator and others to help foster interest in research on Games Legacy. To that end, we have funded two Games-related PhDs through the Scottish Government and ESRC PhD scheme. These will commence in October 2012.
Table 2.1: Key research questions and supplementary questions
| Theme | 6 Key Questions with supplementary questions |
|---|---|
| Active | 1) Has there been a change in sport participation and physical activity?
|
| Flourishing | 2) What effect has the Games had on businesses, employment and volunteering?
|
| Sustainable Active Flourishing | 3) What is the impact of the Games on the lives of the local community in the area of Glasgow's east end most directly affected by the investment?
|
| Connected Flourishing | 4) Has there been a change in cultural engagement?
|
| Connected Flourishing | 5) Has there been a change in civic pride and/or international reputation?
|
| Overarching | 6) Is there a partnership legacy?
|
Methods
2.8 To address the questions set out in Chapter 1 we have designed a programme of research and analysis between now and 2019 (see table 2.4) In principle we plan to make best use of existing data and research including national statistics, national surveys, programme evaluations and other research studies. Where there are clear gaps in evidence we will consider commissioning new work.
2.9 This evaluation will include what might be described as both 'top-down' and 'bottom-up' data and evidence. Our 'top-down' sources include data from national surveys and statistics. These are existing sources from which we have chosen indicators that could, in theory, be affected by the Glasgow 2014 Games investment and delivery. We are also planning further analysis of existing surveys to address the key research questions. Crucially, this will be supplemented by 'bottom-up' evidence from bespoke research and evaluations that should help us ascertain whether the legacy investments and programmes are likely to be contributing to changes in outcomes. The evaluation will comprise of 6 key components:
- Tracking outcome indicators across the 4 national themes
- Secondary analysis of key statistics and surveys
- Prospective Longitudinal Community Study in Glasgow's East End
- Monitoring and Evaluation of key legacy programmes
- Economic Assessment
- Primary research - including on partnership and leverage
A) Tracking Outcome Indicators
2.10 A set of outcome indicators will be tracked from a 2008 baseline to 2019 across all four national themes; active, flourishing, connected and sustainable. The data will be at Scottish, Glasgow, and small area level, where possible and relevant. By small area we are generally meaning the area in the east end of Glasgow with the substantial investment as a result of the Games. For further information on see map 4 in Annex 6. After discussion with GCC colleagues it was decided that a sensible boundary would be that of the GCC East End Local Development Strategy.
2.11 This will give us a broad picture of how we are progressing against legacy outcomes, as well as addressing some of the key research questions. The types of data which could be used as indicators for the Games Legacy in Scotland are:
- Statistical data collected by the Scottish Government (SG) and a range of other organisations or added as a new data collection.
- Survey data collected by the SG and a range of other organisations, or added as a new data collection (including the use of omnibus surveys)
- Indices which provide a ranking. This is normally provided on a yearly basis by private and public organisations to enable comparisons.
2.12 We have consulted widely on the outcomes map and an associated set of indicators for legacy. Some of these data are available now, other measures are in development. A full description of the sources of these indicators is provided in Annex 5. These indicator sets are presented in the themed Annexes of this report. The selection of these indicators is based on a number of key criteria, including: quality, regularity, disaggregation and relevance. Relevance was an informed judgement about whether these are amenable to change due to Glasgow 2014 Games investment. We have also made a judgement about where indicators best fit under themes - but with a clear understanding that many of these cross-cut themes (particularly between the flourishing theme and the others).
2.13 There are clearly data gaps, particularly in the connected and sustainable themes. To address some of those gaps in the short-term we expect to:
- Include new questions in the Scottish Household Survey (SHS) from 2013
- Commission questions on adult and school omnibus surveys
- Develop work with partners to undertake media monitoring
2.14 Worthy of note is that changes in outcome indicators would theoretically be expected at different points in time8 . While we present baseline data from 2008 in the Annexes of this report it is important to note that we would not expect to see a change in many of these indicators due to the Glasgow 2014 Games at this point. We have made clear in the later chapters why indicators have been selected, and when we might expect to influence those indicators.
2.15 Over the coming months we plan to develop web-based reporting for our indicator set which we can update regularly as data become publicly available. As a time-series emerges across the indicators, we will undertake further analysis and testing.
B) Secondary Analysis of Key Surveys and Statistics
2.16 In addition to tracking the key outcomes, we plan to analyse in more detail the underlying trends in the outcome indicators, including spatial analysis and analysis by equalities groups, where possible. Some of this analysis is set out in the themed Annexes, but there is clearly scope for further analysis now, and in the future. In particular, we think there is scope for further analysis of the Scottish Household Survey (SHS) and the Scottish Health Survey (SHeS). It is worth noting that there are additional questions on physical activity and sport in the SHeS in 2012. Further, there will be additional questions on attitudes and engagement with culture in the 2013 SHS. There will also be scope for repetition of these questions in future years.
2.17 However, there are other cross-sectional and longitudinal surveys in Scotland (including UK surveys with a Scottish sample) which we have not drawn outcome indicators from - principally because they provide data less regularly, or for specific age groups. Nonetheless, they could be useful sources in addressing the key questions, either with currently asked questions or with additional questions requested by GLEWG. These potentially include 'Growing Up in Scotland', the UK Household Longitudinal Study (UKHLS), now known as Understanding Society, and the Scottish Health Behaviour in School-aged Children (HBSC) Study. Later in 2012 we plan to commission a 'Data Review in support of the Commonwealth Games Legacy Evaluation'. This will establish how we might use these surveys to address our key research questions, whether new questions on the surveys might be desirable, and at what juncture. This work will be completed by summer 2013.
2.18 In collaboration with the Glasgow Evaluation Working Group, we also plan to explore how we might make best use of data collected (regular or bespoke) through the Glasgow Household Survey (GHS) and the Glasgow Health and Wellbeing Survey (GHWS).
2.19 Finally, there are administrative datasets that might prove helpful including those held by Skills Development Scotland (SDS) and the Department of Work and Pensions (DWP). We will be assessing the feasibility of using these survey and administrative data sources to address questions 1, 2, 4 and 5 in particular. At east end level, these data will also be helpful for the GoWell ecological study described below.
C) Prospective Longitudinal Community Study in Glasgow's East End
2.20 Research question 3 asks how the Games (and broader regeneration activity) has impacted on the lives of the local community in Glasgow's east end where much of the new investment is directed. This is not something we can readily answer from existing studies and sources. To this end we have supported a new study area on the GoWell programme9 as part of the legacy evaluation. This specifically addresses our research question 3 on how the Games (and the broader regeneration activity in the area) has impacted the on the lives of the local community in the area of Glasgow's east most directly affected by the new investment.
2.21 GoWell is an internationally recognised research programme that provides evidence on the impact of regeneration.10 The study generates quantitative and qualitative evidence on how neighbourhood regeneration and housing investment affect individual health and wellbeing; the degree to which places are transformed through regeneration and housing investment; and the effect of regeneration policy on area-based health and inequalities. One of the main benefits of building on the GoWell programme is the scope for comparability with other regeneration areas in Glasgow on some key outcomes. This could allow us to address the question of 'additionality' of the scale and pace of regeneration in Glasgow's east end, if the regeneration effects are positive. Another potential benefit is the ability to collect baseline data at small area level where it is not available through other sources.
2.22 The new study is a longitudinal community study with several components: survey sweeps planned in 2012, 2014 and 2016; a qualitative component; and an ecological analysis of health and deprivation indicators over the study period. In designing the questionnaire for the survey it has been important to ensure consistency in questions with other GoWell study sites. However, there have been new questions added to the questionnaire that are legacy specific. This includes questions on physical activity and sports participation, support for, benefits and expected engagement with the Glasgow 2014 Games, cultural engagement and volunteering. The team are also using a self-completion physical activity questionnaire for children aged 11-16 (at Wave 1) in the respondent households. This makes use of the Youth Physical Activity Questionnaire (Y-PAQ). This instrument was developed by the MRC Epidemiology Unit in Cambridge. It is currently also being used in a study of the impact of the Olympics upon young people in East London. 11 Finally, an ESRC / Scottish Government PhD has just commenced linked to the GoWell programme which is looking at how legacy programmes affect young people's lives and socio-spatial horizons.
2.23 An Advisory Group has been established to support the study.12 They met in May 2012 to agree plans for Sweep 1 and again in July 2012 to review progress with Sweep 1 fieldwork. A questionnaire for Sweep 1 was finalised and went into the field in June 2012. The study area covers 6 communities or sub-areas in Glasgow's east end (Calton, Camlachie, Gallowgate, Parkhead, Dalmarnock, and Bridgeton). It is shown in Annex 6.
2.24 At the time of writing 1,015 interviews had been achieved in sweep 1. Sweep 2 will commence immediately after the Games in 2014. The first sweep results will be available in early 2013 and will be incorporated into our legacy evaluation reporting in spring 2014. In 2013 the GoWell team will be developing the ecological study and conducting qualitative research. Results from the second sweep of the survey will be in the post-Games report in 2015.
D) Monitoring and Evaluation of Key Legacy Programmes
2.25 There are many legacy programmes and projects at national, Glasgow and sub-Glasgow level. Some are completely new; others are embedded in existing strategies and policies. Evaluating every one of these programmes and projects would be a major challenge, and probably disproportionate to the Games investment.
2.26 For the purposes of this evaluation we are keen to champion evaluation studies that will help directly address the research questions set out in Chapter 1. Essentially, those legacy programmes with the potential to influence the intermediate and long-term outcomes. We are currently proposing to use the following criteria to prioritise evaluation work - essentially that a programme/project should be considered for, or included in, an evaluation if:
- The evaluation will help address the key research questions - particularly the supplementary questions under 1, 2 and 4 about the contribution of legacy programmes to change.
- The programme/project is initiated, or very clearly strengthened, by hosting Glasgow 2014.
- The programme/project is of sufficient scale, and uptake, to warrant robust evaluation.
- The programme/project potentially (based on theory or evidence) contributes to the achievement of legacy outcome.
2.27 The Scottish Government and a number of partners have firm or tentative plans to evaluate their legacy investments. For example, Creative Scotland has just commissioned work that includes the development of an evaluation framework for the 2014 Cultural Programme. This will report in 2013. Further, sportscotland plan to evaluate the Community Sports Hubs. We will also be working with other funders (for example Sport Relief) to plan the evaluations of the legacy programmes that they fund.
2.28 An important role for GLEWG in the coming months will be to map current and proposed programme evaluations. This will help us to reach a clear consensus on the strategic priorities for evaluation and to decide where we might best place our efforts and resources. Progress on evaluation will be more fully reported in the next legacy evaluation report in spring 2014. It is worth noting, however, that as part of on-going project management, programmes and projects are generally expected to monitor their activities. Monitoring data will increasingly become available and be reported on the Games portal.
E) Economic Assessment
2.29 We plan to assess the economic contribution of the Games over the study period. Hosting major sporting events involve significant upfront expenditures but, nevertheless, have the potential to leave substantial economic legacies in the form of infrastructure and urban renewal. The economic assessment will seek to help answer key research questions within the Flourishing theme.
2.30 It can often be complex to separate the economic effects of the delivery of a major sporting event from the legacy. With this in mind, the economic assessment sets out to evaluate the economic contribution of the Glasgow 2014 Games over the full legacy period (including delivery) across a number of clearly identifiable strands, which may contribute to economic benefits across both across time (when) and spatial (where) dimensions. Figure 2.2 sets out a number of key element of the Games across the 2009-2019 legacy period.
Figure 2.2. Expected timeline of economic impacts
2.31 The GLEWG assessment was that a 'counterfactual' (what would have happened in the absence of the Games investment) was not feasible (see paragraph 1.23) and, therefore, the economic assessment will not provide an overall measure of the economic impact of the Games legacy. As an alternative, it is proposed that the economic assessment pulls together on a number of sources and evaluation approaches to present a narrative of the economic contribution of the Games over the legacy period on the East End, Glasgow and Scotland as a whole.
2.32 The economic assessment will draw on the key components of the Games legacy evaluation framework. There are a number of economic indicators that will be monitored as part of the tracking outcome indicators strand of the evaluation (A), however, monitoring the identified outcome indicators will not constitute on their own the overall economic assessment. There will be additional relevant data generated through secondary analysis (B) and by the GoWell new study area for the East End (C). Furthermore, it is expected that new survey work with spectators, visitors and businesses will be required (and be subject to agreement).
In some cases, new questions may be added to existing or planned surveys to capture required data (for example, the International Passenger Survey). GLEWG will fully scope out this work in the coming months.
2.33 The economic assessment will also make use of the Scottish Government's Input/Output (I/O) model to estimate the output and employment impacts of expenditure linked to the delivery and legacy of the Games. The I/O model estimates contributions to the economy of expenditure both directly, as well as indirectly in the wider Scottish supply chain. Impacts are expressed in terms of Gross Value Added (GVA - see pdf page 41-42 for more information on GVA) and the number of full time equivalent (FTE) jobs estimated to be supported in each year. Box 2.3 sets out some initial estimates on the number of jobs and GVA supported by the capital investment on the construction and refurbishment of venues and the Athletes Village in preparation for the Games.
Box 2.3. Scottish Government Input-Output Model Economic Impact Estimates
As a first step, an assessment of the economic contribution of the capital investment on the construction and refurbishment of venues and the Athletes Village for the Games has been estimated using the latest Scottish Government Input-Output model of the economy.
The £500 million overall spending on construction and refurbishment of Games venues and the Athletes Village over the six years leading to 2014 is estimated to support on average around 1,000 jobs and contribute £60 million to Scotland's GVA in each year. At its peak in 2012, where £140 million was invested, around 2,000 FTE jobs and £100 million GVA were supported.
However, there may be other aspects of the economic effect of the Games legacy that the Scottish Government I/O model could provide insights for. For example, each £100 million of tourism spending in 2014 is estimated to support around 1,300 FTE jobs and contribute £50 million to Scottish GVA, with around 70% of the supported jobs in the distribution and catering sectors.
F) Primary Research on Partnership and Leverage
We are planning to conduct some primary research on how the Games investment has affected partnership working, across and between sectors in Scotland, how it has impacted on the confidence and ambition of partners, and to tease out the lessons for the future. The baseline year for this will be 2013, and follow-up will take place after the Glasgow 2014 Games, possibly in 2015 and beyond. At this stage we will be able to assess whether established partnerships have continued beyond 2014 and whether partners were in a position to capitalise on the impacts of the Glasgow 2014 Games. This work has yet to commence.
Table 2.4 - Analytical Plan
Key / supplementary questions
| Area | Method | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| A=active; F=flourishing; C=connected; S=sustainable; O =overarching | Scotland | Glasgow | East End | Outcome Indicators | Secondary Analysis | Gowell | Evaluation studies | Monitoring | Economic modelling | Other primary research |
| Has there been a change in sport participation and physical activity? [A] | √ | √ | √ | √ | √ | √ | ||||
| Are sustained changes in physical activity and sports participation evident among particular groups? | √ | √ | √ | √ | √ | √ | ||||
| Is there evidence for a 'demonstration effect' among those already, or recently, physically active? | √ | √ | √ | √ | √ | √ | ||||
| Is there evidence of a 'festival effect' among those who are currently sedentary? | √ | √ | √ | √ | √ | √ | ||||
| Have legacy investments and programmes which aim to increase physical activity contributed to change? What can we learn for the future? | √ | √ | √ | √ | ||||||
| What effect has the Games had on businesses, employment and volunteering? [F] | √ | √ | √ | √ | √ | √ | ||||
| What has been the effect of Games investment on economic outcomes? | √ | √ | √ | √ | √ | √ | ||||
| How do the outcomes of the Games investment compare with other investments? | √ | √ | ||||||||
| Have legacy investments and programmes which aim to support businesses contributed to change? What can we learn for the future? | √ | √ | √ | √ | √ | |||||
| Have legacy investments and programmes which aim to support people into employment, training and volunteering contributed to change? What can we learn for the future? | √ | √ | √ | √ | √ | |||||
| What is the impact of the Games on the lives of the local community in the area of Glasgow's east end most directly affected by the investment? [S, A, F] | √ | √ | ||||||||
| What change is observed over time in key legacy outcomes e.g. (physical activity, cultural engagement, volunteering)? | √ | √ | ||||||||
| Have there been changes in the physical, social and retail environment over time? | √ | √ | ||||||||
| How do changes in key outcomes in this area compare with other similarly disadvantaged areas and regeneration sites in Glasgow? Is there additionality? | √ | √ | ||||||||
| Has there been a change in cultural engagement? [C, F] | √ | √ | √ | √ | √ | |||||
| Are sustained changes in cultural engagement evident among particular groups? | √ | √ | √ | √ | √ | |||||
| Have legacy investments and programmes which aim to increase cultural engagement contributed to change? What can we learn for in the future? | √ | √ | √ | √ | √ | |||||
| Has there been a change in civic pride and / or international reputation? [C, F] | √ | √ | √ | √ | √ | √ | ||||
| Has there been a change in civic pride among particular groups? | √ | √ | √ | √ | √ | √ | ||||
| What aspects of international reputation have changed over time? | √ | √ | √ | |||||||
| Is there a partnership legacy? [O] How have organisations worked together to maximise the Games' opportunities? Did partners capitalise on the impacts of the Games post-2014? What are the lessons for the future? | √ | |||||||||
Contact
Email: Imelda Giarchi
There is a problem
Thanks for your feedback