The Potential of Existing Cross-Sectional and Longitudinal Surveys to Support the Commonwealth Games 2014 Legacy Evaluation

2. METHODOLOGY

2.1 The research comprised two stages. The first was designed to identify possible additional data sources to inform the evaluation of the legacy of the 2014 Commonwealth Games. This was followed by secondary analysis of a number of the surveys identified as valuable sources of information.

Stage one

2.2 Existing surveys were identified through several sources. First, literature that highlighted available longitudinal research in Scotland and the UK was reviewed. This literature included:

• 'UK Longitudinal Research and Analysis Network: Inventory of UK Longitudinal Surveys'
• 'Tracking people: A guide to longitudinal social sources'
• 'Use of longitudinal research in the evaluation of the Scottish Government's National Outcomes.'

2.3 Next, data available on the UK data service site was assessed, general internet searches for relevant longitudinal and cross-sectional surveys were conducted and the expertise and knowledge of Ipsos MORI colleagues in our Edinburgh and London offices was sought.

2.4 Data sources identified through this process were then assessed on the following criteria, taken from the Scottish Government's baseline evaluation document Report 1: Questions, Methods and Baseline^[8]:

• relevance (e.g. does the data cover the topics upon which legacy is focused?)
• quality (e.g. is the sampling method robust?)
• regularity (e.g. does the data cover the time period between 2008 and 2019?)
• disaggregation (does the survey cover Scotland/Glasgow/east end of Glasgow?)

2.5 For each data source that met these criteria key details were recorded, including the survey sponsor, target audience, sampling strategy, geographical coverage and topic coverage (these are included in Appendix B). Surveys that did not meet the criteria listed were discounted from further consideration (see Table 1 in Appendix B).

2.6 An assessment was then made of which of the 'in scope' surveys could best address the legacy evaluation questions. As part of this process, consideration was given to the extent to which the surveys could provide information on key equalities groups. Tables 2 and 3 in Appendix B summarise the outcome of this analysis, ranking the surveys in order of applicability to the evaluation questions.

Stage two

2.7 Three surveys were selected for inclusion in phase two of the research:

• Understanding Society
• The Life Opportunities Survey
• Glasgow Household Survey

2.8 These surveys were chosen as they offered information above and beyond that included in previous analysis conducted by the Scottish Government for the evaluation legacy document Report 1: Questions, Methods and Baseline.

2.9 Understanding Society is the largest UK longitudinal survey, sampling over 40,000 households. The survey uses pre-selected random sampling and the Scottish sample consists of almost 3,000 individual respondents. The survey is conducted annually and comprises an individual computer assisted personal interviewing (CAPI) interview with all adults in the household, a self-completion survey completed by all aged 16+ in the household and a self-completion survey completed by those aged 10-15 in the household. The questionnaire covers a variety of topics including: family, relationships, fertility and child-birth history; health, disability and caring; employment, finance and personal consumption; ethnicity, discrimination, politics and the environment; beliefs and values, and attitudes. Understanding Society is an annual survey with waves planned until 2018.

2.10 The Life Opportunities Survey is a longitudinal study which compares how people with and without an impairment participate in society in a number of areas, including: education and training; employment; transport; leisure; social and cultural activities; and social contact. Around 20,000 addresses are sampled using a pre-selected random sample, with approximately 2,000 interviews completed in Scotland. A face-to-face CAPI interview is conducted with each adult in sampled households. The Life Opportunities Survey is an annual survey. Funding for the survey is granted on a wave by wave basis so it does not have a set lifespan. However, the survey was set up in view of the Government's goal for achieving equality for disabled people in the UK by 2025 so future waves are very likely.

2.11 The Glasgow Household Survey is run on behalf of Glasgow City Council and is an annual cross-sectional survey of Glasgow residents to assess use and perceptions of local services, and views on other local issues. It comprises face-to-face CAPI interviews with a single respondent in the household and uses quota sampling. Approximately 1,000 respondents are interviewed each wave. The Glasgow Household Survey is carried out annually (although it was biannual up until 2013). The next wave is planned for spring 2014 and, budget permitting, will continue to run on an annual basis in the future.

Secondary data analysis

2.12 At the heart of the research objectives is the need to establish the existing data sources that can be used to measure the legacy of the Glasgow 2014 Commonwealth Games in four topic areas: Sports Participation and Physical Activity, Volunteering, Cultural Engagement, and Civic Pride.

2.13 The first report of the Games Legacy Evaluation Working Group (GLEWG) sets out an approach to the evaluation. It notes that while previous similar evaluations have varied in their quality and rigour, most have used a cross-sectional design. It suggests that there may be merit in using longitudinal data.

2.14 It is worth comparing here where the strengths of longitudinal data and cross-sectional data lie in relation to evaluating the legacy of the Games.

2.15 Cross-sectional surveys (and longitudinal surveys) measure statuses at single time points. Both, for example, would give estimates of the percentage of people who undertook some form of sport or exercise in 2012. Similarly, both types of survey allow changes in these statuses over time to be measured among populations and sub-groups. For example, whether more, less, or the same proportion of people in 2012 undertook some form of sport or exercise compared with 2011.

2.16 Repeated cross-sectional surveys are intrinsically no better or worse than longitudinal surveys for measuring changes over time at a population or sub-group level. It is worth noting that some national longitudinal surveys, like the British Cohort Survey 1970 and the National Development Study, follow a single age cohort, so are not representative of adults overall, but of a single age group. Other longitudinal surveys, like Understanding Society, are panel surveys and provide data that is representative of the adult population overall, and not just a single age cohort.

2.17 However, in practise, cross-sectional surveys tend to have larger sample sizes than longitudinal surveys. This means that survey estimates have narrower confidence intervals that allow smaller changes over time to be detected. It also means that they provide more scope for robust analysis of sub-groups.

2.18 Longitudinal surveys, however, unlike cross-sectional surveys, measure changes over time at an individual level by going back and surveying the same people. A number of advantages stem from this.

2.19 Firstly, longitudinal data allows the longevity of a status to be explored. If 20% of people volunteer at any one time, this may mean that 20% of the population always volunteer and 80% never volunteer. Alternatively, it may mean that the whole population volunteer for one fifth of the time. It is not possible to conclude which of these is more accurate without using longitudinal data.

2.20 Secondly, because of the inclusion of a temporal dimension, longitudinal data can be better when undertaking analysis examining why people move in and out of different statuses. The classic example cited is unemployment and ill-health. People who are unemployed tend to have poorer health than those who are in work. Why? This could be because ill-health tends to lead to unemployment, or it could be because unemployment leads to ill-health. Longitudinal data is needed to disentangle the strength of each effect.

2.21 It would not be surprising, for example, to find that people who tend to volunteer in areas unrelated to the Commonwealth Games also volunteer for activities related to the Commonwealth Games. Longitudinal data, unlike cross-sectional data, would throw light on the strength of any demonstration or festival effect to examine how much prior volunteering behaviour leads people to volunteer for the activities related to the Games, and how much volunteering for the Games leads people to continue to volunteer for other organisations and activities beyond the Commonwealth Games.

2.22 Similarly, it may be that people who participate in some form of sport or exercise are more likely than those who are inactive to attend a Commonwealth Games event. This data would again be consistent with two very different hypotheses: that prior involvement in sport or exercise leads to encourage people to attend the Commonwealth Games; and the attendance at a Commonwealth Games event leads to increased participation in sport/exercise. Both are plausible and would require longitudinal data to fully disentangle.

2.23 However, in order to undertake the type of analysis outlined above, not only is longitudinal data required, but the data needs to include some indication of the nature of people's involvement with the Commonwealth Games. The usefulness of the various sources of longitudinal data will, in part, be driven by the questions they include on awareness of, and types of participation in, the Commonwealth Games. Without such measures, the benefit of longitudinal data over cross-sectional data in measuring the legacy of the Commonwealth Games is limited.

2.24 In order to support the analysis of the legacy of the Commonwealth Games, the inclusion of such measures in suitable longitudinal data sources should be a priority for consideration.

2.25 An additional consideration for longitudinal data analysis is the separation of the historic period and the effect of ageing. As noted previously, some longitudinal surveys follow a single cohort of people, while others are representative across different age groups. Analysis of multiple cohorts would be beneficial in order to separate out the effects of the legacy of the Commonwealth Games from aging. As Understanding Society covers all ages rather than a single cohort, it would be a more suitable source of data than the individual cohort surveys, and would preclude the need to analyse multiple cohorts to separate out the effects of the legacy of the Commonwealth Games from aging. Similarly, Growing Up in Scotland would be a suitable source of data because it uses a repeated cohort design.

2.26 The power of using longitudinal data will be in analysing change over time. For this report, however, the analysis presented in this report is limited to baseline measures. The next four chapters of this report detail the pre-Commonwealth Games baseline levels using the key data sources for each of the four research areas.