Scottish Household Survey: response rates, reissuing and survey quality

3 Approach to analysis

3.1 This is a relative non-response bias study, estimating the impact of a change in the response rate rather than assessing the overall level of non-response bias.^[2] At the core of the analysis is the question, 'What impact does reissuing have on the survey estimates?'

3.2 The analysis compares estimates from the weighted full survey sample with estimates from first issue interviews only. It is important to note that the estimates from the first issue interviews were weighted as if they were the final achieved sample.^[3] This analysis is, in effect, showing how the published results of the survey would differ if reissuing had not been used, and the fieldwork had been completed with lower response rates.

3.3 Overall, twelve key survey measures for each of the surveys were selected for analysis at the national level. These are detailed in Table 3.1 along with the sample sizes. These include some of the headline measures as well as measures asked of a subset of the survey.

3.4 Additionally, estimates for the 11 random adult measures were analysed by key sub-groups: gender, age, rurality, deprivation, tenure, area and household type.

3.5 Impact was measured in two ways. Firstly, through the absolute percentage point difference between the final sample estimate and the first issue only sample estimate. The absolute difference gives a good indicator of overall impact on each estimate.

3.6 However, using the absolute difference alone does not give a fair test of the impact of re-issuing as (everything else being equal) we would expect the size of the difference to be largest for estimates around 50% and to decrease as the estimate moves away from 50%. The absolute difference also takes no account of the sample size. Additionally, traditional tests for significance such as a chi squared test or formal hypothesis testing were not appropriate, since the samples are not independent (subsamples of the full sample are compared to the full sample). Alternative tests could be used, but the impact of re-issuing would have to be extreme for a difference to be significant; so they are not very discriminating.

3.7 In order to compare the magnitude of differences across estimates, it was necessary to standardise these in some way. This has been done in different ways in the past. For example, for their assessment of the impact of a lower response rate on the Crime Survey of England and Wales, Williams and Hocekova (2015) converted 'effect sizes' into t-scores.

3.8 Impacts were standardised by calculating the ratio of the absolute difference between the estimate to the standard error of the main estimate. This method of standardising is equivalent to the Bias Ratio method described in Sarndal et al (1993).

3.9 We favoured standardising impacts in this way as the size can be intuitively compared to sampling error. A value of one for this measure means that the difference between the estimates is equal to one standard error of the main estimate.

3.10 Standard errors and confidence intervals were adjusted to take account of the published guidance on assumptions around the expected survey design factors in the SHS and SCJS. The analysis used a design factor assumption of 1.2. The standard errors given throughout this report are after adjustment for the design factor and therefore based on the net effective sample size of the estimates^[5] and do not need further adjustment to calculate the confidence intervals.