Local Area Labour Markets in Scotland - Statistics from the Annual Population Survey 2011

Annex F: Technical note on APS sample size

Response rates for many government sponsored social surveys have been decreasing steadily for several years. Typical response rates for ONS survey in the 1980/1990's was around 80%, but these have declined over the years to their current levels in the region of 60%. These are being driven by increasing refusals and non-contact (where the interviewer fails to make contact with anyone at the target address after many repeated attempts). A multitude of factors have been used to explain the downward trend in response rates. Some of these include:

• falling contact rates attributed to: rise in single person households; rise in households where all adults are in work; rise in controlled access to properties
• increasing interview lengths
• survey overload: the salience of a survey topic has become a more important determinant of response
• external shocks such as data losses, which have impacted on trust in statistics;
• reduced budgets which result in less appetite to administer costly re-issue exercises.

In additional the Annual Population Survey shows higher levels of attrition than in the Labour Force Survey (where respondents who had completed the survey in the first wave, either refuse to take part or are no longer contactable in subsequent waves). This is because respondents in the APS are re-interviewed at annual intervals over four years, compared to the five quarterly interviews over the course of one year for the LFS.

Previously, one of the responses to falling response rates was to increase the overall sample pool, hence ensuring that the final sample of responding households remained constant. However, as government budgets have been reduced, the available funds, both from ONS and the various sponsors of the APS, have impacted. In the fiscal year (April 2011 to March 2012), this meant that ONS held the sample pool for Scotland at a fixed level instead of increasing it as it had in previous years in response to further reductions in response rates. This has resulted in the actual sample size for the APS for Jan-Dec 2011 in Scotland has reduced. The impact at national level is approximately 9%, although there is considerable variation across the various local authorities in Scotland, ranging from an approximate 19% reduction in Glasgow City to an 18% increase in Inverclyde. These variations are in part due to corrections by ONS in response to previous changes in response rates (e.g. corrections to try and ensure ONS achieve the required number of interviews in a particular local authority). The main impacts seen will be in reliability of estimates, with confidence intervals increasing by around 10% for Glasgow, and around 5% for Scotland as a whole (e.g. for employment rate, the confidence interval for Scotland would increase from +0.53 to +0.56 percentage points).

Continued restrictions on government departmental spending, will mean that the actual household sample size for 2012-13 will be reduced by 90 households per quarter from the fourth quarter of 2012. This will have a knock-on effect on the reliability of estimates from the 2012 APS. It is expected that the increase in confidence intervals will be larger for 2012-13 than in 2011-12, as any continued impact of response rate will be increased by the reduced sample size.

ONS have an ongoing program to address the declines in response rates tackling a wide range of issues including; interviewer training, incentives, survey materials, questionnaire reviews, interviewer performance, IT infrastructure and interviewer contracts. In addition, ONS has a programme of research looking at the impact of declining response on survey outputs, including:

• review of reasons for non-response and non-response
• an analysis of attrition bias between waves in the LFS
• refusal follow-up study funded by EUROSTAT
• review of use of incentives
• Census non-response link study
• evaluation of internet data collection for the LFS