Local Area Labour Markets in Scotland - Statistics from the Annual Population Survey 2012

Annex F: Technical note on APS sample size

Response rates for many government sponsored social surveys have been decreasing steadily for several years. Typical response rates for ONS survey in the 1980/1990's was around 80%, but these have declined over the years to their current levels in the region of 60%. These are being driven by increasing refusals and non-contact (where the interviewer fails to make contact with anyone at the target address after many repeated attempts). A multitude of factors have been used to explain the downward trend in response rates. Some of these include:

• falling contact rates attributed to: rise in single person households; rise in households where all adults are in work; rise in controlled access to properties
• increasing interview lengths
• survey overload: the salience of a survey topic has become a more important determinant of response
• external shocks such as data losses, which have impacted on trust in statistics;
• reduced budgets which result in less appetite to administer costly re-issue exercises.

In additional the Annual Population Survey shows higher levels of attrition than in the Labour Force Survey (where respondents who had completed the survey in the first wave, either refuse to take part or are no longer contactable in subsequent waves). This is because respondents in the APS are re-interviewed at annual intervals over four years, compared to the five quarterly interviews over the course of one year for the LFS.

Previously, one of the responses to falling response rates was to increase the overall sample pool, hence ensuring that the final sample of responding households remained constant. However, as government budgets have reduced, this option has become increasingly difficult to support. In the fiscal year (April 2011 to March 2012), this meant that ONS held the sample pool for Scotland at a fixed level instead of increasing it, as it had in previous years, resulting in the actual sample size for the APS for Jan-Dec 2011 in Scotland being reduced by around 90 households each quarter. For the last fiscal year (April 2012- March 2013), a further reduction of 90 households per quarter was made as financial constraints within government departments continued. For the coming financial year (April 2013 - March 2014) the Scottish Government have agreed a settlement with ONS to hold and guarantee the sample size at the level set in April 2012- March 2013, which should ensure no further reduction in the quality of survey estimates.

The impact of these sample size reductions will be seen in both the variability of estimates over time and in the reliability of estimates. Comparing confidence intervals (CI's) for 2011 to 2012's data for employment rates, for example, an increase of around 3% was seen at Scotland level, from +0.54pp to +0.56pp. However, this is not consistent across local authority areas; some areas see decreases (e.g. Moray and Aberdeenshire where CI's decreased by 6% and 11% respectively), while most (22 out of 32) have seen increases, with the highest increases in Highland and Shetland Islands (where CI's have increased by 17% and 43% respectively - from +2.5pp to +2.9pp and from +4.3pp to +6.2pp respectively). It is worth noting that the high increase in Shetland Islands may be more related to poor response rates than any decrease in sample size, as the ONS target for the required number of economically active interviews had not been cut in this area due to its already small level. Table 12 below shows how these targets have been changed. It should be noted, however, that these figures are target levels and that the reductions shown are in part due to sample reductions, but also in part due to rebasing of levels to acknowledge the issues ONS has with lower response rates and achieving target levels which were originally set when response rates were much higher.

ONS have an ongoing program to address the declines in response rates tackling a wide range of issues including; interviewer training, incentives, survey materials, questionnaire reviews, interviewer performance, IT infrastructure and interviewer contracts. In addition, ONS has a programme of research looking at the impact of declining response on survey outputs, including:

• review of reasons for non-response and refusal
• an analysis of attrition bias between waves in the LFS
• refusal follow-up study funded by EUROSTAT
• review of use of incentives
• Census non-response link study
• evaluation of internet data collection for the LFS

Table 12: Change in ONS target for economically active adults interviewed by Scottish local authority