Scotland's Census 2021 Topic Consultation

Annex 1: Census Data Quality

Two aspects of the quality of the 2011 Census data are considered in detail in this section - item non-response rates and Census Quality Survey (CQS) gross agreement rates. Both item non-response and agreement rates are used to understand the quality of the information collected from each of the census questions. Levels of completeness are measured through item non-response rates and respondent error is assessed through the CQS. Using these measures to evaluate the data helps further develop the census questions to improve the quality of data collected in the future.

Completeness is an indirect assessment of how well the self-completion census questionnaire was designed. High levels of completeness indicate that the methods and effort put into qualitative and quantitative testing of individual groups of questions and then the whole questionnaire were successful and had a positive impact on maximising completeness and the accuracy of responses.

Completeness was assessed by measuring how many responses to each census question were missing as a proportion of all of the people who should have responded to that question (but not including persons and whole households that were missed).

Item non-response includes all responses that were missing or not valid, including multi-ticks, out-of-range values and partially answered responses. Item imputation was applied to estimate for a missing value when there was item non-response or where there were inconsistency errors. The latter were correctly recorded values which were considered invalid because they were inconsistent either with other values on the questionnaire, or with auxiliary information or definitions. Inconsistency errors were detected by validating the data against a set of pre-defined edit rules.

Item non-response rates for the main census questions are shown in Table A1. Item non-response rates for the household questions ranged from 1.2 per cent for number of cars and vans to 3.5 per cent for relationship to person one. All household questions had item non-response rates lower than in 2001.^[8]

For individual questions shown in Table A2, the item non-response rate showed wider variation ranging from only 0.7 per cent for age, to 16.9 per cent for the year last worked. Other notably high item non-response rates were 15.2 per cent for the new question on long term health conditions, and high rates for workplace/study address indicator (8.6 per cent) and industry (8.5 per cent), although this is much lower for people who are currently working.

The Census Quality Survey is a small-scale voluntary survey undertaken shortly after the census. It aims to measure the accuracy of answers given to census questions by asking a sample of household the census questions again in a face-to-face interview. By comparing the responses given in the CQS to those given in the census, agreement rates are calculated which provide an indication of how accurately the 2011 Census questionnaire had been completed by the general public. The CQS interview responses are considered to be more accurate as research suggests that answers given to questions in face-to-face interviews tend to result in more accurate answers (for non-sensitive questions) than those given on a self-completion questionnaire.

A clustered sample was selected in three census regions (Glasgow East, Fife and Scottish Borders); each area chosen to represent one of the main area types in Scotland. A sample of 520 households was selected in each area, representative of both paper and internet response modes. Household included in the Census Coverage Survey were excluded and only household with five or fewer residents were selected. The design and methodology used were tailored to ensure a smooth integration with the overall Census operation in Scotland.

In total, 1,760 individuals were interviewed in 787 households; a household response rate of 50 per cent. This was much lower than the target rate of 60 per cent for the survey and the rate achieved in the previous census^[9].

Gross agreement rates for the main census questions are shown alongside the item non-response rates in Tables A1 and A2. The gross agreement rates for the household questions were all above 95 per cent, with the exception of the number of rooms question, which had a relatively poor rate of 86.6 per cent. The lowest gross agreement rates for the individual questions were seen in qualifications (77.8 per cent), long term health conditions (78.7 per cent) and hours worked (84 per cent). Questions on date of arrival in the UK (84.2 per cent) and language skills (84.6 per cent) also showed relatively poor agreement rates.

As part of the quality reporting on Census 2011, further analysis of imputation rates will be published later this year. This analysis will be used to inform aspects of the design of the Census 2021 questionnaire, and will summarise the processes of identifying the response category for both missing and inconsistent variables.

Table A1: 2011 Item non-response rates and CQS Agreement Rates - Household questions

Table A2: 2011 Item non-response rates and CQS Agreement Rates - Individual questions