Developing a modern statistical system - a review of Scotland's Census 2022

Context

As well as being the largest statistical exercise in the Scottish public sector, the census in 2022 was also one of the largest digital programmes.

The early stages of the 2022 census, in line with previous censuses in 2011 and earlier, were led by senior statisticians in NRS with a focus on statistical design. The start of the programme coincided with reduced resource and staffing levels, following the closure of the 2011 Census programme in 2014. The limited number of census statistical resources from 2011 which remained had been deployed on the Statistical Futures programme researching the potential for administrative data to replace a 2021 census. This review starts in 2016 as NRS began topic consultations and the development of an outline business case for a 2021 Census.

There were many changes to the leadership structure over the duration of the Census programme. Between 2016 and 2023 there were three permanent and one interim Chief Executives chairing the 2022 Census Programme Board. Changes in leadership occurred at a times when major decisions, detailed below, were being taken.

A series of assurance reviews between 2018 and 2021 prompted significant changes with the creation of new digital transformation, programme and project management, statistical and operational roles, along with appropriate governance structures to support them.

These assurance recommendations ensured that an integrated, holistic plan and a detailed roadmap were put in place to deliver the 2021 Census. This included clear identification of critical milestones, dependencies and decision points leading up to the Census ‘Collect’ Phase when individuals and households would respond to the census in March 2021.

Background to Census Rehearsals

As the census is a large scale logistical and operational exercise which aims to contact every household in the country, decennial censuses in the UK and US have traditionally included a full end to end rehearsal of operational systems and services two years prior to Census Day. This normally involves a 5-10% sample of households with different geographies and this model was used in Scotland in 1999 and 2009. In countries with 5 year cycles such as Australia, New Zealand and Canada, which also have greater re-use and continuity of staff and services in a shorter cycle, an end to end operational test takes place 12 months before census day.

In 2018, following an assurance review, NRS made the decision to descope the 2019 Rehearsal. This included removal of logistical, operational and other components which were required for the full census in 2021.

Timing of the Census and UK Harmonisation

An agreement between the UK National Statistician and the Registrars General for Scotland and Northern Ireland included an intention to conduct them simultaneously throughout the UK in 2021. A similar agreement was in place in 2011 and included the intention to conduct censuses simultaneously on the same date throughout the UK.

As a result of the COVID-19 Pandemic this agreement was updated to allow Scotland to conduct its census in 2022 citing the challenges the pandemic placed on UK census offices. This included a renewed commitment to harmonise approaches where feasible, alongside continued sharing of skills, expertise and examining the use of administrative data where possible. Collaborative work to align and harmonise Scotland’s Census outputs from 2022 with the rest of the UK is ongoing.

Censuses in the rest of the UK went ahead as planned in March 2021. The other census-taking organisations (ONS, NISRA) had access to alternative sources of administrative data which could be used to supplement the census data if necessary.

These data sources were not available in Scotland. NRS therefore could not assure that potential risks to data quality could be managed if the census went ahead in 2021 during Covid-19 restrictions and design principles were not met.

Due to the one year delay, NRS-were required to solely lead publicity, communications and other activity to support the collect phase in Scotland for a Census in 2022, without the support of similar activity happening in the rest of the UK in 2021.

Concordat of statistics

There is also an overarching agreement which covers harmonisation and coherence on the production of devolved statistics.

The Concordat on Statistics sets out the agreed framework for co-operation between the UK Government (including the UK Statistics Authority and Office for National Statistics) and devolved governments, relationships between statisticians across the UK, statistical standards and the statistics profession.

The Scottish Government’s position on the Concordat is that Scotland should align with UK wide data where possible when it does not conflate with statistics produced for policy areas where policies are divergent to the rest of the UK and undermine users trust in statistics, and does not place excessive burden on statisticians to deliver them.

Statistics Published from the 2022 Census

Despite challenges faced during development of the programme and delivery, NRS published statistics to the level of quality intended which is the objective of a successful census. Statistical outputs were central to the objectives set by NRS at the start of the programme and it is these statistics which are already being used across Scotland to make vital decisions about our economy and society.

In September 2023, the Office for Statistics Regulation confirmed that Scotland’s Census 2022 would be designated as National Statistics.

The OSR stated

“NRS faced unexpected challenges given that the overall response rate for Scotland’s Census 2022 was lower than had been hoped for. However, we have been impressed with the steps that have been taken to ensure the best quality estimates are produced and to provide assurance for users. …NRS has taken a rigorous and methodical approach to the challenges that have arisen; we have noted the dedication and determination that has been demonstrated in exploring alternative options and solutions, as well as how NRS has communicated its decisions, in order to deliver census statistics that meet the needs of users.”

As well as formal engagement with UK statistical assurance processes led by OSR, Scotland’s Census 2022 came under scrutiny from Scottish Parliamentary Committees and generated significant media interest. The first release of census data was published on 14 September 2023 and will continue into 2025.

Many of the outputs from these assurance and review processes have previously been published and were intended to provide advice to NRS which was appropriate for that stage of the programme.

Impact on wider statistical system

In their review of the Census 2022 in Scotland, The Office for Statistical Regulation states that “it is essential that the data and statistics from the censuses are reliable and provide valuable insights, meeting the rigorous standards of trustworthiness, quality and value outlined in the Code of Practice for Statistics.”

It is the potential significance of the major decisions taken by the programme to impact on a source of data and statistics, one which is a key component of this wider statistical system and informs decisions about almost every aspect of life within the UK, which has led the OSR and the Chief Statistician to undertake this review and make recommendations about how large statistical programmes are managed.

Implications for future statistical programmes

All of the strategic objectives set by NRS for the 2022 Census have been assessed by NRS as being met. There were however several challenges for the programme both prior to and following the Covid-19 pandemic. By grouping issues identified from the 2022 Census into three broad themes, this report considers how leadership, decisions taken and influence at different stages might have impacted on delivery of the programme.

It examines at a high level the impact of decisions made and approaches taken by NRS in the context of delivering the objectives of the 2022 Census and considers whether mitigations available to the programme to tackle emerging risks were as strong as would be expected for a programme which underpins the statistical system.

Learning from this, it makes recommendations to ensure appropriate resources are incorporated into the design, planning and delivery of a future household and population statistics system in Scotland which would also be applicable to other cross-cutting statistical components.