Developing a modern statistical system - a review of Scotland's Census 2022

Statistical Decision Making and Resource

This section reviews the decisions, expertise and support for decision making in developing the programme and over the programme lifecycle. It also considers the collaborations that helped lead to a successful Census in 2022.

Developing New Statistical programmes

The programme started in 2016 at the point in any census cycle where resources and staffing are lowest. In these early stages there was a focus on statistical design and the governance structures in place reflected this approach. During the initial period of planning, there was limited experience in delivery of a programme of this size in both the statistics group and wider NRS. There was also limited experience of multi-disciplinary engagement with ownership of the overall delivery model and associated risks was not clearly understood.

Interviews with NRS staff and Assurance Reviews highlighted a limited understanding at an early stage of the impact of the commercial approach chosen on programme design and lifecycle. There was also limited continuity and operational expertise from previous census programmes, partly due to funding challenges in retaining expertise in an ongoing programme between census cycles.

Some of the challenges faced by NRS are not unique to a census. It is not uncommon for programmes to go through cycles as funding for one project or programme ends and another starts. It is also not uncommon that statisticians who have gained experience will be deployed into different roles across the statistical profession.

At the early stages of programme design, user engagement around the topics which would be covered in the Census in 2021 was successful and helped to align benefits with the statistical preparatory work. This is in large part due to a major focus within the statistics profession on meeting the needs of users as part of the Statistics Code of Practice.

However, as many staff with experience of the 2011 Census had moved on or had been redeployed elsewhere, the relationships between statistical disciplines and other professions, including digital expertise, became more important in setting out the programme design and business case. Both continuity from previous similar programmes and stronger multi-disciplinary structures could have helped anticipate future challenges in delivery of this statistical programme.

The topic consultations and other statistics-led activity in the initial stages helped to understand the needs of the programme’s statistical design. While finite resources were focused on this, and examining the future of census, it is likely that other teams in NRS did not have the opportunity to develop experience and resilience in other areas required to deliver a census. This could have been obtained through operational design and planning followed by undertaking a live end to end operation.

This prioritisation and focus, prompted by the resources available to NRS at the time, may have also reduced the ability to use this experience, and experience from previous censuses, to develop the programme, in particular understanding the key logistical components that required professional support outside of statistics.

This lack of understanding of other components may have contributed to the early failure to establish appropriate disciplines required to work alongside statistical leadership and the absence of a structure to effectively support the chosen delivery model, and commercial approach, in the early stages of the programme. The Technology Assurance Framework review found this was a root cause of a number of challenges, resulting in programme delivery delays emerging from 2018 onwards and impacting on the ability to deliver key components.

In 2018 changes were made to governance in terms of both structure and staffing to support decision-making to address emerging issues. These changes reflected the need for multi-disciplinary teams to come together to support a complex programme with commercial, digital, operational and logistical functions as well as statistical. This incorporated appropriate decision making from a wider range of expertise.

As the programme developed, a number of steps were taken to address these issues. Pre-procurement gates were carried out for the major digital procurements and specialist multi-disciplinary expertise and leadership were introduced gradually to the programme. However, by this point some key decisions had already been made which limited scope to implement best practice across different functions. It may have benefitted the programme for these to have been introduced earlier.

The absence of areas of specialist knowledge or experience in the programme team led, based on external advice, to narrow views of impacts and instances of risk adverse approaches being adopted. A more holistic view of managing design decisions and risks overall in the context of the specific requirements of the census programme, and in turn the wider statistical system, may have provided greater flexibility to the programme.

It is noted that the roll-out of ‘Fit for the Future’ Leadership programme since 2021 for the statistics profession, addresses upskilling statisticians at B3 and C1 (UK SEO and Grade 7) with an emphasis on designing new projects and programmes of work.

At the time the census was being planned, ‘The Audit Scotland Principles for a Digital Future, had been published in 2017. This sets out guidance for 5 key areas for public sector projects – planning, governance, a user led approach, clear leadership, and strategic oversight and assurance. Future programmes should also consider the large body of expertise and experience in multi-disciplinary working across major programmes. These principles have been taken forward and evolved through Scottish Government’s digital strategy and assurance functions and are covered in more detail in section on Digital Programme Delivery.

Findings

Strategic leadership roles within both the statistics profession and other relevant professions should be identified at an early stage in the development of new programmes of work for cross-cutting statistical components. This will involve more closely defining this term and identifying projects and programmes which could fall under this definition and therefore be of particular importance for the Chief Statistician in the context of the broader statistical system in Scotland. A suitable governance structure which is effective across disciplines should also be in place at an early stage.

Once these programmes have been identified, the Chief Statistician should conduct an assessment of the broader skills required by a modernised statistics profession to deliver them and take steps to incorporate that in training and development opportunities for the profession. Consideration should be given to building on the ‘Fit for the Future’ programme by providing strategic leadership training to more senior grades. In addition, work should be undertaken to identify the broader range of skills required in delivering cross-cutting statistical components and to target training appropriately. This is likely to include training on collaboration with other experts in the design of new programmes, multi-disciplinary working and digital strategy.

New programmes and projects of statistical work should be in alignment with other established principles for programme delivery. They should also be in alignment with other programmes across Scotland which will implement the Scottish Government’s statistical strategic priorities.

Recommendation 1: The Chief Statistician should consider activities including a task and finish group to define and identify the ‘Cross-cutting Statistical Components’ of the Scottish Statistical system and other projects.

Recommendation 2: The Chief Statistician should consider opportunities to enable planning and training for the wider statistical profession to be linked to the broader skills needed for delivery of Cross-cutting Statistical Components by different organisations across the Scottish statistical system.

Senior Statistical Decision Making

There were four different chairs of the Census Programme Board over the period of this review. There was also a change to the Senior Responsible Owner role, moving it from the Director of Statistics to the Chief Executive of NRS in 2018. This was to ensure appropriate corporate oversight. There was also extensive replanning activity to produce an effective roadmap for delivery. These factors can present risks to programme stability and maintaining business continuity.

Senior management in other NRS functions, outside of the statistics profession, shared their experiences with the review. Overall, there was a common theme that they were required to step in at short notice to coordinate activity and deliver operational and other elements of the programme. These senior leaders in NRS took on leadership of parts of the census delivery in addition to their work elsewhere in NRS. External expertise was brought in to fill roles which were difficult to recruit and to support those in leadership roles.

These changes, in conjunction with changes to governance arrangements, were necessary to implement assurance recommendations. This took place alongside periods of replanning activity to implement a detailed roadmap to deliver on schedule, first to March 2021 and then March 2022, with clear identification of critical milestones, dependencies and decision points.

Senior leaders working at NRS around this time have stated that periods where the census programme lacked stable leadership and firm plans placed additional pressure on all staff and made it harder to implement programme controls.

The census programme had a number of governance structures to support strategic decision making. In the early stages of the programme an absence of a full high level design leading to detailed requirements led to some design decisions effectively being taken by non-NRS external resources who had been engaged to provide specific areas of expertise. Some of these design decisions limited flexibility of the programme to respond to challenges but this strategic decision making function evolved over time to be a strength of the programme in the later stages.

Corporate and delivery structures should demonstrate capacity and capability to deliver large scale statistical programmes, in both the delivery body and organisational partners. These structures should ideally be resilient to withstand changes in leadership or circumstances across organisations and be backed by continuity and expertise in statistical, operational, governance and digital disciplines between relevant programmes.

Representation from ONS and Scottish Government was in place through the Census Programme Board, but there is scope to consider further strengthening structures to provide additional resilience and ensure a wider context and the potential impact of the Scottish Statistical System is considered when significant decisions are made.

There were two such significant decisions taken during this period of instability, the first was to de-scope the Census Rehearsal. The second decision, in light of the COVID-19 Global Pandemic in 2020, was to delay the Census by one year.

Decision to refocus the rehearsal

The aim of a rehearsal is to build confidence and resilience in the census operation and to leave enough time to rectify any issues before scaling up to the full operational service. As an example, the evaluation of the 2009 Rehearsal in Scotland discovered that field force technology was inadequate, but it left enough time for a new solution to be built and deployed, developed in conjunction with user research from field staff who were retained between 2009 and 2011.

For the Census 2021, there were delays in procuring key components of the rehearsal because of an underdeveloped set of requirements, operational design and delivery roadmap. This meant that NRS was unable to deliver a full census rehearsal in autumn 2019 as originally planned.

Identification and rectification of issues, along with subsequent replanning, engaged the focus of the leadership and other resources available to the programme. This occurred at a time when the programme had intended to be preparing for delivery of an end to end rehearsal of systems, services and operational elements alongside learning and refinement of statistical models.

NRS developed a recovery plan which included reducing the scope of the census rehearsal, removing paper capture, coding, logistics and field force. The 2020-21 Audit of the National Records of Scotland found that the revised rehearsal to separately test elements of data collection and data processing was delivered successfully.

However, this rehearsal meant that a significant proportion of census activity, including field force activity to follow up on non-responding households and support for respondents was not included.

Opportunities were also missed to build resilience and knowledge across NRS staff and field force who would be part of the full census. In discussions with NRS staff it was raised that this extended to the building of relationships and trust with suppliers and contractors who had not undertaken a census before. Key components of the census operation were therefore stood up for the first time during the live census leading to challenges in delivery, with risks and issues less well understood than if they had been in place earlier in the programme.

Decision to delay Scotland’s Census to 2022

Under normal circumstances censuses across the UK are conducted on the same date, with an agreement to harmonise to meet the needs of UK data users and reflecting UN-ECE and other international requirements. The impact of Covid-19 led NRS to recommend to Ministers in July 2020 that the Scotland’s Census be moved from 2021 to 2022.

COVID-19 restrictions in place in Scotland impacted on a wide range of preparation activities for the 2021 Census including planning and testing the effectiveness, safety and security of census systems and collection processes to ensure these were ready. NRS requested assurance processes including a Gateway Review and a subsequent options assessment by NRS in April and May 2020. These determined that due to the impact of Covid-19 the census as planned in 2021 was not deliverable.

Ongoing issues in delivery around the time of the descoped 2019 rehearsal had led to a confidence assessment of amber in February 2020 for delivery of the 2021 Census prior to any Covid-19 restrictions being in place. The extraordinary disruption caused by Covid-19 prevented NRS from implementing revisions to the programme plan.

Additional resources from within Scottish Government, including in digital and HR, were deployed to add to the capacity, capability and resilience of NRS at this time. It is impossible to predict what challenges may have been different or would still have been faced by the programme had the pandemic not occurred. Ad-hoc resources from Scottish Government and expertise intended to address those challenges were provided once the decision was taken to delay to 2022.

Findings

Significant decisions made in the delivery of the census programme in Scotland, and other programmes which fall under the definition of a critical statistical components, can have a wider impact on the broader statistical system beyond the objectives of the individual programme. The decisions to descope the 2019 Rehearsal and to delay the Census in Scotland from 2021 to 2022 had impacts felt beyond the census programme.

The outcomes and impacts on the broader statistical system of significant decisions need to be clearly understood. Programmes which could have an impact on the broader statistical system should also as far as possible have strong risk mitigations available to ensure successful delivery as planned.

From the inception of any major statistical programme, there needs to be a clear roadmap and the organisational capacity and capability to prepare, test and deliver a full range of systems and services to deliver the programme as planned. Given the scope for disruption and adverse impacts if the anticipated dates are not met, planning for extraordinary circumstances including financial contingency should be considered.

When an integrated, holistic delivery plan and roadmap for all activities in the programme is in place it should be actively managed to ensure there is sufficient resources, capacity and capability to deliver an effective assessment of operational capabilities and resilience of both organisational structures and staff. This requires a high level design with high level requirements set against quality targets. The approach and build up to an effective end to end test, which is a significant investment in resources, requires to be treated with the same rigour and robust planning activity as any live delivery.

In addition to continued Scottish Government representation on individual Programme Boards, a new Statistical Programme Board with influence over key decisions for cross-cutting statistical components and an effective mechanism to have oversight and a holistic view of risks across the statistical system should also be considered. This would allow important decisions to be made with the perspective of the Chief Statistician on their broader impact on the overall statistical system.

Recommendation 3: The Chief Statistician should ensure that outcomes and impacts on the statistical system of Cross-cutting Statistical Components are clearly understood and agreed with producers of official statistics in Scotland, and that these programmes have strong risk mitigations available to ensure successful delivery.

Recommendation 4: The Chief Statistician should consider what mechanisms are required to provide oversight and assurance of Cross-cutting Statistical Components in a holistic way and escalation of key decisions that could impact an integrated statistical system.