Guide to basic quality assurance in statistics

Explaining changes and discontinuities

Changes and discontinuities in data (e.g. in the form of a step change in a time series) can occur for a number of reasons. It is important however, that these are investigated and that you are able to explain the reasons for them.

 Reasons for discontinuities may include:

• Change in methodology, such as sampling methodology and size, or the method of data analysis used. In the case of surveys, a new contractor may have been used.
• Changes in question wording, or a change in interpretation of a question on the part of the respondent.
• Errors in collating and analysing the data.
• Underlying changes related to the data (e.g change in policy including those impacting on the priority and political importance of the data; economic or environmental changes).

Ensure that those points in Chapters 1-3 of this document have been covered and, ideally have your data peer reviewed.

Policy colleagues may be able to provide possible underlying explanations for any discontinuities (e.g. new legislation or funding streams). Ensure that you do this for both ‘positive’ as well as ‘negative’ changes, ensuring that the conclusions are fully evidenced and do not rely on assumptions.

Contact key data providers to get an understanding of their interpretation of any questionnaires/survey/returns that may have affected response in a particular area.

Principle 4 (Practice 7) of the Code of Practice states that, ‘Where time series are revised or changes made to methods or coverage, produce consistent historical data where possible’ in order to present a clear picture of change over time. This should be done in accordance with the Scottish Government corporate Revisions and Corrections policy. Where it is not possible, the previous year’s figure, at least, should be estimated on the basis of the new method.

Be aware that any one explanation on its own may be spurious so it is important to have an overall picture of your data and the issues associated with it. Also be aware that the data is not context free and some users and providers may have a bias towards a given type of conclusion.

Ensure that an explanation for changes and discontinuities are communicated in your publication. It may help to consider how the statistics are presented in order that the user is able to draw informed conclusions. Footnotes or introductory/explanatory notes should be used to highlight key sources of discontinuity, however additional commentary can provide further context and perhaps assist in outlining the relative significance of various contributory factors. Far from being a source of error, changes and discontinuities can provide a fuller picture of issues relating to a particular dataset. You may also wish to provide an explanation of any key points in any associated minute and news release.