CRIMINAL PROCEEDINGS IN SCOTLAND 2014-15

Annex B: Data Quality, Data Processing and Data confidentiality

Data quality: Data capture

B.1 The Criminal History System (CHS) is an administrative system used to track individuals through the criminal justice system and, as such, was not designed purely for statistical purposes. However, actions and processes have been put in place to ensure that Scottish Government statisticians understand the data.

B.2 Annex A outlines how information is entered on the CHS and that extracts are sent to the Scottish Government from Police Scotland on a monthly basis. The data requirements for these extracts are contained in a joint specification document that has been agreed between Police Scotland and the Scottish Government.

B.3 Monthly extracts are uploaded onto a Scottish Government database at which point validation checks are undertaken to ensure a realistic number of records are added to the database. Checks are also made to ensure values for charges, court locations and disposal type are recognised. If any unexplained patterns or unrecognised codes are identified at the data upload stage, further investigations are undertaken. It may be necessary, at times, to go back to Police Scotland to verify the data.

B.4 Charge codes are the operational codes used to identify the criminal crime or offence and are linked to legislation. New charge codes for crimes and offences under emerging legislation are created by the Crown Office and Procurator Fiscal Service (COPFS) on a monthly basis, and shared with the Scottish Government. If new codes are identified at the data upload stage they are verified and then added to a look-up table of recognised codes.

B.5 The Scottish Government is responsible for mapping each charge code to a crime code, which forms the basis of the crime code classification (see Annex D). There are around 6,000 charge codes which are mapped to around 400 crime types. This mapping is agreed with Police Scotland crime registrars, Police Scotland, the Scottish Police Authority and relevant individuals within partner justice organisations. Once any updates and/or amendments have been agreed for each month, the updated charge code list, together with its mapped crime code, is published by the Scottish Government. The latest version of the charge code list can be accessed here.

Data quality: Data validation during production of the statistical bulletin

B.6 As a court proceeding or police/ COPFS non-court disposal can be made up of more than one offence, production of the statistics at ‘persons’ level requires an intermediary processing stage to be carried out on the CHS data. Where a person is proceeded against for more than one crime or offence in a single proceeding, only the main charge is counted. The main charge is the one receiving the most severe penalty (or disposal) if one or more charges are proved, and is identified using a look-up table which ranks the disposal types in order of importance.

B.7 For example, custody is ranked higher than a monetary fine, so for a proceeding where there was a mixture of these two types of disposal, the main charge counted for this record would be the charge associated with the custody disposal rather than the charge related to the monetary disposal. Once this dataset is created the following types of validation are carried out:

• Automated validation procedures and manual checks to identify any unrealistic data values e.g. long custodial sentences for petty crimes or short sentences for the most serious of crimes. Effort is also made to clean up records for which key information is missing e.g. missing court locations or age/gender of the offender. These are referred back to Police Scotland, Scottish Court and Tribunal Service (SCTS) or COPFS (depending upon the nature of the problem) either for correction or for explanation of any unusual circumstances.
• Other checks are carried out as necessary based on changes to the justice system. For example, between November 2013 and January 2015 some courts have closed with their business transferred to neighbouring courts. Checks were undertaken to ensure the data is consistent with these operational changes. Similarly when new legislation is implemented, checks are undertaken to ensure cases are coming through the system at a realistic rate.
• Trends in the statistics are compared against case processing information published by COPFS and management information provided by SCTS to ensure that the volume of court proceedings are consistent. Information is compared by court type (e.g. high court, sheriff court etc.) to identify any differences.
• Further checks are undertaken by crime type, sentence type and other characteristics to identify any errors. As an extra level of assurance, policy experts within the Scottish Government are consulted to identify why any significant changes may have occurred. Any relevant contextual information is then added to the bulletin.
• Similar consultation is undertaken with COPFS, SCTS and Police Scotland wherein results are shared purely for quality assurance purposes. Insight at an operational level provides invaluable feedback and informs whether further investigation on the statistics is required.
• Further quality assurance and checking is undertaken on the statistics by members of Scottish Government Justice Analytical Services support staff when preparing the tables. Scottish Government statisticians, who have not been involved in the production process, check the results further and highlight issues that may have gone unnoticed.

Data quality: Double counting

B.8 In recent years, we have carried out much more extensive quality assurance with external agencies. The purpose of this is to ensure the accuracy and quality of the statistics published. The COPFS have identified that there may be a small number of court proceedings (often involving multiple charges and of a complex nature) which are being recorded as separate court cases which, in fact, should only be reported as one. The effect of this would be to over-estimate the true number of court proceedings.

B.9 Initial investigations suggest that this affects all crime types, though to varying degrees. Further work will be carried out with a view to quantifying the extent of the problem and identifying whether a change in processing methodology is required.

Data Quality: Police Undertakings

B.10 Please note statistics on police undertakings have not been published this year due to concerns around data quality. We will investigate this issue further and provide an update in next year’s bulletin.

Data confidentiality

B.11 Court proceedings are held in public and may be reported on by the media unless the court orders otherwise, for example where children are involved. However, while our aim is for the statistics in this bulletin to be sufficiently detailed to allow a high level of practical utility, care has been taken to ensure that it is not possible to identify an individual or organisation and obtain any private information relating to them.

B.12 Furthermore, to maintain the security and confidentiality of the data received from the data suppliers, only a small number of Scottish Government employees have access to the datasets outlined in the various stages of processing outlined above. The only personal details received by the Scottish Government in the data extract are those which are essential for the analyses in this bulletin.

Revisions

B.13 The CHS is not designed for statistical purposes and is dependent on receiving timely information from Criminal Justice organisations. A pending case on the CHS should be updated in a timely manner but there are occasions when slight delays happen. Recording delays of this sort generally affect High Court disposals than those of other types of court, as they are the most complex and lengthy trials.

B.14 The figures given in this bulletin reflect the details of court proceedings as recorded on the CHS, that were concluded on or by 31st March 2015, and as provided to the Scottish Government up to the end of September 2015. Any subsequent updates on court disposals made will be incorporated into future bulletins and therefore some figures for 2014-15 (in particular those relating to the High Court) are likely to be subject to minor revisions.

B.15 These recording delays mean that figures for 2014-15 should be considered provisional as future bulletins may provide updates. We estimate that the 2013-14 bulletin contained a small undercount of around 77 people with a charge proved in 2013-14, less than 1 per cent of all people with a charge proved.

B.16 A number of specific revisions have been made to the Criminal Proceedings statistics as described below. Aside from the more substantial revisions some minor revisions were undertaken for this publication including the removal of certain records relating to foreign charges and a reclassification relating to bringing drugs into prisons. Revisions to these statistics comply with Scotland’s Chief Statistician’s current revisions policy.

Stipendiary court – Revision

B.17 During 2014-15, we noted that some Glasgow Stipendiary court records were incorrectly coded as sheriff summary court records. These records mostly related to the years 2009-10 to 2012-13 and have now been corrected. Around 370 to 550 records were affected per year, resulting in an additional 10 per cent stipendiary court convictions between 2009-10 and 2012-13.

Firearms offences – Reclassification

B.18 Since the 2013-14 Criminal Proceedings publication charges relating to possession of a firearm under the Firearms Act 1968 Section 5(1)(B) were moved from the crime of “Handling an offensive weapon” to “Firearms offences” and are now classified alongside other firearms charges. At a crime group level (in terms of the 35 types published in this report) the charges have moved from “Handling offensive weapons” to “Miscellaneous offences”. This change was agreed by justice partners through the process described at Annex B4 - B5.

B.19 This reclassification has been applied through the criminal proceedings series back to 2005-06. The total convictions for handling offensive weapons have decreased by 1 per cent over the period 2005-06 to 2013-14 (138 convictions) as a result of this reclassification.

B.20 Although the changes in the totals are fairly small, these figures are high profile and therefore the change has been highlighted. Please note that the overall trend of a decrease in the number of convictions for “Handling offensive weapons” remains at a similar rate to before the reclassification. In addition the length of custodial sentence and percentage of those convicted being given a custodial sentence remain at similar levels.

Early and Effective Interventions – new disposal information

B.21 In the 2013-14 publication we took out statistics on restorative justice warnings, a police disposal specifically used for dealing with young people.
This was because it was identified that we were excluding other types of youth diversions which meant that our reporting was incomplete and potentially misleading.

B.22 A methodological change was implemented for this year’s publication to estimate statistics on Early and Effective Interventions (EEIs). EEIs are measures used by the police to redirect juveniles away from the adult courts and the Scottish Children’s Reporter Administration (SCRA). This is the first time these statistics have been published and they can be seen in section 14.

B.23 It was identified that EEI records coming through on the monthly files from the Criminal History System were not being picked up at the data processing stage outlined above. This is because the EEI disposals were not correctly specified in the look up table that ranks the disposals in order of importance.

B.24 Please note that there may be an element of improved recording of EEIs on the CHS as time progresses. This happens as the new codes become more familiar to the police in terms of recording the information on the CHS and can be an issue when a new disposal is introduced.

B.25 We have incorporated statistics on restorative justice warnings back into the publication this year as the new information on EEIs helps to explain the trends we previously felt were misleading. Please note that restorative justice warnings and EEIs are only two of a wide range of interventions the police and other organisations use to deal with young people involved in offending behaviour so there are still data gaps in terms of quantifying the full levels.