Scottish prison population statistics technical manual

CellWise Data Construction

• CellWise reconstruction of prison occupancy
  • Stint
  • Analytical period
  • Occupancy period
  • Quality assurance and error-checking
• Prisoner characteristics
• Geographic data
• Legal information and court warrants
  • Warrant span
  • Sentencing data
  • Legal status heirarchy
  • Index offence derivation
  • Sentenced Index Offences
  • Remand Index Offences
  • Index Sentence derivation
  • Validation of Index Sentence population sizes
  • Statutory Throughcare flags
  • Validation of Statutory Throughcare markers
• Liberation data

The unique construction of cellWise data from the prisoner records system leads to a number of considerations that users must take into account when interpreting these statistics.

Data from PR2 for analysis is extracted three months after the analytical period has ended. This allows time for the information held on prisoners to be corrected and confirmed before analysis begins.

In this section we explain the construction of the longitudinal prison database:

• the cellWise reconstruction of prison populations, and its subdivision for analysis within a given analytical period
• personal attributes of people in custody from the Prisoner Manifest
• the addition of Geographical Data
• the augmentation of the population spine with Legal information and court warrants
• how Liberation data is integrated, and how it should be interpreted

CellWise reconstruction of prison occupancy

The most complete record of a prisoner’s presence in a prison establishment that is retained in sufficient detail historically on PR2 is the occupancy by prisoners of specific cells in the estate. (A cell is any space that a prisoner can be registered as resident in over the course of their stay, including temporary holding areas during reception, etc.).

This section describes the construction of a longitudinal data spine from cell occupancy data, and how that spine is subdivided to produce statistics within an analytical period.

Subsequent sections discuss the methods by which additional information is joined to this spine.

Stint

A stint in prison is defined as the period in which a prisoner has an allocated cell (or series of cells) in a single SPS establishment. Stints are maintained at establishment level to allow for prison-level analysis of the population.

A stint has five attributes:

• The prisoner number identifying an individual
• The establishment in which it occurred
• The date it started – when the person entered the establishment, either from the community or from another prison
• The date it ended – when the person left the establishment, either to the community or for another prison
• The duration in days

The data in Table 1.1 provides an example of the data of cell allocations and vacations for a single prisoner. These individual cell occupancies are combined into stints in Table 1.2. In Figure 1 the resulting stints are visualised as blocks of time over the course of four years.

Table 1.1: Example cellWise data for a person in prison

Establishment	Cell Allocation Date	Cell Vacated Date
HMP Maidup	25/04/2001	25/04/2001
	25/04/2001	02/05/2001
HMP Barloney	02/05/2001	12/05/2001
	12/05/2001	26/07/2001
	26/07/2001	07/08/2001
	07/08/2001	02/10/2001
	02/10/2001	19/10/2001
	19/10/2001	12/02/2002
HMP Careloch	24/09/2003	04/10/2003
	04/10/2003	14/10/2003
	14/10/2003	24/10/2003
	24/10/2003	03/11/2003
HMP Glen Nochil	03/11/2003	23/02/2004
	23/02/2004	14/06/2004

Table 1.2: Table 1.1 data compressed into stints

Stint	Establishment	Date In	Date Out	Prisoner Days
1	HMP Maidup	25/04/2001	02/05/2001	7
2	HMP Barloney	02/05/2001	12/02/2002	286
3	HMP Careloch	24/09/2003	03/11/2003	40
4	HMP Glen Nochil	03/11/2003	14/06/2004	224
			Total:	557

Figure 1.1: Table 1.1 data visualised as stints

Analytical period

An analytical period can effectively be of any length. Using the derived Stint information, we can determine how many prisoners were present for (some part of) any day or combination of days. In general, when we refer to analytical periods in the Official Statistics, we are talking about calendar or financial years.

The intersection of stint and analytical period allows us to make statements about the population in that period:

• How many Individuals spent any time in an establishment over that period
• What was the average occupancy – or Average Daily Population (ADP) – over an analytical period
• How many prison stints started in the period, i.e. how many Arrival and transfers between establishments occurred
• How many prison stints ended in the period, i.e. how many Departure and transfers between establishments occurred

In Table 1.3 the cellwise data from ` are used to calculate the number of prisoner days in each stint as they intersect with financial year analytical periods. The weight for each stint – each stint’s contribution to the average daily population – is then calculated based on the number of intersecting prisoner days divided by the total days in the analytical period.

Table 1.3: Stints within analytical periods

Analytical Period	Stint	Establishment	Intersecting Prisoner Days	Weight
2001/02	1	HMP Maidup	7	0.019
2000/01	2	HMP Barloney	286	0.784
2002/03	-	-	-	-
2003/04	3	HMP Careloch	40	0.109
2003/04	4	HMP Glen Nochil	150	0.410
2004/05	4	HMP Glen Nochil	74	0.203
		Total:	557

Occupancy period

If a prisoner is not allocated a cell on a certain day between two stints, they are considered absent from prison for that day. The preceding stint ends in a departure, while the subsequent stint begins with an arrival.

Occupancy periods are effectively Stints joined end-to-end with no gaps. An occupancy period begins with an Arrival and ends with a Departure (if the individual has left custody for one or more nights).

Where a prisoner moves from one Scottish prison to another within the space of a day, i.e. a prisoner transfer, this is part of the same continuous occupancy period. Combining stints into occupancy periods removes the distinction between establishments.

Occupancy periods are used as the matching basis for information from:

• Geographical Data at the beginning of the occupancy period
• Legal information and court warrants, including information about offences, sentences, etc.
• Liberation data, at the end of the occupancy period.

Returning to the example data from Table 1.1, the individual establishment-level stints may be combined into two occupancy periods, as shown in Figure 2.

Figure 2: Table 1.1 data compiled as stints and as occupancy periods

The first two stints are combined into an occupancy period entirely within the analytical period 2001-02. The latter two form an occupancy period that straddles the analytical periods 2003-04 and 2004-05. This means that Occupancy Period 1 contributes an arrival and a departure to the total counts in 2001-02. Occupancy Period 2 contributes an arrival in 2003-03 and a departure in 2004-05.

Figure 3 provides some further example occupancy periods for notional prisoners within an analytical period:

1. A full year prisoner, who was in custody prior to the beginning of the analytical period and remains in custody throughout that period and beyond, has no arrival or departure within an analytical period. They may move between prisons.
2. An individual whose occupancy period begins prior to the analytical period but departs before the end of the analytical period. One departure.
3. An individual who arrives and departs prison within the analytical period. One arrival and one departure.
4. An individual who arrives in the analytical period but does not depart in the same period. One arrival.

These occupancy patterns can appear in combination, meaning a single prisoner can arrive and depart multiple times in the space of a single analytical period.

Figure 3: Patterns of occupancy for individual prisoners in the space of an analytical period

Quality assurance and error-checking

Cell allocation data are not perfect and require some quality assurance before sequencing. The sorts of errors arising in these data which must be resolved include:

• Atemporal records: the allocation date occurs after the vacation date
• Coincidental occupancy: an individual is allocated to multiple cells in a prison at the same time
• Co-location: an individual is allocated cells in multiple prisons at the same time

These issues are identified in the process of producing the stint records.

External data sources for validation are generally not available. To resolve these issues in the data build process, analysts refer to other tables on PR2 – specifically records of Admissions and Liberations – to validate and/or correct the cell allocation data. This is one of the more time-consuming manual tasks in the production process.

For more information about data quality and quality assurance, see the Data Quality section.

Prisoner manifest

Many of the factors associated with prisoner characteristics held on PR2 do not change over time. For those that do, the historic information held about prisoners is over-written with corrected values. The prisoner characteristics joined to the cellWise data are therefore the most up-to-date values for each prisoner as extracted. Those details include a number of self-declared attributes of prisoners, and details taken from documents passed to SPS on admission.

The prisoner manifest is the source for the following prisoner characteristics:

• Prisoner Date of Birth: used to determine Prisoner Age, and banded age groups
• Prisoner Gender: used to determine Gender
• Prisoner Forces Service: output as Armed Forces
• Prisoner Disability: output as Disability
• Ethnicity Group & Subgroup: output as Ethnicity
• Prisoner Nationality: output as Nationality
• Prisoner Marital Status: output as Marital Status
• Prisoner Sexual Orientation: presented in analytical papers only due to small numbers of minority groups. Output as Sexual Orientation.
• Prisoner Religion: excluded from analysis due to inconsistencies and errors in collection.

Data Quality

In the Sexual orientation field, the number of “Missing” cases has fallen from 2015-16 onwards, but sigificant numbers of individuals still decline to log a response when asked about their sexual orientation. No inference about these individuals can be made on the basis of non-response.

Similarly for Disability, the number of “Missing” cases has fallen from 2014-15 onwards.

Geographic data

Geographic data are time-related: the address associated with an individual is liable to change over the course of their stay. However, updates to the current address of a prisoner are not maintained systematically. All prisoners provide an address (where possible) at admission. We therefore select the geographic record closest to the arrival date to characterise an individual for each of their Occupancy Periods.

The following information is drawn from the address information provided by prisoners at or near their admission date:

• Postcode: the basis for Local Authority attribution, and SIMD 2016 Decile Group, via matching to a postcode database of 2016 datazones
• Country: where valid postcode information is not provided, we can use this country field to identify the usual abode of prisoners
• No Fixed Abode: on arrival the person reports being Of No Fixed Abode

Legal information and court warrants

In this section we explain the method for overlaying court warrant and sentencing information held on PR2 with the cellWise population spine. Based on the resulting combined data, a number of new measurements of the prison population become possible. These are designed to be internally consistent and comparable across time and between population segments.

Information provided by the Scottish Courts and Tribunals Service indicates the basis for individuals’ imprisonment and (in the case of determinate sentences) the associated sentence length.

This information is held across multiple tables in PR2, the most complete of which is the Court Warrants data. Where there is information missing from this table, it is supplemented by data from additional tables which hold information about sentences and recall warrants.

A number of steps are required to link the previously derived occupancy data with information held on PR2 about legal status, offences, and sentences.

1. Occupancy Periods are derived based on continuous occupancy in any SPS establishment
2. Warrant Spans are derived based on the start and (nominal) end date of each warrant held on PR2.
3. Warrant spans are joined to occupancy periods by date overlap
4. The first warrant of each type defines the start of that Legal Status within each period according to the Legal Status Hierarchy

This process is visualised in Figure 4.

Figure 4: Joining warrant spans to the population spine

Warrant span

The end date of a warrant’s effect is not uniformly included in the data held on PR2. When a person presents at prison with sentenced warrants, the admissions officer calculates an overall sentence length based on guidance from the Court, and uses that to calculate a prisoners current Earliest Date of Liberation (EDL). However, this is a live value liable to being updated over the course of a prisoner’s stay, and whether sentences are to be served consecutively or concurrently is not retained on the PR2 system. As a result, there is ambiguity on the end date of warrant sentences.

To attach warrant information to Occupancy Periods, we must determine a span for each warrant to match them individually to the population spine (see Figure 4). The value chosen for a warrant’s length of effect varies depending on the warrant type:

• For remand-type warrants, a flat 10-day length is attributed to the warrant.
• For determinate sentencing warrants, the full length of the sentence is used to match the warrant details to the population spine.
• For indeterminate sentences, an effectively infinite duration is attributed.

For individuals who have received an indeterminate sentence warrant, this will affect any further occupancy periods (where these exist). This reflects the nature of indeterminate sentences, where any individual given an indeterminate sentence is subject to conditions pertaining to that sentence on release from custody.

Sentencing data

As well as in the warrants data table, PR2 also holds a separate table of information about the sentences served by people in prison. They do not however include information about the offence committed. In cases where no warrant data are available for matching, gaps in sentencing information are filled with information from this secondary table. These data provide the length and type of sentence and when that sentence started to be served, and exclusively backfills information on indeterminate sentences.

Legal status heirarchy

Time within an Occupancy Period may be divided into three broad types of occupancy in prison:

• U: Untried: Unconvicted individuals held in prison awaiting court proceedings
• A: Convicted Awaiting Sentence: Time spent convicted of an offence but awaiting a sentence. Sometimes shortened to “CAS”.
• S: Sentenced: Time spent in prison with a custodial sentence. This includes individuals with determinate sentences, indeterminate sentences like Life sentences, and those recalled to prison after previously being released.

An individual may hold one or more of these legal statuses over the course of their imprisonment.  It is also possible that individuals may come to the end of a sentenced period in custody but be remanded immediately on the basis of a separate criminal proceeding,  We are not able to attribute the times spent on remand after a sentenced period with the current data design due to the ambiguity around the end dates of Warrant Spans.

U- or A-time indicates the length of a period after the first designation in each category before entering a subsequent status, as shown in Figure 4. It is therefore not possible to attribute backwards movement through the U-A-S hierarchy. This is evident in occupancy periods ending in a sentenced status being liberated “On Bail”, or other liberation types associated with remand prisoners.

Index Offence Derivation

People are often imprisoned for a number of offences at the same time.  This may be a series of offences committed during a single incident, or a series of offences committed over a given course of time.

Due to the structure of the warrant file in PR2, it is not always possible to disentangle which offences are active or inactive at what stage of an Occupancy Period.

We therefore provide statistics on Index Offence. Index offences are intended to represent the most serious offence for which a prisoner is imprisoned, and are determined per occupancy period:

Sentenced Index Offences

For the collated sentencing warrants applicable over an occupancy period, we must pick the index offence based on the following prioritisation:

1. the offence associated with a life sentence
2. where there are no life sentences, the offence associated with the longest determinate sentence
3. where multiple offences have the same sentence, the index offence is the offence associated with the longest average sentence length, as shown in Table 1.4
4. where information on the offence is missing but there if a life sentence or other indeterminate sentence associated with a sentenced period in custody, the label “98: Indeterminate Sent. - Unknown Offence” is applied

The assumptions  made in the process of assigning an index offence introduce potential for error in attribution, which must be borne in mind in interpreting and using the published statistics:

• Individuals may be sentenced for a less serious offence first, and spend several months serving that sentence while awaiting trial, receiving the heavier sentence later in the occupancy period. Particularly where the occupancy periods overlap multiple years, we may therefore over-count certain offences in-year when compared with custodial sentence statistics from criminal proceedings statistics.
• Individuals might serve a series of consecutive sentences for lesser offences. It may appear that individuals remain in prison for extended periods for what looks like a single, lesser offence. However this extended period might be justified by multiple sentences served consecutively.
• Offence categories are quite broad, and there is a good deal of variability in sentence lengths for the 50 offence groups. The average sentence length is arguably a poor measure of the actual severity of each of these categories taken together.

Remand Index Offences

Where warrants do not have associated sentences, the determination of the index offence is based solely on the average sentence for each category, as shown in Table 1.4.

The prison statistics publication covering 2021-22 preceded the publication of Criminal Proceedings for that period. The offences are therefore ordered based on average sentences held in the warrant files on PR2 for each offence category.

Statistical publications up to and including financial year 2020-21 used the former set of offence categorisations listed in Table 1.5. The average sentences used for this reordering are provided by Scottish Government criminal proceedings statistics.

Table 1.4: Offence Categories ordered by Average Length of Sentence

#	Scottish Government Top 50 Offence Category	Order by Average Length
1	Murder and culpable homicide	1
8	Rape & attempted rape	2
98	Indeterminate Sentence - Unknown Offence	3
2	Causing death by driving dangerously	4
9	Sexual assault	5
3	Serious assault and attempted murder	6
5	Robbery	7
15	Other sexual crimes	8
13	Indecent photos of children	9
14	Crimes associated with prostitution	10
40	Licensing offences	11
24	Fire-raising	12
30	Drugs - Supply	13
11	Communicating indecently	14
6	Domestic Abuse (Scotland) Act 2018	15
7	Other non-sexual violence	16
10	Causing to view sexual activity or images	17
12	Threatening to or disclosing intimate images	18
16	Housebreaking	19
32	Other crimes against society	20
4	Common assault	21
17	Theft by opening lockfast places	22
19	Theft of a motor vehicle	23
26	Reckless conduct	24
28	Weapons possession (not used)	25
42	Other misc. offences	26
43	Dangerous and careless driving	27
22	Fraud	28
18	Theft from a motor vehicle	29
20	Shoplifting	30
21	Other theft	31
23	Other dishonesty	32
33	Coronavirus restrictions	33
35	Racially aggravated conduct	34
41	Wildlife offences	35
25	Vandalism	36
27	Crimes against public justice	37
34	Threatening and abusive behaviour	38
44	Driving under the influence	39
38	Community and public order offences	40
31	Drugs - Possession	41
77	Other Jurisdiction Charge	42
46	Unlawful use of vehicle	43
39	Environmental offences	44
50	Other road traffic offences	45
47	Vehicle defect offences	46
36	Drunkenness and other disorderly conduct	47
37	Urinating etc.	48
45	Speeding	49
48	Seat belt offences	50
49	Mobile phone offences	51
88	Other - not classified	52

Table 1.5: Offence Categories ordered by Average Length of Sentence

Scottish Government Top 35 Offence Category	Order by Average Length
1    Homicide etc.	1
5    Rape and attempted rape	2
4    Other non-sexual crimes of violence	3
2    Attempted murder and serious assault	4
6    Sexual assault	5
3    Robbery	6
8    Other sexual crimes	7
17    Fire-raising	8
22    Other other crimes	9
21    Drugs	10
9    Housebreaking	11
20    Handling offensive weapons	12
15    Fraud	13
28    Dangerous and careless driving	14
27    Other miscellaneous offences	15
26    Urinating etc.	16
23    Common assault	17
11    Theft from a motor vehicle (OLP)	18
12    Theft of motor vehicle	19
10    Theft by opening lockfast places	20
16    Other dishonesty	21
18    Vandalism etc.	22
31    Unlawful use of motor vehicle	23
14    Other theft	24
24    Breach of the peace etc.	25
29    Driving under the influence	26
19    Crimes against public justice	27
13    Shoplifting	28
35    Other motor vehicle offences.	29
33    Seat belt offences	30
34    Mobile phone offences	31
7    Crimes associated with prostitution	32
25    Drunkenness and other disorderly conduct	33
30    Speeding	34
32    Vehicle defect offences	35

Index Sentence derivation

Sentencing warrants are associated with Occupancy Periods. However, information about how these discrete records should be processed and combined is largely absent from the system.

Determining the overall length of sentence a prisoner is serving in a given occupancy period has therefor not been possible with these data.

On arrival to prison, an officer calculates the overall sentence length of the prisoner based on a combination of the sentences they arrive with. This includes information from the Court about whether sentences are to be served consecutively or concurrently. This information is not transferred to PR2 and therefore cannot be used in our calculations. Overall sentence length can also change over the course of a period in prison, as it is recalculated if more convictions and sentences are added during that period.

The longest sentence from a Warrant Span intersecting an Occupancy Period is used to characterise the sentenced portion of that period.

Where, for example, a prisoner is serving a overall sentence of multiple years as a result of several consecutive shorter sentences, they will be categorised by index sentence length as the longest of those constituent sentences.

Added to this information on determinate sentences, we have included the available information about life sentences and certain recall types from additional Sentencing data.

Validation of Index Sentence population sizes

We can compare the distribution of prisoners by sentence length in 2013-14 from this newly build data set with previously published statistics as shown in Figure 5.

Figure 5: Sentence Band Comparison of Index Sentence Length population estimates from cellWise data from 2013-14 with Overall Sentence Length bandings from previously published National Statistics

The previous statistics publication was based on a collation of daily snapshots, and therefore retained the overall sentence length information from the live PR2 system.

The following differences between sources can be observed:

• A greater number of prisoners sentenced to under 6 months but fewer in the longer sentence length categories when using the cellWise method.
  This is likely a direct result of our inability using the cellWise data to reliably combine sentences, whether consecutively or concurrently
• A greater number of life prisoners when using the cellWise method: Life sentences are given precedence over sentences in the cellWise build, and may legitimately over-ride determinate sentenced stints for some prisoners.
• Fewer recalled prisoners using the cellWise method: Recall is given precedence in this categorical description over determinate sentence lengths. Individuals recalled to custody may quickly complete their sentence on recall then transition to an alternative sentence or remand group. This subsequent movement would not be detected, as described in Legal Status Hierarchy.

As a result of these shortcomings in the utility of these data, we cannot say that the newly constructed data reflect the population in the same way as the previous data set. Throughout reporting we therefore refer to the “index” sentence, rather than overall sentence length, to differentiate these two different metrics.

Statutory Throughcare flags

Indicators of the presence of a Supervised Release Order (SRO) or an Extended Sentence disposal from the court are held on PR2 along with the date that order was applied.

For more information on these types of statutory throughcare, see the Scottish Sentencing Council website.

These disposals are associated with an occupancy period were the date the disposal was applied is on or after the date of the first sentenced warrant in that period. Only the presence or absence of such a flag is noted. Statutory throughcare will not always take place on departure if other more stringent conditions are applied.

 

Validation of Statutory Throughcare markers

Comparing results from the 2023-24 release with the information provided by Justice Social Work statistics we see that the trends track each other closely, although there is some degree of variation between the two counts.

Figure 6: Comparing Justice Social Work Statistics and Prison Population Statistics between 2014-15 and 2023-24

 

The number of individuals departing an occupancy period associated with a Supervised Release Order (SRO) peaked at 355 in 2016-17, when they comprised 3.7% of departures. However, the reduced number of departures in 2022-23 meant that the smaller number of such releases (302) accounted for a greater proportion of those entering the community (5.5%) in 2022-23. The 297 individuals departing with an SRO flag in 2023-24 made up 5% of all people departing custody.

Comparing these figures with the number of community commencements each year, we observe a systematic over-estimate in the prison data. This is to some extent expected: SROs are given as part of a short sentence with a fixed end date. If a more stringent condition or additional sentence is applied to that person in custody that would disrupt the planned SRO, that Order would be overridden and not commenced.

So for example there was a single individual who departed from an occupancy period in 2023-24 which was associated with both an SRO and an extended sentence. In this case the extended sentence would be the focus of the justice social work data collection and would be the only commencement counted in these statistics. Other similar interactions are the likely cause of the apparent over-count of those departing with an SRO in the prison statistics.

Liberation data

When a prisoner leaves prison and is not expected to return under the same legal basis, their liberation is recorded on the SPS management information system.

We have made a distinction here between Departures – defined as a stint in prison ending and not immediately followed by a further stint – and “liberations”, where an individual has finished their period in custody and is released. A person in prison can have multiple departures and liberations within a year if they are released for short periods but subsequently return.

Where a departure coincides with a Liberation record (+/- 1 day) that departure is counted as a Liberation. Where no Liberation is available, they are simply counted as departures and Liberation Type recorded as “(Missing)”.

People may leave prison without a liberation being recorded if their absence is temporary, for example if they are on leave to return home for a night or more. In these cases no liberation is recorded – the prisoner is still under the authority of the Scottish Prison Service. These “non-liberation departures” are relatively few each year (1-2% of departures).

If a liberation record occurs but does not coincide with a departure it is not not reported on in statistical outputs.

Liberations to court are included – while they generally occur in the middle of a stint in prison, after which a person would return to prison at the end of the day, where they coincide with a departure they are listed as such.

The categories of Liberation Type reported upon are listed in the Analytical Factors section of this document.