Evaluation of the Reforms to Summary Criminal Legal Assistance and Disclosure

Appendix C - Research Methods

Summary of research design

This study used quantitative and qualitative methods to explore the practice and operation of the summary justice system and participants within it pre- and post-reform of SCLA and disclosure, perceptions of the reforms amongst key participants involved in the summary justice system, the impact of the reforms on case trajectories, timescales and outcomes, and the financial impact of the reforms.

Our approach sought to explore, in detail, the impact on the reforms on the behaviour and practice of key criminal justice personnel, identify facilitators and barriers to the reforms achieving their objectives and assess the contribution of the reforms to the overall aims of SJR.

The design was based on a number of, separate, but linked, research elements, and involved the collection of primary qualitative data from depth and semi-structured face to face and telephone interviews, alongside detailed analysis of existing aggregate national performance/monitoring data and economic data and individual case-level data. The principal elements involved:

• Qualitative exploration with key stakeholders of the perceived objectives, operation and outcomes of reforms to SCLA, disclosure and SJR as a whole;
• Quantitative analysis of the 'impact' of the reforms on case timings, trajectories and outcomes at both a national and, where possible, individual court level;
• Qualitative examination of the local operation and impact of SCLA and disclosure in four Sheriff courts;
• Analysis of the content and quality of a sample of 200 disclosable summaries and consideration of data on their subsequent case trajectories and stage of conclusion;
• Quantitative analysis of data recorded in selected courts on the reasons for continuation of IDs;
• Depth face-to-face interviews with accused persons, where possible with experience of the summary system before and after the reforms;
• Exploration of 'national level' views on the reforms amongst defence solicitors - via a postal survey and follow-up telephone interviews - and amongst Depute Procurators Fiscal via telephone interviews;
• Analysis of case trajectory and cost data to identify typical court, prosecution, sentencing and legal aid costs associated with processing Sheriff court cases before and after reforms.

Stakeholder interviews

Scoping interviews with key players in the initiation of the reforms were conducted to examine perceptions of the original problem which the reforms would address and perspectives on what the reforms hoped to achieve as well as exploring understandings of 'fairness' and 'effectiveness' in relation to the broader SJR aims. Interviewees from COPFS, SLAB and ACPOS were also asked about the monitoring data collected by these agencies. Eight interviews were undertaken in total involving 10 individuals, drawn from the stakeholder agencies/organisations listed below.

• COPFS
• ACPOS
• An experienced Sheriff
• SLAB (A small group discussion was held with three senior SLAB officials)
• Law Society (2 representatives of the Law Society were interviewed)
• Bar Association (representatives of 2 Bar Associations were interviewed)

Analysis of quantitative monitoring and case trajectory data

Analysis was undertaken of SJR monitoring and Key Performance Indicator (KPI) data to compare over time the differences in the outcome of PDs, IDs and TDs, differences in the stage of conclusion of summary cases by court, and agency performance against target timescales set out in the Summary Justice Model Paper. The data was also used to explore broader changes in the nature and distribution of summary business and to examine changes in applications for and grants of criminal legal aid.

The majority of data in the KPIs relevant to the analysis of SCLA and disclosure is available as monthly counts from April 2006. Data was extracted corresponding to the period April 2006 to March 2011. To consider the impact of the changes to SCLA and disclosure on case trajectories and outcomes, a series of average monthly plea rates were calculated for each of the KPIs used. The rates indicate the proportion of each diet held in each month which resulted in a particular outcome - e.g. guilty plea offered, adjourned. These average rates correspond to the following periods:

• The period prior to the introduction of the disclosure reforms (April 2006 to September 2007);
• The period between the introduction of the disclosure reforms and the introduction of the legal aid reforms (October 2007 to June 2008).
• The first year after the legal aid reforms were introduced (July 2008 to June 2009)
• The second year after the legal aid reforms were introduced (July 2009 to June 2010)
• The first three quarters of the third year after the legal aid reforms were introduced (July 2010 to March 2011).

The data was arranged in this manner in order to tentatively examine: first, the impact of changes to disclosure; second, the combined impact of subsequent changes to legal aid; and third, whether any change which appears to have resulted from the reforms has been sustained and whether outcomes and trajectories have stayed the same post-reform or continued to change.

Where possible, local variations in these data were explored at individual Sheriff court level for the four Sheriff courts selected for the qualitative case study. However, not all indicators of interest were available at individual court level.

As data was not available from JP/District courts prior to unification, it is not possible to compare pre- and post-reform rates on these indicators.

• In-depth face-to-face interviews with a representative of each agency sitting on the SCLA Monitoring and Evaluation Group.
• Analysis to identify any changes in case timings, trajectories and outcomes since the two-stage onset of the reforms, and differences in the magnitude of any observed changes between courts. This early analysis was used to select the four courts necessary for the second stage of the research.

Qualitative examination of the local operation and impact of SCLA and disclosure reforms in four courts

This stage of the research involved observation of court proceedings along with qualitative, in-depth interviews with key justice personnel in four Sheriff courts. A range of monitoring data was extracted and analysed to influence the selection of the four courts. Selection criteria for the courts, including the justification for considering each criterion, are detailed in Table A.1 below.

Table A.1 Criteria for selection of Sheriff courts for case study

Criteria	Detail	Reasons for inclusion
Volume and mix of court business	- Number of total and different types court diets	- The total number of cases heard, and the nature of those cases all potentially impact on the ability of agencies involved to prepare and respond as required.
Change in guilty plea rate at pleading diet	- % of pleading diets where guilty plea is offered	- An increase in Guilty plea rate suggests both an impact of reforms to SCLA through the new case disposal fee, and an impact of disclosure reforms where disclosable summaries allow defence solicitors to have more informed discussions with PFs and clients about the most suitable plea to be entered. A greater increase, and one which has been sustained, may suggest greater impact of the reforms.
Continuations and other outcomes at intermediate diet	- % of intermediate diets which are continued % of intermediate diets with other outcomes[28]	- A key objective of the reforms to disclosure is to permit more 'effective' intermediate diets. Effective intermediate diets may be considered to be those which are 'terminal' - allowing the trial diet to be cancelled - or which do not affect the existing date for the trial diet. Previous research (Leverick and Duff, 2001; Stephen and Tata, 2006) also indicates that one of the main reasons for intermediate diets being continued is that the defence is still waiting on statements from the Crown. Thus, an increase in 'terminal' outcomes at intermediate diet, and a decrease in continuations may suggest greater impact of the reforms on practice and procedure.
Average number of diets per case		- Each of the reforms seeks to create a more efficient and effective summary justice system. A reduction in the average number of diets suggests more effective diets, and a more efficient system.
Change in volume of legal aid grant applications	- Number of grants of Criminal Advice and Assistance Number of grants of ABWOR Number of grants of SCLA	- A knock-on effect of any increase in guilty plea rates at pleading diet will be a subsequent decrease in applications for summary criminal legal aid. Examination of changes in the volume and types of legal aid applications made will give some indication of the success of the reforms against key objectives.
Geographic composition of Sheriffdom/location of court	- Sheriffdoms which cover areas which are predominantly urban or rural Courts located in urban or rural areas	- To allow exploration of variations in operation of reforms which may result from particular geographic issues, for example related to rurality - e.g. access, payment methods.

Relevant data to explore each of these criteria were downloaded from the CJBMI system or provided by the SCS. These data were then analysed and the relevant courts selected. To protect the identities of interviewees, the names of all courts have been anonymised.

Volume and mix of court business

Data on the number of PDs, IDs and TDs heard in each of the 49 Sheriff courts in each month from April 2007 to December 2009 was considered. This data was believed to provide a suitable measure of court throughput. Monthly data for the whole period was combined and a figure calculated indicating the average number of diets heard in each court per month. This figure ranged from 10 to 3625. To filter out courts with very low throughput - which were considered too small to be included in the research - only courts with more than 150 diets per month were considered for selection. After this filter was applied, 25 courts remained available for selection.

Change in guilty plea rate at pleading diet

KPI25 on the CJBMI system records, for each month and in each Sheriff court, the proportion of accused who pled guilty at PD. To consider change in guilty plea rate, two average monthly guilty plea rates were calculated: one referring to the period prior to the introduction of the SCLA reforms (April 2006 to June 2008); the other referring to the period after the introduction of the reforms (July 2008 to December 2009). Amongst the busier courts included in the analysis, the pre-reform rate ranged from 10% to 43%. Notably, the courts for whom the average pre-reform guilty plea rate was lowest were the largest and busiest courts. The post-reform rate ranged from 23% to 50%.

These data were used, amongst those courts with more than 150 diets per month, to differentiate between two different groups:

• Courts with a relatively high proportion of guilty plea rates at PD in the pre-reform period.
• Courts with a relatively low proportion of guilty plea rates at PD in the pre-reform period.

The 25 busier courts were split into four equal groups (quartiles) according to their average pre-reform monthly guilty plea rate. Any court with a pre-reform guilty plea rate in the top quartile of busier courts was considered to be in the high group (equal to a pre-reform guilty plea rate of 25% or higher). Any court with a pre-reform rate in the bottom quartile was considered to be in the low group (equal to a pre-reform rate of less than 18%). Six courts fell into the high group and five fell into the low group.

To examine the change in guilty plea rate, the difference between the pre-reform rate and the post-reform rate was calculated in percentage points illustrating the 'impact' of the reforms. This difference ranged from 5 to 17 points. However, for comparative purposes, the relative 'rate' of change was then also calculated. This figure indicates the relative increase in average guilty plea rates between the pre- and post-reform periods. It permits an exploration, among courts which saw a similar change in rates as measured by percentage points, of the magnitude of that change relative to the pre-reform rate. For example, say guilty plea rates in Court X and Court Y both increased by 11 points. Because Court X started from a lower pre-reform rate of 16, compared with 26 in Court Y, the rate of change is thus higher for Court X at 1.6 compared with 1.4 in Court Y. This data on the rate of change was used to further differentiate courts in the high and low groups identifying courts in each group where there was a high increase in guilty plea rate at PD or a low increase relative to the pre-reform rate following the introduction of the legal aid reforms.

Our aim was to select, from each group, one court displaying a low rate of change and one court displaying a high rate of change. Furthermore, the selected courts were required to reflect variations in volume of business and geographic areas. The courts selected had the following characteristics:

• Court A: This is the largest Sheriff court selected and has the lowest % guilty plea at PD, but high increase in plea rate. Court A represents a very large urban area.
• Court B. This is a medium-sized court with a relatively low % guilty plea at PD pre-reform and low impact. Court B represents a medium-sized urban area located within predominantly rural surroundings.
• Court C: This is a larger court with a relatively high % guilty plea at PD pre-reform, and high impact. Court C represents a large urban area.
• Court D: This is a smaller court with a relatively high % guilty plea at PD pre-reform, but low reform impact. Court D represents a relatively rural area.

Data illustrating the remaining criteria for selection were then considered in order to compare other areas of potential impact. Overall, the selected courts are believed to represent a reasonable variation of reform impact according to these additional criteria. Furthermore, several of the proposed courts were being included in the various local court recording exercises which are discussed below. This would potentially generate detailed data on reasons for continuation in these courts.

Observation of court proceedings were undertaken for each case study court. In addition, for each of the 4 courts at least 7 face-to-face interviews were conducted with key personnel some of whom had been involved in the observed cases: Sheriffs/JPs (2), Depute PFs (2), defence lawyers (2) and court clerks (1). Face-to-face interviews with 2 senior police officers in each case study area were also conducted. These interviews were conducted between July and September 2010 and covered how the reforms were impacting on the practice of key criminal justice personnel and their perceived effect on case trajectories and outcomes.

Interviews with persons accused

In-depth face-to-face interviews were conducted with 16 persons accused (4 in each case study area) between October and November 2010. The sample was recruited via a variety of sources including defence lawyers, social workers and the police. Four interviews took place in the respondents' homes with the remainder conducted within agency offices (e.g. Arrest Referral Scheme offices, NHS Substance Misuse agency offices) or whilst the respondent was in custody. All respondents had experience of the Sheriff court process pre- and post-reform. Interviews covered views and experiences of court appearances before and after the reforms were introduced as well as perceptions of fairness and views on the actions of their lawyers.

Analysis of disclosable summaries

An analysis of 192 anonymised disclosable summaries supplied by COPFS was carried out in October and November of 2010. The sample comprised 50 disclosable summaries from each of three summary Sheriff courts which were investigated in detail and 42 from the remaining court (reflecting the number supplied rather than the number requested). The sample was selected at random simply by taking the first 50 relevant^[29] cases in each court from 1^st July 2009. This date was selected on the basis that it was sufficiently distanced from the introduction of the reforms to allow for any initial teething problems to have been addressed and for the new processes to have 'bedded in'. It was also far enough in the past from the current date so that the majority of cases would have reached a conclusion thus allowing consideration of that conclusion in the additional trajectory data received on the cases.

For each of the selected cases, the disclosable summary was printed and supplied to the research team. The names of the accused, witnesses and any other identifying details, such as addresses, were redacted from the summaries by COPFS before the disclosable summaries were handed over to the research team.

The disclosable summaries were assessed against a number of criteria, based on the ACPOS guidance contained in the ACPOS Disclosure Manual. Summaries were thus scored according to a 5 point scale - a '5' indicating that the information provided was entirely satisfactory. The ACPOS guidance explains that the COPFS 'summary of evidence' should be drawn from the following sub-sections of the police report: description of locus; description of events; police interview/text of admission; caution and charge/reply; and medical evidence. In addition to assessing the disclosable summaries against these criteria, an overall rating was given to each one and various other data was recorded about the nature of the case. These included: the broad type of crime involved; whether the accused was remanded in custody until the next court; and the PF case number. The ratings and other data were recorded in an Excel spreadsheet.

A number of defence solicitors were recruited in order to test the validity of the research team's ratings. These were each given a sample of ten disclosable summaries and asked to give them an overall rating on the same 5 point scale already used. Each solicitor was given some satisfactory and some less than satisfactory summaries. When, in their judgement, there was insufficient information in the disclosable summary, they were asked to indicate briefly what was missing that they would have found helpful in advising the accused if he/she were their client. As a result of the low number of summaries rated as less than entirely satisfactory (see below), it was possible to include the great majority of these in the summaries given to defence solicitors.

Methodological points

An important methodological issue arose very early on in relation to this exercise. Essentially, the task was to determine how satisfactory the disclosable summaries were in supplying sufficient information to the accused and, more important, his/her solicitor to be able to decide upon a plea at the earliest possible stage. The question was - what standard should disclosable summaries be measured against: an 'ideal' standard; or a 'realistic' one? This dilemma can be illustrated simply. The disclosable summary will not contain information about the laboratory analysis of suspected drugs found on the accused. In an ideal world, such information might be vital in determining whether the accused should plead guilty or not guilty. If the accused informs his solicitor that the material found in his possession was not prohibited drugs, then the solicitor will be bound to advise the client to plead not guilty; obviously, if the laboratory analysis reveals that the substance was prohibited drugs, there is likely to be a late change of plea. In practice, however, at the time when the disclosable summary is generated from the police report, a laboratory analysis of the suspected drugs will not have been carried out, nor can it reasonably be expected that it will have been done. Very often, the disclosable summary will have to be given to the accused, and thus his solicitor, the morning after he has been arrested for his appearance from custody in court. In many other cases the period immediately before the scheduled court hearing may be the first opportunity the solicitor and client have had to discuss the case.

This gives rise to a related point, namely that the disclosable summaries were evaluated on what one might call a prima facie basis, i.e. in the abstract, as it were, without knowing what the accused might tell his defence solicitor. For instance, continuing with the illustration above, it is very unusual for the accused to claim that the 'drugs' in his possession are not in fact prescribed drugs and one cannot expect the police to anticipate every possible defence in the disclosable summary. Similarly, it is always possible that a 'drunk' driver will instruct his solicitor that he suffers from diabetes or some other medical condition which explains the apparent level of alcohol in his blood or urine. This can only be proved or disproved by a battery of medical tests and one cannot reasonably expect these to be done prior to the preparation of the disclosable summary, particularly if the accused does not reveal this defence until he consults his solicitor. In short, it is simply not possible that the disclosable summary can or could provide all the information that the defence solicitor might need; this cannot always be foretold in advance. Thus, in evaluating disclosable summaries, the standard used was the 'realistic' one: does the summary of evidence include on a prima facie basis all the information that the accused/defence solicitor could reasonably expect to have been included, given the time constraints and the possibility of the accused coming up with an unusual or unsustainable line of defence. This was judged to be a more informative approach than rating summaries against an 'ideal' standard which in practice could never be met in all cases.

In brief, therefore, the research team assessed how satisfactory the disclosable summaries were in terms of the information they might reasonably have been expected to include. Similarly, the defence solicitors who carried out the review of a small number of disclosable summaries were asked to assess them against the same standard.

Analysis of the case trajectories for the disclosable summary sample

To permit a consideration of the potential impact that the quality and content of a summary of evidence may have on case outcomes and stage of conclusion, the research requested case trajectory and outcome data on each case for which a disclosable summary had been provided.

The data supplied by COPFS provided a summary of the date, nature - e.g. PD, ID - and outcome - e.g. plea given, whether the appearance was 'final' - of each individual court appearance related to the disclosable summary cases. Using this data, the research team were able to calculate the stage at which each case was concluded - i.e. PD, ID or TD - and the number of times the case had called in court.

A single disclosable summary may refer to multiple accused. The case trajectory data, on the other hand, concerns only a single accused person. Thus the number of cases on which trajectory information was supplied was greater than the number of disclosable summaries considered (214 individuals from 192 summaries).

Analysis of reasons for the continuation of intermediate diets

Centralised management information is not available on reasons for continuation but we were fortunate that a number of reasonably large-scale local recording exercises were planned or already underway in many of the larger Sheriff courts. Whilst these exercises were focussed on local implementation and were being led by local SJR co-ordinators, they were to produce information useful to the evaluation, not least because they involved obtaining data from some of our proposed case study courts. Thus, whilst not a component of the evaluation research, the exercises would potentially produce data useful to triangulate findings from other elements of the research. The exercises were recording information for a large number of cases detailing the outcome of the case at ID along with the reason for that outcome. The range of outcomes, for example, include:

• Case moves forward to next stage
• Continuations
• Disposal
• Warrant
• Adjournment for enquiry

Where the outcome was categorised as 'continuations', the reason for the continuation was also to be recorded. Data was to be recorded on an electronic proforma by local practitioners who were attending ID courts. For the most part, the proforma being used was to be standard across all of the courts.

The research team received data of this nature referring to eight Sheriff courts and the outcomes of around 2000 cases at ID. However, in practice and due to local issues, the various recording exercises did not follow a standard procedure in each of the courts from which data were obtained. For some courts, the recording exercise was already underway when the proforma described above was being specified meaning the content of the data is slightly different. Of the eight courts from which data have been obtained, four applied the standard proforma described above. For some courts the raw data was supplied, in others a written summary of analysis of the data was provided.

Ultimately, it was possible to consider the outcomes of around 500 IDs across 4 Sheriff courts. To permit this, detailed reasons in the proformas were grouped under 6 broad headings. These headings, and the more detailed reasons which they incorporate, are detailed in Table A.2. All data was recorded during May and June 2010.

Table C.2 Grouped reasons for continuation of IDs

Attendance of accused/co-accused: Accused ill Accused not available Accused not brought from custody Accused now present Co-accused failed to appear Personal appearance of accused Failure to appear	Attendance of witness/witness citation: Civilian witness unavailable Crown witness not available Police (or police witness) unavailable Check witness citation position Check/ cite PF Witness Check/ cite DEF Witness Essential defence witness absent
Further investigation/lack of preparation: Ascertain position on other matters Further enquiries To obtain further evidence Not prepared for trial Further prep Investigate/ enquire For clarification	Legal aid/representation/change of agency: For accused to obtain legal advice / assistance Legal aid issues Change of agency No Legal Aid Lack of instructions Solicitor unavailable
Disclosure: Defence medical report not available Lack of disclosure - statements Lack of disclosure - other Lack of disclosure - video / CCTV	Other: For resolution New evidence identified To call with other cases To await the outcome of other cases Other reason Child/ Vulner Witness Interpreter Other matters calling Poss Resolution Lack of court time

National perspectives on SCLA and disclosure reform amongst defence lawyers and Procurators Fiscal

A postal survey of defence solicitors was conducted in March 2011 to collect their views on how the summary justice reforms to legal aid and disclosure have impacted on their practice and more broadly, whether there has been any perceived impact in terms of efficiency, effectiveness and fairness of the summary justice system as a whole.

A list of defence solicitors who had applied for summary criminal legal aid work between April 2009 and September 2010 was obtained from the Scottish Legal Aid Board. A sample of 700 solicitors from the 1019 on the list was randomly selected. The sample was selected to reflect the spread of cases in different sizes of court, so was therefore stratified by the size of court the solicitors worked in. A judgement was made about the definition of large, medium and small courts such that large courts dealt with 700 or more cases per month, medium courts between 300 and 699 cases per month and smaller courts less than 300 cases per month.

Monitoring data from COPFS was used to ascertain the split of cases across large, medium and small Sheriff courts across Scotland in a month. Based on this definition, the monitoring data indicted that of the 18,237 court cases that took place in Sheriff courts across Scotland in one month, 53% were dealt with in the largest courts, 31% in medium sized courts, and 15% in smaller courts. Each solicitor on the mailing list was allocated a court based on the nearest Sheriff court to their office, and subsequently, a court size (large, medium or small). The research team were therefore able to select the sample to reflect the spread of cases in different sizes of court (53% of the sample worked in large courts, 31% in medium courts and 15% in smaller courts). All 16 PDSO solicitors in the list from SLAB were selected, to ensure that views of PDSOs were invited.

An advance letter explaining the study and a paper questionnaire were sent to the 700 solicitors in the sample. After two reminders, 202 completed questionnaires were returned, giving a response rate of 29%.

Responses to the postal questionnaire were input into and analysed using SPSS statistical software.

Further in-depth telephone interviews were carried out with a sub-sample of 17 defence lawyers who responded to the postal survey to allow a more in-depth exploration of the key issues raised by the postal survey and of some of the more qualitative questions raised around plea advice and negotiation.

In addition, in-depth telephone interviews were conducted with 10 Depute PFs who were not consulted as part of the case study fieldwork, to obtain their views on how the reforms have impacted on their practice, as well as perceptions of any impact on efficiency, effectiveness and fairness of the summary justice system.

Identification of the economic impact of SCLA and disclosure reforms

A full description of the economic analysis undertaken is included in Appendix B