Self-directed Support Implementation Study 2018: report 4

3. Research findings: Evidence assessment for self-directed support

After creating the SDS Change Map, an assessment of existing evidence helped to create a revised set of research questions for future monitoring and evaluation, as well as looking at the economics/resource implications of self-directed approaches. The assessment of existing evidence (fully documented in Report 2)^[11] included:

• a literature review of other evaluations of similar programmes;
• a review of current data collections in Scotland that potentially relate to self-directed support; and
• the production of case studies to explore the various ways self-directed support is being implemented across user groups and the potential for scaling up and replicating.

Literature review^[12]

To help inform the options appraisal we undertook an ambitious, international literature review to highlight approaches that could inform evaluation of self-directed support in Scotland. Given the wide range of terms that are used to describe models for social care where personalisation, choice and control are core, a broad search strategy with high sensitivity and low specificity was developed and applied to local, national and international systems. These searches retrieved 6,120 unique records, which were analysed for relevant evaluations of existing schemes at a national level, pilot programmes, and local evaluations of national policies and programmes and local pilots.

The review sought to identify methodological approaches and limitations from these evaluations, and revealed a number of challenges in designing effective evaluation. It highlighted the additional challenges involved in economic evaluation of approaches like self-directed support due to an inability to identify suitable comparator cohorts in pilot studies or no comparator being possible where changes are system wide. Collecting costs on the time taken to actually undertake assessments or undertake processes associated with choice and control was rarely undertaken or the information was not robust when collected. Outcomes in social care – especially when personalised – can be difficult to capture, monitor and analyse at an aggregate level and as such was rarely attempted.

Only six studies were identified that attempted to address any aspect of economic impact, with four of these only looking at the costs of delivering individual budgets or direct payments with no exploration of outcomes. One study did calculate the cost per satisfied family (as judged by a survey) but this was without any form of counterfactual.

Only the sixth study could be considered a full economic evaluation of costs and benefits, with cost-effectiveness ratios calculated based upon responses to the General Health Questionnaire (GHQ) and the Adult Social Care Outcomes Toolkit (ASCOT). Whilst the methodology is not directly replicable in the current context in Scotland, as it was an evaluation of a pilot project, it highlighted how a holistic high-level outcomes tool, such as ASCOT, could be used in an evaluation of choice and control and personalised outcomes.

No evaluation attempted to provide evidence on how to evaluate or monitor a “good conversation”. However, a number of pilot evaluations did seek to use existing tools to measure high-level outcomes in social care and health– such as the ASCOT, GHQ-12 or EQ5-D, and bespoke survey questions.

A key theme arising from the literature review was the challenge in ensuring that self-directed support approaches as a means of social care delivery is working across highly diverse groups. Any ongoing monitoring or evaluation in Scotland must therefore capture the characteristics of supported people and carers to ensure that social care is working for everyone. In addition, as almost all published reports were audits or ad hoc evaluation, the review found little information on undertaking ongoing national evaluation or monitoring.

Current and proposed data collections

Data is collected on aspects of self-directed support as part of the Social Care Survey and the Health and Care Experience Survey. An initial SDS User Experience Survey has also been undertaken. As part of the research, an assessment was made of the effectiveness of using evidence from each of these collections for monitoring and evaluation against the SDS Change Map.^[13]

Social Care Survey (Scottish Government national survey)^[14]

Strengths: An established, annual census gathering information from local authorities . The survey questions include the self-directed support option chosen, gross budget and who contributes to it, and what it is spent on. The data is read-through indexed which means that it can potentially be linked to other data sources, such as health data (subject to approval).

Weaknesses: As highlighted by the case studies reports, there are differences in how self-directed support is delivered across the country so it can be difficult to compare data across Scotland. The survey does not cover the extent to which supported people have ‘good conversations’ during their assessment.

Health and Care Experience Survey (Scottish Government national survey)^[15]

Strengths: A large scale, bi-annual survey that gathers information about health and care service users’ experience of care, including their awareness of the help, care and support options available to them, whether they had a say in how their help, care or support was provided, and whether they felt treated with respect and compassion.

Weaknesses: The survey is not targeted specifically at people eligible for social care and does not ask people directly about social care. The sample is drawn from all adults in Scotland registered with a GP practice. As a blanket survey there is a chance of self-selection bias.

SDS User Experience Survey (SDSS)^[16]

Strengths: Contains quantitative data about whether supported people are having their options explained to them and qualitative data about users’ views of the conversations they have had with social care workers.

Weaknesses: This was a survey originally piloted with a sample of supported people from three local authority areas (in 2018-19 being repeated in eight authorities). In the pilot, the low response rates meant that insufficient data was collected to enable statistically significant conclusions to be made about supported people’s experiences across Scotland.

Summary of current evidence against the SDS Change Map

The Health and Care Experience Survey and the Social Care Survey currently collect limited information that directly evidence, or provide proxy data for, the SDS Change Map. In the SDS user survey, there were questions that, in their current form, would provide evidence against outcomes 5, 6, 7 and 8 in the SDS Change Map. For Outcome 9 – Outcome 17 there are no questions in any of the current surveys that could be used as proxies for impact against those changes.

Evidence from the Case studies

As part of the wider research conducted around self-directed support, thirteen case studies were produced to highlight how self-directed approaches were working in practice across a sample of six local authorities in Scotland.

The case studies were not intended to be evaluative. In the process of their design and delivery, they have provided useful context and information that have implications for any future evaluation.

Case study approach as an evaluation activity

The aim was to undertake case studies exemplifying social care, accessed under the four options, across a diverse population. This ambition was tempered slightly by the ability to identify and secure participation in the primary data-collection.

Undertaking the case studies was resource intensive and took several months to complete, which is important to be aware of when considering a case study approach as part of future research and evaluation of social care. The involvement of supported people, their families/carers, social workers and providers provided detailed insight into people’s experiences and, if this had been an evaluation, the case studies would help to understand the extent to which self-directed support is meeting its policy outcome in this small number of cases.

Case studies could be a valuable option for evaluation of specific aspects of social care, but the resource intensity required to do them thoroughly means they would be difficult to include as part of ongoing evaluation and monitoring.

Local authority engagement 

Whilst some local authorities were keen to engage with the research, in other areas this was challenging and the researchers required considerable effort to gather sufficient information. The case studies were voluntary, not designed to be evaluative and did not require the level of challenge and evidence as evaluation would. It was difficult to obtain sufficient information for some of them, largely because it was not easily accessible or in some cases not available at all. This highlights how difficult evidence gathering and engagement for evaluation may prove to be.

Variation of delivery between and within local authorities

As expected, implementation of self-directed support had not been uniform between the case study areas. A national evaluation would account for such variation, especially if the approach were based upon a robust change map. What is potentially more challenging is variation in practices within local authorities.

In some areas, this variation was because choice, control and personalisation were more readily available to some groups, such as younger adults with disabilities or parents of children with disabilities. A full evaluation would need to take account of and explore local variation in experience for different groups.

The individualisation, choice and control a supported person experiences is shaped by their social worker’s practice and how they can operate within the system.

An evaluation needs to have sufficient depth to be able to unpick issues of variance in the offer of choice and control that may both explain and be masked by aggregate statistics.

The need for review and changing circumstances – evaluation considerations

Several of the case studies highlighted how changing circumstances altered not only the support needed for individuals to achieve their outcomes, but also which outcomes they wanted to achieve, the priority of those outcomes and/or how much responsibility they wanted in terms of managing their support. Several case studies highlighted how circumstances in terms of funding available and priorities for funding can change rapidly within an authority. The changes that can take place for individuals and in authorities indicate the importance of regular monitoring, and point to the need to consider longitudinal studies with regular monitoring points with a cohort of individuals followed over time.

Sufficient supply and ensuring quality of supply – evaluation considerations

Challenges in the social care market in terms of the supply of care workers are well documented. The case studies provided further evidence of the impact of supply of the care workforce on the outcomes individuals are trying to achieve. Even where individuals had a budget to recruit a personal assistant (PA), it was not always possible to find someone suitable. It is potentially concerning that non-traditional methods of recruitment are being used and it is unclear how appropriate checks on the quality of care a PA can provide (as opposed to basic safety through criminal records checks and registration with the Scottish Social Services Council) are being undertaken. It is not clear how people employed through Option 1, or in some cases Option 2, receive ongoing training and support to ensure the quality of support they offer. The case studies reinforced the importance for evaluation to gather information on how local care markets are developing and being managed, and on how quality and stability of provision are assured.

Evaluating the economics of self-directed support

Understanding the difference in local authority resource use and costs associated with implementing self-directed support requires an understanding of the current difference in resource use compared to previous social care practices. The case studies highlighted that the additional administrative and social work resource needed is predominantly in the assessment process and in resource allocation panels. Ascertaining resource use in either panels or in the assessment process proved difficult to gather, with only one site providing information that allows the costs of both the assessment and allocation process to be estimated.

A full evaluation of the economic and resource implications of self-directed support would ideally include a comparison of current estimates of local authority resource use on assessment and allocation activities compared to previous practice. Such a comparison would require research into time spent on assessment and allocation processes before 2014 and an understanding on a case-by-case basis of how much the local authority was already engaging in practices to support choice and control and personalisation.

Furthermore, any attempt to evaluate whether self-directed support has led to health and social care system-wide costs and benefits will inevitably run into potentially intractable difficulties, not least those arising from the range of services and support that can be provided to achieve an individual’s personalised outcomes. In addition, compared, for example, to many medical interventions or narrower service interventions with less diverse target populations, it is extremely difficult to monetise social care and social care outcomes or estimate returns on investment .

The case studies therefore reinforced how challenging it will be to undertake an economic evaluation of self-directed approaches to social care.

More details of the case studies’ findings about experiences of self-directed support element are summarised in the next chapter.