Self-directed Support Implementation Study 2018: report 2

5. Recommendations for future evaluation and ongoing monitoring

Building on the findings from the literature review, the case study research and the appraisal of existing evidence collected, this chapter presents options and recommendations for modifying and adapting existing data sources for ongoing monitoring (and potentially to feed into future evaluation) as well as recommendations for future ad hoc research and evaluation(s).

Ongoing monitoring

The Health and Care Experience Survey is not currently a useful mechanism for collecting data on self-directed support. The questionnaire is largely about people’s experience with GPs. Additional questions would be required rather than re-phrasing of current questions to ensure that people were aware that they were responding in relation to social care only. As the target audience for this survey is the general public, it may be difficult to ensure that a sufficient number of those using social care would answer this survey.

Similarly, the Social Care Survey does not provide a useful mechanism to gather information on self-directed support. The question on implementation can be considered as a proxy for the percentage of supported people who have had a ‘good conversation’ but it will not be as good as the kinds of information whay was collected in the Self-directed Support User Experience Survey.

In addition, as evidenced by the case studies there is an inconsistency in the way the questions are interpreted across local authority areas and it is not a reliable indicator as to whether a conversation actually took place and was truly helpful. Who is included in the denominator may be different in different areas because of either interpretation or local need profile. There is no indication about the quality of the assessment, for example, if the merits of different choices for the individual were discussed in an open and unbiased manner. As shown by the case studies, it is not clear that a focus on option choice is in any case meaningful where the case studies showed that in some areas many people on Option 2 had very limited choice and control whilst in other areas those on Option 3 had substantial choice and control.

The Self-directed Support User Experience Survey provides a more targeted and robust opportunity to explore the same issues and to evidence the outcomes in the SDS Change Map. However, the survey has previously focused on the numbers receiving various social care supports rather than the quality of that care.

Given the availability of the Self-directed Support User Experience Survey and a willing organisation in SDS Scotland, we would recommend that the key information on the Outcome 5 to Outcome 8 be drawn from a social care user experience survey modelled along those lines – rather than adapt the Health and Care Experience Survey or the Social Care Survey. To do this, we recommend that the following questions are added to a user experience survey in the future to provide a fuller picture of change in terms of Outcome 5 to Outcome 8:

• Were there any questions you had that they couldn’t answer? (Outcome 5)
• Were you satisfied with the outcome of the meetings you had with staff? (Outcome 5)
• Was this information helpful in understanding your options and what would work best for you? (Outcome 6)
• Were there any questions that you were unable to find answers to? (Outcome 6)
• Were you aware of the availability of independent advocates to support you? (Outcome 7)

As the case studies highlighted, the circumstances of both supported people and local authorities can change rapidly and we would recommend the survey should be undertaken annually with a longitudinal element following a sample of people over several years. Analysis of the responses against individual characteristics should be undertaken to ensure that self-directed support is working for all groups.

Given the move to personalised outcomes that could potentially be hard to collate and monitor in an aggregate form over time, questions on high-level outcomes such as those found in ASCOT should be added to the user experience survey.

Options for addressing the data gaps

Currently, the key gaps for ongoing monitoring against the SDS Change Map are in the availability of information on:

• Outcome 9-Outcome 11 which require the views of workers to be captured; and,
• Outcome 12-Outcome 17 which require the views of Statutory Authorities – in particular, heads of services in local authorities.

To address the gaps for Outcome 9-Outcome 11, and Outcome 12-Outcome 17 we recommend there are several activities that need to be undertaken.

Survey of workers across social work and providers of social care 

The survey should ask questions around the quality of the conversation workers feel they can have with people (evidence for Outcome 9), how they can work with people to develop a suitable package of support (Outcome 10) and the extent to which they are able to act creatively and with autonomy (Outcome 11).

Suggested worker survey questions 

• To what extent do they feel they have clear and consistent information and time to provide appropriate support? (Outcome 9)
• How confident are they in discussing Self-directed Support with service users and in helping service users to make informed decisions? (Outcome 9)
• How adequate is the training and support they receive to implement self-directed support? (Outcome 9)
• To what extent are they able to identify people’s experience, knowledge and connections and draw on this to help each person create an appropriate package of support? (Outcome 10)
• To what extent do they have clear delegated authority in making decisions? (Outcome 11)

Reporting by all local authorities

Routine reporting by all local authorities needs to be more robust and comprehensive than is currently occurring, and should also include an assessment of:

• Their approach to strategic planning and engagement with supported people and the community

Reporting should evidence how strategic planning creates the conditions to enable choice and control at a systems level by, for example, co-ordinating support and developing and implementing appropriate needs assessments.  Evidence should also be gathered on how strategic planning has improved the quality of and choice on offer in social care, and how it improves the ease of decision-making for both workers and supported people (Outcome 12).

The extent to which – and how - communities and supported people are able to influence the planning, commissioning, procuring and administering of social care and support should also be gathered (Outcome 13).

• How they are facilitating creative solutions

Reporting should evidence the extent to which decision-making approaches facilitate creative approaches in the delivery of social care and support, both with the individual and with service providers (Outcome 14). The extent to which the contractual and audit arrangements between the commissioner, provider and supported people are flexible, proportionate and outcome-based should be described and evidenced (Outcome 15) as well as the extent to which procurement and commissioning processes help choices to be made with flexibility, personalisation and creativity (Outcome 17). From an economic perspective, the roles of social workers, senior social workers and team leaders, management and local authority finance officers in assessing individual cases is required. Information therefore should be collected on who is involved in these decisions, the process (for example, whether panels are involved) and how much time is spent on the process by those involved.

Engagement with independent and third sector providers providing support through self-directed support

We recommend that a voluntary survey circulated by a separate body – such as SDS Scotland or the Scottish Government - of all service providers commissioned by statutory services (primarily the local authorities), is required to cover several aspects of the SDS Change Map. The survey should consider the support providers can offer  to individuals and their relationship with commissioners of their services.

The individual perspective

The survey needs to evidence how providers are able to plan in a way that encourages person-centred and person-led social care support, improves the quality of and choice in the offer, improves the ease of decision-making for both workers and supported people. Barriers to choice at an individual level should also be captured, such as the ability to find staff, but any question should be open to allow providers to articulate the challenges they experience rather than leading them to state barriers that the survey writers could anticipate (Outcome 12).

The commissioner relationship

The survey should cover the extent to which providers feel they can operate in a way that facilitates creative approaches in the delivery of social care and support and evidence of where this has occurred (Outcome 14). This should cover the extent to which they feel that contractual and audit requirements are flexible, proportionate and outcome-based (Outcome 15), how far procurement and commissioning processes help choices to be made with flexibility, personalisation and creativity (Outcome 17), and the extent to which relationships between statutory, independent and third sector health and social care organisations are trusting and collaborative (Outcome 16).

Ad hoc evaluation studies

Improving national level monitoring as described above will provide a rich evidence base to understand the extent to which self-directed support is generating the changes and outcomes it was intended to achieve, and how this develops over time. However, both the case studies and the literature review highlighted that the implications and challenges of introducing choice and control are many, varied and potentially unforeseen. There are potential challenges for specific groups or areas of specific delivery – such as market management – where national monitoring may only ever be able to give a partial picture.  Ad hoc evaluation or research in specific areas as stand-alone activities (or, for example, as part of other activities such as thematic inspections by the Care Inspectorate) will be required to understand and report in more depth on specific issues in the SDS Change Map. The methodologies employed in the existing evaluations identified in the literature – surveys and qualitative interviews – should be the focus of such evaluations.

Case studies may also have a role for deep analysis of specific issues, but our experience of the difficulty in undertaking them suggests they would be very resource intensive and so therefore should only be considered if they provide information against the SDS Change Map that cannot be addressed through more routine qualitative collection such as ad hoc surveys and interviews or through the routine monitoring data collection proposed, including any longitudinal surveys.

Economic evaluation

Economic evaluation requires information on both resource use and outcomes, evidence that is currently entirely lacking in a form that could be used in evaluation or monitoring. It was challenging to locate relevant information in previous studies identified in the literature review.

Collecting data on outcomes through an annual staff survey as suggested previously will start to plug this outcomes gap in evidence. When considering the use of resources, the case studies showed – albeit in a small sample - that there were differences across authorities in the methods employed to enable supported people to identify the outcomes that are important to them and identify the support needed to meet those outcomes and also in the means to assess whether that support package would be funded. The case studies also showed that not all areas were readily able to provide resource use information so any questions asked should take into account potential difficulties or sensitivities around providing this information.  We think that these questions are best answered through asking workers simple questions about time spent on assessment, and social care partnerships asking about the processes and time involved in allocating resources – either through panels or some other mechanism.

For social workers, we suggest asking the following questions in the proposed worker survey to assist with economic evaluation:

• How long do you normally spend undertaking an assessment that involves a quality conversation?  What is the shortest and longest this can take?  Which parts of the assessment take the longest?
• How long would you spend undertaking an annual review? What is the shortest and longest this can take?  Which parts of the assessment take the longest?

Asking social workers for this information will provide direct information on the time actually involved in assessment and review.  It will also allow differences to be explored within and between authorities as well as between different types of social care (e.g. Children’s and Adults’ services).

Reporting by all local authorities needs to include information on the roles of social workers, senior social workers and team leaders, and management and local authority finance officers in assessing individual cases. Information should be collected on who is involved in these decisions, the process (eg whether panels are involved) and how much time is spent on the process by those involved.

Resource implications

The proposed additional evidence gathering has funding and other resource implications.

Continuing a slightly expanded version of the Self-directed Support User Experience Survey will require resource for SDS Scotland or another body for the survey’s dissemination, promotion, analysis and reporting.

The addition of a survey of staff who administer self-directed support in local authorities will require input from:

• local authorities through staff time to help with the distribution and to encourage staff participation; and
• an organisation to design, administer, promote, analyse and report on the survey.

There are marginal resource implications of collecting the data proposed if a social worker survey is undertaken and additional questions are added to reporting by local authorities. This should be minimal although these questions – as indeed all changes to collection proposed – should be trialled to ensure not only that they are clear and unambiguous but also that they are not overly burdensome in terms of the time they take to complete.

If a survey is conducted of independent and third sector organisations which hold service contracts with statutory services. This will require input from:

• independent and third sector organisations; and
• an organisation to design, administer, promote, analyse and report on the survey.

A reporting requirement should be introduced for local authorities to evidence and self-assess their performance against Outcomes 12-17. This will require input from:

• local authorities, through staff time to complete this assessment and buy-in from senior management to hold themselves accountable to their assessment and actions.
• An organisation to design, administer, promote, analyse and report on the survey.