Devolution of disability benefits: evaluation strategy

Framework for evaluating disability benefits

We have established that the initial changes to Disability Assistance are focussed on the way the benefit is delivered, the way that entitlement and awards are decided, and the way that benefit delivery is experienced by applicants and recipients.

Some of the impacts of the way that the benefit is delivered, such as whether benefit recipients feel they have been treated with dignity and respect, will be examined through process evaluation and the activities of Social Security Scotland in gathering information on the experiences of applicants, clients and staff.

The questions for policy evaluation, then, are:

• What are the implications of these changes for the clients themselves?
• How do we capture and assess the intended changes?
• How are they related to individual client outcomes?
• How do we identify the contribution to wider social security and government objectives?

Evaluation framework

Given that many of the changes are process orientated, overlap and cut across the individual disability benefits, we have decided to use a thematic approach to evaluation rather than a benefit-by-benefit approach (Figure 1).

It will be possible to identify lessons relevant to indvidual benefits through management information, and by categorising experiences gathered in qualitative research by the relevant benefit ^[8].

Two new policies (the Accessible Vehicles and Equipment Scheme and Child Winter Heating Assistance) will sit outside of this evaluation framework but will be evaluated as bespoke evaluation projects.

Our evaluation approach has three main strands, which will be outlined in further detail below:

1. Thematic evaluation projects

2. Monitoring management information and high-level trends

3. Learning and further research

Thematic evaluation projects

Thematic evaluation projects will be commissioned to examine the impact of key policy changes on the delivery of disability benefits and how this has been experienced by the client.

Initially there will be four core evaluation projects, looking at the following areas of significant policy change across Disability Assistance:

• Decision making
• Case Transfer
• Terminal Illness
• Eligibility for Child Disability Payment

Decision making encompasses a host of different policy changes, and so assessing the impacts of those changes, and decision making generally, will be much more complicated relative to some of the other projects. Therefore, it is likely to consist of various 'sub-evaluations' or 'sub-projects' which seek to explore the impact of particular changes to the decision-making process.

These sub-projects may explore:

• The use of supporting information in assessing entitlement
• The use of client consultations in assessing entitlement
• Decisions on award type, level and duration
• The use of light-touch reviews

Given the cross-cutting importance of decision making across the different forms of Disability Assistance we also intend to conduct a 'sub-evaluation' focussed on decision making itself which will incorporate each of the processes above as well as other aspects of the decision making process. However, it is important that research is timetabled at the most productive point. Too early will fail to take account of the experience of enough clients, across the different forms of Disability Assistance, or the relevant stages in benefit implementation. Too late will mean that the research results may not feed into improvement work or the future development of disability benefits at the appropriate point.

Case Transfer is also complex and evaluation activity will likely overlap with that of other thematic projects (such as the eligibility change for Child Disability Payment and decision making). This, as well as the ongoing nature of the case transfer process, means that evaluation activity will be phased. We will need to ensure that projects are planned and timetabled carefully to allow for relevant learning to be incorprated into the transfer process in the shorter term, but also allow for interaction with parallel projects that may have longer timescales, and for an overall picture of the full Case Transfer process to emerge.

The scope of these specific evaluation projects, as well as the methods used and timescales involved, will depend on particular circumstances, such as the aims, objectives and research questions governing the project. However, it is intended that these individual thematic evaluation projects (and sub-projects) will follow a theory-of-change approach which will allow us to identify the relationships between what we intended to achieve by the individual policy changes, and what the outcomes have been.

They may also be able to assess the likely contribution of policy changes to high-level trends, or wider social security outcomes and government outcomes. However, given that these longer-term outcomes take time to determine and are affected by a range of factors, it is often difficult to measure and attribute change to specific policy changes. That said, we can reasonably expect that if success against short- and medium-term policy outcomes has been achieved, then this could contribute (to some extent) to better outcomes in the future.

Monitoring management information and high-level trends

We will monitor management information from Social Security Scotland in order to identify patterns and trends across and within the individual benefit caseloads. This may include analysis on caseload, conditions, awards, and expenditure, and can be compared to other data sources and evidence to assess the degree to which the patterns and trends we expected to see are occurring or not. We can also look at equalities data collected in applications to explore issues raised in impact assessments completed at initial policy development phases, picking up human rights elements and issues of intersectionality where relevant. Analysis of management information will also be a key component of the individual evaluation projects outlined above.

We will also look at existing data sources, such as population-level surveys, which can track wider, long-term outcomes across society and how these are distributed depending on how people report disability.

Learning and further research

Monitoring and evaluation activity is part of the ongoing cycle of policy development.

As the evaluation work outlined above is completed, we will learn more about the way the benefit has been implemented, and what the impacts have been, and can identify shortcomings and unintended consequences. The implications of these findings will be considered by the Scottish Government and can set the groundwork for policy improvement and/or further policy development, which may necessitate further data monitoring, evidence gathering and research as appropriate.

Evaluation tools

With any evaluation activity there is a trade-off between investment in data collection (time and money) and the depth of insight (quality of findings).

It is intended that each evaluation project, and sub-project, makes use of a range of relevant methods and data sources in order to build up the picture of effectiveness. Though the exact approach of the thematic evaluation projects are still to be defined, it is expected that a combination of the following sources of data will be utilised:

• Management Information which can provide immediate information on effectiveness of delivery and client outcomes.
• Social Security Scotland research activity which will include data collected for the purposes of gathering client and staff insights and socio-economic characteristics. This includes a Client Survey of all applicants who have reached a decision, the annual People Survey of staff, and programmes of ad-hoc research projects.
• Survey data which can measure long-term outcomes across society. In themselves they are not likely to be sufficient to disentangle the overall impact of disability benefits from other factors. However, it may be that additional questions can be added to population surveys to tailor them to policy-specific questions subject to space.
• Commissioned research can fill gaps in management and survey data, deploying tools such as qualitative research, which can explore user experience and outcomes. It may be that surveys are also deployed which use representative samples of benefit clients and explore their experiences in detail allowing for comparison across groups of clients - though there are also financial costs and trade-offs to consider. Where relevant, commissioned research will also give an opportunity for clients to recount their experience of disability benefits generally, and the impact of disability benefits on their quality of life specifically.

Whether qualitative or quantitative in nature, commissioned research will allow us to engage with individuals who have a lived experience of Disability Assistance, and organisations representing disabled people or people with long-term health conditions, in order to assess the effectiveness of the social security by involving the people who access it.

In addition, the following approach will be scoped in order to assess the usefulness and viability in answering specific research questions and/or tracking the impact of disability benefits on wider outcomes, such as employment.

• Data linkage with DWP, Her Majesty's Revenue and Customs (HMRC) or NHS data could help track longer-term outcomes for clients, including comparable outcomes for those on existing benefits. There are significant issues to resolve with regard to ethics, data protection and security as well as ensuring ongoing resourcing. Permissions and agreement to share data with partner organisations would also have to be secured. However, there are potential benefits of being able to track long-term outcomes, and allow for detailed analysis by different groups.

Bespoke evaluation projects

The Accessible Vehicles and Equipment Scheme and Child Winter Heating Assistance will be evaluated as bespoke evaluation projects, because of the difference in scope comparative to the other policies included in this strategy.

These two policies are designed for a discrete purpose, and are forms of passported assistance, determined through entitlement to disability benefits. Unlike the other policies included in this framework, they are not linked directly to the delivery of the three main types of Disability Assistance, and clients will have limited interaction with Social Security Scotland when accessing these forms of assistance.

Though the scope of these evaluation projects is still be defined, it is intended that these projects also use a theory of change to identify the chain of intended impacts, and make use of the relevant evaluation tools outlined above, though they are likely to be smaller in scale.