Disability and Carers Benefits Expert Advisory Group - disability evaluation strategy: advice

Disability assistance evaluation strategy

We were grateful for the request made by your officials to provide you with advice on the draft evaluation strategy for disability assistance. 

We welcome the commitment by the Scottish Government to ensure safe and secure delivery of the newly devolved benefits, and to improve the application process for all clients. We understand that our advice may incur risks, implications and challenges for the Scottish Government. These will be made explicit in this advice note and we aim, where possible, to signpost a proposed solution. 

We have worked with officials to ensure that our advice is informed by information available at this time and relevant to the current policy landscape. We met with officials on 11 May 2021 to discuss this issue. We would like to especially thank the officials who worked with us on this policy area. Their input was clear, concise, and greatly aided us in focusing this advice. 

Clearly, information and the current policy landscape may change in ways that cannot be foreseen at this time, so the advice we give now is with the caveat that this too may change in light of developments. We set out below some issues arising from our discussions which we wish to draw to your attention. Our recommendations are summarised at the end of this letter.

Background

Scottish Government officials leading on developing the evaluation strategy for disability assistance provided us with a draft of the strategy document and a comprehensive overview at our meeting on 11 May 2021. The presentation covered:

• benefits in and out of scope of the evaluation
• role of policy impact evaluation, noting some impacts of the way the benefit is delivered will be examined through process evaluation and the activities of Social Security Scotland
• the five proposed key thematic evaluation projects within the evaluation framework (decision making, case transfer, ‘rising 16s’, terminal illness, and the Accessible Vehicles and Equipment Scheme)
• process for monitoring management information and high level trends, learning and further research
• evaluation tools and
• timescales for the evaluation

The Group welcome the opportunity to set out what they hope to see reflected in a mixed method approach to understanding the experiences of those in receipt of disability assistance and to consider how this evaluation sits alongside consideration of the planned review of Adult Disability Payment and longer term plans to improve disability assistance following the completion of safe and secure case transfer. Our discussions have highlighted a number of areas for further consideration that we hope will help officials in finalising the strategy.

Purpose of the evaluation and scope

Evaluation itself is a key commitment within the Scottish Government’s social security charter. In discussing the overall purpose and scope of the disability assistance evaluation, the Group notes the importance of ensuring the key aim is to evaluate the purpose, adequacy, eligibility and format of each disability benefits. The Group believes that evaluating the ways in which the forms of Scottish assistance differ from the existing benefits they replace should be an important aspect of the strategy, but not the main focus of the strategy as the draft and presentation from officials suggested.  

Recommendation 1: Set out more clearly within the disability assistance evaluation strategy document the stated purpose(s) of each form of assistance under evaluation and the ways in which the strategy will evaluate the degree to which the purpose(s) are being achieved. If this is not to be evaluated as part of this strategy, it would be helpful to include information on how this will be evaluated and how any relevant findings from this evaluation will inform that wider evaluation.

While acknowledging the point made in the strategy document that the disability benefits are more complex than the benefits delivered in ‘Wave 1’, the Group feels the logic model for the first wave of social security assistance may be a helpful example to follow in this regard. They also believe that learning from evaluations of Scottish Child Payment in terms of large numbers claiming a 4 weekly payment should be taken into account.

Recommendation 2: Consider where learning from the evaluation of Scottish Child Payment could be taken into account, given the similarities of volume and frequency of payment.

If the intention is to have a narrow focus on policy divergence for this particular piece of work, then the Group suggest that more information is provided on the wider plans to evaluate the effectiveness of disability assistance. The Group would also welcome reassurance that any findings from the proposed evaluation that may inform understanding of the effectiveness of the purpose, eligibility and format of the disability benefits are effectively captured.

The Group also identified the need for evaluation of disability benefits to map across to the achievement of the eight legislative social security principles, including how far the duty on Ministers to boost benefit take-up among those eligible is being achieved and how far Disability Assistance is contributing to poverty reduction. Since payment rates are not changing in the short-term and we know that disabled people have a higher poverty rate than the population as a whole, higher take-up is a possible pathway to reduced poverty for those not currently claiming. This seems to be a significant issue to explore through the evaluation strategy.     

The Group, therefore, feels it is important that this disability assistance evaluation strategy includes specific evaluation of take up and poverty reduction for each of the disability benefits within scope - or make clearer the connections that will be made with any separate evaluation of take up and poverty reduction. For example, it would be interesting to ensure this strategy or a combination of strategies allowed comparison of the take-up of Scottish forms of assistance to the UK equivalent benefits to see if any initial improvement in take-up was maintained or eroded over several years and the impact that had on achieving overall outcomes.

Recommendation 3: Include evaluation of take up and poverty reduction for each of the disability benefits within scope of the disability evaluation strategy, or make clearer the connections that will be made with any separate evaluation of take up and poverty reduction.

A further issue that the Group feels may be missing from the scope at present is an evaluation of the efficacy of signposting and provision of advocacy support. This is an important and new part of the social security legislation and is meant to enable and support disabled people in making applications and will be crucial for those with any sort of communication impairment. The Group would welcome further clarification on how this will be evaluated within the disability evaluation strategy. 

Recommendation 4: Include evaluation of the efficacy of signposting and provision of advocacy support for each of the disability benefits within scope of the disability evaluation strategy, or make clearer the connections that will be made with any separate evaluation of the efficacy of signposting and provision of advocacy support.

The strategy document notes that some of the impacts of the way that the benefit is delivered (such as whether benefit recipients feel they have been treated with dignity and respect) will be examined through process evaluation and the activities of Social Security Scotland in gathering information on the experiences of applicants, clients and staff. However, the Group believes it would be good to see client wellbeing as a ‘key consideration’ within the thematic evaluation projects of the disability assistance evaluation. It was acknowledged that this is in some ways implicit within the strategy document, but it would be helpful to have it explicitly stated. The Group would also welcome further information on how supporting wellbeing and wellbeing outcomes will be measured and how the separate evaluation of Social Security Scotland’s activities will be used to inform the evaluation of disability assistance.

Recommendation 5: Set out more clearly within the disability evaluation strategy document how client wellbeing will be considered, specifically within each of the thematic projects.

Taking a human rights based approach

One of the Scottish Government’s key social security principles is acknowledging that social security is itself a human right and essential to the realisation of other human rights. The strategy document notes this point and that the Scottish Government is committed to ensuring that those who interact with the social security system are treated with fairness, dignity and respect. With this context in mind, the Group consider it would be helpful to provide greater clarity within the strategy document that it is seeking to take a rights based approach to the evaluation itself. This could be in terms of both the approach to the evaluation and to what is being monitored. It could include reference to specific rights of relevance or the rights based approach within the social security system following availability, accessibility, acceptability and quality, and the PANEL principles.

Recommendation 6: Provide greater clarity within the disability assistance evaluation strategy document that it is seeking to take a rights based approach and how this will be achieved.

The strategy document further notes the Scottish Government’s intention to develop "a person-centred approach" to social security, which treats clients as individuals, taking into account their personal circumstances and needs when making decisions on entitlement and awards. The Group feels pulling this ‘person-centred’ approach through into the approach to the evaluation is key. This could be achieved by ensuring there is space for personal testimony on how people feel about whether the purpose(s) of the disability assistance they are receiving is being met. 

Recommendation 7: Consider specific focus on the impact of the policy divergences having been co-designed with experts including those with lived experience.

The Group also believes that while the document stresses that policy changes to date have focused on the way that the entitlement and awards are assessed, it will also be important to evaluate the impact of each of these elements having been co-designed with experts by experience as a distinctive feature. The Group would welcome further information on the ways in which lived experience and direct feedback from disabled people will inform the evaluation of disability assistance.  

Recommendation 8: Ensure there is space within the disability assistance evaluation strategy for personal testimony from disabled people.

Evaluation of general decision making

The Group welcomes the proposal to have general decision making as one of the thematic projects and noted that it would be important to ensure there is enough scope to explore experiences across equality groups. This would require having a big enough sample size, complemented by qualitative ‘deep dives’ into diverse groups of disabled people. The Group feels it would be difficult to gauge the contribution of Disability Assistance to the social security human rights principles and the extent to which human rights have been upheld in practice without this. The Group suggests considering starting the ‘general decision making’ project sooner and stretching for longer across this timescale in order to better address these points.

Recommendation 9: Consider starting the evaluation of general decision making earlier in the overall and extending the timescale for this project in order to ensure there is enough scope to explore experiences across equality groups and the extent to which human rights have been upheld in practice.

Further learning and development

The Group feels the disability assistance evaluation presents an opportunity to further explore potential academic collaborations. The Group would welcome more information on the Scottish Government’s plans to engage further with academic institutions in order to get the most from shared areas of interest.

Recommendation 10: The Group would welcome further information on the Scottish Government’s intentions to use the disability assistance evaluation to explore opportunities for collaboration with academic institutions.

Finally, the Group notes that from the timeline set out in the strategy document, it looks likely that the planned review of Adult Disability Payment will begin prior to evaluation of key work streams within the disability evaluation strategy such as ‘general decision making’. The Group would welcome further information on the intended timescales for the review of Adult Disability Payment and how that review will take on board the conclusion of the disability assistance evaluation.

Recommendation 11: The Group would welcome further information on the intended timescales for the review of Adult Disability Payment and how that review will take on board the conclusion of the disability assistance evaluation.

Conclusion 

We support the Scottish Government’s overall approach to evaluation of the social security system to date and the commitment that this is done in line with the Scottish Government’s social security principles – as well as helping to measure the extent to which these principles are being met. We hope our recommendations on areas for further consideration for the proposed disability evaluation strategy will help achieve these aims.

Recommendations

Please see below our key recommendations.

Recommendation 1: Set out more clearly within the disability assistance evaluation strategy document the stated purpose(s) of each form of assistance under evaluation and the ways in which the strategy will evaluate the degree to which the purpose(s) are being achieved. If this is not to be evaluated as part of this strategy, it would be helpful to include information on how this will be evaluated and how any relevant findings from this evaluation will inform that wider evaluation.

Recommendation 2: Consider where learning from the evaluation of Scottish Child Payment could be taken into account, given the similarities of volume and frequency of payment.

Recommendation 3: Include evaluation of take up and poverty reduction for each of the disability benefits within scope of the disability evaluation strategy, or make clearer the connections that will be made with any separate evaluation of take up and poverty reduction.

Recommendation 4: Include evaluation of the efficacy of signposting and provision of advocacy support for each of the disability benefits within scope of the disability evaluation strategy, or make clearer the connections that will be made with any separate evaluation of the efficacy of signposting and provision of advocacy support.

Recommendation 5: Set out more clearly within the disability evaluation strategy document how client wellbeing will be considered, specifically within each of the thematic projects. 

Recommendation 6: Provide greater clarity within the disability assistance evaluation strategy document that it is seeking to take a rights based approach and how this will be achieved.

Recommendation 7: Consider specific focus on the impact of the policy divergences having been co-designed with experts including those with lived experience.

Recommendation 8: Ensure there is space within the disability assistance evaluation strategy for personal testimony from disabled people.

Recommendation 9: Consider starting the evaluation of general decision making earlier in the overall and extending the timescale for this project in order to ensure there is enough scope to explore experiences across equality groups and the extent to which human rights have been upheld in practice.

Recommendation 10: The Group would welcome further information on the Scottish Government’s intentions to use the disability assistance evaluation to explore opportunities for collaboration with academic institutions.

Recommendation 11: The Group would welcome further information on the intended timescales for the review of Adult Disability Payment and how that review will take on board the conclusion of the disability assistance evaluation.

I hope this is helpful. I look forward to your response and we would be pleased to discuss this further with officials. 

With best wishes,

Dr. Jim McCormick
Chair