Adapting for Change: evaluation

Chapter 1: Introduction

This report presents the findings of an independent process evaluation of the Adapting for Change ( AfC) initiative commissioned by the Scottish Government. The AfC initiative stemmed from the work of an Adaptations Working Group ( AWG) which was established by the Scottish Government and tasked with reviewing housing adaptations practice and providing recommendations for change.

The AWG's 'Adapting for Change' report was published in November 2012 ^[1] and recommended that fundamental changes were needed to the delivery and funding arrangements for housing adaptations. The report's overall recommendations focused on: strategic leadership sitting with the local housing authority; services being centred around the individual; services taking a broader, outcomes-focused approach; and partnership governance arrangements. Further information on the AWG's recommendations is set out in Chapter 2.

The AWG proposed that their recommendations be tested as pilots, within several sites, prior to large-scale roll-out. Through five test sites, the AfC initiative has been developing and testing ways of improving the housing adaptations process. The Initiative ran from the Autumn of 2014 and came to a formal conclusion at the end of March 2017.

The test sites (Aberdeen, Falkirk, Fife, Lochaber and Scottish Borders) received no funding from the Scottish Government for being a test site and staff taking the work forward have been doing so as part of their existing roles. The test sites have however, received a package of consultancy-type support through the Improvement Hub (ihub) ^[2] .

Study objectives

There was a clear Ministerial commitment to undertake an independent evaluation, and to use the findings from the study to inform the development of new guidance and further stakeholder consultation. Given these timescales, the focus of the evaluation has been on process and the study has taken place during the latter stages of the AfC test period.

A process evaluation focuses on assessing how a programme or initiative is being implemented. The overall aim of this process evaluation was to assess whether, and how, the activities at the five AfC test sites have led to fundamental changes in the delivery and funding arrangements of housing adaptations. The specific objectives were to:

• Understand the 'real-life' implementation and delivery of the AWG recommendations, as taken forward by the five sites.
• Explore the extent to which the different approaches adopted achieved the overall recommendations (or the extent to which it is possible to assess this).
• Assess how implementation compares to the approach outlined in the Working Group recommendations.
• Assess the role of different factors which helped or hindered implementation and delivery.
• Deliver policy and practice focused outputs.

Evaluation approach

In response to challenges encountered, the approach taken to the evaluation evolved over the study period. In summary, the approach has involved:

• Desk-based analysis and document review; this included a review of any performance or other related data available, minutes and other information associated with the Project Boards for each area and materials developed by or through the AfC initiative.
• Interviews with key stakeholders; a series of in-depth, semi-structured interviews was conducted with a range of key stakeholders from across the test sites. The substantial majority of these key stakeholders were members of a Project Board ^[3] within a test site. The remaining interviews included senior management or frontline staff with an interest in AfC but who were not Project Board members. Representatives of the ihub also took part. In total, 56 interviews were carried out, with the number of interviews per test site ranging from 9 to 14.
• Interviews with current or former users of adaptations services; these interviews were also semi-structured. The research team worked with four of the test site areas to identify possible interviewees. The focus was on speaking to people with a range of different housing circumstances, for example those living in the owner occupied or social rented sector. Within each of these groups, the test sites then looked to identify people needing a more complex and/or urgent adaptation through to those needing a smaller and/or less urgent adaptation. In total, 59 interviews were carried out across four of the test sites. The number of interviews per test site ranged from 9 to 20. Of those interviewed, 19 lived in a local authority property, 12 lived in a housing association property and 3 were renting in the private sector. The remaining 25 interviewees lived in the owner-occupied sector. The interviewees included those whose cases had been very complex and involved major works through to those who had required only a small and straightforward adaptation.
• Observation; members of the research team attended a learning event for test sites organised by the ihub and one member of the team attended a Project Board meeting for one of the test sites.

Research challenges

This has proved to be a challenging evaluation to conduct for a range of reasons, some of which were anticipated in advance but also because of others which arose during the study period. Key points to note are:

• Different key stakeholders have sometimes had contrasting views on the current state of play and what has been undertaken within their test site as part of the AfC initiative. This includes because work to make changes was sometimes underway before becoming a test site. Also, there may be a range of other work, such as that to address delayed discharge for example, which is very closely connected to, but not necessarily part of the immediate remit of, AfC.
• Due to a range of unforeseen challenges, the programme of work is sometimes behind the original schedule. On occasion, this has meant that changes expected to be made during the AfC initiative are still underway.
• Both past and present approaches to recording information within the tests sites mean there is generally an absence of accessible, performance-related data and information. This has been a challenge for both the evaluation and the process of change and is discussed further at Chapter 6.
• It has proved difficult to engage a small number of key stakeholders with the research process, including members of Project Boards.
• A number of key stakeholders offered considerable assistance to the study team in recruiting people to take part in the current or former service user interviews ^[4] . However, because many of the changes are still in the process of being developed or rolled-out, current or former service users generally have experience of the service prior to the changes being fully implemented.
• It was clear from the outset that the evaluation could touch on a range of sensitive and difficult issues. This applied not only to service users but also to key stakeholders and, in particular, to the working relationships between key stakeholders. Both the Scottish Government and the research team were clear that this study should not undermine the work being carried out in the test sites, but that it was important for participants to feel able to speak openly. To this end, research participants were offered a very high level of confidentiality, including that neither they nor their service would be identified within the report, and that no information beyond that set out in this report would be shared outwith the study team.

Over and above these specific challenges, it should be borne in mind that the AfC initiative and, by extension this evaluation, is part of a wider drive to look at how housing adaptations are being delivered. Within the test sites, the AfC initiative may have defined start and end dates, but this does not and should not equate to the start and end of the process of looking at adaptations locally. A small number of key stakeholders within each of the test sites had already been looking at adaptations; these key stakeholders were generally then involved in applying to become an AfC test site. They included housing, social care and third sector stakeholders. In the fifth test site, Scottish Borders, much of the work was already underway before becoming a test site ^[5] so here, in many respects, the period of the AfC initiative has represented more of a bedding-in phase. Given the process-focus of this study, this does mean that the study findings are generally more likely to apply to one or more of the other four test sites.

Structure of the report

The next chapter of the report sets out the context for the AfC initiative, and in particular, considers the role of the AWG in framing the issues to be explored through the AfC initiative.

The middle section of the report ( Chapters 3 -7), is structured around the main recommendations made by the AWG in their Adapting for Change report (and as set out in Chapter 2). At the beginning of each chapter, the original AWG recommendation and brief further information is set out within a text box. Each chapter then presents the main research evidence from this study. Please note that current and former clients of adaptations services are referred to as service users. The professionals who contributed to the research are referred to as key stakeholders. If there was a clear pattern in terms of the sector within which key stakeholders worked and the issues raised this has been set out. However, it should be noted that the focus of the evaluation is qualitative and on summarising the range of views expressed. The latter part of each of Chapters 3 to 7 presents summary conclusions.

The final chapter sets out a brief overview of the work taken forward in the latter stages of the AfC initiative and the future plans for each of the test sites. Final summary conclusions from the study are also set out.

Owing to the nature of the report, no Executive Summary is provided, instead a research findings paper focusing on key learning points, is available as a separate document http://www.gov.scot/adapting-for-change-rf.