The Opportunities and Challenges of the Changing Public Services Landscape for the Third Sector in Scotland: A Longitudinal Study: Year Two Report

4. PERFORMANCE AND OUTCOME MEASURES

Chapter Summary

This section examines perceived trends in performance and outcomes measurement of TSOs by funders, including a move towards measuring 'soft' outcomes. In Year One, there had been limited use of the Social Return on Investment (SROI) measurement tool promoted by the Scottish Government, except by one organisation involved in its development. There had been no further take up of the SROI tool by Year Two.

A number of TSOs felt that funders had become more focused on measuring outcomes within the last year, and in particular 'soft' outcomes. As TSOs felt this was an area where they could particularly add value, this was generally welcomed.

In order to demonstrate the value they added to the client experience, some organisations provided additional evidence to funders on the impact their service had on clients over and above what was formally required.

A number of organisations continued to explore innovative ways to demonstrate client progress to clients and funders. This was a particular challenge for clients with complex issues or where literacy and numeracy was low. Also there was no standardised way of measuring outcomes. Some had looked at using existing tools while others were involved in the development of new tools.

Measuring 'soft' outcomes

4.1 In Year One many TSOs felt funders had focused on measuring 'hard' outcomes at the expense of 'soft' outcomes²⁹ . Since many TSOs worked with vulnerable or hard-to-reach clients who were some distance from being 'job-ready', the way in which many of the organisations added value was through 'soft' outcomes, such as increases in confidence and moving them along the route towards possible employment.

4.2 By Year Two, a number of TSOs felt that funders had become more focused on measuring outcomes within the last year. By this they meant what had happened to a client (e.g. getting into a job) during their time with the programme rather than outputs (such as a client undertaking a development training programme).

4.3 In particular, it was in measuring the 'soft' outcomes that some TSOs often noted the change, with funders wanting to identify the impact that TSOs had on clients:

I think a lot of funders now are much more sophisticated and much more interested in knowing the difference that you make.

Senior Manager, Local Learning Provider

4.4 This perceived trend was often welcomed because it enabled TSOs to demonstrate the value of the services they provided. One respondent thought that this trend was linked to the move towards personalisation, where there was more emphasis on what the service users would like to achieve. This translated into more emphasis on measuring the extent to which service users achieved their goals.

Providing additional evidence to funders

4.5 Some organisations provided additional evidence to funders on the impact of the service on the client over and above what was formally required. They believed that this helped to show funders what they do, the impact they were having and the value they added. This was perceived to be very important in gaining renewed funding, although it required additional investment:

We've got evidence of the impact of what we do, but you can't underestimate the time just going out and doing all of that.

Senior Manager, Local Learning Provider

4.6 TSOs who provided this evidence believed that funders welcomed this additional information because it showed the added value they were getting from the TSO.

Using additional measures to demonstrate impacts

4.7 As the Year One report noted, much of the work carried out by TSOs was often with clients who were hard-to-reach and/or who had complex issues, focused on prevention and had wider social impacts which were harder to measure. Some TSOs were continuing to look at innovative ways to demonstrate their impact both to clients and to funders. Some had, or were considering, using existing tools (e.g. Rickter scale), while others were involved in the development of new tools. A small number of TSOs had carried out research or evaluations in the past and also found these useful in obtaining further funding. However, these required resources and one TSO had looked into this route but found this kind of research evidence to be prohibitively expensive.

4.8 As the Year One report noted, funders required TSOs to report performance and outcomes in different formats and using different measures, leading to multiple measurement devices being employed. However, a couple of organisations were attempting to introduce some standardisation across their TSOs. In one case, a TSO working in the social care and health field had developed an internal measurement tool which was used across services for care planning and to measure how far clients were achieving the outcomes they wanted. This evidence-based and outcome-based tool also aimed to demonstrate some of the softer outcomes that can be hard to quantify.

4.9 Another organisation had sought to develop a common performance management framework across multiple contracts in the organisation and key performance indicators to the projects. They employed a performance and finance team to monitor all projects on a regular basis and created a red, amber, and green way of flagging up issues on any particular contracts in key performance areas so they could then intervene early and deal with them.

4.10 One TSO faced challenges in measuring and demonstrating the progress of clients with low levels of literacy and numeracy. Questionnaires in these cases were inappropriate, so they adapted existing software (Comic Life) in order to create an accessible tool. The software programme was used to create a picture book comic with captions, so project workers could work with clients to take photographs and add appropriate captions so that the client could tell a story of their experience and progress. This was perceived to create something that was more involving and more meaningful to the client.

Case Study Four: Measuring the Client Journey: Logic Modelling

This case study looks at how LEAD Scotland was involved in the development of an innovative measurement tool which aimed to measure the progress made by clients who often had complex and difficult to track journeys. The 'Logic Modelling' tool was used within the organisation in order to demonstrate the importance of the work they do with clients and map a coherent story of the client journey.

CASE STUDY FOUR

Lead Scotland

'Linking Education and Disability Scotland' (LEAD Scotland) is a specialist Scottish-based third sector organisation that seeks to widen access to learning for disabled people and carers of disabled people. LEAD Scotland provides services in eight local authority areas across Scotland working with disabled people, carers of disabled people and other socially and/or educationally excluded or isolated people.

LEAD Scotland supports disabled young people and adults and carers to access learning. Some of their clients do not progress straightforwardly on a linear path from A to B, sometimes their journey involves many stages, sometimes going backwards before going forwards, for instance, they may have to take a break for family reasons and return at a point before they were when they left. This journey and the impact of interventions at different stages is often hard to capture simply and briefly, without using a long narrative record. This is why LEAD Scotland were keen to be involved in a pilot project carried out by Learning Link Scotland and Evaluation Support Scotland (funded by the Scottish Government in 2009). The project, called "Explaining the Difference", aimed to articulate the outcomes of the Scottish voluntary adult education and learning sector. LEAD Scotland was a participant in this development process in order to develop "an accredited programme to enable practitioners in the voluntary adult learning sector to gain professional development in evaluating impacts and outcomes of their work". This programme is now available to Learning Link Scotland members and subscribers via their website³⁰ .

Out of this process emerged a Logic Modelling process. This helps to visualise and capture client journeys through identifying what happens at different stages of their journey and the interventions which they receive from LEAD Scotland. This is summed up in a one-page visual map of the client journey which is accessible to funders.

The Logic Modelling process is not a wholesale tool to be used in the same way for all projects across different TSOs. In practice, the tool needs to be tailored to each individual project because of different requirements.

We've tailored it for each individual project...I think we've used it three times with funders and it's been different each time because the need in each areas is different...so as soon as that's different the interventions have to be different, and the outcomes have to be different, and it's very much an outcomes focused tool rather than on output, so that's quite helpful because all the time we're measuring outcomes.

Officer

One of the strengths of the tool is to be able to identify engagement and preparation work done with a client before they are ready to start. Sometimes this part of the process of working with a client is not recognised or funded by funders. Therefore the Logic Modelling process is a way to demonstrate the importance of this work and map a coherent story of the client journey.

Conclusion

4.11 A number of TSOs were rising to the challenge of measuring 'soft' outcomes, particularly in relation to clients whose literacy and numeracy were limited. Some had made significant steps towards adapting and/or developing tools which would both demonstrate progress to the client themselves as well as the added value of the service to funders. While there was still no standardised way of measuring outcomes, some TSOs were making important steps towards developing tools flexible enough to be applicable across a number of different projects and funders.