Review of Autism Network Scotland

6 defining and Measuring outcomes

6.1 This chapter explores the issue of defining and measuring outcomes for ANS - or any other national autism network - in the context of the 'Scottish Strategy for Autism Outcomes Approach: Priorities 2015-2017' document which was developed in 2015.

Current approach to defining and measuring outcomes for Autism Network Scotland

6.2 As has been described earlier in this report the current grant award letters set objectives for ANS which are framed in terms of activities which the network is expected to carry out. (The range of activities undertaken by the network has been described in detail in Annex 1.)

6.3 No formal 'key performance indicators' or other similar metrics have been developed for the network. ANS progress reports have therefore focused to a large extent on describing the network's activities. More recently, the progress reports (in March 2015 and December 2015) have attempted to draw out impacts and outcomes more explicitly, albeit without any formally stated requirements for monitoring and / or evaluation.

6.4 ANS does invite delegates attending conferences and learning events to complete evaluation forms to capture feedback, to assess how useful participants found the events and to measure overall satisfaction. In general, these show high levels of satisfaction. For example, at the 4 ^th annual national strategy conference in December 2015, 62 delegates (out of around 200 who attended) completed an evaluation form. Around half of these found both the keynote presentations and the workshops 'very useful' whilst the other half described them as 'useful'. There was unanimous agreement that there were enough opportunities on the day to network and connect, and almost 90% of those returning forms said they had made a useful contact at the conference. Virtually no negative comments were made by those who completed an evaluation form.

Future approach to defining and measuring outcomes for Autism Network Scotland - SharePoint Management Information System

6.5 Within the University of Strathclyde - and specifically within the Centre for excellence for looked after children in Scotland ( CELCIS) - the SharePoint management information system is used to report on outcomes. This system has been developed over a substantial period and now provides a sophisticated management tool as well as mechanisms for interrogating and presenting inputs, activities, outputs and outcomes from the work of the centre.

6.6 The Project Lead for SharePoint is currently working with ANS staff to adapt the system for use within ANS. In order to maximise the usefulness of this exercise, a clear understanding of the inputs, activities, outputs and (short medium and long term) outcomes of ANS is required.

Scottish Strategy for Autism Outcomes Approach

6.7 In 2015, the Scottish Government developed its 'Scottish Strategy for Autism Outcomes Approach: Priorities 2015-2017' document. This specifies four Strategic Outcomes ( SOs) with a descriptor of what requires to be done to achieve each of the outcomes as follows:

• SO 1: 'A Healthy Life: People with autism enjoy the highest attainable standard of living, health and family life and have timely access to diagnostic assessment and integrated support services'. The action is identified as 'Improve access to integrated service provision across the multi-dimensional aspects of autism'
• SO 2: 'Choice and Control: People with autism are treated with dignity and respect and services are able to identify their needs and are responsive to meet those needs'. The action is identified as 'Consistent adoption of good practice guidance in key areas of education, health and social care across local authority areas'
• SO 3: 'Independence: People with autism are able to live independently in the community with equal access to all aspects of society. Services have the capacity and awareness to ensure that people are met with recognition and understanding'. The action is identified as 'Capacity and awareness building in mainstream services to ensure people are met with recognition and understanding of autism'
• SO 4: 'Active Citizenship: People with autism are able to participate in all aspects of community and society by successfully transitioning from school into meaningful educational or employment opportunities'. The action is identified as 'Improve access to appropriate transition planning across the lifespan'.

6.8 The current version of the Outcomes Approach document (December 2015) sets out what has been achieved, and what will be done next, in relation to each of these four outcomes. Scrutiny of this document shows that ANS has contributed to the outcomes including in the following ways:

• SO 1: by i) facilitating the rollout of the Menu of Interventions and ii) working strategically with local authorities in the development of local autism strategies and plans
• SO 2: by i) disseminating and sharing the web based Autism toolbox developed by Scottish Autism, ii) running a wide range of conferences and events aimed at sharing good practice in autism and iii) developing and facilitating networks and events for professionals (e.g. in employment, early years, speech and language therapy, and education)
• SO 3: by i) delivering a symposium in collaboration with NHS Education for Scotland to promote the NES Autism Training Framework which identifies training provision and gaps for further development ii) providing a national information hub for autism iii) developing and facilitating networks for professionals and iv) running a lead collaborative officers event on complex needs and out of area placements which was attended by 55 delegates
• SO 4: by i) facilitating the (10) 'Digging Deep: Exploring Transitions' roadshows across Scotland which were attended by 200 delegates and which focused on promoting the Principles of Good Transitions for professionals ii) encouraging employers to increase opportunities for people with autism and to promote positively evaluated models of supported employment through the autism and employment network and iii) providing opportunities for employment through working in collaboration with Project Search.

6.9 Much of the work described in paragraph 6.8 above is ongoing, and will continue to contribute to the achievement of the outcomes identified in the future.

6.10 It is clear from the analysis in paragraph 6.8 above that the activities of ANS (or of any reformulated national autism network or other organisation) underpin the achievement of all the identified outcomes. In particular, networking and sharing good practice ( paragraphs 5.4- 5.8), provision of information and resources ( paragraphs 5.9- 5.16) and raising awareness ( paragraphs 5.17- 5.18) are all necessary activities in relation to any / all of the identified outcomes.

Stakeholder views of the defining and measuring of outcomes for Autism Network Scotland

6.11 In general, stakeholders were unable to comment in relation to the defining and measuring of outcomes. There were two main reasons for this:

• First, as set out above ( paragraphs 5.68- 5.73), stakeholders were not clear about the remit of ANS and were therefore not able to comment either on what outcomes they were expected to achieve nor on how these could be measured
• Second, stakeholders in general were not familiar with the Scottish Strategy for Autism Outcomes Approach and did not therefore feel able to comment on the extent to which the activities of ANS were well aligned to this. However, one stakeholder did express the view that 'the strategic outcomes are so broad that the work of our organisation can fit under any of the outcomes'.