National autism post-diagnostic support service - pilot programme: evaluation

Consolidated report

Introduction and overview

1. Preliminary discussions took place in September and October 2020. This project started in November 2020, in preparation for the first intake in December 2020. It was proposed that evaluation data would be generated and provided to AT-Autism throughout 2021. The project has generally proceeded in accordance with the planned outline. AT-Autism was asked to provide preliminary findings September 2021. This report is therefore earlier than planned.

2. Initial scoping meetings were held with Scottish Autism and Scottish Government to devise a formal structured evaluation of all NPDS programmes, which would demonstrate the degree of efficacy of the various programmes, collectively and individually, in the context of public accountability.

3. Several follow- on meetings involved the whole Service Provider (SP) group. to discuss the rationale and approach to evaluation and to clarify format and processes.

4. It was agreed that this evaluation would use a hypothesis-testing model that drew on Participatory Action Research (PAR) methods. This would reflect the NIHR PICO framework and would generate both quantitative and qualitative data.

5. Data would be obtained primarily from users of services, post intervention via a confidential and anonymous survey. The opportunity to withdraw from the programme or from the evaluation was agreed. All SP organisations agreed to provide information and data to ScotGov and to AT-Autism as required, and to emphasise to the user the importance and value of participation in the independent evaluation.

6. Several meetings of the whole group including all SP's were dedicated to designing, defining, refining and simplifying the processes to be used, which would require the minimum of effort to participant SP organisations. SP orgs were also advised not to undertake their own evaluation of the programme during this pilot as this would detract from, and could undermine, the independent evaluation.

7. A DELPHI process was used to develop and refine questions for the survey from the SP group, with ScotGov and Scottish Autism acting as core reference group. Questions were circulated to the SP group as part of ongoing consultation.

8. Based on this and the returns from group members relating to a summary description of their services and the PICO framework, questions were developed to be applicable to all services. The final questions for the evaluation were then agreed and signed off by the group, and in turn, by ScotGov.

9. These questions were turned into a platform using SNAP software, which is held by AT-Autism. In May 2021 one SP belatedly asked for changes to the survey platform. This request was denied as it would have reversed the agreed SP group and ScotGov position obtained via the DELPHI, but amendments to the introduction were accommodated. It would also have been disruptive at a technical level at this late stage.

10. It was agreed by the SP group to provide information to AT-Autism for all users registered post December 2020 and to keep records of attendance, goals, and completions.

11. It was initially agreed that contact emails for users to complete the evaluation would be provided to AT-Autism by all SP organisations once the user had completed the programme. At the request of the SP group this was amended, and all SP members would now send these details (including the link to the evaluation survey) to users. This to ensure all users are contacted and to maximise uptake. The survey would be confidential and anonymous.

12. SP members also agreed to maintain and safeguard a record of location and race, gender, and ethnicity. These would remain confidential to the service but disclosed to ScotGov as required or requested.

13. AT-Autism opened the survey in February 2021 and received 149 responses concerning all organisations. All health Regions are represented except Orkney. The responses comprised quantitative and qualitative data, and comments from respondents, many of which were extensive.

14. SP organisations were asked for

a. Number of users registered post 1 December 2020.

b. Number of users completed the programme by end of July 2021.

c. Number of evaluation links sent out.

Organisation	Total	Completed	Links sent	Responses
AI	139	73	62	33
ARGH	26	NK	16	5
AMASE	89	NK	36	13
NAS	144	121	120	24
SA	115	30	29	25
SWAN	224	224	NK	30
TA-OSS	229	20	72	19
Total				149

15. This report contains responses in respect of all organisations and are extensive in scope and range. The quality of responses from participants is high, with a wealth of lived experience captured within them. They are produced in full, without editing or redaction.

16. The responses support the perceived value of the range of post-diagnostic support services across Scotland in the eyes of the users of these services.

17. The data also suggest areas where individually and collectively, providers can address issues of importance to users or where performance is below the standard expected.

18. The true value of the report will be as a mechanism for reflection and action on the findings presented.

19. A composite report collates data from across all respondents plus additional reports for each of the service providers.

20. The full set of reports plus individual reports are the property of the Scottish Government.

21. It is proposed to make the composite report, plus their own report available to the individual service providers.

Preliminary conclusions

1. The responses to the survey indicate that for most respondents, the aims and outcomes set out by services providers are substantially achieved, and in many instances exceeded.

2. The services reviewed offer a valuable alternative and complementary addition to statutory services.

3. These limited data offer a valuable insight into how these services are experienced by users, and a steer on future direction. They also offer a baseline, a benchmark for comparison, and a mechanism for reflection and change.

4. I propose they also serve as a basis for discussion on future direction of these services. We are prepared to participate and assist in this.

Richard Mills

AT-Autism

26 August 2021