Connecting Scotland: phase 1 evaluation

Conclusions

Positive Engagement with the Programme

Taken together, the data from the phase 1 impact and experience survey, and the responses in qualitative interviews, indicate that users have engaged with, and derived significant positive benefit from participating in the Connecting Scotland programme. Almost without exception, people are regularly using their devices and the internet (60% more than once a day), are improving their skills, and appear to be appreciating the opportunity given them:

"We are very pleased to have the device as it has opened a lot of doors for us and has changed our lives."

Asked, in the impact survey, to sum up their experience of the programme in just 3 words, people's responses were overwhelmingly positive:

Device and internet use

Phase 1 users have, so far, principally used their devices and connection for keeping in touch with others and entertainment; these have been valuable resources throuhgout the pandemic and lockdown, reducing isolation and helping to maintain wellbeing.

Overall, the impact survey suggests limitations in the wider use of the internet, however interview data indicate ambition to learn more in the case of several respondents. Some respondents acknowledged their relatively limited use but seemed content with the purposes for which they had been using their device.

Nevertheless, there are indications of scope for further learning and engagement which are considered in the final section of this report.

Limitations

The data collected for phase 1, through surveys and from interviews, presents a consistent picture of how people are using their devices, their levels of confidence and the benefits deriving from being connected. However, it ought to be noted that the sample of participants included in the research might reflect biases resulting from the means used to engage participants.

• Because surveys are conducted online, those more confident using online and digital technologies are likely to be overrepresented in the sample
• To promote responsiveness, digital champions were asked to highlight research activities and help people return surveys where necessary. It is therefore likely that users who enjoy a positive relationship with their digital champion are overrepresented

The telephone top-up survey is one mitigatory measure to balance the sample of participants; other channels will also be considered for future research activities.

Moving Forward

This is an early stage report written to provide a snapshot of the findings that have emerged from the research and evaluation activities for phase 1 of Connecting Scotland to date. Moving forward, the insights from this report will be developed in the following ways:

• Develop the analysis to show how the findings align with Scottish Government priority policy areas, for example, relating to poverty and families, health, employment and education
• Develop the analysis of the phase 2 welcome survey and integrate evidence from the phase 2 interviews and the phase 2 impact survey when in completes in April 2022.
• Undertake a more detailed analysis of the survey data, e.g. breaking it down by age group, region and gender, to gain additional insights and linking with mobile usage data.

Analysis and Evaluation

We are learning lessons from the research and evaluation activities in phase 1 (and currently in phase 2) and constantly developing our research approach to get closer to demonstrating the impact of Connecting Scotland on users. Lessons learned include:

• the need for finer grained detail on who the users are so that we can make comparisons between surveys more directly and have greater confidence that we are measuring real changes
• to have customer data gathered in a timely way when citizens join Connecting Scotland, so that we can more effectively enroll participants into research activities
• the importance of being able to link together datasets to allow more detailed analysis so that we can triangulate between data sources and draw more robust conclusions
• finding ways to boost survey sample sizes where possible
• the need to use more than one channel to approach participants so that we include people who will not be in a position to respond to a survey delivered digitally
• the need to focus the evaluation around measurable outcomes that can be linked to benefits and enumerated

Accordingly we have taken the following actions:

• implemented the collection of postcodes from phase 2 and phase 3 users to allow more fine-grained comparisons
• altered the process for awarding devices so that data is collected prior to devices being released
• brought together all of our data sources onto the a single platform to be able to link them together and perform more sophisticated analysis (e.g. breaking down survey responses by gender)
• we are in the process of recruiting a data engineer to manage Connecting Scotland data and create reports
• we are seeking to get further statistical resource for Connecting Scotland analysis
• we are supplementing our digital survey with a telephone survey to ensure a balanced sample
• we have attempted to boost sample sizes through different strategies, included strategically timed reminder emails to organizations, and texting individual beneficiaries directly, and will continue exploring ways to facilitate research participation
• we have focused our evaluation for phase 3 on capturing the progress made towards employment during the time supported by Connecting Scotland, to be able to demonstrate impact
• we are exploring use of 'Social Return On Investment' as a way of demonstrating the financial benefit to the individual and to society of outcomes such as those delivered by Connecting Scotland. E.g. by being enabled to access health information online

Programme Delivery

Several phase 1 respondents reported issues with having a limited data allowance; running out of data and not being able to check usage. This has been addressed and, from phase 2 onwards, ends-users are given an unlimited data allowance for 2 years.

A small number of interview respondents said they felt that, in certain respects, the devices issued might not be the most appropriate for some uses. For the further roll-out of Connecting Scotland, as the range of users broadens, it may be worth considerating whether different kinds of devices can be made available and whether users might have greater input into what they receive.

Several interviewees indicated they would be interested in attending face-to-face training for using their devices and the internet. Connecting Scotland could work with organisations, particulalry those applying on behalf of clients, to determine if and how such resource could be coordinated.

While mostly positive, people's experience of digital champion support varied; from being unaware of support, to having strong relationships with advisers. The programme could consider ways to ensure consistency of the support offer, either through training or resourcing, so that all users can realise the full potential of devices and connectivity.

A number of respondents, both to surveys and in interviews, indicated that they had concerns about sharing personal data and financial information, which could be a barrier to engaging in some online activities. Connecting Scotland could consider ways to provide training and information specifically about managing such issues to increase user confidence.