Connecting Scotland: phase 2 evaluation

Conclusions and Learning

Digital Confidence

Compared to phase 1 users, the majority of people receiving devices in phase 2 of Connecting Scotland had more experience of using digital technology and being online. This reflects an overall younger cohort of users in phase 2, the majority of whom live in a household with children. Most described themselves as confident internet users and a substantial number of respondents said that they had not needed to meet with a digital champion for support. Instead, the primary needs of phase 2 users were having access to: a device suitable for different tasks; enough devices for all household members to undertake tasks simultaneously; and a stable connection without limits on data.

This raises questions about the type of digital support required for a generally younger group of Connecting Scotland beneficiaries. If people say they are, on the whole, comfortable with digital technology and can set up and use devices themselves, then what should be the role of digital champions?

People might not actively seek support from digital champions, though it is unlikely that they will have no gaps in skills or knowledge. Perhaps people who feel they are broadly competent would benefit from specific workshops or learning events to enhance their existing skills, rather than considering support primarily as a response to encountering difficulties. Of course, the needs of the user should dictate their engagement with support.

Devices

The vast majority of research participants were positive about their devices and grateful for their provision, however, a small number of people had concerns (some more or less severe than others) over the capacity of their devices to perform certain tasks. Several of these expressed a preference for a laptop over what they had received.

To present, Connecting Scotland has provided users with one of two devices (iPad or Chromebook), plus connection via MiFi device if needed. Given the diversity of users' needs expected as the programme expands, and based upon some of the feedback in the evaluation of both phase 2 and phase 1, there may be scope to consider a broader selection of devices to offer. This could potentially include laptops and mobile phones.

It is also possible that some of people's dissatisfaction with their devices might be explained by insufficient engagement with digital champions, or another source of digital skills support. It may not, in every case, be that devices are inadequate, but that people need greater understanding of those devices in order to be able to do the things with them that they would like.

The balance between suitability of device, and sufficiency of user competence, will be further explored as part of Connecting Scotland's future provision.

Supporting Families

The evaluative research highlighted the importance of the internet to day to day family life, especially with the increased prevalence, since the pandemic, of working and studying from home. Though not, for the most part, entirely excluded, families experienced digital marginalisation by not having access to appropriate devices and/or did not have enough devices to be comfortably used by the whole household.

These issues are quite different to those faced by beneficiaries of phase 1 who received devices in the first national 'lockdown'; many of whom lived alone and who primarily wanted devices to maintain contact with loved ones, and in some cases, support services. Phase 2 users' needs were more practically focussed, requiring a device and connection to make sure children could keep up with schooling, to search and apply for jobs, to participate in adult education and to work from home.

Devices and connection from Connecting Scotland alleviated some of the pressure on families in these situations, as reflected in the majority of comments in the impact survey concerning 'the biggest change' since receiving a device:

-" The biggest change was that my son could complete his home work efficiently and comfortably."

-" It's made studying and accessing online classes so much easier"

-" It's helped my daughter mainly with her school work. I use it for bills, internet banking and online grocery shopping."

A few respondents in the qualitative interviews indicated that, though grateful for support, having access to more devices would be necessary to enabling their entire household to do all that they needed to online in a timely way.

It may not be feasible for Connecting Scotland to provide households with enough devices for each person. However, the fact that there are other services providing various levels of technical support may present opportunities for coordination to ensure maximum coverage.

Evaluation Approach

The second phase of Connecting Scotland was rolled out soon after the first and followed generally the same approach to evaluation. Indeed, evaluative research activities for phase 1 were ongoing when phase 2 research began. As such, many of the lessons learned from phase 2 are the same as those from the first phase (which are presented in the phase 1 evaluation) though some changes were implemented for phase 2.

• we collected postcodes (voluntarily supplied) from survey respondents to check coverage of the programme and enable analysis by SIMD (Scottish Index of Multiple Deprivation)
• we included a question in the phase 2 impact survey to enable analysis of household groups known to be at greater risk of poverty, including those identified in the latest child poverty delivery plan

One limitation to the surveys, in both phase 1 and 2, is that it has proven difficult to achieve high response rates (generally around 5% of registered users). While this does not invalidate the survey results, or our findings, we would have greater confidence in our conclusions if the sample of research participants was boosted. Lower response rates can reflect bias in that those positively engaging with the service may be more motivated to take part.

To this end, we have implemented various strategies in the phase 3 research (ongoing) with the aim of increasing participation levels. These include:

• contacting participants directly using the registration data held by SCVO, rather than solely relying on organisations to promote surveys
• sending strategically timed reminders to organisations to encourage their clients to complete surveys
• reviewing response rates and targeting areas where rates appear to be lower than elsewhere, or lower than expected

The early signs from the phase 3 welcome surveys shows that these strategies have achieved some success, though boosting participation remains a priority for the programme as planning for the next stage continues.

Part of the difficulty may have to do with the inconsistent programme recognition as mentioned earlier. A strong and visible communications strategy may be a key part of increasing participation in research. There is also scope to explore additional research methods, such as focus groups or longitudinal panels.