Connecting Scotland: phase 1 evaluation

Findings

Headline findings from the Phase 1 impact survey

The principal aim of Connecting Scotland Phase 1 was to help those shielding, or otherwise vulnerable, to cope mentally and practically with the isolation imposed by lockdown. The results of the impact survey indicate for a large majority of respondents that this aim has been met.

89% of respondents agree or strongly agree that access to the internet has helped them to cope with being at home more due to COVID-19 restrictions. Only 1% disagreed with this statement.

The impact survey included a series of questions to understand more specifically how Connecting Scotland had helped. During the initial phases of the pandemic, concerns centred around how social isolation and the inability to access normal daily activities could lead to a widespread decline in mental health^[22]. Impact survey results indicate highly positive outcomes on each of three metrics relating to staying in touch, staying occupied, and improvements in mental health.

86% of respondents reported an improvement in their ability to stay in touch with each other

83% of respondents reported an improvement in being able to find interests

74% reported an improvement in their mental health

The impact survey also asked about how gaining devices and connectivity via Connecting Scotland supported people to be able to shield effectively whilst continuing to live their lives during lockdown. This included asking about people's ability to continue to access public health information, find jobs and access services.

Again, survey respondents felt that Connecting Scotland support had helped them in these respects; crucially in being able to access public health information in an evolving lockdown situation. 71% of respondents said that their ability to find advice and guidance on important issues was much (38%), or a little (33%) better.

Building digital confidence and skills

Another significant measure for the success of Connecting Scotland is whether it proved an effective intervention at getting people online by building their confidence and developing their skills.

In the original baseline survey for phase 1, a majority of respondents reported they could not use the internet, would need help to use it or could only use it with some difficulty. In contrast, after being involved in Connecting Scotland, 68% of respondents in the phase 1 impact survey said that they were confident or fairly confident internet users and 86% said that there digital skills had improved.

64% of respondents in the phase 1 baseline survey said they can't use the internet, would need help to use it or could only use it with some difficulty

68% of respondents in the phase 1 impact survey said that they were confident or fairly confident internet users.

86% of respondents in the phase 1 impact survey answered that their digital skills had improved since becoming involved in Connecting Scotland

Engagement with digital champions

An important part of the Connecting Scotland delivery model is the support provided by digital champions to enable people to gain the skills and confidence they need to use their iPad or Chromebook online.

48% of respondents have met with their digital champion regularly or at least a few times.

99% receiving support from their digital champion rated that support as good or very good.

The three main sources of support to get online and use devices were:

• digital champions (46%),
• family members (37%)
• learning by themselves (37%)

Summary

There is substantial evidence that Connecting Scotland has met its aims in phase 1 of enabling some of those most at risk from COVID-19 to cope both mentally and practically with the lockdown or shielding restrictions imposed on them. There is evidence of Connecting Scotland clients acquiring digital skills, being able to remain connected with support networks, and access important public health information during the pandemic, borne out by the following interview responses:

-"I'm fairly confident with what I'm doing at the moment. I've been on the iPad every day so my confidence is far more."

-"People will Facetime just to say 'Hi, How are you?'. It's been an absolute blessing. I don't know what I'd do without it."

-"I've used it for the Scottish Government. Keeping up to date with the Scottish Government and what's going on with this COVID. I do that quite a lot."

There is evidence too that the support and training provided with digital champions has been effective for those engaging with it.

"It was hard at first but my worker was brilliant. Any questions I had I gave her a call and she helped me. I've grown in confidence with it."

Some of the findings suggest further areas to explore, such as where a large minority of respondents report that their ability to use online government health services has not changed since getting support from Connecting Scotland. This may indicate areas where more focused help from Connecting Scotland could be beneficial for users, or else that there are external or situational factors that are preventing wider use of these services.

Reported Impacts

The issues affecting phase 1 end-users mean that, for the most part, digital support was seen as a means of reducing social isolation during lockdown, with several applications referring to the risk of people being cut-off from family, friends, community and sources of support.

It is important to note that end users were not given devices with the expectation that they would use them for specific purposes, and support was provided in an individualised way, responding to the needs, goals and skill levels of end users, rather than setting expectations around the use of devices.

The following outlines the primary impacts of the programme reported by phase 1 end users. Quotations are from qualitative interviews with users, unless otherwise stated.

Digital skills

A number of measures in the experience and impact survey indicate that users had developed competence in using their internet connected devices. For instance:

• Over 80% of respondents reported they were using the internet a lot (62%), or a little (25%), more than they had done before receiving equipment from Connecting Scotland
• 49% said their digital skills had improved 'a little', and 37% said they had improved 'a lot'
• 66% of respondents reported they were now 'very', or 'fairly confident' internet users
• In comparison, 64% of respondents to the phase 1 welcome survey said they either couldn't use the internet, would have difficulty doing so or would need help

Although we should be cautious in comparing these results with the findings of the welcome survey^[23], the contrast between the levels of digital skills and confidence expressed in the welcome survey and those reported in the impact survey is significant enough to suggest that users have, at the very least, engaged positively with the programme. The responses from qualitative interviews too, suggest a notable improvement in digital competence:

"I'm quite happy and quite proud. For my age I would say a 6 or 7 out of 10 and pat my own back. Before it was maybe a 3. I knew how to turn it on!"

Engagement with digital champions and developing online skills

An important part of the Connecting Scotland delivery model is the support provided by digital champions to enable people to gain the skills and confidence they need to use their devices online.

• Overall, 48% of respondents had met with their digital champion regularly or at least a few times.

A quarter of respondents (24%) told us that they met with their digital champion regularly. Another quarter (24%) said that they have met^[24] with their digital champions a few times. 15% of respondents said that they did not need to meet with their digital champion, and 14% said that they have never met with their digital champion. 8.1% responded to say that they don't know if they have met with their digital champion or not.

Of those who have had support from their digital champion, 99% rated the support as good or very good, with 1% responding that they don't know.

Three main sources of support identified by respondents were from digital champions (46%), family members (37%) and learning by themselves (37%). Of course, people may have developed their skills through a combination of each of these means, too.

The qualitative interviews with phase 1 end users provide further insight into how people have been learning to use their devices and the internet, including relationships with digital champions.

It appears that the extent of digital champion support has varied considerably among users; some having a consistent, and even close, relationship with their digital champion, others reporting that they were not familiar with the term, or the role.

In most cases, people who said they had been supported by their digital champion were satisfied with this support. Interviewees praised, in particular, the responsiveness and patience of their digital champions:

"I think she's just brilliant. I don't think you could improve on her. Because she's took time to sit and explain things to me."

Several other respondents were aware that digital champion support was available but chose, for various reasons, not to make regular contact. For the most part, this was because people had others, such as family and friends, who were able to help them, or they believed themselves capable enough not to require additional assistance:

"I set it up myself. A girl phoned me to offer training but I said I did it myself, no bother."

Where people said they were mostly 'self-taught', they had simply experimented with the device, working things out through regular use:

"I just keep trying to get what is needed from it, using trial and error without assistance."

There is some evidence that those relying on learning for themselves may be limited in the things they are able to do with their devices:

-" I know this thing can do all sorts of things I don't know, but I don't need to know the things; it doesn't concern me."

-"It is simple. I don't understand other things and don't touch it. I use just Zoom and Youtube, and sometimes read my college books and emails."

A small proportion of interviewees knew about digital champions, but seemed reticent in asking for help, saying they didn't wish to bother, or 'burden' somebody.

A significant minority (around 1/5) of interviewees seemed to have little awareness of who their digital champion was, or the kind of support that might be on offer. Consequently, some of these reported that they had not made much progress in developing their skills and confidence. One interviewee presumed that someone would go through how to get the device up and running, but commented that the person who had dropped off the device 'was distant' and explained that they weren't comfortable using their device:

"If I was confident in using it, then I would be more likely to use it."

Another user said that they weren't familiar with the concept of a digital champion but would have got in touch to ask for help had they been aware.

Amongst the sample, there was a strong preference for face-to-face learning, rather than the remote support necessitated by the COVID-19 restrictions. A number of interviewees suggested that the resumption of in-person services would provide greater opportunities for learning.

-"I would like one-to-one tuition to demonstrate how things work and to allay fears"

-"Face-to-face would be advantageous"

Finally, a small number of interviewees expressed an interest in having written materials, in a physical format, that they could refer to when learning how to use their devices:

"The tablet didn't come with a manual. I've looked for one but none was provided. It would have been helpful to have a manual. Pictures with explanations would be preferable. I would like the manual to describe how it operates, how to change password, how to access certain screens/apps. I would prefer the manual to prevent having to contact someone."

Connecting with others

Both the phase 1 impact survey, and the qualitative interviews show that users primarily used their devices for keeping in touch with others in a variety of ways, including:

• email (72%)
• video calls (51%)
• instant messaging services (44%)
• social media sites (42%)

A considerable majority of survey respondents (87%) reported that their ability to stay in touch with others was either much, or a little better since getting support from Connecting Scotland.

The phase 1 impact survey suggests keeping in touch may have positively impacted upon people's mental health, with almost 75% of people reporting that their mental health was either 'a little' (40%) or 'a lot' better (33%) since receiving support from Connecting Scotland. 89% of respondents agreed (41%) or strongly agreed (48%) that "Access to the internet has helped me to cope with being at home more due to COVID-19 restrictions".

The data collected in qualitative interviews allows us to further examine the connection between staying connected and mental health, as people told us how things had changed for them since receiving support. Several appreciated their newfound ability to stay in touch with others. In the following, a user describes a direct link between the ability to connect with others and their mental wellbeing:

"That's my lifeline. It might not sound a lot, but in my mind it makes me in my body feel better, not as tense as what I am. If that wee green button is on, I can contact anyone at the touch of a button."

Entertainment

The next most reported use of the internet, after staying connected with people, was watching videos, films and television programmes (58%) which, again, might be supposed to contribute to the positive impact on mental health which the survey reveals.

The qualitative interviews indicate that people's access to entertainment was helping to keep their minds occupied, especially during lockdown restrictions – "It stops you climbing the wall" - but also that, in some cases, the scope of material available on the internet meant people were able to enjoy a more personalised and stimulating experience:

"But for my own personal use I can type in 1964 and a song will come up. It's all I use it for, it lets me reminisce. It lets me go back into the good times."

Finding information and using services

While 71% of respondents to the impact survey rated their 'ability to find advice and guidance on important issues' as much (38%), or a little (33%) better, just under half of respondents also reported going online to seek out information, either by reading the news (45%), or looking up advice and guidance on important issues (45%). Only around a quarter said they used the internet to access public, or health, services for themselves, which might appear surprising given the potential for the internet to simplify aspects of healthcare (such as ordering repeat prescriptions, or arranging GP appointments).

The qualitative interview data show, perhaps more encouragingly, that some people are planning to expand their device and internet usage to access public services when they are more accustomed to their devices, and have further developed their skills:

"I intend to use it for this. I haven't had the device for very long and have only possessed it for nine or ten weeks."

It may also be the case that people are anxious about using the internet for things where they have to enter their personal information – respondents' main concerns were 'being a victim of fraud' (49%) and 'sharing personal data' (41%).

Similarly, half of the respondents said that they hadn't used the internet to save or make money and use of internet banking was relatively low (24%, compared to 73% in the wider population^[25]). Again, this likely reflects people's concerns about online safety, as exemplified by one interviewee:

"I'm very security conscious as there's that many scammers and hackers out there…I try to be as cautious as I can be"

Some of the survey questions asking about changes in people's abilities to access various services show high incidences of people answering 'not relevant for me' or indicating 'no change' (see table 2). For instance, 51% of people selected the 'not relevant' response when asked about applying for jobs. This is unsurprising given the demographics of the phase 1 cohort, in which nearly a third of people are retired, and a further 23% are not in work due to a long-term disability.

Perhaps more unexpectedly, when asked about their ability to access health services, a relatively high proportion of respondents reported 'no change' (29%) or 'not relevant for me' (27%), with similar results when questioned about their ability to access a service such as a social security benefit.

There could be a number of reasons for this: it may be that respondents already had a solution independent of getting online, for instance, preferring to use the phone (which was reported by a few of those taking part in qualitative interviews). In the case of accessing a public service, such as social security entitlements, it might be that this simply had not coincided with the period of support from Connecting Scotland. Also, the survey question specifies Universal Credit as an example of a public service; it may be that people not specifically claiming this chose to discount the question.

However, these ambiguous answers could be due to a factor that Connecting Scotland arguably should be helping with (such as acquiring appropriate digital skills to use a specific service). In which case, further research, with both users and digital champions, should aim to understand more about why people seem to be relatively disengaged from these activities.

Table 2 Phase 1 impact survey - How respondents ability to access information and services has improved since getting support from Connecting Scotland

Outcome	% reporting an improvement in phase 1	% reporting an improvement with 'not relevant to me' answers excluded	% reporting no change	% reporting the question is not relevant to them
Your ability to search and apply for jobs	32%	51.3%	16%	51%
Your ability to access online public services (for example a benefit like Universal Credit)	44%	67%	19%	37%
Your ability to access health services (e.g. GP appointments or prescriptions)	42%	58%	29%	27%

Continued Use

Several of those taking part in qualitative interviews acknowledged that, while their skills and confidence had improved since receiving equipment from Connecting Scotland, there remained scope for further development:

"I'm a lot better but I have a long way to go."

Many identified things that they were aiming to be able to use their devices for in the future:

"The next thing I thought I'd do would be look at the weather forecast, that's a goal for me."

This arguably demonstrates that people are recognizing the potential of the internet for them, and are invested in learning more about its uses and applications.

A small number of interviewees questioned whether the device that they received was optimal for certain kinds of uses and some said there were certain apps or programmes that were unsupported:

"If the restrictions are still here next year the online education part is restrictive with the Chromebook. It's good for basic stuff but not ideal for educational stuff…not a good hard drive, no office and not very powerful. If I was going to continue with online education I would prefer something like a cheap laptop"

Expanding the range of devices offered may be beneficial to some users as Connecting Scotland reaches more people. However, it is worth also considering whether, where people expressed dissatisfaction with a device, this was due to insufficient knowledge and understanding, rather than inadequacy of the device, per se.

While almost 30% of people responding to the impact survey said that they wouldn't be able to afford to pay for their own internet after provision from Connecting Scotland ends, nearly half of people (44%) said they did plan to pay for it themselves. It is possible that the relatively high proportion of people planning to pay for their own internet indicates increased appreciation of the benefits of online access and a desire to maintain their connection, in spite of the costs involved.

As with any survey, these responses are a snap shot of people's thoughts and intentions at the time; it should be noted that the impact survey was administered before recent dramatic inflation in prices of goods and services. Given that beneficiaries of the programme are on low incomes, it is possible that people have since reappraised the affordability of maintaining an internet connection.

Analysis of the Telephone Top-up Survey

The demographic profile of top-up survey respondents was similar to that of those responding to the online survey, with the distribution of age, ethnicity and gender broadly consistent.

Fewer people in the top-up survey reported being unemployed which is reflected in greater numbers of retirees (38% compared to 29%) and people in employment (19% to 6%). The proportions of respondents not working due to a long term condition is roughly the same for both surveys.

The similarity in the demographic profile of respondents indicates that any differences in responses are more likely to be due to other factors – such as less interaction with a digital champion.

We asked the top-up survey respondents why they hadn't completed the online survey. Almost all (97%, n=55) responded that this was because 'no one had told them about the survey'. Many more people in the top-up survey had not met with their digital champions (66.5% of top-up survey respondents saying they had never met with their digital champion, compared to 14% in the main sample). These results indicate that the sampling strategy for the top-up survey was reaching a group of people who did not get the opportunity to complete the main online survey. Either because they would not have received the link to the survey directly having no e-mail address, or would not have been supported to complete it by their digital champion as they had no ongoing contact with them.

In other words, the top up sample addressed some of the types of bias we anticipated may be present in the group responding to main online survey. A last point of difference between the responses to the top-up survey and the main sample is that the top-up sample reported having their devices longer than those in the main sample, which can be accounted for by the fact that top-up survey was implemented after the main survey had closed.

On the main metrics for phase 1, the results from the top-up survey were often similar to those of the main online survey, providing a degree of reassurance that the findings from the main sample are valid. In particular, the high level of agreement that support from Connecting Scotland has helped respondents to cope more with being at home during the pandemic (89% main survey / 87% top-up survey agreed that it helped).

However, the postive ratings in the top-up survey are systematically lower than those in the main survey, indicating that the main survey may be slightly biased towards reporting more postive outcomes, although not to the extent that would invalidate the overall conclusion of this report. The table at annex B shows the variation in responses between the online and telephone survey, for the main metrics.

Because of the lower number of participants in the top-up survey there will be larger confidence intervals around these results than for the main survey sample, which may go some way to explaining discrepencies between the findings from each.