Scottish 4G Infill Programme Evaluation

Executive Summary

Introduction

This report sets out an evaluation of the Scottish 4G Infill (S4GI) Programme. S4GI is a £28.75 million funded programme which was developed by Scottish Government and the Scottish Futures Trust (SFT) which aims to deliver 4G mobile infrastructure and services in locations where they would not otherwise be provided by the market. The investment objectives established for S4GI are as follows:

1) Reduce the Digital Divide by improving available levels of connectivity for people and communities in areas of deprivation, and for people and businesses in rural areas. This includes:

• Improve resilience in areas with poorer quality or less reliable wired internet connections.
• Strengthen communities, including in rural areas, by offering enhanced opportunities to work, trade, and deliver public services.

2) Improve wellbeing of people across Scotland by improving available levels of mobile connectivity.

3) Improve business performance in Scotland by improving available levels of connectivity and improved internet speeds.

4) Enable use of technology, or facilitate changes in behaviour, which support a reduction in Greenhouse Gas emissions in Scotland.

The primary aim of the evaluation was to evaluate the S4GI Programme against the above investment objectives. In addition, the evaluation also aimed to expand the evidence base in terms of how businesses and households use 4G and its associated benefits.

Methodology

Having completed a literature review on the benefits of enhanced digital connectivity, a series of logic maps specifying the range of potential outputs, outcomes and impacts which may result from the delivery of the S4GI Programme were developed.

Primary research was then undertaken in order to gather evidence of the potential changes. The primary research involved two stages as follows:

• Stage 1: Online and paper-based survey of residents and businesses in the areas affected by the delivery of the S4GI Programme
• Stage 2: Follow up focus groups with residents and businesses identified during Stage 1

Each of these is discussed further below.

Stage 1: Surveys

A series of surveys were developed aimed at residents and businesses based in the areas affected by the S4GI Programme. Four separate surveys were developed as follows:

• Resident Survey (‘before’) – aimed at residents in the areas which were scheduled for S4GI investment
• Business Survey (‘before’) – aimed at businesses in the areas which were scheduled for S4GI investment
• Resident Survey (‘before AND after combined’) – aimed at residents in the areas where masts had already been installed
• Business Survey (‘before AND after combined’) – aimed at residents in the areas where masts had already been installed

The surveys were structured around both the Investment Objectives and the potential outputs, outcomes, and impacts identified within the logic maps. A broad overview of the topic areas covered in each of the surveys is set out in Table 1.

The ‘before’ surveys necessarily focused on satisfaction and use before the delivery of the mast. These aimed to establish a pre-scheme baseline prior to mast delivery. The ‘before AND after combined’ surveys included questions on satisfaction and use of services before AND after the delivery of the programme. These surveys therefore established the pre-scheme baseline and the post-scheme position via the same survey. While this is an established and commonly used approach, it is noted that it relies on people remembering their past behaviour and therefore can lead to some distortion of the pre-scheme baseline. In this case, the ‘before AND after combined’ surveys were undertaken an average of 18 months after the delivery of the mast. The distortion due to misremembering is therefore likely to be relatively limited.

Resident Survey (‘before’)

Awareness of S4GI

• Broadband:
  • Home broadband connectivity prior to the delivery of S4GI
  • Satisfaction with broadband
  • Frequency and use of broadband
  • Activities undertaken when using home broadband
• Mobile phone access:
  • Mobile phone access prior to the delivery of S4GI
  • Level of mobile phone connectivity inside (only where respondent does not have broadband) and outside the home
  • Level of mobile phone reliability inside (only where respondent does not have broadband) and outside the home
  • Use of mobile services
• Behavioural impacts of S4GI on:
  • Decision to move to the area
  • General wellbeing
• About you

Resident Survey (‘before AND after combined’)

Awareness of S4GI

• Broadband:
  • Home broadband connectivity prior to and any changes since the delivery of S4GI
  • Satisfaction with broadband
  • Frequency and use of broadband
  • Activities undertaken when using home broadband
• Mobile connectivity:
  • o Mobile phone access prior to and any changes since the delivery of S4GI
  • o Level of mobile phone connectivity prior to and since the delivery of S4GI – inside (only where respondent does not have broadband) and outside the home
  • o Level of mobile phone reliability prior to and since the delivery of S4GI – inside (only where respondent does not have broadband) and outside the home
  • o Use of mobile services and impact of S4GI on frequency of use
• Behavioural impacts of S4GI on:
  • Decision to move to the area
  • Nature of employment and access to learning opportunities
  • Social activities, wellbeing and physical activity
  • General wellbeing
• About you

Business Survey (‘before’)

Awareness of S4GI

• Broadband:
  • Business broadband connectivity
  • Importance of broadband connectivity on business activities / business performance
  • Satisfaction with broadband
• Mobile phone connectivity:
  • Level of mobile phone connectivity and satisfaction inside (only where respondent does not have broadband) and outside the business
  • Importance of mobile services and impact on business activities / business performance
• Behavioural impacts of S4GI on:
  • Decision to operate in the area
• About the business

Business Survey (‘before AND after combined’)

Awareness of S4GI

• Broadband:
  • Business broadband connectivity prior to and any changes since the delivery of S4GI
  • Satisfaction with broadband
  • Frequency and use of broadband
  • Impact on business activities and business performance (those signing up to new/ upgrading business broadband)
• Mobile connectivity:
  • Level of mobile phone connectivity and reliability prior to and since the delivery of S4GI – inside (only where respondent does not have broadband) and outside business premises
  • Impact of S4GI on frequency of use of mobile services and impact on business activities / business performance
• Behavioural impacts of S4GI on:
  • Decision to operate in the area
  • Impact of 4G on employees working from home
  • Impact of 4G on source of supplies / raw materials
• About the business

WHP Telecoms Ltd supplied the Scottish Government with the addresses of properties which would be affected by the delivery of 93% of the masts in the S4GI Programme (51 out of the total 55 mast locations) along with the date that these masts were scheduled to be delivered / had been delivered.

Based on the later, the addresses provided were allocated into either i) areas which were scheduled for S4GI investment or ii) areas where masts had already been installed. The relevant ‘before’ or ‘before AND after combined’ surveys along with a letter introducing the research and a pre-paid return envelope were then issued by post directly to the individual addresses provided.

The surveys were issued in several rounds over a 12-month period from January to December 2023. Table 5‑1 shows the total number of responses received to each of the surveys.

Stage 2: Focus Groups

In addition to the above surveys, a series of online focus groups / one-to-one interviews with residents and businesses affected by the S4GI Programme were also undertaken. The aim of these focus groups / interviews was to explore the topic areas covered within the survey in more depth and gather deeper information and individual experiences.

Focus group attendees / interviewees were recruited via a question in the above surveys which asked respondents whether they would be interested in taking part in follow up focus groups. In total ten participants took part in this aspect of the research.

The focus groups / interviews were delivered virtually via Microsoft Teams and lasted up to one hour each. There were two researchers in attendance, one steering the discussion and one scribe. In advance of the focus groups / interviews, separate Resident and Business Topic Guides were prepared which were used as a broad framework to help guide the discussions.

Key Findings

The key findings from the evaluation with respect to each of the S4GI investment objectives are summarised below.

Reduce the Digital Divide

This objective is focused on reducing the digital divide between i) rural areas and other locations and between ii) deprived areas and other locations by improving connectivity in the former in each case.

All of the mast sites are in rural areas and fall into the 10% most deprived datazones in Scotland in terms of access. The programme has therefore effectively targeted these locations.

The results of the survey suggest that the S4GI Programme has led to improvements in connectivity for both people and businesses in the areas where it has been implemented. These include:

• improvements in mobile connectivity and reliability inside the home / business premises for those without home / business broadband. While based on a small sample, the data suggests that those without broadband saw improved mobile connectivity and were more satisfied with mobile signal reliability inside their premises as a result of the S4GI Programme.
• improvements in mobile connectivity outside the home / business premises with 81% (n=95) of residents with access to a smartphone stating they had 4G connectivity following the S4GI Programme compared to just 4% (n=5) prior to the masts being installed and 83% (n=29) of businesses stating that they were aware of improvements in mobile connectivity in their local area compared to 17% (n= 6) who were unaware
• improvements in the reliability of mobile connections outside the home, with average satisfaction with reliability increasing from 2.4 to 7.6 amongst residents and from 1.9 to 8 amongst businesses following the delivery of the S4GI Programme

In addition to improvements in mobile connectivity, the data suggests that around 20% of both business and residents made a change to their broadband connection (either upgrading or purchasing a new contract having not previously had one) because of the S4GI Programme. In addition to enhanced mobile connectivity, the delivery of S4GI has therefore allowed a proportion of those affected to enhance their home / business broadband connectivity so that speeds and reliability are more in line with more populated locations, resulting in additional benefits.

While on the whole connectivity has improved, it is also important to note that some people / businesses continue to experience difficulties with connectivity, with issues associated with topography and / or weather highlighted in both the open text survey responses and the focus groups / interviews. In addition, participants raised concerns around the need to use the MNO linked to the mast site and the more limited consumer choice and higher costs compared to more populated locations as a result.

Key Point: Overall, the research suggests that the S4GI Programme has helped to reduce the digital divide and deliver this objective. However, while on the whole connectivity has improved, some people / businesses continue to experience difficulties in part due to topography and / or weather and it is evident that S4GI had at the time the research was carried out not been able to deliver / facilitate the same level of consumer choice in terms of the number of operators as that of more populated locations with the result that there remains a level of inequality. It is noted though that since the research was carried out, customer choice has improved. At the date of publication, more than half of the 55 masts now offer services from at least 3 mobile operators, and this figure is set to rise even further.

Improve Resilience

This sub-objective focuses on improving resilience in areas with poorer quality or less reliable wired internet connections.

Amongst the businesses surveyed, half (50%, n=18) identified ‘resilience when there is a broadband outage’ as a positive impact of the S4GI Programme, higher than any other aspect. The resilience benefits of 4G were also identified by residents and businesses in the open text survey responses and the focus groups / interviews, with respondents noting the importance of a back-up during storms, poor weather and other incidents. The ability of 4G to act as a back-up when broadband is unavailable corroborates some of the findings identified in the S4GI Use Cases^[1] and wider literature review which identified improved resilience as a key benefit, particularly for businesses.

While potentially slightly beyond the intended scope of this objective, the research also suggests that the S4GI Programme has led to communities organising online, bringing improvements in terms of community resilience (broadly understood as the cohesion and strength of a community). This was picked up in both the surveys and the focus groups. These aspects are discussed further in the section on wellbeing below.

Key Point: The research suggests that the S4GI Programme has resulted in improved resilience for those who have benefited from the enhanced connectivity provided and therefore the programme has delivered this objective.

Strengthen Communities

This sub-objective focuses on strengthening communities by offering enhanced opportunities to work, trade, and deliver public services.

The results of the survey and focus groups suggest that the delivery of the S4GI Programme has provided enhanced opportunities in terms of work for both residents and businesses. While based on a small sample, 25% of residents surveyed (n=9) said they were able to work from home more frequently. There was also some evidence that S4GI had enabled some respondents to set up a new business, take on an additional job, or move to higher paid employment although it is noted that sample sizes were small and therefore it’s difficult to draw firm conclusions on these aspects.

A range of benefits with respect to work were also identified in the business survey. Positive impacts on business activities included:

• being able to access online servers (identified by 42% of businesses surveys as a positive impact of 4G)
• the ability of staff to access online resources when outside of business premises (identified by 42% of businesses surveys as a positive impact)
• improved communication with staff and the ability to undertake video conferencing / Zoom / Teams calls (identified by 36% of businesses surveys as a positive impact)

In addition to employment, the research also suggests that the availability of 4G has led to an increase in people taking up both formal and informal online learning opportunities.

In terms of providing enhanced opportunities to trade, while sample sizes were relatively small, there was some evidence from the business survey that the delivery of 4G had had positive impacts in this respect with:

• 30% (n=11) of businesses stating that the availability of 4G had had a positive impact on ‘selling products / services’
• 28% (n=10) stating that it had had a positive impact on business promotion, including via social media
• 25% (n=9) stating that it had had a positive impact on the product, or services provided

In addition, findings from the resident survey suggest that the availability of 4G has led to an overall increase in the level of online shopping and the use of online and mobile banking. In the open text responses to the survey, respondents particularly highlighted that the availability of 4G had allowed them to receive security confirmation texts / verification codes without which they were previously unable to access applications which require this security check.

While the research did not explicitly cover the delivery of public services, 20% (n=25) of the respondents to the resident survey noted that the availability of 4G had led to an increase in them undertaking GP / hospital appointments via a mobile connection. In terms of providing such services, 4G connectivity may therefore contribute to the viability of such services and potentially lead to cost savings for the NHS as well as businesses as a result of reduced sick days.

In terms of strengthening communities, while the sample sizes were small and therefore it is difficult to draw firm conclusions, the results from the resident survey indicate that the future availability of 4G did influence some people’s decision to move to their current property. The benefits for the long-term sustainability of rural areas were also highlighted in the focus groups, with several participants noting that improved connectivity (and particularly the ability to work from home) had contributed to young people moving back to the area and / or been a factor in people’s decision to stay.

Key Point: Overall, the results suggest that the delivery of the S4GI Programme has resulted in positive impacts for the communities benefiting from the programme, particularly with respect to opportunities to work and the delivery of public services. While sample sizes are small, there is also some indication that the availability of 4G has influenced people’s decisions regarding moving to / remaining in the area and therefore that 4G has contributed to a strengthening of the communities affected.

Improve Wellbeing

This objective focuses on improving wellbeing as a result of improved mobile connectivity. To help identify wellbeing impacts, two questions were included in the resident survey as follows:

• a question asking respondents to indicate the impact of the availability of 4G on their wellbeing
• a question which replicated the format of the questions on personal wellbeing included in the UK’s APS^[2] whereby respondents are asked one question covering each of the four measures of personal wellbeing (namely life satisfaction, feeling the things done in life are worthwhile, happiness, and anxiety) - this approach to capturing wellbeing impacts is well established in the literature and has been used in previous evaluations of similar schemes, although attributing any change identified to the investment being evaluated can be challenging.

In terms of the former, 39% (n=14) of residents surveyed indicated that the delivery of 4G had had a positive impact on their wellbeing compared to 8% (n=3) who said it had a negative impact. It is unclear as to why the latter felt 4G had had a negative impact with potential reasons including the connection not being adequate (i.e. due to topography) or a potential knock-on reduction in-person contact. The latter is discussed further below.

In terms of the UK APS wellbeing question, the results showed that wellbeing was relatively high in the areas surveyed. However, while there were some changes between the ‘before’ and ‘before AND after combined’ surveys, these differences were marginal, and it is difficult to attribute any changes directly to the S4GI Programme.

Beyond the specific questions on wellbeing, a range of positive impacts of 4G which may have an overall positive impact on wellbeing were identified within the survey. For example, a high proportion of respondents reported positive impacts on:

• keeping in touch with family and friends (56% recording positive benefits, n=20)
• feeling of connectedness to the wider world (53% recording positive benefits, n=19)
• finding out about what's going on in the local area (42% recording positive benefits, n = 15)
• feelings of personal safety / personal safety when out and about (39% recording positive benefits, n=14)
• feelings of connectedness to the local community (36% recording positive benefits, n=13)

These findings were also replicated in the focus groups / interviews. Participants in the focus groups also highlighted that enhanced connectivity had enabled the local community to organise online. It was noted that new WhatsApp groups and online message boards had been set up to share information; offer support; and set up group activities. As noted above, as well as potentially improving individual wellbeing such activities could also lead to improvements in community resilience (broadly understood as the cohesion and strength of a community).

There was however a recognition amongst some focus group / interview participants that more online activity has the potential to result in a reduction in face-to-face contact / local services in some areas which could have negative impacts.

Key Point: Overall, while the results of the specific question on personal wellbeing were inconclusive, the wider research suggests that the availability of 4G has, on the whole, had a positive impact on wellbeing with a range of positive impacts identified. However, a number of disbenefits including a decline in face-to-face contact and local services were also identified. The overall balance of benefits / disbenefits in this respect is likely to differ depending on both the community in question and the individual. For example, with respect to the latter, there may be more disbenefits for those who are digitally excluded due to age / level of ability and / or economic situation. It may also be the case that some disbenefits (such as a decline in local services) take longer to materialise.

Improve Business Performance

This objective is focused on improving business performance by improving available levels of connectivity and improving internet speeds.

As noted above, data from the business survey suggests that the majority of businesses saw an improvement in mobile connectivity and reliability as a result of the delivery of 4G and around 20% also made a change to their broadband connection because of the programme. The research therefore confirms that S4GI has generally led to improvements in connectivity and internet speeds for businesses in the areas affected.

While sample sizes were relatively small and therefore it is difficult to draw firm conclusions, a range of positive benefits in terms of business performance were identified within the business survey. The aspects of business performance for which the highest proportion of businesses recorded positive impacts included:

• ‘customer base’ (33% recording positive benefits, n=12)
• ‘employee productivity / efficiency of working patterns’ (28% recording positive benefits, n=10)
• ‘product / services provided’ (25% recording positive benefits, n=9)

A range of benefits of 4G were also identified in the open text responses to the survey and the focus groups / interviews. These included employee productivity / efficiency gains due to enhanced communication and improved levels of safety; a widened customer base for tourism accommodation businesses; promotional benefits as a result of being able to promote attractions / the area via social media; and enhanced safety for visitors e.g., hillwalkers etc. While beyond the scope of this research, there is potential for such benefits to result in higher visitor numbers, with consequential impacts on visitor spend and business turnover.

Key Point: Overall, the results of the research suggest that the delivery of 4G has led to improved business performance in the communities affected by the scheme with a range of positive impacts identified including a wider customer base, improved employee productivity / working patterns, and improvements to the product / services provided.

Reduce Greenhouse Gas Emissions

This objective is focused on enabling use of technology or facilitating changes in behaviour which support a reduction in Greenhouse Gas emissions. In terms of broadband and 4G connectivity, the main mechanism identified for a reduction in carbon emissions is a reduction in travel as a result of home working and / or increased use of online services. As noted above, while based on a small sample, 25% of residents surveyed (n=9) said they were able to work from home more frequently as a result of 4G being available. There was also evidence of increased online shopping, use of online and mobile banking, and online GP / health appointments as well as online attendance at community council and other community meetings. However, while the above may lead to a reduction in travel and therefore reduced greenhouse gas emissions, it is also the case that home working and the delivery of online shopping / services will themselves generate carbon emissions (e.g., through additional domestic energy consumption, more delivery drivers) which will in part offset the savings generated. In addition, the research identified evidence of increased community activities as a result of the ability to organise online e.g., yoga classes and badminton groups which may contribute to additional travel albeit potentially more locally based.

Key Point: Overall, the results suggest that the delivery of 4G has enabled people to use online services and work from home more, thereby suggesting a reduction in resident travel at the local level. Some of these savings will however be offset by carbon emissions generated via other activities with more detailed analysis required to determine the overall impact.