Scottish 4G Infill Programme Evaluation

5 Methodology

5.1 Overview

5.1.1 This chapter sets out the approach to conducting the primary research with residents and businesses which aims to gather evidence of the potential outcomes and impacts identified within the logic maps. The chapter includes details of the overall structure of the research, the methods employed, and the samples achieved.

5.2 Research Structure

5.2.1 The research involved two broad stages as follows:

• Stage 1: Online and paper-based survey of residents and businesses in the areas affected by the delivery of the S4GI Programme
• Stage 2: Follow up focus groups with residents and businesses identified during Stage 1

5.2.2 The approach to the delivery of each of these stages is set out below, with full details of the results and analysis of each stage summarised in the subsequent report chapters.

5.3 Stage 1: Surveys

5.3.1 When the research commenced, several masts had already been delivered while others were scheduled for delivery. In recognition of this and the differential potential impacts on residents and businesses, four separate surveys were developed as follows:

• Resident Survey (‘before’) – areas which were scheduled for S4GI investment
• Business Survey (‘before’) – areas which were scheduled for S4GI investment
• Resident Survey (‘before AND after combined’) – areas where masts had already been installed
• Business Survey (‘before AND after combined’) – areas where masts had already been installed

Survey Content

5.3.2 The surveys were structured around both the Investment Objectives and the logic maps. A broad overview of the topic areas covered in each of the above four surveys is set out below.

Resident Survey (‘before’)

Awareness of S4GI

• Broadband:
  • Home broadband connectivity prior to the delivery of S4GI
  • Satisfaction with broadband
  • Frequency and use of broadband
  • Activities undertaken when using home broadband
• Mobile phone access:
  • Mobile phone access prior to the delivery of S4GI
  • Level of mobile phone connectivity inside (only where respondent does not have broadband) and outside the home
  • Level of mobile phone reliability inside (only where respondent does not have broadband) and outside the home
  • Use of mobile services
• Behavioural impacts of S4GI on:
  • Decision to move to the area
  • General wellbeing
• About you

Resident Survey (‘before AND after combined’)

Awareness of S4GI

• Broadband:
  • Home broadband connectivity prior to and any changes since the delivery of S4GI
  • Satisfaction with broadband
  • Frequency and use of broadband
  • Activities undertaken when using home broadband
• Mobile connectivity:
  • Mobile phone access prior to and any changes since the delivery of S4GI
  • Level of mobile phone connectivity prior to and since the delivery of S4GI – inside (only where respondent does not have broadband) and outside the home
  • Level of mobile phone reliability prior to and since the delivery of S4GI – inside (only where respondent does not have broadband) and outside the home
  • Use of mobile services and impact of S4GI on frequency of use
• Behavioural impacts of S4GI on:
  • Decision to move to the area
  • Nature of employment and access to learning opportunities
  • Social activities, wellbeing and physical activity
  • General wellbeing
• About you

Business Survey (‘before’)

Awareness of S4GI

• Broadband:
  • Business broadband connectivity
  • Importance of broadband connectivity on business activities / business performance
  • Satisfaction with broadband
• Mobile phone connectivity:
  • Level of mobile phone connectivity and satisfaction inside (only where respondent does not have broadband) and outside the business
  • Importance of mobile services and impact on business activities / business performance
• Behavioural impacts of S4GI on:
  • Decision to operate in the area
• About the business

Business Survey (‘before AND after combined’)

Awareness of S4GI

• Broadband:
  • Business broadband connectivity prior to and any changes since the delivery of S4GI
  • Satisfaction with broadband
  • Frequency and use of broadband
  • Impact on business activities and business performance (those signing up to new/ upgrading business broadband)
• Mobile connectivity:
  • Level of mobile phone connectivity and reliability prior to and since the delivery of S4GI – inside (only where respondent does not have broadband) and outside business premises
  • Impact of S4GI on frequency of use of mobile services and impact on business activities / business performance
• Behavioural impacts of S4GI on:
  • Decision to operate in the area
  • Impact of 4G on employees working from home
  • Impact of 4G on source of supplies / raw materials
• About the business

5.3.3 As indicated above, the ‘before’ surveys necessarily focused on satisfaction and use before the delivery of the mast. These aimed to establish a pre-scheme baseline prior to mast delivery. Initially, it was intended that follow-up ‘after’ surveys would be issued directly to the individuals who responded to the ‘before’ surveys in order to establish the post-scheme position and the level of change. However, this was not taken forward due to:

i. the limited number of addresses provided for properties in the scheduled locations; and

ii. difficulties associated with directly linking responses from the ‘before’ surveys to any subsequent ‘after’ survey due to the UK’s General Data Protection Regulation (GDPR). This is discussed further in Section 5.3.14 below.

5.3.4 The ‘before AND after combined’ surveys included questions on satisfaction and use of services before AND after the delivery of the programme. These surveys therefore established the pre-scheme baseline and the post-scheme position via the same survey. While this is an established and commonly used approach, it is noted that it relies on people remembering their past behaviour and therefore can lead to some distortion of the pre-scheme baseline. In this case, the ‘before AND after combined’ surveys were undertaken an average of 18 months after the delivery of the mast (see Section 5.3.8 below). The distortion due to misremembering is therefore likely to be relatively limited.

Survey Implementation and Analysis

5.3.5 WHP Telecoms Ltd supplied the Scottish Government with the addresses of properties which would be affected by the delivery of 93% of the masts in the S4GI Programme (51 out of the total 55 mast locations) along with the date these masts would be delivered / had been delivered. Details of the locations for which addresses were provided are included in Appendix C .

5.3.6 Based on the date each mast was scheduled to be delivered / had been delivered, the addresses provided were allocated into either i) areas which were scheduled for S4GI investment or ii) areas where masts had already been installed. The relevant ‘before’ or ‘before AND after combined’ surveys along with a letter introducing the research and a pre-paid return envelope were then issued by post directly to the individual addresses provided. The letter also included a link to an online version of the surveys enabling respondents to complete the survey online if they preferred. An option for a free telephone call back service to go through the survey over the phone was also provided.

5.3.7 It is noted that information on whether the property was a business or a residency was not provided by WHP Telecoms and therefore copies of the appropriate resident and business survey (i.e. scheduled or mast installed) were issued to each address.

5.3.8 The list of sites to which surveys were issued, the delivery date of the mast in each location and the survey dates are shown in Appendix D . For the ‘mast delivered locations’, the number of months between the delivery of the mast and the survey date is also shown for reference. The mast delivered locations were surveyed between 5 - 36 months following the delivery of the mast, with most (68%) surveyed within 2 years of the masts being delivered.

5.3.9 The surveys were issued in several rounds over a 12-month period from January to December 2023. This is because WHP Telecoms provided the addresses incrementally over this period. As such, the status of some mast locations (and therefore some properties) changed (from scheduled to installed) over the course of the survey period as the masts were delivered. Some properties were therefore issued with both ‘before’ and ‘before AND after combined surveys’.

5.3.10 Appendix C also provides details of the MNOs active on each mast at the time when the surveys were undertaken. As with the S4GI Programme as a whole, EE was the dominant provider amongst the mast locations surveyed.

5.3.11 It is noted that the above approach is self-selecting in that the individuals responding to the surveys have chosen to do so. This may introduce an element of bias where those who are particularly happy or unhappy with the outcomes from the S4GI Programme are more likely to respond.

Achieved Sample

Number of Responses

5.3.12 Table 5‑1 shows the total number of responses received to each of the surveys. A detailed breakdown of the number of responses received in each mast location is provided in Appendix B.

5.3.13 Table 5‑2 shows the response rate. Given that information on whether the address was a business or residency was not available, it is only possible to provide a combined rate (resident and business together) for the installed and scheduled locations.

5.3.14 As discussed above, initially it was intended that follow-up ‘after’ surveys would be issued to all those who responded to the ‘before’ surveys. However, due to GDPR, this was only possible where respondents provided their name and contact details specifically for this purpose. In total of the 53 ‘before’ responses received from residents 20 provided their name and contact details for this purpose and of the 9 ‘before’ responses received from businesses 4 fell into this category. Given that not all of those issued with a follow up survey would have responded, it was reasoned that the total sample would likely be too small a sample to provide any meaningful results and therefore ‘after’ surveys were not issued.

Demographic and Business Profile

5.3.15 Information on the demographic profile of respondents to the resident survey and background information on the businesses who responded to the business survey is included in Appendix E . Key points are as follows:

Resident:

• There was a 47% / 48% split between males and females.
• While responses were received from all age groups, the largest proportion of responses were received from the 55 – 64 age group (accounting for 32% of the sample), followed by 65-74 (accounting for 25% of the sample). This likely reflects the generally older age demographic in rural areas^[51]
• Nearly half (46%) of the sample were retired – again this likely reflects the profile of residents in rural areas with relatively high proportions of retirees compared to elsewhere.
• In terms of employment, 28% were employed or self-employed full time and 14% were employed or self-employed part-time.
• In terms of S4GI, just less than half of respondents (48%, n=88) were aware of the S4GI Programme, with, 52% (n=96) unaware.

Business:

• Businesses in the agriculture / forestry / fishing industry accounted for the largest proportion of respondents, making up just over one third of the sample (34%, n=11). This was followed by businesses in the tourism sector (16%, n=5), accommodation and food service activities (13%, n=4), and manufacturing (6%, n=2).
• The businesses who responded were generally small - nearly a third of businesses had an annual turnover of less than 10k with nearly half turning over less than 30k.
• Over 40% (44%, n=15) operated as sole traders with remaining businesses employing an average of approximately 3-4 full time equivalents.
• Three of the businesses employed seasonal workers with an average of 2 seasonal workers employed per business in this category
• Over one third (38%, n=13) of businesses have staff that work out in the local community during the course of their job (e.g., trades people, delivery drivers etc), hence they do not always have access to the business broadband. A level of reliance on 4G can be expected for these employees.
• Over half (58%, n=21) of the businesses were aware of the S4GI Programme, while 42% (n=15) were unaware.

Survey Analysis

The results of the survey are set out in Chapters 6 and 7. Given that much of the pre-scheme baseline established via the ‘before’ surveys only becomes meaningful when compared with the post-scheme position, rather than a separate narrative covering these surveys, Chapter 6 and 7 draw on the results of the ‘before’ surveys where relevant.

5.4 Stage 2: Focus Groups / Interviews

5.4.1 In addition to the above surveys, a series of online focus groups / one-to-one interviews with residents and businesses affected by the S4GI Programme were also undertaken. The aim of these focus groups / interviews was to explore the topic areas covered within the survey and gather deeper information and individual experiences.

Approach

5.4.2 In order to gather potential focus group attendees / interviewees, a question asking respondents whether they would be interested in taking part in this aspect of the research was included within each of the above surveys.

5.4.3 In total, 44 respondents (36 residents and 8 businesses) stated that they would be interested in taking part in a focus group and provided their contact details for this purpose. Emails were issued to all 44 individuals asking for their availability across a range of dates. In order to encourage uptake, an incentive (a £30 Love2Shop e-voucher) was offered to those taking part in the focus groups.

5.4.4 Of the 35 initial respondents, 15 responded to the email, including three declining the focus groups. Those who accepted and chose date options were allocated to five focus groups, or individual interviews where only one participant could attend. The total number of attendees was 10, with two participants who had indicated they could attend not responding to meeting links. The number of attendees at each focus group / interview split by whether they were a resident, or a business is shown in Table 5‑3: Focus Group and Interview Attendees by type below.

5.4.5 Geographically, the areas represented by the attendees were: Applecross, Blairmore, Cairngarroch, Crinan, Manish, Strathconon, and Skerray. It is noted that only one business attended and comments relating to business benefits / disbenefits should be considered in this context. The more limited participation by business representatives is perhaps to be expected given the monetary costs associated with attendance for businesses.

5.4.6 The focus groups / interviews were delivered virtually via Microsoft Teams and lasted up to one hour each. There were two researchers in attendance, one steering the discussion and one scribe.

5.4.7 In advance of the focus groups, separate Resident and Business Topic Guides were prepared (see Appendix F ). Rather than a script, these were used as a broad framework to help guide the discussions. As such, all topic areas were not necessarily discussed in each focus group but rather the discussions were allowed to unfold as per the interest and knowledge of the participants in each case.

5.4.8 Information from the focus groups / interviews was captured in a series of tables, containing key questions and findings. The focus groups / interviews were also recorded to ensure all comments were captured and the recordings were deleted once the material was analysed.

5.4.9 The findings from the focus groups / interviews are summarised in Chapter 8. All comments have been anonymised, and identifiable details removed, including location, sex, and age.