Islands Bond: consultation responses analysis

12. Measuring the impact of the Bond (Q11)

12.1 The consultation paper mentioned the collection of data in relation to (i) monitoring the Bond application process and (ii) measuring the wider impacts of the Bond scheme. Question 11 asked respondents to suggest methods (beyond those outlined in the consultation paper) for measuring the progress and impact of the scheme. This chapter presents an analysis of these suggestions.

Question 11: Beyond those outlined in the Consultation Paper, what other methods of measuring the progress and impact of the Islands Bond might we adopt?

12.2 This was an open question, and it should be noted that, in general, respondents' comments focused on measuring the (potential) wider impacts of the bonds. There was limited comment on 'process indicators' relating to the application for a bond (e.g. numbers of applications received, dropout rates, success rates, etc.). The main issue raised by respondents in this regard was the need to have a full account of the awards made in terms of island location, whether the award was for retention or relocation, and the specific purpose of the award.

12.3 Respondents who were opposed to the introduction of an Islands Bond, or who thought that the amount available (£5m in total) was insufficient to create any meaningful impact, generally indicated that they did not support any effort to measure progress or impact.

12.4 Substantive responses to Question 11 from respondents who thought that measuring impacts was worthwhile – and indeed important – were wide ranging and included comments about:

• General principles and considerations which should inform the approach to measuring the progress and impact of the Islands Bond
• Methodological approaches and research designs which were best suited to capturing the kinds of data which would be useful for assessing progress and impact
• Specific measures / indicators which it would be beneficial and / or important to capture
• Other issues.

12.5 Each of these is discussed in turn below.

General principles and considerations

12.6 Respondents suggested the following principles and considerations should inform the development of the approach to measuring the progress and impact of the Islands Bond:

• Measurements should relate to the criteria for allocating the bonds. So, for example, if bonds are allocated to help with retention, returning and relocation, measurements should capture all these specific elements.
• Any measurement would need to be undertaken over an appropriate timeframe. Respondents made the point that it was unlikely that impacts would be detectable in the short term. This did not mean that the measurement should not start early. On the contrary, measurement should be undertaken at regular intervals throughout the period of assessment, so that any learning or adjustments to the implementation of the scheme could be incorporated. However, any assessment of whether, for example, those who relocated were retained in the community could not be undertaken for 5 or 10 years.
• Measurements would have to capture impacts both on recipients and non-recipients of bonds. In particular, it would be vital to capture the impacts on the wider community in which bond recipients live.
• Consideration would need to be given to the appropriate level of aggregation for any measurement of impact. For example, would impact be measured for a single island, sub-group of islands, or all island groups? The discussion of this point was linked to comments about how bonds would be distributed / allocated.
• It will be very challenging to isolate the impacts of the Island Bond scheme from the impacts of other contemporaneous infrastructure developments relating to, for example, transport, digital connectivity and housing.

Methodological approaches

12.7 Respondents agreed with the proposal set out in the consultation paper that any measurement framework should include both quantitative and qualitative approaches. Some respondents suggested that qualitative approaches were more relevant given the likely timeframe for quantitative measures to change.

12.8 Respondents suggested a wide variety of methodological approaches would be appropriate, and provided reasons for their suggested approaches as follows:

• Longitudinal: Given that the impacts of the Islands Bond could play out over a long timeframe, respondents suggested that a longitudinal approach, with measurements carried out at appropriate timepoints over a number of years would be useful.
• Case studies: Respondents suggested that case studies of bond recipients and of non-recipients could provide insights about the reasons why a particular award did or did not achieve the expected impacts.
• Comparative studies: Respondents suggested that comparing the social, economic and financial 'success' of recipients of bonds with non-recipients would help to determine the 'added value' of the Islands Bond.
• Surveys: Respondents suggested that annual surveys of residents could be useful, especially in determining impacts on local residents and on the local culture.
• Census / audits: Respondents suggested that continuous recording of the moves between, within, from and to island communities would be useful. This could also cover audits of businesses and other significant infrastructure, improvements to roads, other amenities, etc.
• Progress reports (to be submitted at regular intervals by bond recipients): These would account for the use of the funds as well as offering a perspective on any impacts, both positive and negative.
• Annual events within communities to elicit feedback on the impacts of the scheme.

12.9 In addition, there was mention of extant toolkits or measurement frameworks used elsewhere (e.g. Community Mapping Toolkit) as well as suggestions for a TV documentary to publicise the scheme.

Specific measurements

12.10 Respondents offered a wide range of suggestions for specific indicators that they thought would be useful to measure in assessing impact. These covered in particular:

• Demographic: e.g. population numbers by age and sex, birth rates, school rolls, household size, number of individuals and families returning
• Economic: e.g. employment (number of jobs, number of jobs created, number of vacancies, number of new businesses), skills shortages, vibrancy of local economy, local investment, house prices, pressure on health, healthcare and education services, household income, local expenditure, number of visitors / tourists, transport, ferry services
• Social: e.g. housing (number of new affordable homes, ratio of first to second homes, underused / unoccupied crofts, number of planning applications), quality of life, satisfaction, happiness, wellbeing, community connections, social cohesion, knowledge sharing and exchange, volunteering
• Environmental: e.g. environmental degradation or improvement, water quality, carbon emissions
• Other: e.g. number of Gaelic speakers, numbers attending Gaelic classes.

Other issues

12.11 It was suggested that a range of other aspects should be investigated in relation to measuring impacts. These covered: the introduction of innovative practices to the islands, the impact on crofting, and the maintenance of traditional skills and crafts.

12.12 Some respondents who wished to receive a bond offered to provide some social media content (in the form of a video blog or other relevant format) to advertise and promote the scheme and to demonstrate its impact on their situation.