Impact Evaluation of the Community Right to Buy

Annex B: Methodology

The first stage of the impact evaluation of the Community Right to Buy consisted of a desk-based review of relevant documentation and in-depth scoping interviews with stakeholders. Stakeholder interviews were conducted face to face and by telephone with representatives of: the Scottish Government, Big Lottery Fund, Development Trusts Association Scotland (DTAS), and Highlands and Islands Enterprise (HIE). Stakeholders were identified based on their involvement with the CRtB and their role in the wider field of community ownership of land and assets. Interviews explored stakeholders' views on the aims of the CRtB, the outcomes, challenges or barriers to achieving these outcomes, and the inputs that have contributed to the process. Stakeholders were also asked to review and comment on the draft logic model.

Data was analysed to identify the number of community bodies involved at each stage of the process and the nature of their involvement at each stage. Data was provided by the Scottish Government Community Right to Buy Branch and was available through the Register of Community Interests in Land (RCIL).

An online survey was carried out with community bodies involved at each stage of the process. The survey sample was identified as a result of the data analysis outlined above. Those without a valid e-mail address were excluded from the sample as they would be unable to receive the online survey. The survey was carried out between 11^th February and 6^th March 2015.

Responses were received from 65 community bodies, from a sample of 186 (response rate of 37% after adjusting for 29 non-contacts). The rate of response is closely linked to when the community body was set up (the more recently set, the higher the response rate). Table B.1 shows the sample broken down by survey response status and by the year the community body was incorporated.

Table B.1: Survey response by year of incorporation

Responses also differed by how far a community body had got in the formal process, but not by a great deal. Forty eight percent of those whose registered interest was currently active completed the online survey. Table B.2 shows the sample broken down by survey response status and by the furthest stage they reached in the process.

Table B.2: Survey response by stage reached in the process

Following the survey, case studies were selected for more in-depth, qualitative research. Case studies were selected to include a broad spectrum of communities involved at each stage of the process. Selection criteria were based on the stages reached in the process, agreed with the Scottish Government in advance. Case studies were then selected at random from the original survey sample (derived from analysis of administrative) to fit each criterion.

Recruitment of case studies first involved an e-mail to the named contact from each group, explaining the purpose of the evaluation and inviting them to participate in the research as a case study. The initial e-mail was followed up with a telephone call from a member of the evaluation team to discuss the research further and confirm whether or not the group was willing to take part. Where participants agreed to take part, arrangements were made for fieldwork to take place.

Sixteen case studies were carried out (Table B.3). Eight of these were "full" case studies involving site visits and face-to-face fieldwork, while eight with "light" case studies that involved fieldwork conducted by telephone.

Table B.3: Case study selection

Fieldwork at full case studies consisted of a site visit and a focus group with members of the community body. In some cases these focus groups were substituted with individual or paired depth interviews depending on the circumstances of each group and the number of people still involved. Where possible and where relevant, full case studies also included depth interviews with other community representatives (e.g. community councils) and with the current or former landowner (although the latter proved difficult in most cases). Fieldwork at light case studies consisted of telephone depth interviews with representatives of the community bodies. Relevant monitoring data (e.g. application form and supporting documents) was also reviewed for each case study.

Methodological critique

The development and application of a four stage methodology (stakeholder interviews, data review, survey and case studies) has produced a diverse set of quantitative and qualitative findings. The survey provided a detailed overview of community bodies which have engaged with the CRtB, with a reasonably high return rate (37%). Respondents represented a range of scales and types of assets and included community bodies from all stages of the CRtB process (Table B.2).

The survey respondent group was skewed towards more recently established community bodies, with the highest level of non-response (59%) evident in those established pre-2008 (Table B.1). This may be due to a higher number within this group having been less recently active and therefore less aware/able to remember the CRtB process and some which may have disbanded/become effectively dormant. The online approach also led to the exclusion of community bodies with no current obtainable email address and it should be noted the survey results represent a snapshot of views at a single point in time. To address these factors a more regular structured survey approach may be required, linked to data gathering related to an established set of "progress indicators". Integrating such an approach with part of the wider administrative framework relating to the CRtB and community asset management is one possible approach to increasing return rates and developing a long term monitoring approach for community bodies utilising the CRtB process.

The case study dataset represents a broad spectrum of CRtB examples representative of all the main stages of the process. However, the inclusion of case studies which had progressed only to earlier stages of CRtB/were unsuccessful in acquiring assets, limited the numbers of case studies which had reached later stages, and specifically those which had acquired assets. This approach allowed for the gathering of detailed data on the CRtB process and the exploration of outcomes from different CRtB stages (a key objective of the evaluation). However, the approach limited the number of case studies which demonstrated the full potential outcomes of the CRtB (e.g. those related to asset ownership). Furthermore, despite an original objective to include previous/current landowners within the case studies, this proved difficult in most cases due to these stakeholders being difficult to access (and in some cases identify), resulting in this viewpoint being largely absent from the data.

Case studies have been fully anonymised, which protects the identity of respondents and minimises sensitivities around individual viewpoints and existing ("live") CRtB processes. However, this approach removes the potential for a fuller representation of individual cases and "stories" and the presentation of fuller and more detailed examples across the report. Critically, it is not always possible to infer which outcomes are directly or solely related to the CRtB legislation/process (due to the complex context within which the process occurs), although this has been indicated where possible. The results have been structured to reflect the inputs, activities and outcomes of the logic model (Figure A) and the conclusions section notes key areas where results indicate a departure from the model, with potential alterations to the model proposed in response to the data presented in this report.