Community-led regeneration approach: review

Appendix A: Detailed Study Method and Limitations

The study integrated a number of elements from the following research approaches:

• Theory of Change ( ToC) Review;
• Realist Evaluation;
• Qualitative Comparative Assessment (at programme level); and
• Attribution Analysis.

ToC Review : The rationale and logic of the ToC was jointly reviewed with the Study Research Advisory Group at the outset and at the end of the study. This included a detailed consideration of all ToC aspects as presented in the PCF ToC such as Inputs, Outputs (activities and participation) and Change (learning, action, outcomes).

The findings of the first ToC workshop further clarified the theoretical assumptions underpinning the programme and the success criteria, and therefore provided the basis for the project sample and fieldwork tools/material to be developed in line with Realist Evaluation principles.

Realist Evaluation: Realist approaches are suited to exploring causality and contribution, and to understanding why interventions do or do not work for specific groups in different contexts rather than simply whether they work ^[10] . The approach therefore fitted the study requirements well. Realist Evaluation and theories of change identify, test and refine the theories that underlie interventions ^[11] .

Realist Evaluators use the terminology of contexts, mechanism and outcomes (please also see Glossary):

• Contexts include factors such as the: setting, groups targeted, resources and, the types of activities or strategies and approaches used in interventions;
• Mechanisms are reasoning/psychological responses of the participants (or organisations) to the resources/changes in context brought about by the intervention (Wong et al., 2013); and
• Outcomes are the intentional or unintentional changes brought about by the intervention for individuals, organisations or the environment.

PCF Project Themes/Types

Advice/Support Service
Community Facility
Confidence and Skills Building
Diversionary Plus
Employability Training
Energy Efficiency/Fuel Poverty
Family Support
Health Improvement
Healthy Eating/Cooking
Intergenerational Work
IT Skills Training
Older Peoples Project
Peer Mentoring
Training and Upskilling
Volunteering

The programme theories uncovered in Realist Evaluation are described as Context, Mechanism and Outcome Configurations ( CMO-Cs). Crucially, CMO-Cs postulate how and why an intervention works . It explains how by intervening and altering a context, a mechanism is triggered causing/contributing to the intended or unintended outcome ^[12]

Based on the range of themes funded by PCF, there are many potential theories within the PCF. The study team therefore, needed to agree with the Scottish Government in the ToC workshop the most prevalent and promising theories to be tested and refined to determine an appropriate sampling framework, methods and analysis (see sample frame description provided opposite)

Qualitative Comparative Assessment : While relevant quantitative measures were mostly extracted from PCF documentation (project application and/or monitoring reports), the fieldwork programme focused primarily on capturing the qualitative evidence from project staff, benificiaries and partners involved in PCF. This included the qualitative dimension measuring intangible project outcomes, learning journeys and social impact often associated with personal development, social capital, positive changes in relationships, attitudes and behaviours, skills, health and well-being, and quality of life improvements.

Contribution Analysis: In addition to assessing the extent to which change has been achieved, the fieldwork explored perceptions of how much of the achievement was contributed by/attributable to the PCF approach in view of other funding sources used by the community organisations or other factors potentially contributing towards the changes that have been reported.

In the context of a qualitative impact assessment, contribution analysis needs to make the prioritsed assumptions and theories explicit and use these to inform decisions about the focus of the subsequent analysis ^[13] . This aids transparancy. The primary research sought to understand the contextual factors and conditions associated with particular mechanisms and outcomes.

A substantial amount of information was gathered, which enabled the exploration of patterns of contexts [including PCF strategies/activities] and mechanisms associated with self-reported outcomes. The findings from the primary research have been compared with the intervention logic ( ToC) of the PCF and presented in the second ToC workshop with the Study Research Advisory Team.

Limitations regarding a pure Realist Evaluation approach

By applying the Realist Evaluation approach in a practical grass-roots setting, the review has been subject to a small number of limitations.

• In light of the underlying values, principles and aims of co-production within PCF, cross-participant group work was a required aspect of the fieldwork programme. In each workshop therefore one element of data collection was completed as a whole group ( e.g. included partners, anchor organisation staff and projects beneficiaries). This supported the sharing of project experiences and thereby triggered a process of individual reflection and recollection of everyone's own learning journey. Although the fieldwork also included detailed and intensive individual work to capture each person's own experiences, the data and findings of the group element of the fieldwork could not be individualised for the purposes of the Realist Evaluation approach (which works on the basis of highly individualised responses and views group work as a methodological weakness). The group findings were used to report at programme level and to be a consistency check with the individual data, whereas the individualised findings (delivered by the 'Mini Interview' session) were the source for coding findings in a more Realist fashion.
• A total of four researchers were involved in coding. This required a detailed understanding of the Realist Evaluation terminology and care to calibrate each other's interpretation of the terms in light of the mini interview reports from 120 individuals participating in this aspect of the study. Considerable effort was taken to prepare all researchers in-depth and to find a joint level of understanding of the terms used to avoid variations in coding. The 'mini interviews' were conducted and written up individually or in pairs by participants as part of the workshop in the presence of researchers but using the participants/beneficiaries own language. These were subsequently coded by the researchers according to the outcomes and mechanism identified in the prioritised CMO-C/theories. Participants told their own personal outcome stories/journeys rather than being asked exactly the same questions. This meant there was not full and consistent data sets for each CMO-C/theory. This, as well as further limitations in timing and funding, meant that the analysis sought out patterns in contexts and mechanisms according to the presence or absence of outcomes. It did not use emergent methods such as Realist Evaluation-specific Qualitative Comparative Analysis or Contribution Tracing to further validate these patterns/associations. Data on projects' contexts were gathered from the mini interviews as well as the preparatory interviews with the project main contact and/or from project applications and monitoring reports. As such our findings formulate theories that will require further research and validation.
• The study included beneficiaries who had only recently joined the PCF initiative ^[14] . Although they were able to report on early improvements, they all expected more change to happen in future (this included aspirations such as taking up a voluntary position in the Community Anchor, pursuing training objectives, finding a job, setting up their own social enterprise).
• Not all mechanisms or outcomes may have been explicitly reported in the mini interviews, but may still have been experienced and reported in the group sessions of the workshops as part of the study (or remained unreported).
• The same range of mechanisms were mostly present whether an outcome was reported or not - however the frequency of them varied. Our analysis is based on the frequency not present in every case or complete absence of the mechanism. The association would be stronger were the latter the case, however, in a complex multifaceted intervention, such as PCF, this may never occur. The same issues are true in relation to the analysis of contexts linking to mechanisms. We should, therefore, acknowledge the parallel contributions from the other reported mechanisms and contexts to outcomes triggering mechanisms respectively.
• As agreed in the first ToC workshop, the Realist Analysis focused on the short-term (or early) outcomes as anticipated by the PCF.
• Beneficiaries attending the workshops were selected by the projects and all reported positive outcomes. This confirms that many positive changes can be achieved with engaged participants. Study data/analysis, however, tells little about the impact on the less engaged, or in terms of project reach or scalability. Many of the participants were embedded within projects and had long-term relationships (over one year long). It is likely that Community Anchor organisations would need substantial time to achieve a relationship of this nature. Additional timeframes would also be necessary if these relationships are being delivered as part of sustained services alongside wider reach and population impact. Delivering intense bespoke support to large numbers of clients, target groups or wider populations is likely to be very challenging.