Child poverty pathfinders - early implementation process: evaluation

Chapter 2: Evaluation methodology

This section describes the design of the evaluation and the methods used. The first part explores the methods used to engage with Pathfinder partners and stakeholders. It then outlines the methods used to engage with families and explores the limitations of the research.

2.1 Partner and stakeholder interviews

The evaluation team worked with the Scottish Government to identify and define stakeholder groups. Partners were defined as those who work in key strategic roles within organisations who had direct involvement in the strategic board involved in the development of each Pathfinder. Stakeholders were individuals who occupied key roles within local delivery services. While most stakeholders did not have direct involvement in board level decisions, they had high levels of involvement in the development of the Pathfinders at a local level.

Rocket Science then worked with Scottish Government to identify a list of key stakeholders and partners to approach for research interviews. The lists were sent to Pathfinder leadership teams to secure agreement concerning who to approach, to maximise participation. Pathfinder leads were also involved in the identification of key people to include in the partner and stakeholder interviews. Therefore, the participant selection was identified purposively with the people most likely to be able to answer the research questions selected for inclusion. The number of stakeholder and partners who agreed to take part are presented in table 1 below.

The Rocket Science team developed interview topic guides for use with Pathfinder partners and stakeholders. The topic guides for each group are provided in appendix 2. The interviews followed a semi-structured design. This meant that while the topic guide was there to ensure that all relevant topics were discussed, there was sufficient flexibility built in, so that both groups (partners and stakeholders) could share their views on what had worked well, and what had not worked so well in the development of the Pathfinders. The interviews explored experiences of developing the programme and examined how learning was being used to influence replicability. The interviews also examined views on the scalability and sustainability of the Pathfinders.

Those identified for interview were contacted by Rocket Science researchers, and invited to attend an online interview. The interviews lasted approximately 45 minutes to 1 hour and took place between November 2022 and February 2023. In total, 38 stakeholders and partners received invitations to participate in interviews. 30 stakeholders and partner interviews were conducted across both Pathfinders. Participation in qualitative interviews was higher in Dundee (n=18) than in Glasgow (n=12). One additional source of data collection occurred as one partner could not find time for interview but expressed a desire to contribute views in writing. In this case, the interview topic guide was provided via email, and the written contributions have been included in the qualitative analysis.

Researchers took detailed notes during all interviews. This included conducting live verbatim transcribing and writing detailed reflective notes immediately after the interviews had taken place, to capture key insights. Transcripts and researcher notes were uploaded into software package MaxQDA. This enabled the evaluation team to engage in line by line thematic coding, which was developed into a coding hierarchy. The evaluation team then met to discuss themes and codes, which supported the refinement and further development of the thematic coding structure.

2.2 Partnership Scorecard

Rocket Science developed a bespoke partnership scorecard for the Pathfinder programme (see appendix 3). Partnership scorecards are online questionnaires that follow a standard framework, to track performance against pre-set, pre-agreed research questions. The partnership scorecard was developed via collaborative discussions with the leads of each Pathfinder. The scorecard was designed to allow partners to rate, reflect on and provide feedback on collaboration and joint working, the aims of the Pathfinder, the aspects that had been working well, key learning and how the Pathfinder could be improved.

The scorecard was scripted and then uploaded onto the online survey platform, SmartSurvey. A link to the Partnership scorecard was sent via email to Pathfinder partners at national and local level on 25 November 2022. Follow-up reminders were sent on two occasions, in December 2022 and January 2023, to those who had not yet responded. The Partnership scorecard received responses from 25th November 2022 to 18th January 2023.

Responses to the partnership scorecard were higher among Glasgow Pathfinder partners than among Dundee Pathfinder partners. Invitations to complete a scorecard were sent to 29 partners across both sites and 16 participants took part in the partnership scorecard as outlined in Table 2, below.

The higher participation in Glasgow, compared to Dundee should be taken into account when reading the results of the partnership scorecard data.

The Partnership scorecards consisted of a series of Likert scales comprising graded statements from which people could select a response based upon their experiences. The aim was to analyse responses as percentages, enabling the provision of charts and other data visualisation methods. However, the scorecards were not sent to large numbers of people, but rather, were sent to specific partners that were identified by Scottish Government and Pathfinder leads.

These participants were purposively sampled and were selected because they were close to the Pathfinder and would have knowledge of the design, set up and delivery phase of the Pathfinders and would be able to share insights on how systems change was occurring. Because of the low numbers of participants, quantitative analysis was not possible, yet the insights shared are valuable to understanding the development of the Pathfinder. To that end, the partnership scorecard data was analysed qualitatively and iteratively to identify how it sits within, and supports, the qualitative findings. The scorecard results have been used in the discussion of strengths and areas for development.

2.3 Parent/carer interviews

As described in chapter 1, the Pathfinders aimed to reduce child poverty, and so the perspectives of parents/carers were crucial to capture. Throughout this report, we use the term 'family' when referring to the household impact of the efforts being made via the Pathfinder. We use the term parent/carer to refer to people who have dependents aged 16 years or under, and conceptualise this broadly to include parents, carers, guardians, kinship carers and foster carers.

At the Dundee site, a researcher visited the Pathfinder hub on two occasions and invited individuals attending to consent to interview. In Glasgow, Pathfinder staff identified and invited engaged individuals to be interviewed.

In both sites, NIHR Guidance was followed on compensating people with lived experience of services for their time spent taking part in research. This meant that all research participants received £25 in gift vouchers, to compensate for their time and to thank them for participating in the evaluation.

In total, 36 interviews were completed with parents/carers (n=20 in Dundee, n=16 in Glasgow). Further details on the approach used to recruit parents/carers in the Glasgow and Dundee Pathfinders, as well as the demographics of the parents/carers, is included in appendix 4.

2.4 Qualitative analysis

Researchers took detailed notes during all interviews. This included conducting live verbatim transcribing and writing detailed reflective notes immediately after the interviews had taken place, to capture key insights. Transcripts and researcher notes were uploaded into the software package MaxQDA. This enabled the evaluation team to engage in line by line thematic coding, which was developed into a coding hierarchy. The evaluation team then met to discuss themes and codes, which supported the refinement and further development of the thematic coding structure.

Reflections on limitations to the evaluation are included as appendix 5.