The Opportunities and Challenges of the Changing Public Services Landscape for the Third Sector in Scotland: A Longitudinal Study Year Three Report (2009-2012)

APPENDIX C: METHODOLOGY

The methodology involved qualitative research within 20 third sector organisations based in Scotland. The methodology involved two key components: (1) in-depth case studies with eight TSOs and; (2) three focus groups involving twelve additional TSOs.

A qualitative longitudinal approach was used to ensure that the complex and fluid experiences of participants over a three-year period were reflected in the report. A qualitative approach ensured that interviewers could explore important and sensitive issues in depth with research participants. The importance of understanding and reporting changes over time within third sector organisations and how those changes are embedded in patterns of social, economic and political change has meant that a broad range of issues were explored during interviews between researchers and participants.

Case studies for year one (baseline) were carried out between December 2009 and May 2010 and for the focus groups between April and June 2010. The results were reported in the year one report.

Case studies for year two were carried out approximately one year after the first visit with organisations (between January and June 2011). Focus groups were carried out at six monthly intervals following the baseline meeting, between October and November 2010 and April and May 2011. In addition, a workshop was carried out in June 2011 at Edinburgh University Business School, which is also used as evidence for year two.

Case studies for year three were carried out between January and May 2012. Focus groups were carried out between October and November 2011 and between January and March 2012.

Initial selection of case study and group work organisations

Following discussions with the Scottish Government and the Research Advisory Group, a framework was developed for the selection of research participants. This was designed to ensure the establishment of a purposive sample of organisations working in different:

• policy areas (with a mix of social care, healthcare, and employability/economic development/regeneration providers)
• geographies (based in different locations across Scotland)
• scales (with a mix of larger and smaller organisations included)
• and to include some social enterprises

The selection of focus groups was based on similar lines with individual focus groups bringing together organisations with strong agendas in the following areas: (a) equalities; (b) social care and health care, and (c) employability/economic development/regeneration.

Potential participants were identified through a database of 685 possible organisations provided through the Scottish Council for Voluntary Organisations (SCVO). The final selection of possible organisations was made in order to achieve the balance required by the framework (above). All organisations were then contacted and invited to take part in the research for a period of three years as either: (1) a case study carried out once a year; or (2) to participate in a focus group carried out twice a year. Most first choice organisations were happy to participate, with the few who declined being replaced by other suitable organisations. In this way, the baseline sample of organisations was obtained.

In-depth case studies

The case studies included face-to-face interviews with staff at different levels of the organisation. These included: chief executives; other senior officers/managers; research/policy officers; business/planning managers; operational and line managers; front line staff delivering services. The selection of staff for interview was decided in consultation with the main contact from the organisation (usually the chief executive or another member of the senior management team) and actual staff interviewed varied depending on the size of the organisation and availability of appropriate functions. A list of interviews carried out within each organisation for each year can be found in Appendix D. A copy of the main interview schedule used in year three is attached in Appendix F.

In years two and three the main contact in each organisation was approached and asked for an interview and permission to follow up staff contacted in year one. If original staff were not available for interview, where possible, other staff covering a similar role were identified and interviewed. In one organisation access was restricted to a smaller number of participants due to resource issues. A very small number of staff either did not reply to invitations to be interviewed, or there were difficulties in arranging available interview dates within the research timeframe. All eight organisations originally involved were able to participate for all three years.

Focus groups

Twelve organisations were divided into three focus groups of four participants. Each focus group pulled together organisations with strong interests in particular areas. These included: (a) equalities; (b) social care and health care, and (c) employability/economic development/regeneration^[41].

One representative from each organisation (usually the Chief Executive or a member of the senior management team) attended the focus groups. Where an organisational representative was unable to attend the focus group, telephone interviews were conducted. A common discussion framework was used - see Appendix F.

Over the three years, two organisations were unable to continue their original participation. Following the focus groups in year one, one participant left the TSO and her successor was unable to continue the participation on behalf of that organisation. However, another organisation was approached and agreed to take part in the focus group research. In year two, one organisation had merged with another TSO not involved in the research. The new TSO was approached and agreed to take part in the research in year three on behalf of the original organisation.

Anonymity

In order to protect the anonymity of individual respondents who took part in the research, quotes have been labelled with generic job titles (e.g. Senior Manager, Manager, and Officer). A brief description of the type of organisation is also provided after each quote. Additional background information on the participating organisations is provided in Appendix E. This is intended to give context to the overall report and individual quotes without revealing the identity of participating organisations. All organisations were happy to be identified as taking part in the research (although not necessarily to have particular opinions credited to them). All participating organisations approved the approach to anonymisation that has been used. A full list of participating organisations is available on the project website at http://www.thirdsectorproject.org.

Analysis

Data from each year was analysed using the Framework analysis method^[42]. This is a 'matrix-based method for ordering and summarising data' (Lewis, 2007: 550). Key themes from findings are identified and divided into sub-topics. Matrices are then drawn up in Microsoft Excel, each representing a different sub-topic. Columns represent sub-topics and rows represent participant interviews.

The first stage of the qualitative longitudinal research was to apply the Framework and identify significant topics and sub-topics. This provided a flexible common core framework that enabled 'comparability over time and between projects' for which 'the use of common data collection tools and reproducible modes of analysis are suggested'^[43]. Thematic analysis has been carried out on individual focus group and case study data collected in each year. This analysis provided the basis for the longitudinal analysis as well as providing more detailed cross-sectional data, including quotes, which are used in the report.

For the longitudinal analysis, later data were then added to the earlier data within the Framework enabling accounts provided by different respondents at different points in times to be compared.

Individual focus groups and case studies were subjected to a longitudinal analysis comparing data collected in the waves. This was done for each group/case study in Microsoft Excel. This data was then integrated (in Microsoft Excel) across all groups and case studies. In this way, a summary analysis of key changes between the waves was enabled and forms the basis of this report.

Methodological challenges

The longitudinal qualitative nature of the research presents an unparalleled opportunity to track the dynamic of change over time. However, this also presents challenges including issues of attribution, policy changes and attrition.

Attribution is being able to attribute changes to a specific cause. However, TSOs in Scotland operate within a complex and changing policy context, with policy emerging from different levels. The UK and Scotland level are particularly important for policy, but also policy emerging from (and interpreted through) local authorities, regulatory bodies and Europe form part of a complex background. There is also often a time lag between policy announcements and the actual impact on TSOs since these can be mediated via other bodies (e.g. local authorities in particular). Where possible the report identifies the key links, but it would be too simplistic to assume that causal links always exist or that they are straightforward in nature.

Over the course of a number of years, it is likely that the circumstances of organisations or individuals may change in a way that means they can no longer continue in the research (attrition). Fortunately, attrition had been minimal. Of the 20 organisations taking part in year one (baseline), only two focus group participants were unable to continue and replacements were found for both.

There was some attrition of interviewees within case study organisations. This was because of staff leaving the organisation, being unable to set up interviews within the timeframe or because access was restricted by the key contact. Where possible, alternative participants were interviewed.