External Review of the National Carer Organisations (NCOs)

Annex 4: Methodology

The methodology used for the review was designed to reflect the overall aims and objectives, and specifically the need to identify options for the way forward to meet the related and complementary needs of carers, the NCOs and the Scottish Government.

It was also important to reflect the complexity and diversity of the nature and operation of the NCOs and the pattern of support to carers, and to recognise the need to explore the operation of individual NCOs alongside the operation of the Network. It would have been impossible to achieve the objectives of this review without taking an overview of the general nature of support provided to carers in Scotland (and identifying any points of overlap and distinction), as well as the nature of individual NCOs' input to relevant policy (both local and national).

In order to do this, there was a need to ensure that the views of all of the relevant stakeholders were identified. As such, a number of complementary research techniques were used, with a combination of quantitative and qualitative work. The methods included:

• Examination of documentary information.
• Development of a template and database.
• Interviews with a range of stakeholders.
• Group discussions with a range of stakeholders.
• A written "consultation".
• Analysis and reporting.

Each aspect of the methodology is described in more detail below.

Documentary evidence

The first stage in the process involved examining existing key documentary information, to inform the development of the research tools. It was vital, at the earliest stage, to take account of the initial scoping study report prepared by the Scottish Government consultancy team. This provided specific information about the function and operation of the NCOs, the current Network, relationships and joint working, the scope for rationalisation, and the funding relationships. It also helped to avoid duplication in the information gathering.

A range of other documentary material was also examined in the course of the review, providing further contextual and operational information (both at a general level and in relation to particular NCOs). This included, for example:

• Policy and strategy documents.
• Minutes of meetings.
• Annual reports, evaluations, research.
• Funding applications and agreements.
• Website information.

Template for comparison

As noted above, it was considered important, as part of this work, to examine the actual operation of the individual NCOs and to be able to compare the services provided (particularly in terms of identifying points of overlap and distinction). In order to do this, information was collected from a range of sources. Some was gathered from existing documentary information (e.g. the scoping study). Some was gathered in discussion with the NCOs. Other information was gathered using a template which was provided to each of the NCOs.

The information sought via the template included, for example (where such material was not already available from other sources), data about issues such as:

• Structure and governance.
• Perceived "national" and "local" roles, and perceived "representational" roles.
• Resources available (staff, practical and financial), and premises.
• Sources and amounts of funding, as well as revenue generation.
• Size and nature of their membership (individuals and organisations).
• Nature of clients, service provision and use (include revenue earning services).
• Online communication methods.
• Research capacity.

The material from the template was then analysed and compared. From this, RHA developed a matrix as a means of understanding the information. This provided a clear exposition of which NCOs were involved in each type of work, which were fulfilling particular roles etc.

It had originally been hoped also to examine and analyse the NCOs' existing database information and map their membership. This proved impossible, however, as a result of data protection issues. Instead, membership patterns and key issues were explored through the written consultation process described later.

Interviews with stakeholders

A series of face to face interviews was held to explore the overall views of the senior managers of the NCOs and other relevant stakeholders.

NCOs

Face to face interviews with the overall Manager / Chief Executive (or equivalent) of each of the NCOs formed the first stage of gathering stakeholder views. These interviews explored issues relating to the overall strategic direction of support to carers, in the context of this review. The interviews covered issues such as:

• Overall purpose and expectations of the NCOs.
• Role of the NCOs in the pattern of provision of support to carers and input to policy.
• Links between NCOs, and links to other organisations.
• The current NCO Network, its role and effectiveness.
• Key requirements of Government and service providers.
• Current issues in service provision (e.g. efficiency and effectiveness) and any recent changes impacting on NCOs.
• Funding issues.
• Constraints faced, potential future developments and improvements.

Other stakeholders

Interviews were also carried out with a sample of other stakeholders, to identify their views of the operation of the NCOs, particularly their interface in the context of carers' needs. Participants were chosen to reflect different types of organisation, as well as different geographical areas and different service user groups (e.g. disabled people; elderly people; children and young people; ethnic minority groups etc.) and organisations with a specific relevant focus. Interviews were also carried out with representatives of local authorities and the NHS.

These interviews covered issues such as:

• The role of the organisation in the overall pattern of provision of support to carers and input to carer policy.
• Overall purpose and expectations of the NCOs.
• The organisation's links to NCOs and cross-referral.
• Perceptions of gaps and distinctions in provision.
• Perceptions of the current NCO Network, its role and effectiveness.
• Impact of the Network on other stakeholders.
• Current issues in service provision to carers and carer policy.
• Potential future developments and improvements.

Group discussions

In addition to the interviews, a number of workshops and group discussions were carried out to gather the views of a range of stakeholders, including:

• National staff, Boards and management groups of NCOs.
• Other workers and carers.
• The NCO Network.

These were facilitated by RHA, and detailed notes taken in each case.

National staff, Boards and management groups of NCOs

Group discussions were held with the national staff members of NCOs, as well as the NCOs' Boards and management groups.

Separate meetings were held with the staff and Board members or equivalent (brought together with the assistance of the NCOs). These discussions covered issues such as:

• The strengths of the NCO.
• The purpose of the NCO, and the specific and distinct services they provide.
• Patterns of membership (both individual and organisational) and perceptions of strengths, weaknesses and any silent voices.
• Perceptions of gaps in provision to carers in Scotland.
• The role of the organisation in the overall pattern of provision of support to carers and input to carer policy at national, and local levels.
• Perceptions of the representational role of their own, and other NCOs.
• Relationships with other NCOs and cross-referral / signposting.
• Current issues for service providers and clients (including constraints).
• Awareness and perceptions of the current NCO Network, its role and effectiveness.
• Impact of the Network on carers and other stakeholders.
• Key aspects of future provision.
• Current issues and concerns in service provision to carers and carer policy.
• Potential future developments and improvements.

Other workers and carers

Opportunities were also provided for other relevant workers and carers to input their views, through a series of local workshops and group discussions held in different parts of Scotland. These were arranged with the help of the NCO Network and individual local carer organisations. Meetings were held in the following areas:

• Dumfries.
• Dundee.
• Edinburgh.
• Galashiels.
• Glasgow.
• Kilmarnock.
• Perth.

Discussions were also held with young carers; workers with BME carers; and workers with carers in remote and rural areas.

These discussions involved a total of 94 participants. The issues examined were:

• Points and means of entry to receipt of support and the 'carer journey'.
• Overall purpose, expectations and requirements of information and support.
• The provision of a "voice" for carers.
• Constraints and limitations to provision.
• The role of NCOs in support and representation of carers.
• Suggested improvements to the means or pattern of provision by NCOs.
• Views of the way forward.

The NCO Network

Three joint discussions were held with members of the NCO Network, and a specific discussion was held with the Research Sub-group to enable their input to the written consultation (described below). The Network provided comments on the draft survey materials, as well as assistance with their circulation.

The meetings focused on providing an opportunity for the NCOs to input to the research process as a group (in addition to the interviews and discussions held with the individual NCOs), but also (particularly in the case of the final meetings) on exploring their views of the way forward.

These meetings were arranged through the administrative process for bringing the Network together, and facilitated by RHA.

Written consultation

It was also considered vital to ensure that all of those with a potential interest in the issues were given the opportunity to contribute to the review. For this reason, in addition to the targeted approaches set out above, a wider written survey or "consultation" was undertaken.

The written consultation was carried out through the use of an online "SurveyMonkey" questionnaire method between early April and mid-May 2014 (to ensure that this strand of the work did not coincide with the Scottish Government consultation on carer legislation held in early 2014).

Two separate questionnaires were made available: one to individuals with an interest in carer issues (both carers and workers); and one to organisations working with carers. These were designed in draft and discussed with the NCO Research Sub-group, whose comments were incorporated. Printed versions of the questionnaires were also made available on request.

The online questionnaires focused on the following issues:

• Information, support and the carer "voice".
• Knowledge of, involvement with, and perceptions of NCOs.
• Optimum role and functions of the NCOs.
• Suggestions for the future.

Details of the surveys were circulated widely by the NCOs, and a total of 271 responses were received (231 from individual carers or workers; and 40 from organisations).

Analysis, identification of options and report writing

Following the collection of all of the information, the material was analysed using a combination of quantitative and qualitative techniques. Some of the information (e.g. the audit and documentary information) involved a straightforward summary of the material, in order to provide the general context. The comparative material from the NCOs was analysed using a matrix created for this purpose. A small amount of the online survey material was analysed quantitatively.

Most of the material gathered, however, (e.g. material from the individual interviews and workshops / group discussions, as well as much of the material from the online surveys) required a qualitative approach. This involved working systematically through the material gathered from each source relating to each of the issues explored in the review. A series of themes and sub-themes was identified for each of the issues, including arguments and suggestions made in each case.

This approach enabled the range and depth of the qualitative views to be reflected accurately and comprehensively, and ensured that the analysis followed the material, rather than being fitted into pre-defined categories. A wealth of detail was identified from this qualitative analysis which informed the final report.

All of the themes raised were presented qualitatively, to reflect the nature and purpose of the review (i.e. to identify the range of views and suggestions about the NCOs). Any attempt to "count" or "weigh" the views of specific issues within the overall themes would have been inappropriate (and might have undermined the importance of views of smaller groups of respondents, including the NCOs themselves). Within this overall approach, however, the nature of respondents raising particular issues was identified wherever appropriate.

From the findings of the review, a number of options and suggestions for the future were identified and presented within the report.