Third Sector Interface network model and Voluntary Action Scotland: evaluation

5. How effectively is Voluntary Action Scotland (VAS) fulfilling its role as the intermediary body for the Third Sector Interface (TSI) Network?

Overview

In addition to evaluating the effectiveness of the Third Sector Interface model, we were tasked with evaluating the effectiveness of Voluntary Action Scotland ( VAS) - the intermediary body representing the 32 Third Sector Interfaces - and to consider what form any TSI network intermediary body would need to take in future. In this chapter we outline our findings in relation to VAS's effectiveness to date, and in the final chapter of this report we explore the significance of these findings for any future model of support.

Background

VAS's role is to develop, support and represent the Third Sector Interface network through:

• promoting the positive impact that the Third Sector Interfaces have at local level;
• encouraging good practice;
• raising the profile of the Third Sector Interfaces at national level; and
• facilitating peer support to the network.

VAS was first conceived in 2009, not long after the TSI model itself was introduced.

The organisation receives approximately £250,000 annually from Scottish Government. VAS is a small organisation and although it has existed for seven years, compared to other national intermediary organisations it is still a relatively young organisation.

VAS's vision is for "A Scotland based on fairness and equity with a thriving Third Sector at its heart." Its mission is "Championing, connecting and developing the network of TSIs to support their contribution to a Scotland based on fairness and equity." It seeks to deliver this vision through three key objectives which are outlined in the next section.

Key objectives

Voluntary Action Scotland has three key objectives which are to:

• champion the role, impact and interests of the TSIs, Third Sector and volunteers they support;
• connect TSIs to each other, national stakeholders, Government and key policy issues; and
• develop VAS as an effective peer-led network organisation able to support and challenge TSIs.

Is Voluntary Action Scotland ( VAS) achieving its objectives?

In this section we explore the extent to which VAS is delivering on these objectives and any challenges that are being encountered.

Views from the TSI network

First and foremost, VAS is accountable to its members and we therefore sought to establish the extent to which members feel that it is meeting its objectives. We also consulted with staff and boards in the indepth study areas.

The results of the survey we conducted with Chief Executive Officers and Chairs of TSIs during our consultation process indicate that views are very mixed in relation to the effectiveness of VAS in delivering on its objectives. As would be expected, amongst the 32 TSIs, individual experiences of the services being provided by VAS vary depending on whether the services meet their needs, what services they and their staff have accessed to date, and the closeness of their relationship with VAS. Some TSIs have very pro-actively engaged with VAS, others less so - sometimes through choice, and sometimes because they do not think that VAS has reached out to them enough.

The annual conference was seen as a helpful way of meeting others in the network, as were the practice forums for some, but attendance at VAS events did not appear to be a priority among those interviewed, and there was a strong message from research participants that VAS events were not sufficiently accessible to TSIs outside the central belt.

Some of the TSI staff and board members in the depth study areas often had little knowledge about VAS and what it did.

Lead officers have the greatest connection to VAS, and welcomed the re-introduction of more regular Chief Officer meetings, but there were mixed views of how useful participation in the recently revived TSI Chairs' network has been.

We have endeavoured within these next sections to give a sense of the range of these views.

Objective 1: Championing the role, impact and interests of the TSIs, Third Sector and volunteers they support

In our survey of TSI Chief Executive Officers and Chairs we asked them to specify the extent to which they agree with the statements in the table below:

Table 5.1 TSIs' views of impact in championing the role, impact and interests of the TSIs, Third Sector and volunteers they support

	Base	Not at all	A little	To some extent	To a great extent
VAS has increased the profile and credibility of the TSI Network with national stakeholders	66	2 3%	21 32%	37 56%	6 9%
The contribution of the TSI Network is better understood by national stakeholders because of the role of VAS	66	3 5%	19 29%	37 56%	7 11%
The policy interests and priorities of TSIs are better represented as a result of VAS's role	65	6 9%	21 32%	24 37%	14 22%
The value and role of volunteers in local delivery is better understood at a national level as a result of VAS's input	64	10 16%	20 31%	30 47%	4 6%

The responses received through the survey show a very mixed response to perceptions of VAS's impact in relation to championing the role, impact and interests of the TSIs, Third Sector and volunteers:

• Slightly more than half of respondents (56%) indicated that they felt VAS has to some extent increased their profile and credibility with national stakeholders and helped them to better understand the contribution that the TSI network makes. Only a minority felt that VAS had achieved either of these impacts to a great extent.
• 59% indicated that as a result of VAS's role their policy interests and priorities are better represented, but 41% thought this had only happened on a limited scale or not at all.
• 53% indicated that VAS had some success in helping the value and role of volunteers in local delivery to be better understood at a national level, but 47% thought VAS had had little or no success in doing so.

Stakeholders and policy influencers also expressed concern that VAS had made little impact in championing the role of the TSI although it was recognised that there were examples of good practice ( e.g. Supplementary Guidance on the role of TSIs in Health and Social Care Integration).

Objective 2: Connecting TSIs to each other, national stakeholders, Government and key policy issues

Table 5.2 TSIs' views of impact in connecting TSIs to each other, national stakeholders, Government and key policy issues

	Base	Not at all	A little	To some extent	To a great extent
TSIs are more connected and know more about the work of other TSIs in the Network	66	6 9%	19 29%	24 36%	17 26%
TSIs have greater access to expertise from within the TSI Network	66	5 8%	22 33%	27 41%	12 18%
TSIs are more connected with national Third Sector intermediary organisations/initiatives that support the Third Sector	66	8 12%	28 42%	25 38%	5 8%
TSIs have greater access to expertise (to support the Third Sector) from national Third Sector organisations (and national bodies) that support the Third Sector	66	12 18%	22 33%	29 44%	3 5%
TSIs have increased access to knowledge and expertise on national policy priorities	66	7 11%	20 30%	25 38%	14 21%
TSIs have been able to contribute to national policy development	65	5 8%	25 38%	25 38%	10 15%
TSIs have been able to influence national policy development	65	7 11%	24 37%	29 45%	5 8%

Again, responses are very mixed showing very disparate views about the effectiveness of VAS in this role, notably:

• 54% indicate that VAS has had little or no success in connecting TSIs to national Third Sector intermediary organisations or initiatives;
• 46% indicate that VAS has had little or no success in supporting TSIs able to contribute to national policy development; and,
• 48% indicate that VAS has had little or no success in supporting TSIs to able to influence national policy development.

Objective 3: Developing as an effective peer led network organisation able to support and challenge TSIs

We asked Chief Executive Officers and Chairs to rate the extent to which VAS's work has influenced the way their TSI works.

Table 5.3 TSIs' views of impact in developing as an effective peer led network organisation able to support and challenge TSIs

	Base	Not at all	A little	To some extent	To a great extent
We share our learning/practice with other TSIs	65	5 8%	22 34%	27 42%	11 17%
We have learned from other TSIs	64	4 6%	25 39%	25 39%	10 16%
We have changed our practice as a result of things we've learned from the network	63	16 25%	25 40%	13 21%	9 14%
We have collaborated with other TSIs to develop new projects/processes/services	65	12 18%	23 35%	22 34%	8 12%
Learning from the network has increased our capacity to support the Third Sector at the local level	65	12 18%	26 40%	21 32%	6 9%

Again, the responses are mixed, but while more felt that VAS had been to an extent successful in supporting TSIs to share their learning (59%), there was little evidence of changes in practice - 65% of respondents indicated that there had been little or no change to their practice as a result of things they have learned from the network, and 58% did not think that it had significantly impacted on their capacity to support the Third Sector locally.

Quality of Voluntary Action Scotland services

Through our survey we sought to establish levels of satisfaction with the quality of services being delivered by VAS. The table below shows the ratings given to each of the services by respondents to the survey.

Table 5.4 Quality of VAS services

	Base	Very good	Good	Fair	Poor	The TSI does not engage with this service
TSI Practice Forums	61	13 21%	32 52%	13 21%	1 2%	2 3%
Other practice events	54	10 19%	21 39%	16 30%	1 2%	6 11%
Conferences	64	17 27%	27 42%	14 22%	4 6%	2 3%
Ad-hoc advice and support	62	13 21%	21 34%	12 19%	2 3%	14 23%
Use of the Knowledge Hub	57	3 5%	8 14%	17 30%	10 18%	19 33%
Use of policy briefings	62	18 29%	16 26%	20 32%	6 10%	2 3%
Newsletter/bulletin	63	14 22%	30 48%	17 27%	1 2%	1 2%

The majority of responses are clustered between good and fair ratings, but with a significant minority of respondents rating the Knowledge Hub, for example, as poor.

A significant percentage do not currently use some of the services on offer - particularly the ad hoc advice and support, and the knowledge hub:

• Almost a quarter do not engage with VAS for ad hoc advice and support and a further fifth regarded this service as fair to poor.
• This suggest a lack of confidence in the quality of advice and knowledge held at VAS.

In VAS's own progress survey, last conducted in 2015, around half of respondents rated the services overall as a 4 out of 5. This is an improving picture since the survey was first conducted in 2013, but poorer ratings were given in critical areas:

• practice development support needs (only 37.71% rating the support highly), and
• supporting the TSI to play a role in community planning locally (only 38.71% rating it highly).

We do not expect these findings to be of surprise to VAS. Staff and board members consulted during our evaluation recognise that further development of services is needed to better meet the needs of the network and indicated that these were a priority for development.

In the next section we summarise those areas where VAS has been most successful and note areas where further improvement is required. The summary reflects views gathered across the research participants we consulted, including TSI survey respondents, depth study area participants and stakeholders (where they were sufficiently familiar with VAS to comment).

Key achievements

Throughout the evaluation we asked participants to share with us their experiences of the support provided by VAS. Key achievements identified by them included:

• Some pro-active development work with partners such as the work with Community Planning Partnerships and the Alliance to broker better understanding of the TSIs among national Third Sector organisations, although more needs to be done in this area.
• The development of additional guidance to Health and Social Care Partnerships on the role of the TSI in relation to integration activities.
• Production of written publications that are of a high standard e.g. high quality report on "Collaborating for Community Impact".
• Work with Quality Scotland to help TSIs to achieve the Committed to Excellence Award.
• Practice forums which are seen to be relevant and improving.
• A high quality, well-organised annual conference that is both informative and, more importantly perhaps, an opportunity for a wide range of people involved in the TSI at different levels and in varying roles to come together and exchange views and experiences, although as we noted previously the priority placed on attending the conference varies from one TSI to another.
• Improving connections to civil servants and politicians.
• Improving understanding within Community Planning Partnerships of TSIs through the Community Planning Improvement Programme.

While progress has not been as fast as most stakeholders would have wished, it is notable that most stakeholders described VAS's performance as "improving".

Areas for improvement

Research participants also identified a number of areas where they believed VAS had performed less well, and are areas for improvement:

Connecting the network

• VAS needs to better connect with each TSI, but there is also a need for VAS to engender a greater sense of a connected network. Many TSIs did not feel part of a bigger network.
• Some regional connections exist between neighbouring TSIs and this is something that could be fostered by VAS.
• More presence is needed in local areas and more contact from VAS - "having been on the board, that seems to be the privileged position, for the majority of the network this is fleeting at best, connection is very much part of individual effort and connectivity".
• VAS is too central-belt focused.

Promoting the network and having an influencing role

• A sense that VAS had not achieved parity of status with other national intermediaries and did not have influence.
• Lack of success in increasing the knowledge, awareness and understanding of national Third Sector intermediaries about the role of TSIs in local areas.
• Need for more focus on policy development and influencing policy change.
• Absence of a well-defined communication strategy between VAS and the network and externally.

Supporting and developing the network

• Need for more sharing of good practice within the network to improve quality.
• Concern about the lack of support for failing TSIs. VASs role and capacity to intervene is unclear but is a priority for the network.

Effectiveness of the organisational structure

Governance

VAS's board comprises a mix of Chief Officers or trustees of TSIs who are elected by the network (each TSI having one vote) and independent members who bring specialist expertise to the board, for example in strategy development. Independent members are recruited through an open recruitment process. At the time of writing we understand that plans are in place to recruit additional independent members imminently.

VAS staff and board members themselves acknowledge that historical governance arrangements have not served them as well as they hoped and that improvements are necessary. This view is strongly reinforced by those members of the network we heard from.

This has led to a situation where self-interest is sometimes preventing vital discussions being pursued (for example, outcomes of discussions around the allocation of Scottish Government funding to TSIs would directly affect individual TSIs and so an objective discussion at Board-level becomes impossible), and agendas being driven which do not necessarily reflect the views of the broader membership.

Challenges in relation to governance have to date included ensuring a clear operational and strategic split of responsibilities; ensuring there is no geographical bias in membership; and ensuring that members are acting on behalf of the network and not representing their own interests.

We know that action is already underway to address some of these concerns including:

• At a recent board meeting, the decision was taken to recruit more independent members.
• To date, board membership has been reviewed annually, but at a recent Annual General Meeting a motion to appoint members for a three-year period was agreed.
• Trying to ensure diversity of experience on the board, with new members being sought to address gaps in expertise.
• Gradually re-introducing portfolios for board members to ensure that they can tap into expertise and share the burden of representation at events etc..
• Implementation of 3 short-term working groups to address governance, policy and health and safety. The membership of the working groups also draws on the expertise of the wider membership.
• Plans to introduce a corporate policy sub-group of the Board.

It is too early to say whether these actions will be effective, but they give a sense that the VAS board recognises the challenges and is taking responsibility for improving governance.

Accountability

As a membership body, VAS's accountability is to its members. We asked Chief Executive Officers and TSI Chairs, and staff in the selected study areas whether they feel that VAS is accountable to its members. Feedback was very mixed.

• Some respondents were positive about VAS's accountability, with 22% stating that it was accountable "to a great extent";
• " VAS is excellent with respect to being accountable. It fully involves TSIs in all aspects of its work and governance";
• 51% indicated that they felt that VAS was accountable to some extent;
• Others were more negative, with 27% of respondents stating that they thought VAS was only accountable to its members to a little extent or not at all; and,
• "As a Chief Officer not involved in the VAS board I feel somewhat disconnected from decision-making and scrutiny of VAS".

Issues and challenges for Voluntary Action Scotland

We have identified a number of issues and challenges which have impacted on VAS's performance. These are issues which need to be addressed in any future intermediary model.

Organisational maturity

There is a sense of VAS as a relatively new organisation. TSI representatives and stakeholders alike recognised that it can take time for a new organisation to embed itself, but there is a sense that the organisation lacks leadership and strategic direction.

Governance

In the period since its inception, VAS has experienced frequent change at board level (due to the current, but we understand changing, requirement for board members to be renewed annually) which will inevitably have had an impact on the achievement of the organisation's strategic and operational goals. There is a need for VAS to develop its leadership and governance structure to support improved performance.

Leadership

If VAS is to improve its credibility as an intermediary body for the TSI network it is essential that it shows strong leadership. There is an opportunity to address some of the challenges highlighted in this evaluation report and to drive forward the progress already being made.

Lack of clarity of purpose

Stakeholders and TSI interviewees reported that VAS's purpose was not distinct and clearly defined, and the scope of its apparent remit in comparison to its size meant that it was 'pulled in too many different directions'.
Its positioning in the wider landscape of intermediaries, also needs to be reviewed:

" VAS needs to stop being pulled in lots of different directions and re-focus on supporting TSIs and the big impact issues" ( TSI Chief Executive Officer)

" VAS is operating in a busy field, full of organisations that are very territorial. It is hard for a small organisation like VAS to carve a niche" (Stakeholder)

Moving forward, it will be important to focus on key areas of work and target resources on delivering critical outcomes.

Expertise and understanding of TSIs

TSIs feel that at a fundamental level, VAS does not have the in-depth knowledge of the TSI work and environment in which they operate.

To be an effective intermediary, VAS needs to be seen as having expertise that goes beyond that of the TSIs. Some TSIs did not feel that VAS had the necessary expertise to offer:

"You would only go to someone for advice and support if you felt that they knew more about it that you did." ( TSI Representative)

This is a critical gap that must be addressed if VAS is to act as an intermediary for the TSI network.

Behaviour of TSI network

Some research participants identified a culture of resistance to change within the TSI network which VAS has struggled to dispel, and noted that the TSI network itself needed to improve its behaviour in order to ensure that VAS can increase its impact.

To date they have not succeeded in engendering a sense of a connected network - "There are 32 opinions of what VAS should look like" and VAS is challenged in meeting the needs of all 32 TSIs.

As a membership organisation, more TSIs need to get behind VAS and play their part in promoting, supporting and developing their own network.

Chapter Conclusions

• To date, the quality of support delivered by VAS has been inconsistent and has not addressed all of the TSIs' needs but many members of the TSI network, and external stakeholders, believe that support is improving and has the potential to improve further.
• VAS is not considered to have achieved the 'positioning' and credibility to achieve its role in championing and policy influence.
• The criticisms of VAS recognise that VAS is a small and relatively young organisation, operating with a small team of staff to service 32 TSIs with a wide range of expectations and needs, and operating in a complex environment. There is a need for greater clarity of purpose and increased focus on a smaller range of critical activities.
• The organisational structure and leadership of the organisation needs to be improved to support improvements in the operation of the organisation.
• There are a number of governance challenges which need to be addressed including issues relating to the balance of independent and TSI members, and tenure of membership.
• While it was recognised that VAS has not yet fulfilled its core functions, there is broad agreement that the TSIs need an intermediary body.