The Realigning Children's Services Programme: evaluation

Chapter 2: Experiences of the Realigning Children's Services Programme

Introduction

In this chapter we discuss the interviewees' experiences of the different elements of the RCS programme and explore the themes that emerged about the implementation and delivery. We also identify how the programme has evolved over the course of two tranches and in the current support to CPPs in Tranche 3.

The discussion about the experience of the programme is framed around the following key themes:

• Communication and engagement with RCS programme principles;
• Data and evidence: collecting and processing;
• Data and evidence: key outputs;
• Programme support;
• Workshops and learning exchanges; and
• RCS national team

Communication and engagement with RCS programme principles:

The programme aims to provide support and challenge to CPPs to drive improvement in their joint strategic commissioning of children's services. The CPPs selected in Tranche 1 and Tranche 2 were aware of the Dartington model used in other CPP areas, understood the key elements of the programme and were motivated, for different reasons, to be a part of RCS. For some, this was in response to Children's Services inspections and the need for change. Others wanted and recognised the value of having whole school population data to help inform decisions.

At a senior level, there was an understanding of and commitment to the RCS programme across all the CPPs, and the local decision structures were adapted or created to ensure that the work and evidence it produced could influence the local policy and practice landscape.

The extent to which that understanding and commitment was retained at middle management or with certain stakeholders, e.g. education or health, varied. The RCS team recognised that senior and middle managers were tasked with delivering the main elements of the programme and that there was sometimes a disconnect between the two layers of management. For local stakeholders, like head teachers in schools or third sector organisations funded to deliver services, they were often focused on one element of the programme, (e.g. the survey element or the mapping exercise) without fully appreciating how it fitted into a much wider programme.

Clarifying the message

The RCS team members explained that, in order to provide clarity and fully reflect the work, the original programme emphasis on joint strategic commissioning of children's services has been adjusted. The team recognised that many CPPs were and are not currently in a position to undertake joint strategic commissioning; that there seemed to be a heavy focus on the survey element, because it was a complex activity to coordinate and deliver, and; that this resulted in local stakeholders concentrating their attention and effort on the surveys rather than all aspects of RCS. Therefore the programme and key messages were flexed.

The adjustment in the messaging and communication about RCS has helped to clarify and emphasise the aim of the programme, which is now described as supporting CPPs to make better joint strategic decisions about how to improve outcomes for children and families.

As identified by local stakeholders and the RCS team, CPPs are supported and challenged to do this by:

• Bringing together organisational stakeholders to share information, build collective understanding, agree priorities and develop joint plans of action.
• Using data and evidence to map current need, services and expenditure; identify priority outcomes and ways of addressing those, and evaluate subsequent changes to policy and practice.
• Identifying ways to shift investment 'upstream' to allow prevention and early intervention and reduce the need for high intensity, high cost services.
• Finding meaningful and effective ways of involving children, families and frontline practitioners in the process of service redesign.
• Using the lessons of implementation science and improvement methodology to bring about effective, sustainable and scalable change.

Latterly, this description about the programme and the five elements has been used consistently in communications with all three tranches. Not only should this help to increase the general understanding across all stakeholders but it should also assist in establishing the links between the key activities and the other aspects of the programme.

Data and evidence: collection and processing

The data and evidence to map current need, services and expenditure is drawn from the Child Wellbeing Survey Programme, the data linkage and the mapping exercise.

The Child Wellbeing Survey Programme

The survey programme involved the primary school survey of P4-6 pupils, the secondary school survey of S1-4 pupils and a face to face survey of parents of 0-8 year olds. Both school surveys were web-based. The primary school survey was designed for the RCS programme and the secondary survey was based on the Scottish Adolescent Lifestyle and Substance Use Survey (SALSUS), with a wellbeing boost which asked additional questions related to, for example, healthy living and physical activity. All aspects of the programme regarding data collection, analysis and reporting were contracted to ScotCen, with the exception of the data collection in secondary schools during Tranche 1.

Securing the involvement of the majority of primary and secondary school head teachers to participate in the survey was a significant task, although this was successfully achieved in all CPPs with assistance from the RCS team. Each CPP approached the awareness raising and engagement of the survey programme in a different way. In one area, they worked closely with colleagues in education to communicate with head teachers about the process through various means including area briefings and written summary guides and information. Whilst this helped to engage many head teachers, the CPP still felt they needed to build in more time and more face to face communication to ensure buy-in.

Another area, in contrast, struggled to get their head teachers on board with the survey. It came at a time when the message from government was about reducing bureaucracy and emphasising autonomy. Their involvement was eventually secured with support from the RCS team, who attended a meeting with head teachers, discussed and addressed their concerns, and helped them understand the benefits to their school in having detailed wellbeing data from several year groups.

The process of undertaking the survey showed what was possible to do on that scale, and while it went quite smoothly in most areas, interviewees in CPPs reported some minor challenges. These included:

• the timing of the survey in the school calendar year (summer term) for some of the primary schools in Tranche 2;
• a minor challenge in the generated unique identifiers for the surveys in one school that had the same/similar name to another school, and this issue was quickly resolved; and
• the focus of surveying pupils in mainstream schools.

Building on this point, some local stakeholders were disappointed that pupils in special schools and/or some pupils with disabilities or special needs did not take part in the survey. The RCS team was clear that, as a group of pupils with additional support needs, there would be a greater range of existing data about these pupils compared with the whole school population. It was also felt that as a standardised tool, the survey would need to be adapted or the pupils would need additional support to complete it. The CPPs could opt to do this but they needed to find the time and resources to administer it in that way.

The parents of 0-8 year olds were selected using a tightly-controlled form of quota sampling and a small sample of 400-500 were surveyed face to face within their homes. This was a resource intensive and costly element of the survey programme and one CPP opted out of the parent survey, preferring instead for the resource to contribute to three strategic needs assessments. The RCS team recognised the limitations of the parent survey; its small sample size meant the data were less robust than data from the school surveys. In addition, it was relatively costly to administer. The parent survey has been removed for Tranche 3 and the resource is redirected to the school surveys and a part-time evidence officer in the CPP.

Where wider strategic needs assessments were carried out, these involved survey work, desk based reviews, focus groups and interviews with a range of stakeholders, including practitioners and frontline staff. This aimed to gather their views on the themes of community justice, looked after children and early intervention and prevention. The RCS team then facilitated consideration and discussion of the broader picture of need alongside the survey data.

Data Linkage

The self-reported data from each survey response were linked to a set of additional fields^[3] drawn from Social Work and Education records in each area. This process was conducted securely and anonymously by the electronic Data Research and Innovation Service (eDRIS) and the National Records of Scotland (NRS). This created a single data set that contained information about child wellbeing and need, and the receipt of children's services.

Mapping exercise

The mapping exercise aimed to comprehensively map children's services in each CPP and the amount of expenditure on statutory services across education, social work, health, leisure and culture, as well as funding to voluntary organisations to deliver projects focused on children and families.

The information collected as part of the mapping included: the age group(s) that the services provided for, the total number of children provided for, the percentage of the budget associated with different tiers of intervention and; third sector services and the origin of the funding.

Whilst some CPPs did eventually complete the exercise, all the CPPs reported that they had struggled to prioritise or to undertake the mapping. This appears to be as a result of a number of factors:

• the perceived overemphasis on the survey programme;
• understanding the purpose and value of such an exercise for those contributing the information;
• difficulties engaging the local stakeholders or partners that needed to have an input;
• challenges for health colleagues in sharing budgetary information and unpicking NHS Board level budgets to align with the CPP area; and
• for third sector organisations delivering services were not always comfortable divulging commercially sensitive information about their contracts.

Despite the challenges, there are a few examples of successful contributions to the exercise. In one area, they had a contact in their NHS board who was able to progress the financial unpicking of services and identify what was delivered in their CPP. In another, the Third Sector Interface (TSI) worked with the voluntary sector organisations to capture the information, which, as an activity, they found helpful to understand what funded work was being carried out.

In one area, now in receipt of the survey data, they are undertaking the mapping exercise. With a clearer picture of the wide range of funded services, they now know what is being delivered as well as how and to what extent it is meeting needs. They can use this information to inform decisions about funding services.

The RCS team has reflected on the mapping exercise and recognise that making clearer links to how CPPs could invest in preventative work and make long term savings would be more useful and worthwhile for those bringing the information together. To do this, there would need to be support from an economist who could carry out some financial modelling.

Data and evidence: Key outputs

There were several outputs from the data and evidence element of the programme.

Survey reports

The reports included a set of bespoke school reports, cluster reports for groupings of schools and an overview report for each CPP. For Tranche 2, these were produced within six months of the surveys being completed, which was the earliest possible timeframe to undertake the analysis, link the data and create the reports. A summary at the start of each CPP report was written by a member of the RCS team to help identify themes and messages within it. With this survey data several local stakeholders acknowledged that it was the first time that their CPP had detailed, linked information on the characteristics, needs and perceptions of the wider population of children and young people.

All of the stakeholders in CPPs who were interviewed commented on the quality of the summaries in the reports in distilling key messages and found that the survey responses, linked to existing local evidence, had created a picture of service use and need that was hugely helpful for informing decision making at a local, area and CPP level.

Data set and visualisation tool

The data from all of the surveys is collated and given to each CPP as a full anonymised dataset. A visualisation tool, created within Tableau software, is provided to help interrogate the data. This means that the data is an ongoing resource that can be explored and presented in a user friendly way.

At the time of the evaluation, CPPs in Tranche 1 had access to the full anonymised dataset and had been brought together for a learning and practice exchange to go through the functionality of the tool. These CPPs were beginning to conduct further ad hoc analysis and create visuals of the data.

Programme support

Alongside the collection of new evidence, to inform joint strategic decision making, the development programme works with CPP stakeholders to create the environment and conditions to effect the change needed for improved and more targeted children services.

Development days

The three development days were important elements of the programme support. They brought together key stakeholders to:

• learn about the use of data and evidence to inform decision-making;
• be inspired by experts on improvement methodology and implementation; and
• understand how to lead effective change management.

The development days were described as high quality in their content and delivery. The participants enjoyed and welcomed the rare opportunity to learn about and discuss a wide range of issues with local colleagues. Interviewees highlighted that the exercises provided challenges stimulated debate and helped them to understand how aspects of joint strategic decision making and/or commissioning could work in practice.

Many of the interviewees involved in the development days felt that the timing of the three days needed to change slightly so that they occurred at a point in the programme when the participants could apply the learning. This is something that the RCS team had acknowledged and, in Tranche 3, the days are spread over a longer period and will be held within a timeframe that is most relevant to the stages of the programme.

Support and facilitation

Each CPP received support from the RCS team and this was tailored to their needs. The support could range from facilitating a meeting with local partners to move forward discussions, or inputting into children services planning sessions, or engaging with senior leaders to challenge inactivity within the CPP. The RCS team members fulfilled this fluid role and, as a team, were described as engaging, knowledgeable, flexible, hands-on and responsive. The CPPs appreciated the support that the team provided and considered it to be essential in maintaining the momentum locally, helping them to get to the point of having local data but also assisting them to understand how to use the new evidence to shape the decision making about children's services.

The one slight frustration with the RCS team was the changing personnel and the need to build new working relationships as new team members joined. Nevertheless, these regular changes did not detract from the widely held view that it was a high quality and effective team. At the time of the evaluation, the loss of so many of the team members-particularly exacerbated by the vacant RCS manager post had led to a temporary reduction in the support provided by the team.

Once back to full strength, all CPPs in both tranches were keen to see the RCS team involved over a longer period to maintain a connection, even if this was limited to six monthly contact and annual updates. They also would welcome more joining up of the RCS work with other Scottish Government teams, like those involved in Children's Service Planning.

Workshops and learning exchanges

These meetings occurred twice a year and brought together the key local stakeholders involved in the RCS programme, as well as experts and speakers, like Dartington Social Research Unit, to explore and discuss relevant issues. These events were enjoyed by the CPP representatives and they are keen for more opportunities to connect with each other and the CPPs in other tranches, so they can regularly share learning and offer mutual support.

RCS national team

Initially, the RCS team was located alongside the looked after team within the Children's & Families Directorate. They were later moved to be with the Better Life Chances team, where the policy fit seemed more relevant. The RCS team was described as 'nimble' with a level of freedom that enabled them to be responsive and adapt to the needs of the CPPs and the programme. The RCS manager had a leadership style that was engaging and open to challenge and encouraged a culture of learning and reflection which led to a regular refinement of the approach and to adjusting some elements of the programme.

As a team not only involved in policy design but also policy implementation, most of the team members needed to adapt their approach and work in a less conventional way so that they could provide the proactive and reactive support, challenge and encouragement to the CPPs.

This way of working was a steep learning curve for some of the team, but on the basis of their testimony and wider evidence, they adjusted well. This gave them an insight and understanding that benefitted the RCS programme but also that could support the work of other teams. Several interviewees mentioned the potentially important role that the RCS team could play in feeding into other Scottish Government programmes and agendas, for example those responsible for Children's Services Planning, to maximise the learning and inform ongoing policy development and implementation. Stakeholders were unsure as to whether this was happening.

Team changes

As noted above, over the last three years, several members have left the team and moved into new roles. This turnover had not, until this point, impacted on the support provided to the CPPs. Whilst the posts have been advertised, with the RCS manager role vacant, the team is staffed with an analyst, an intern and a consultant working one day per week. The team appears vulnerable in its ability to continue to support Tranche 2 and the new CPPs in Tranche 3 in the responsive and engaged way that has been a trademark of its approach.

Summary

In this chapter we have described the different elements of the RCS programme and how they successfully engaged, encouraged and enabled CPP partners and local stakeholders to gather and use data for local service planning and improvement. In Chapter 3, we explore the changes and outcomes that have resulted from the new knowledge, skills and understanding of local evidence, and the learning from the RCS programme.