Exploring the evidence base for integrated children's services

4. Evidence supporting integrated working

This section considers available evidence in support of integrating services. This includes negative evidence, where a failure to work in an integrated manner has resulted in poor outcomes for children as well as evidence where attempts have been made to measure the impact of integrated working on outcomes for children and young people, including perceptions from professionals about the benefits of integrated working. The economic implications of integration and the counter evidence for integration are also briefly discussed.

Child Protection

Strong arguments in favour of better integrated children's services have emerged from inquiries into child protection cases, most recently from reports into the deaths of Victoria Climbie and Caleb Ness. It was concluded from these inquiries that a lack of integration contributed to poor outcomes for these children. A failure to share information between different agencies was identified as a fundamental factor contributing to the death of Caleb Ness (O'Brien et al. 2003). Many of the recommendations from the inquiry into the death of Caleb urged agencies to review their protocols on child protection and information sharing and recommended joint working practices between departments within social services be reviewed as a matter of urgency. All agencies were to make collaboration a priority. Similarly a lack of joined-up working was identified by the inquiry into the death of Victoria Climbie with similar issues around information sharing and collaboration between agencies identified as significant in leading to such a tragic outcome (Laming 2003). Such high profile cases and other instances where people have "fallen through the net" have provided strong evidence that services should be working better together and have contributed to current policy emphasis on integrating children's services (van Eyk & Baum 2002).

Measuring Impact

We know what can happen when services are not integrated and assume therefore that when services are better integrated there will be positive outcomes across a range of areas. The integrated children's services agenda is driven by a shared vision for children and young people:

"in order to become confident individuals, effective contributors, successful learning and responsible citizens, all Scotland's children need to be: safe, nurtured, healthy, achieving, active, included, respected and responsible" (Scottish Executive 2005)

This section considers evidence that integration is positively impacting on those outcomes.

Webb & Vuilliamy's (2001) study of inter-agency working reported a reduction in exclusions from school of young people with challenging behaviours. Other benefits reported include a positive impact on the quality of the service being provided, information sharing between agencies, and the potential through the introduction of the support worker in the school to avoid inter-agency disputes over responsibilities and resource allocation. The study differentiated how the support worker built inter-agency links with school-focussed agencies (such as educational social workers, behaviour support teachers, educational psychologists and school nurses) and with agencies external to the school (such as social workers, police officers and health service professionals). The research concluded that the possibilities and constraints influencing collaborative working differed depending upon whether the professional was from a school-focussed agency or an agency external to the school. Such findings may be useful to bear in mind when considering suitable methodologies to evaluate the impact of integrated children's services and the role of the school in this agenda.

This particular project did report actual evidence on outcomes - a 25% reduction in permanent exclusions across the project schools (26 pupils). Given the strong link between exclusion and criminal offences committed over an individual's lifetime, this project is likely to also have a significant impact on reducing future offences and the financial and human costs associated with such crimes, although this study cannot confirm such assumptions. As with many of the studies examining integrated working, important lessons were learned about the dynamic process of inter-agency working as well as the length of time required for results to appear, i.e. no positive impact was identified until the second and third years of the project (Webb and Vuillamy 2004).

The Integrated Care Network's guide on Integrated Working points out the limits to what can be generalised from studies on joint or integrated working. One experimental study that the Network is confident about reporting positive outcomes as a result of integrated working is a study into joint working between social services and the NHS. This study reported positive outcomes for the client group, in this case older people, as well positive benefits to organisations from pooling expertise, establishing a new way of operating and opening up direct communication. The Network argues for more experimental studies such as this one to build up a robust evidence base within the field of integration.

It is apparent from reviewing the literature on partnership working and integrating services for this paper, that the majority of studies focus on the process of integrated working rather than the outcomes achieved from such working (Cameron and Lart 2003; Sloper 2004). Even when studies do focus on outcomes these tend to be narrowly focussed, for instance, impact on exclusions or try to address outcomes that might only be shifted in the longer term????are difficult to measure over short time periods, for instance attainment or health. While the process involved in integrated working should not be dismissed, there is a danger that it can detract from focussing on the principle purpose of integrating services, which is for public services to achieve better outcomes. The Integrated Care Network (2004) questions whether too much emphasis is being placed on the structure and input to integrated working rather than on outcomes.

Studies commonly contain discussion about how to overcome the well-rehearsed barriers to joined up working as well as perceptions from professionals about the benefits of integrated working. Despite this, there is little in the way of how these processes relate to outcomes (Sloper 2004). The literature contains much assumption rather than hard evidence on improved outcomes as a result of such changes in the way services are delivered and planned. Full-service schooling lists many benefits from taking an holistic approach to the needs of children and young people and there is much literature explaining what is happening in such schools and how it is improving the well-being of its pupils (Dryfoos 1996). But as Smith (2000, 2004) writes, such listings leave many questions to be answered about the trustworthiness of the data and the measures that have been used. Dryfoos herself, replying to the question "But do they work?" answered "I wish I could give an unequivocal 'yes' to the question. I have to report a strong 'maybe' " (Dryfoos 1998).

Stewart et al (2003) caution against the assumption that integrated working is the preferred option to deliver effective services without a robust evidence base to justify such thinking. While accepting that the process of integrated working is essential, the authors warn that the complexities of integrated working are unlikely to be overcome to produce its intended benefits unless a clear and sustained focus on the long-term outcomes for clients is maintained. Similarly, Percy-Smith's (2005) discussion of strategic partnerships, which if we accept as a given for integrated working, warns about the assumption that partnership working is "a good thing" and states that definitive answers to the central question "what is the impact of partnership working?" are very difficult to find. Setting up and making partnerships work requires significant investment in time and resources and needs to be justified in terms of the benefits such partnerships deliver. This needs to focus not only on outcomes but also on the extent such outcomes can be attributed to partnership working.

This requires clarity about the outcomes that integrated working should deliver. Sammons et al (2003) comment in their evaluation of Integrated Community School pilots that it is unsurprising that no significant impact on measures such as attainment and attendance was found over a 3-year period. Tisdall, Wallace, McGregor, Mullen and Bell (2005) comment in their review of integrated community schools and family service centres that most of the impacts are not regularly quantifiable or captured by standardised evaluation evidence measures. This raises fundamental questions about what outcomes we should be measuring and the methodologies that should be adopted to gauge the success of such integrated working

Sloper (2004) also states that there is a clear need for methodologically sound local evaluations of multi-agency services on outcomes as well as an exploration of the cost effectiveness of such ways of organising services. As Johnston et al (2003) report from their study of joint working between Social Services and the NHS all case studies could report extensive planning meetings to develop collaborative activity but few could identify actual instances where joint working was currently operative.

Wilkin et al's (2003) review of the literature on extended schooling and other related concepts such as multi-agency working and joint-working concluded that there is little systematic and rigorous evaluation of the concept and its implementation. Similarly, Sullivan & Skeltcher (2002) argue that the need for adequate frameworks to monitor, understand and assess collaborative activity will only increase further as more collaborative working becomes embedded in public services. They suggest a distinction will need to be made between implementation, outputs and outcomes to assess what outcomes have been achieved and trace the activities that have contributed to this achievement. It is also suggested that an assessment of partners' capacity to learn should be incorporated into any evaluation activity of collaborative working as the experience itself can provide partners with valuable insights around the achievement of collaborative outcomes. Examining the implementation process and involving a range of stakeholders to look at power relationships between them was also identified as important for any evaluation activity around collaborative working.

Economic implications of integration

The principle driver for the integration of children's services is to improve outcomes for children and families. However, there may also be financial benefits from such policies and this is one issue to consider when looking at the evidence base on integration.

As earlier discussed, the rational for integrating services is that no one agency or professional working in isolation will be able to meet all the needs of an individual child, thereby limiting opportunities to improve life chances. The costs therefore arising from children whose needs are not met at an early stage, potentially as a result of agencies not working effectively together, are large and fall on a wide range of agencies over an individual's lifetime: the negative effects of low attainment are recognised as lower lifetime earnings, higher chances of being unemployed and poorer health of individuals. It is estimated that by age 28, costs to society of individuals identified as having conduct disorder ¹ at age 10 are estimated to be 10 times higher than those with no problems (Scott, Knapp, Henderson and Maughan, 2001). It has been estimated that a programme of supportive multi-agency interventions for children who demonstrate challenging behaviour from an early age could save over £100,000 in direct costs incurred to age 16 (Audit Commission 2004), as well as preventing crime and the associated costs of social exclusion throughout that individual's lifetime.

There may also be financial implications on organisations from integrating services, for instance, savings over time as a result of efficiencies from multi-agency working practices. These savings may be in areas such as staff time, improved communication, technological advancements, as well as simply better ways of doing things. An example of such benefits is the Home Office funded project Meeting Need and Challenging Crime in Partnership with Schools, which placed five full time social work trained support workers in secondary schools for three years targeting children identified as likely to be excluded. The savings associated with such integrated working accrued to various groups of beneficiaries, including: reduced teacher and senior management time spent on dealing with 'problem' children, reduced costs to Local Education Authorities of alternative education provision, and the personal costs to families and young people of exclusion (Webb and Vuillamy, 2004).

As well as possible economic gains from integrating services there may also be costs, both directly and indirectly from integration. Whereas it may be relatively straightforward to identify the direct costs of integrating services, such as extra personnel or new processes, the indirect costs may be more difficult to capture. For instance, the cost of redirecting resources to integrate services, providing integrated services for all children rather than focussing integration for more vulnerable groups, future costs where children and families continue to require support despite integrated service provision. These indirect costs of integration are not easily quantifiable yet some attempt to illustrate the trade-off between pursuing integrated working and maintaining the status quo is an area for future consideration to determine the best use of public resources.

Economic Analysis of Integration

If the economic implications of integration are to be considered then a suitable methodology will be important to accommodate the many outputs and outcomes. However, this is not straightforward due to several difficulties inherent in reliably measuring outcomes. These difficulties are discussed below:

• Measuring the added value (the improved outcomes over and above those that would have occurred had the policy not been in place) of integrated service delivery must require a counterfactual case: a cohort with similar characteristics amongst whom the policy was not implemented. This is almost impossible for research involving children as the control group will never be characteristically identical to the cohort targeted with integrated services.
• The second major barrier to measuring the impact of integration is that causality can not be accurately attributed. For example, the factors affecting a pupil's academic performance are multiple and interconnected, which makes it almost impossible to isolate the effect of integrated service delivery on attainment. As was shown in the evaluation report on the early integrated community school pilots, attainment in all Scottish schools increased over the pilot period so we cannot conclude that this policy alone was the casual factor in the pilot schools.
• Timing of outcomes is also an issue when evaluating any policy. It is highly likely that the total economic benefits of any policy aimed at for instance, improving educational outcomes, will not be evident until several years, if not decades, after the intervention occurred. This is especially relevant in the case of increased earnings due to higher educational attainment. On the other hand, the costs are incurred in the year of the intervention.
• There may also be significant indirect or spill over benefits of outcomes from integrated services that are not easily determined. For example, a significant proportion of the cost of crime and anti-social behaviour falls on the victims of crime and the wider community.

Developing an appropriate framework to overcome this problem of causality and enable conclusions to be reached will be a critical part of the evaluation process and is likely to require the collection of both quantitative and qualitative data as well as a range of methods.

Perceptions about the benefits of integration

There is more abundant evidence based on the perceptions of professionals but this tends to focus on the process of service provision rather than outcomes for children and families. An audit of multi-agency activity through a survey with all Local Education Authorities reported that staff perceived there to be a wide range of benefits to children and families from such multi-agency working (Atkinson et al 2001; 2002). Advantages also centred around opportunities for professional development through working with other agencies, such as a broader perspective and better understanding of other agencies' roles. Similar findings from professionals were later reported by Sammons et al (2003) in the evaluation of the early integrated community school pilots. Rushmer and Pallis (2002) report that organisations and professionals have much to gain from inter-professional working such as wider expertise, knowledge of the skills of other professionals, opportunities to contribute to other developments, reducing the duplication of work and improved communication in organisations.

Although Townsley et al (2004) focussed more on the process of multi-agency working in services for disabled children with complex health needs rather than on outcomes, the study did conclude that the structures that professionals had worked hard to put in place would bring about significant change in the future and was making a difference to families to a certain extent at the current time. There was again acknowledgement by professionals that although it would take time, positive evidence from such a way of working would eventually emerge. However, such perceptions from professionals can only be theoretical about the outcomes for clients often referring to potential benefits for service delivery rather than actual hard evidence of improved outcomes.

Other findings from studies looking at the impact of integrated working report increased feelings of support among professionals, improved job satisfaction from working collaboratively with other agencies and positive attitudes from professionals towards the aims and benefits of collaboration in general (Rushmer & Pallis 2002; Anderson-Butcher & Ashton 2004; van Eyck & Baum 2002, Borrill et al 2002). While improving job satisfaction and professional development is to be welcomed, especially given some of the recruitment and retention difficulties in certain services, it is not the primary purpose of integrating services.

Whilst there is abundant research reporting professsionals' views, there is limited evidence based on client perceptions and again what exists tends to focus on views of service provision rather than improved outcomes. Liabo, Newman, Stephens and Lowe (2001) reviewed multi-agency key worker systems for disabled children with health needs and found some evidence that families with a key worker reported more positive outcomes, such as quicker access to services and reduced levels of stress compared to those families who did not have such a key worker to join up services.

Counter Evidence

Collaboration may be only one of many solutions to delivering effective services and as McCulloch, Tett and Crowther (2004) suggest, may be best avoided in some instances. From their assessment of professional attitudes towards collaborative working in response to the integrated community schools agenda, they suggest that when time, energy and support are very limited, collaborative working may not be the best option. The study focussed on professional attitudes rather than looking at data on outcomes for children and families so makes a limited contribution to any evidence base in support of integrated working. It did raise issues from professionals about the difficulty in evaluating success and developing indicators of success.

An interesting finding by Glisson & Hemmelgarn (1998) was that increased service co-ordination actually had a negative impact on outcomes for children and families. Examining the effects of both organisational characteristics and inter-agency co-ordination on children's psychosocial functioning, led the researchers to recommend that further attention should be given to improving the organisational climate of agencies rather than increasing organisational co-ordination. Organisations where staff reported greater job satisfaction, role clarity and fair organisational practices were found to deliver significantly better outcomes for children and families (measured as independent descriptions of children's behaviours by their teachers) than those organisations with poor climates. The authors suggest that there should be further research to explore the relationship between outcomes for children and families and factors such as stress, workload and low job satisfaction in professionals working with them. This is echoed by Gardner (2003) who comments that as yet there is insufficient evidence to argue that greater collaboration between services will undoubtedly produce better outcomes for all children and families

Conclusion

Inquiries into child protection cases have provided strong support for improving joint working between services. From this review however, there appears to be limited positive evidence on outcomes from integrated working with much of the current research focussing on the process of integrated working and perceptions from professionals about the impact of such services both on clients and on their own professional development. This gap in the evidence base has led to calls for more outcome focussed investigation to build up a more robust evidence base on integration. Exploring the economic implications, whether service users experience better services through integration and when integration may not be the best solution would also contribute to the evidence.