Disability and Carers Benefits Expert Advisory Group: review

4. Effectiveness and impact of the Group

Regardless of the uncertainties around the remit and purpose of the Group highlighted above, most members who responded to the survey questionnaire believed that the Group had been effective in realising its stated purpose: ten (out of sixteen) respondents agreed with the statement that the Group had been effective and further three respondents said that it had been very effective. Only three respondents were unsure about the effectiveness but no respondent disagreed with the statement that the Group had been effective. Likewise, most respondents believed that the Group had had a demonstrable impact: twelve respondents (out of sixteen who responded to this question) agreed with this statement and one strongly agreed; three respondents were unsure but no respondent disagreed with the view that the Group had had a demonstrable impact.

In line with this, interview data suggests that Scottish Government officials have found the advice produced by the Group very useful in advancing policy making in various policy areas. Expertise and insights from the Group were highly valued by all officials interviewed. However, in that context there were also critical views and it is those critical views that are discussed in this chapter in order to identify areas where improvements could be made. Four such areas were identified by both officials and Group members themselves and these are discussed in separate chapters below.

4.1. Degree of challenge

There was a view in the Government that some of the advice given by the Group was not particularly challenging. However, the Group themselves thought that there was a high level of consensus among members around the direction of social security reforms. It was suggested that members were on board with the main principles of much of the legislation around the creation of the new social security system and that indeed much of what was being proposed by the Scottish Government was what members of the Group had campaigned for through their organisations for many years. Moreover, it was suggested that the Group may have seen itself as part of a progressive change that it sought to assist rather than challenge.

One member pointed also at the fact that the Group gave advice on the social security system that was being developed from scratch and explained that more challenging advice would have been given if the benefits system already in place was being changed. In the latter case, the process of change would inevitably produce winners and losers which would in turn raise a range of dilemmas and trade-offs that were simply not there because the system was being designed starting with a 'blank paper'. Following from this, it was indicated that more challenging advice is likely in the future as the policy process is moving into a new phase.

The two points above suggest that a lack of challenge in some advice may have been a time-limited and a context-specific problem related to the social security landscape in Scotland at a particular time.

Other explanations given by Group members were related to the strategic, long-term choices made by the Group. For example, it was argued that the Group tried to be as objective as possible in order not to become another lobbying body driven by interests of different organisations which its members were affiliated to. This pursuit of objectivity may have made the advice appear less challenging to some officials in the Government. Related to this, two interviewees explained that the Group had to be smart and strategic about selecting areas where it wanted to be more challenging in order to build a working long-term relationship with Ministers. It was suggested that being critical to everything that the Government proposed, especially in the early phase after the Group was created, could have had a negative long-term impact on relationships with officials and Ministers. A member of the Group argued that it could have also generated 'noise' that would overpower the voice of the Group at times when there were really important issues to advise on. This would reduce the influence of the Group in the long-run because its critical voice might no longer be heard when this was really needed

Another type of explanation for the lack of challenge that emerged from the interview data pointed at the lack of access to relevant information. Some members felt that there were times when key information reached the Group too late to enable it to develop a challenging voice. There was also a view among some members that the Group was not consulted but informed about decisions after they had already been made. One member noted that there were instances when some information came to light from sources other than the Government. An argument was made by one member that giving expert advice in a rapidly evolving situation, when the Group did not always have access to the sort of information it needed, meant that the Group may not have been equipped to make stronger statements. However, this argument was immediately followed by a clarification that this was more of a problem in the early stages of the Group's functioning and that things had improved over time thanks to maturing relationships between the Group and officials in the Scottish Government.

The critical views above indirectly relate to what has been said by some officials who explained that engaging with the Group was more about making sure that the Government had not missed something important and that officials had consulted relevant stakeholders. If some officials used the Group to gain such reassurance, then it is likely that some information reached the Group at relatively advanced stages of a policy making process.

Importantly, interviewees indicated ways in which the Group could become better at giving more challenging advice. For example, it was suggested that the Group could engage more closely and frequently with key officials within the Government and ask for information more proactively. It was also proposed by one member that the Group may want to focus on "upstream" advice to ministers. Such upstream work would involve working with officials to give advice before the regulations came forward to Parliament, as well as working around long-term policy options in order to play a more proactive function in addition to simply responding to the Government's requests for advice.

On the other hand, some members felt that where advice was challenging, it had not necessarily been acted upon by the Government. It was proposed by Government officials that the Group could proactively ask the Government when it feels that the advice was not acted upon to clarify why that may have been the case.

4.2. Degree of detail

There were views within the Government that some of the advice lacked sufficient detail and depth. It was indicated that there were instances where the advice produced by the Group was too 'high level' to add significantly to the analytical work that had already been carried out by analysts in the Scottish Government to a high degree of nuance. It was suggested that 'high level' advice could be useful only if it provided fundamentally new thinking, which was not always the case.

It has to be noted, however, that one of the two instances of advice that were seen as too high level was potentially not within the remit of the Group.

Within the Group itself, there were nine respondents (out of fifteen who answered this question) who agreed with the statement that the group had been able to produce in-depth and detailed analysis and recommendations and four who strongly agreed. Only two respondents were unsure about this and none disagreed. At the same time, qualitative survey and interview data with members indicate that improvements could be made in three areas to help the Group provide more detailed advice.

First of all, it was noted in the chapter on remit and purpose (see above) that some members may consider highly detailed advice to fall outside of the Group's remit and may prefer to focus on giving more 'high level' advice on directions of the policy reform. Therefore, clarity of expectations among both Group members and officials could possibly help the Group to be more effective in providing the type of advice that is required.

Secondly, one member expressed a view that the Group should become more skilled at developing more specific (including technical/operational) advice. It was proposed that there could be two strands within the Group: one working on more specific and technical issues of social security policy delivery and the other one looking at the long-term, strategic and transformative aspects of social security reform. Another interviewee noted that the Group may not have sufficient skills around the table to provide very technical advice.

Thirdly, there was a view in the Group that the degree to which advice can be specific depends on how specific the ask from the Scottish Government is. This highlights the importance of effective communication. In relation to this, the majority of respondents agreed or strongly agreed (seven and two out of sixteen respondents respectively) that expectations of the Scottish Government from the Group had been clear to them. However, six respondents were unsure about this statement and one respondent disagreed with it. This represents a somewhat higher rate of 'unsure' responses compared to other questions.

Qualitative survey and interview data corroborated the findings that several members of the Group thought that the steer from the Government had not always been clear and that a clearer steer would have been welcomed. At the same time, some Group members felt that a lack of strictly defined steer gave the Group a fairly free hand to define its focus. Moreover, several survey respondents wrote that the steer, lines of communication, expectations, vision and brief from Ministers had been clear or reasonably clear.

There is a chance that the perception of the lack of clear steer may have been influenced by the lack of clarity on the terms of engagement that accompanied cases where the Group sought to provide advice proactively, but was perhaps also seeking some kind of "signal" or suggestion from the Scottish Government to aid its thinking. This touches upon a larger problem of how the Group engages with officials in the Government when it seeks to give advice proactively and this is discussed in a separate chapter of this report (chapter 5.4.3).

With regard to the advice that was specifically requested by officials, the view in the government was that the Group was intentionally given a fairly free hand in terms of working within its remit and that even when advice on a specific topic was requested, the room was intentionally left for the Group to decide which aspects of the larger problem they would like to focus on. Officials said that they were not aware that the Group had found some of the steer unclear and explained that requests for a more specific steer would have been welcomed. A comment was made that officials can provide more specific steer in the future if this is what the Group needs. Another official suggested that this is already happening and that the Government is trying to be more prescriptive in terms of asks and timescales

Last but not least, it was also indicated by one member of the Group that the degree of detail to which advice can be developed will ultimately depend on the nature of the issue - some issues are quite specific in their nature and thus may involve more specific advice than others. This represents limitations that are not avoidable.

4.3. Evidence base

There were views in the Government that some advice could benefit from the exploration of additional sources of evidence; that it was not so much grounded in the analysis of actual evidence as it was in discussions during meetings where members expressed their personal views and opinions, or views of the organisations they belonged to. It has to be noted that these inputs were still seen as valuable in their own right and officials recognised that views of members were informed by a good understanding of evidence and of lived experiences of individuals affected by particular policies, even though the link to the sources of evidence may not have been apparent. Related to this, officials said that they liked the 'light-touch' way in which advice was conveyed. However, there were also views that some advice could have been better grounded in evidence.

There were diverse views among the interviewed DACBEAG members as to what evidence the Group could and should draw on. One view expressed in a survey questionnaire was that the key inhibitors to Group's effectiveness were 'a lack of systematic review of objective evidence' with a simultaneous strong influence of advocacy that represented particular interests of benefit claimants. However, a number of other members emphasised the importance of knowledge of lived experience of benefit claimants in informing the advice produced by the Group. Among them, one interviewee stated that one of the strongest points about the new social security system was that it was being developed by acknowledging and valuing lived experience. It was also suggested that a perception among officials that some advice was not adequately evidence-based could have been caused by misperception about what constitutes evidence. Thus, one interviewee said that 'what officials can sometimes mean by evidence is quantitative data', pointing out that the evidence drawn from lived experiences of benefit recipients is qualitative in nature. This respondent suggested that in the future the Group could pay more attention to pinpointing where lived experience should be taken as robust evidence of what will work and what will not work, and where it is simply a matter of personal opinion of an individual.

Another interviewee proposed that the Group could also draw its evidence more extensively from the experience of front-end workers involved in the delivery of benefits as opposed to relying predominantly on the lived experience of benefit recipients. It was suggested that getting more of the views of those on the delivery side would expand the evidence base, helping to strike a better balance between views of different stakeholders thus making the process of developing policy advice and recommendations more democratic.

Beyond this, there was no clarity among interviewees what other sources of evidence they could draw upon, given the capacity issues (see chapter 5.5) that make it unrealistic for the Group to conduct its own analyses of evidence. Nevertheless, it was acknowledged that the Group could be more proactive in asking the Government to conduct specific analyses of evidence for them. Moreover, some members proposed that the Group could commission work to external experts or that it could be more active in organising events to engage outside experts, including experts in other countries.

4.4. Collective voice

Related to the issue of the evidence base, there was a view in the Government that at times the Group seemed to operate as a collection of people with their own views and interests rather than as a group with a collective voice in its own right. It was noted that this may prevent the Group from adding value because the Government already engages with various stakeholder organisations.

As noted in chapter 4.1, several interviewees within the Group felt that there was a high level of consensus within the Group around the direction of social security reforms. At the same time, several DACBEAG members also noted that narrower organisational interests did occasionally spill over into the Group's discussions. It was indicated that this was something that may be hard to avoid in a group that is composed of members who are full-time employees at other organisations, including advocacy groups, and who can volunteer only a small proportion of their working time to activities related to DACBEAG. Nevertheless, it was also noted that it was important for the Group to mitigate such spill over in order to prevent DACBEAG from becoming yet another lobby group.

4.5. Chapter summary

To sum up, this chapter highlighted some of the key areas related to the nature of advice produced by the Group where improvements could be made to increase the effectiveness and impact of this advice, or at least to make them more consistent. In addition to the four areas discussed above, survey respondents and interviewees identified also factors of a more generic nature that may facilitate or inhibit the Group's ability to produce effective advice. These factors are discussed in the next chapter.