Social Security Experience Panels: legacy report

5. Reflections and Learnings

Across the evidence review, interviews and feedback survey in this research, several learnings and reflections have been identified. These have related to all aspects of conducting the Experience Panels including setting up the panels, conducting the panels and using panel research data. Specifically:

• Reflections on sampling of panel members
• Importance of building trust
• Managing expectations of panel members
• Offering multiple ways to participate
• Feeding back to panel members
• Importance of culture around social security

5.1 Running the Experience Panels

Reflections on sampling of panel members

Various positive and negative opinions regarding the demographic make-up of the panels were shared during the research for this legacy report.

As the Experience Panels required ongoing input from panel members, it was by necessity a self-selecting group of people. Some people involved in this legacy research regarded this as a positive because it meant panel members were taking part because they voluntarily wanted to share their experiences. Others appreciated that having a ready resource of people willing to participate in research allowed them to circumvent the recruitment phase. One member of staff also highlighted that people can experience participation-fatigue within smaller panels, so having a large group of people who could be invited was valuable.

However, a few members of staff discussed the self-selecting nature of the panel as a challenge because it could mean that the same people were volunteering repeatedly, which could lead to some individuals having disproportionate input to research.

Another concern was if the opinions of these participants, who were often very experienced and knowledgeable of the benefits system, could be regarded as ‘typical’ of others. However, some policy colleagues also said that because panel members represented a sample of people with lived experience of the benefits system, they were less likely to express prejudiced or politicised views regarding benefits (as was reportedly a challenge with some public consultation responses).

The demographical representativeness of the Experience Panels was also considered by some staff. One colleague mentioned that the early Experience Panels events had greater representation of people of a certain age, gender and ethnicity. As noted earlier, this was recognised and addressed through a second wave of recruitment to the panels, together with a more targeted approach to seldom heard groups through the Seldom Heard Voices project of research.

While the aims and objectives of each research project may dictate how recruitment and sampling are done, these reflections emphasise the need for researchers to be mindful of ensuring that participants are representative of those who may be affected by the research subject. For panel research in particular, this might be useful to consider for both recruitment to the panel and for targeted recruitment to individual research activities, to ensure that a diverse range of voices are represented. This would need to be balanced against the risk of over-sampling some participants and causing research fatigue. Depending on the size of the panel, it might also be useful to maintain a record of participation in research events to ensure that views from a range of panel members are captured.

Importance of building trust

At the outset, the Experience Panels team felt that it was important to take the time to build trust with panel members, given that they were participating on a voluntary basis and the programme was still new and relatively unknown. Several steps were taken by the Experience Panels team to build trust with participants.

In the registration form for the Experience Panels that was sent to potential members, only information that was required to confirm eligibility and contact details were collected. There were two reasons for this. First, to keep personal information separate from sensitive personal information or any other research findings. And second, to keep registration as simple as possible for panel members.

After recruitment had closed in 2017, a survey was conducted with panel members called ‘About Your Benefits and You’. A draft of this survey was tested with a group of panel members, which were selected to include people with a range of characteristics and experiences. Findings were published in the Experience Panels survey testing session report.

During that session, participants stated that they felt that time should be taken to build trust with panel members before all equality questions were asked. This added to existing awareness in the research team that many panel members were still suspicious of government research. For example, people registering over the phone often asked if the Experience Panels would share their information with the DWP.

The decision was taken not to ask all questions in the first About Your Benefits and You survey in 2017. Instead, a working relationship was built with panel members through undertaking research activities across the country and running surveys, and publishing reports which raised the profile of the Experience Panels. The full set of equalities questions were only asked once the Experience Panels were better established and trusted by participants. The researchers involved in Experience Panels events were also careful to build trust and rapport with participants during events.

“The experience panel team were very at ease and welcoming, I didn’t feel so nervous after meeting them. It was good to hear other participants expressing their views and opinions in their words. That then helped me make statements and contribute.” (Feedback survey respondent)

Offering multiple ways to participate

The onset of the Covid-19 pandemic in 2020 disrupted the Experience Panels and led to a shift in the way research was undertaken. During the pandemic, surveys and phone/video interviews were used to keep panel members and staff safe and to ensure the research could continue. Separate research on the impact of Covid-19 on communication preferences was conducted with members of the Experience and Client Panels. This highlighted that more people considered video calls to be a helpful way to communicate. Following the pandemic, in-person events did return, but there was a shift towards greater inclusion of phone, video and survey methods during research.

The feedback survey for this research reflects the findings from this previous research, with some respondents stating that participating via video call made it easier for them to access research events. However, many respondents said they thought in-person research should continue to be used and that they felt it could not be replaced.

This has emphasised the importance of future research participants being offered a choice in how they take part in research. This broadens not only the appeal of research, in that people can participate in their preferred way, but accessible options for taking part. This is particularly relevant where participants may have conditions which make in-person events more difficult, such as limited mobility or anxiety.

Managing expectations of panel members

A challenge raised by some of the social researchers interviewed (many of whom worked on other panels) was that managing the expectations of their research participants was difficult. They explained that some of their participants expected that by taking part in the research, their specific views would go on to directly influence policy decisions. However, the development of policy is complex and while research participants’ views do feed into the process, it is likely that other factors will also be involved in final decisions. This was a challenge that was also evident in the Experience Panels research and which some panel members commented on through the feedback survey. For example, one survey respondent expressed disappointment that something their focus group had agreed on had not been implemented.

“We almost 100 per cent agreed…what should and shouldn’t happen…that did not happen…shocking and am so disappointed.” (Feedback survey respondent)

It is understandable that panel members might feel disappointed when their specific individual contributions are not taken forward. It is therefore important to manage these expectations at the outset of participatory research, particularly where it may be used to inform decision making within a policy or service design context. Common understanding should be established between the research team and participants to ensure that participants are aware of the context surrounding decision making, and the extent to which their contributions may influence that process.

5.2 Use of Experience Panels research

Feeding back to panel members

A recurring challenge throughout the research for this report was identifying how the data produced from the Experience Panels had fed into the decision-making process. As noted above, there are many different factors that can feed into decisions that are made within policy contexts, so it is understandable that there would not be in every case a clear ‘line’ that could be traced from Experience Panels’ research through to outcomes within policy or Social Security Scotland.

Staff who had been involved in research conducted during the early stages of the Experience Panels explained that the process for doing research with panel members was initially needs-based, responding to the requirements of colleagues as they began to design the new social security agency and develop the benefits. This meant that sometimes there was little supporting information that could be drawn on to understand the research journey from inception to use, such as clearly stated research aims or documented next steps from the end users of research findings.

This could also mean that colleagues who had joined their teams in the years after Experience Panels research had been conducted were sometimes unable to disentangle how the research findings had been used compared to other forms of evidence. Again, this is partly due to the decision-making process being complex and involving different intersecting forms of information. However, in other cases staff were recollecting second-hand information from what they had been told by previous colleagues. In these cases, it seemed that some knowledge about decision-making processes was held within individuals, rather than documented.

Related to this, a common theme identified in the feedback survey responses was the desire for more transparency around how research contributions were used in decision making. During interviews with social researchers who were running other panels, several colleagues noted that feeding back how research participants’ contributions had been used was part of their agreement with their participants. For Experience Panels research this was done to an extent through reports and newsletters, but in many cases this feedback was not explicit.

For future research, establishing clear steps to document how research findings are being used and their input within decision-making processes would be advisable. This would also allow more transparency when communicating with research participants about how their contributions have been used, particularly within areas such as service design where participants may be directly affected by the decisions reached.

Relatedly, the Experience Panels did not have a clearly defined set of objectives outwith the core aim of ensuring that people with lived experience of the previous UK benefits system had direct input into the design of the new Scottish social security system. The varied nature of the research projects that were run by the Experience Panels perhaps precluded specific research objectives, but a more defined set of goals for the panels overall could have aided in assessing their outcomes and achievements once they had reached their conclusion. This could have also been used to provide more specific feedback to panel members as to how their contributions helped to achieve these goals, and potentially addressing some of the concerns outlined above.

5.3 Outcome of the Experience Panels research

Importance of culture around social security

A repeated finding throughout the legacy research was that one of the biggest impacts of the Experience Panels was the creation of a culture that embodies dignity, fairness and respect. Through the development of ‘Our Charter’, the language used in benefit names, Social Security Scotland staff training materials, a commitment to inclusive communications and accessible research, and the development of needs-based localised support, the Experience Panels have set a high standard for how participatory research and service development should be done. A panel member in the feedback survey remarked on this shift.

“The fact that [the Scottish Government] are treating people with respect and taking a human-rights based approach when dealing with people, makes a tremendous difference.” (Feedback survey respondent)

Some policy team members interviewed for this report noted that when they were developing policies many of the changes to benefits were often minor. However, the way clients were treated through benefit names or language used in the policies made an importance difference to clients because this was often an area where social security services were able to demonstrate respect for clients, or increase clients’ feelings of being treated with dignity. For example, panel members explained that benefit names that include the word ‘allowance’, or ‘low-income’ are patronising and stigmatising. Choosing new benefit names with input from Experience Panels members was a simple but effective step to reduce feelings of stigma, and promote Social Security Scotland aims of treating clients with dignity and respect.

A significant learning from the Experience Panels research was therefore the importance of culture for clients and the ways in which small changes within client-staff interactions, language and communication can significantly improve clients’ experiences of social security services.