Public participation in policymaking: exploring and understanding impact

What evaluation frameworks might be useful in a Scottish Government setting?

This section considers how public participation processes taking place across Scottish Government might be better evaluated and understood. First, it details some of the themes from the interviewee contributions related to evaluation. Then, it details the reflections and discussions from a workshop convened with civil society representatives and civil servants (N=5) who regularly support and deliver public participation processes. The workshop drew on the attendees' expertise, knowledge and practice experience to consider what is important when trying to evaluate the impact of public participation processes and the different methods and tools which might be useful for doing so. Please see Appendix A for more detail on the session.

There was consensus across interviewees that evaluation of the various impacts which can arise from public participation processes was at once urgent and essential, as well as challenging in practice:

It's bloody difficult. Int 5

We tend to look at process as opposed to outcome because it's easier to do that. Int 14

It's very difficult to empirically assess a lot of these impacts. Int 16

The challenging nature of understanding impact has been highlighted in literature which specifically considers evaluating democratic innovations (Geissel & Newton, 2012). One of the reasons suggested for this is that public participation processes often have a wide range of aims and aspirations associated with them, which can be difficult to track and analyse in great detail (Geissel, 2012):

Democratic innovations are expected to fulfil a wide variety of hopes, some of them contradictory. Actors involved expect participatory innovations to solve a mixture of problems, such as enhancing citizens' competence, cutting back the budget of a municipality or providing more redistributive and sustainable policies. (Geissel, 2012, p.209)

Alongside participatory theorists, stakeholders advocated for understanding the wider context in which public participation processes are taking place (Elstub & Escobar, 2019). For some, this meant developing a range of tools or frameworks which were applicable across a variety of settings and modes of participation. Indeed, one interviewee detailed what they perceived to be the limitations of some of the existing models or spectra (see for example, Arnstein, 1969; Fung, 2006; IAP2, 2018) which are used to understand participation processes:

It's just really tricky … I'm not totally convinced there is like a framework that would work for everyone … I'm not a mad fan of Arnstein's ladder and the IAP2, I know they're not frameworks for evaluation, but I think they're used for evaluation. I think they're really helpful but they're too simplistic, especially for more complex types of participation. Int 15

Understanding this 'complexity' was largely considered necessary. Despite the recognition that evaluation of impact was challenging, across accounts, interviewees expressed a strong desire to understand, in more detail, the outcomes of public participation and provided compelling arguments for why Scottish Government needed to focus energy and resource on robust evaluation.

This included being able to explicitly demonstrate the value and benefits of working in this way, as well as providing evidence and justification for the continued and expanded use of public participation processes within decision-making. This was thought to be particularly beneficial for convincing senior leaders that public participation was 'worthwhile':

I guess it's that kind of triangulation of, did the staff think it was worthwhile? Did the Members think it was worthwhile? Do the people who took part think it was worthwhile? Do the partners we worked with think it was worthwhile? And each of them will have their own kind of biases on it. But a sort of collective all-round opinion. Int 6

And I think a lack of evidence to say that public engagement is a good thing. Again, a lot of it's quite anecdotal or quite spread out. We don't have a data database saying here are all the impacts. Int 8

I think it's great that we do as much as we do, but we need a proof of concept with it. We need to be able to demonstrate to people that actually it's a legitimate way of doing things, and therefore it's absolutely essential for it to be effectively resourced. Int 11

Linked to this, was the need to share 'good news stories' about positive and impactful experiences of public participation. Interviewees expressed concern that, despite lots of citizen participation happening across the organisation, there was not a shared understanding of where it was working well, or the opportunity to learn from examples of success:

You've got someone coming back to you saying that work was phenomenal, and it meant something, and we would like to do more of it basically. Int 14

That's about telling our story, and we're dreadful at telling our story. But there's good stories out there that aren't being told because we don't know about them. Int 12

I just think there's a wee bit of a gap there with sharing impact and like what's happened and like good stories really isn't it? It's positive stories and it can inspire others to do it. Int 1

Charting these stories of impact was thought to contribute to improved participatory practice within the organisation but also provide an opportunity to raise awareness amongst the general public about the opportunities available to participate in decision- making:

There's a genuine issue around the public just not knowing … I think the public doesn't know that they can, they can be involved, they could engage with the Scottish Government … And I think there's a there's a real responsibility on Scottish Government to communicate a consistent message of this is this is how you engage with us or here's where you can do it and how you can do it. Int 9

Better evaluation of impact was also considered to provide crucial insight into whether or not public participation processes were achieving their intended aims and objectives. This was thought to be particularly helpful in the context of achieving impact at a policy level:

There's a danger that we say, oh, let's run this thing. We'd like policy impact, but it doesn't matter if it doesn't get it because, but if you know if the promise made is policy impact, then we should definitely be judging it in those terms. Int 7

Indeed, understanding impact on policymaking processes as a result of public participation was thought to sit clearly within the gift and responsibility of Scottish Government, with interviewees describing the institution as being best placed to analyse and scrutinise where and how public participation processes were influencing policy outcomes and decision- making processes:

Influence on the policies or priorities or resourcing or sometimes even the understanding of decision-makers about issues, those at a policymaker level, is a difficult thing to evidence … because we are outsider organisations. So sometimes Scottish Government will say to us well show us the impact you've had, and our reaction is to think well you tell us. Int 4

I don't think we've then gone back and looked to say actually as a result of all this participation, our cumulative learning is X, which means these are the changes we need to make to all our policymaking processes.

Int 11

In short, there was a lot of support from interviewees for developing better ways of evaluating and understanding impact. In order to unpick and explore some of the themes identified above, the workshop with participatory practitioners focused on thinking about what is important in evaluating impact in the context of public participation as well as how Scottish Government might begin to develop tools or frameworks to do so. The discussions and reflections from this session are detailed below.

Adapting existing tools: overview of stakeholder workshop

This workshop focussed on gathering participant perspectives on evaluating public participation processes and considered how an adapted version of KNOCA's impact evaluation framework (Demski & Capstick, 2022) might be used in practice (please see Appendix A for more detail). Attendees echoed many of the sentiments of interview participants, particularly around the urgent need to better understand and evaluate public participation taking place, as well providing some helpful comments and additional questions for consideration.

Key themes in participant discussions included evaluating impacts on policy 'changes' as well as how public participation functions within and connects to policymaking settings. In addition, attendees highlighted what they thought to be important steps to consider either before embarking on evaluation of impact or alongside. This included:

• investigating and understanding the purpose of public participation within policymaking settings. One attendee advised that there is not always a shared understanding of why, and to what end, public participation processes have been convened. This meant it was necessary to be clear about the aims and objectives of the process. For example, is the purpose for citizens to shape and inform policy proposals or does the process have more qualitative, research-based, aims of understanding citizen perspectives around a particular issue?
• ensuring that any evaluation of impact was preceded by preparatory work with relevant policy actors. This was thought to ensure better understanding of the actual policy windows, and area of governance, open to citizen input, and therefore better understanding of whether the participation process had had any influence in that particular space.

The second half of the session focused on considering the use or development of an evaluation framework for understanding public participation across Scottish Government. Attendees were asked to spend some time thinking about how they would apply the framework to a particular participation process. This involved considering the following questions:

• Which areas of impact would you plan to evaluate for this example of public participation?
• What methods would you use to understand or evaluate these areas?

The exercise stimulated some helpful comments and insights from the group, detailed in the post-it notes below.

It is important to think about the interconnectedness of the different areas and types of impact within the framework.

There appears to be a need to think about both external and internal evaluation of public participation processes.

For example, alongside understanding policy impact, how does the convening of one process build organisational and individual learning?

Timescales are very important.

There is a need for short-, medium-, and long-term plans for evaluating impact of public participation processes. This needs to be articulated in the designing of participation processes.

Understanding longer-term policy legacies of public participation is complicated and requires dedicated staff resource and planning.

Developing some baseline data and metrics around the use of public participation across Scottish Government alongside evaluations of individual processes would be helpful.

Linked to this, finding ways to track the various uses of the outputs from participation processes, across Scottish Government, would be helpful in capturing both intended and unintended impacts.

Recognising different types of impact, outside of direct policy change was considered important, for example influencing wider political debate.

It was acknowledged that it was difficult to measure / keep track of how participation processes 'impacted the general policy conversation'. Similarly, it was important to acknowledge any 'unknowns', for example when policy action is attributed to citizen influence when that action might have happened in any case.

Evaluation needs to consider, in an in- depth way, the conceptual impacts on policy stakeholders, particularly those in positions of power.

This could be illuminating in the context of Scottish Government to help understand how public participation processes are perceived and experienced by the people they are seeking to influence.

When trying to understand impact, it was considered important that evaluation of the process connected to evaluation of the eventual policy outcomes and whether it 'addressed the concerns that were raised in the first place by the participants in that deliberative process'

It was suggested that citizen participants need to be considered in more detail within any evaluation framework:

What is the longer-term impact on participants, particularly in the context of further participation and action on their part?

Citizen participants could / should be involved in designing evaluation, what metrics would they like to see measured and assessed?

In developing any evaluation framework, it is important to have a focus on resourcing.

The ways identified for evaluating impact were varied, in-depth and time- consuming. For example, qualitative and quantitative methods, multiple surveys and in-depth policy analysis and evaluation.

Thinking about how evaluation would be resourced needs to be considered at the point of process design.

There was broad agreement that the exercise, while useful, did not provide enough time to explore the subject of impact and evaluation in enough detail. The framework used was considered to be a helpful starting point for developing thinking within Scottish

Government around how to understand public participation processes taking place across the organisation. Attendees were interested in and motivated to be involved in future discussions around evaluation and impact. Given the range of expertise and experience which exists in this group, there is great potential to explore, investigate and develop the ideas shared above and work towards better understanding of the impact of public participation on policymaking in Scotland.

Key insights: developing an evaluation framework

Key insight 14: Developing robust evaluation of the impact of public participation processes is considered urgent and essential work. The responsibility for this sits squarely with Scottish Government, particularly when understanding how and if public participation processes influence policymaking.

Key insight 15: There is a lot of enthusiasm and interest in developing evaluation framework(s) which might be helpful in a Scottish Government setting. Expert stakeholders from both within and outwith the public administration are able, and willing, to support the development of this work.