Fair Work action plan: evidence plan

2. Monitoring progress and impacts

2.1 This chapter sets out the approach the Scottish Government intends to take in monitoring Scotland’s progress towards being a leading Fair Work nation and monitoring and evaluating the impacts of the actions within the Fair Work Action Plan.

Understanding the impact of Fair Work

2.2 The Fair Work Action Plan set out the detailed actions that will be undertaken to support delivery of the Scottish Government’s Fair Work objectives. These are articulated in the Fair Work Action Plan under five headline actions, covering:

• Public sector and the role of leadership;
• Our ask of employers and support available;
• Support for people to prepare for, access, and sustain Fair Work;
• Making Fair Work the norm; and
• Evidence, monitoring and reporting.

2.3 Monitoring progress against these actions will be an important factor in understanding the impact of the Action Plan on Fair Work in Scotland. However, it is important to recognise that Fair Work is multi-dimensional, and that Scotland’s progress towards being a leading Fair Work nation will be influenced by factors in addition to the delivery of the Fair Work Action Plan (such as the broader macroeconomic environment). The actions set out in the Action Plan will have different contributions towards Fair Work in terms of their scope and attribution. The approach to monitoring and evaluation recognises this and more detail is provided in the following chapters.

2.4 The Fair Work Action Plan recognises that women, disabled people and racialised minorities are less likely to access Fair Work. Their experiences, along with those with intersecting characteristics, are central to understanding the impact of Fair Work. This is therefore intended to be a key component of the monitoring approach set out in the Evidence Plan. It will also be an important feature of the evaluations taken forward and has informed the focus of intended future work set out in Section 3.

Analytical framework

2.5 The Fair Work Action Plan included an initial measurement framework. Figure 1 is a diagram that shows the broader analytical framework underpinning the Evidence Plan. This was developed from the initial framework set out in the Action Plan. The analytical framework sets out:

• High level indicators of Scotland’s progress around Fair Work;
• Progress indicators around the high-level Fair Work actions and the medium-term outcomes consistent with the Fair Work Action Plan; and
• The approach being adopted to monitoring and evaluation within the Fair Work Action Plan.

2.6 The Action Plan also recognised the importance of continuing to develop the broader evidence base around Fair Work. Considering this, we will also look to develop the available data and evidence to support this framework.

Monitoring the Fair Work context, and Scotland’s progress towards being a leading Fair Work nation

2.7 The Scottish Government’s view of what constitutes Fair Work is informed by the framework developed by the Fair Work Convention.^[8] This sets out a conception of Fair Work based on five dimensions: effective voice; opportunities; security; fulfilment; and respect.

2.8 Assessing Scotland’s progress towards being a leading Fair Work nation involves considering data on how Scotland’s performance has changed over time and comparing this performance to that of other countries, where such data is available.

Monitoring Fair Work in Scotland over time

2.9 There are currently several frameworks in place that monitor Fair Work performance within Scotland. The Fair Work Convention has developed a monitoring framework that contains 44 indicators covering the five dimensions of Fair Work.^[9] In most cases, these indicators offer insights into Scotland’s performance since 2016.

2.10 The Scottish Government also has several frameworks that support monitoring Fair Work outcomes within Scotland. There are currently nine national indicators within Scotland’s National Performance Framework^[10] that correspond to aspects of Fair Work. The Scottish Government’s Wellbeing Economy Monitor^[11] also contains Fair Work indicators. These are summarised in Annex 2.

2.11 As there are different dimensions to Fair Work, no single indicator will be sufficient to encapsulate progress in Fair Work. Combining indicators into composite indicators can, in principle, provide a route to generating a single Fair Work indicator. In this case the range of dimensions, and the different aspects within these, may serve to obscure key elements of Scotland’s performance and reduce transparency. Consequently, monitoring Scotland’s progress in Fair Work will require monitoring change across several indicators. This is also important for understanding progress across indicators for different groups within Scotland’s labour market.

2.12 The Scottish Government will work with the Fair Work Convention to support future updates to its Scottish measurement framework in the period to 2026. This is particularly important if new data relevant to the dimensions of Fair Work are developed. The Scottish Government will also work collaboratively with the Fair Work Convention to develop a common narrative around Scotland’s progress on the different dimensions of Fair Work. Given timeframes for producing data and official statistics, it is anticipated that this will require joint working through to 2026 or 2027.

Monitoring how Scotland compares to other countries

2.13 The encouragement of Fair Work, Good Work, or Decent Work is a feature of several countries’ and international organisations’ approaches to labour market development. However, there are different definitions across approaches,^[12] which creates challenges for common international measurement frameworks.

2.14 There are also different approaches to making international comparisons and benchmarking performance against other countries. Comparisons against a broader range of countries may offer a helpful perspective on Scotland’s performance in some areas, but may lead to partial or even no comparison, given differences in data availability, and economic and social characteristics of each country. Conversely, benchmarking against a smaller set of countries may offer richer insights into how Scotland compares internationally but may still only offer a partial perspective on Scotland’s overall international position.

2.15 The Fair Work Convention have recently developed an international framework to allow for comparisons between Scotland and a set of countries identified as being potentially comparable to Scotland^[13] across a subset of Fair Work indicators^[14] for which comparable cross-country data was available. The Scottish Government will support the Fair Work Convention to develop this framework further, and will work collaboratively to agree an approach to reflect Scotland’s progress compared to the countries in the framework.

Monitoring progress for the Fair Work Action Plan

2.16 The actions set out in the Fair Work Action Plan^[15] are intended to deliver the five headline actions set out in paragraph 2.2. Figure 2 sets out the theory of change, showing how successful delivery of the headline actions will contribute towards delivery of the following medium-term outcomes:

• Scottish Government and other public sector organisations demonstrate leadership in Fair Work through embedded Fair Work practice;
• Fair Work is embedded in all forms of public sector financial support;
• Employers are supported to offer Fair Work; and
• People are supported to prepare for, access, and sustain Fair Work.

2.17 The actions, outputs and outcomes set out in the theory of change correspond to and follow on from the actions and outcomes set out in the Fair Work Action Plan. Some actions are general and apply across labour market groups; others are targeted towards specific groups. As such, the actions in the theory of change include those that apply across different groups within Scotland’s labour market, and also those that specifically relate to gender, disabled people, and people from racialised minority backgrounds. The outcomes in the theory of change also align with the equality dimensions that underpin the Action Plan.

Monitoring progress towards outcomes

2.18 Progress on delivering the medium-term outcomes will be monitored through a suite of indicators. There are two types of indicators: those selected to show the direct impact of actions within the Action Plan where attribution is possible; and those which indicate progress towards Fair Work aspirations but which cannot be directly attributed to a single or group of actions.

2.19 Where an action in the Fair Work Action Plan has a direct impact on a Fair Work outcome, the progress indicator for that particular action becomes an indicator to monitor progress towards medium-term outcomes.^[16] Other indicators which capture progress towards Fair Work aspirations are included. These are where we cannot directly attribute impact from a particular action or group of actions, but would anticipate a positive contribution to Fair Work as those actions are delivered.

2.20 Figure 3 sets out the indicators identified for monitoring progress towards medium-term outcomes. It is recognised that there are some limitations to this approach; indicators may not fully capture all dimensions of the outcomes. In such cases, they offer a guide to areas where the evidence base could be developed further. Areas where the Scottish Government will look to further develop the evidence base are articulated in Chapter 3.

2.21 It is also recognised that individual indicators for specific outcomes should not be viewed in isolation, but in conjunction with others for that indicator. Focusing on single indicators in isolation may obscure greater nuance and could be open to misinterpretation. For instance, it is recognised that interpretation of pay gap indicators should be combined with consideration of indicators of employment gaps, and a wider contextual awareness of concentration of employment across sectors and occupations.

2.22 During preparation of this Evidence Plan, individual indicators that could illustrate progress, and where rapid development may be feasible, have been identified. Consequently, several potential indicators are currently being explored. Where this is the case, we will confirm the approach in the Fair Work Annual Report. The only exception to this is where the data is expected to be sourced from the Transformed Labour Force Survey,^[17] where timings may be uncertain.

2.23 A summary of current and envisaged indicators for monitoring progress towards outcomes is set out in Figure 3, and a more detailed list of current and envisaged indicators and their sources are provided in Annex 3. Envisaged indicators, for which investigations as to robust data sources and definitions are currently being explored, include: FE/HE workforce by equality group, public sector workforce by equality group and intersectional gender employment gaps (all based on Annual Population Survey data); low pay in Fair Work agreement sectors other than social care and trade union membership (based on Annual Survey of Hours and Earnings); number of grants awarded that include a Fair Work First criteria; public sector employers publishing pay gap and action plans for equality groups (published as part of PSED reporting); equality groups supported through employability support; and employability progression by equality group.

Understanding impact of the individual actions

2.24 The Action Plan explicitly recognises the importance of understanding the impacts of the policies being taken forward to support Fair Work. However, the activities set out in the Action Plan are of differing scales and duration. This necessitates a flexible approach towards understanding their impacts as they are delivered.

2.25 For actions that contribute towards Fair Work aspirations more generally, narrative reporting will provide a description of achievements and perceived impact. Where it is anticipated that actions are likely to impact directly on achievement of Fair Work outcomes, impacts will be evidenced through progress indicators which capture change around real-world improvements. This approach is set out in Annex 4.

2.26 More detailed understanding of the impact of specific actions would be generated by evaluation activities. It is recognised that the range of actions in the Action Plan, and their differing nature, preclude specific evaluation for each activity individually. The focus of evaluation activity will depend on the scale of the action, its length of time in operation, and the anticipated timeframes to realise outcomes. The focus of evaluation activity will also be influenced by the learning being sought: for instance, understanding how the action has been delivered and the experiences of participants and delivery agents requires process evaluation, while understanding what outcomes have been achieved would require impact evaluations.

2.27 Progress indicators, MI data and narrative reporting will be provided through the Action Plan’s Annual Progress Reports, with the first of these scheduled for the first half of 2024.

2.28 It is also recognised that there are several policies in place to support the delivery of specific actions. With this in mind, it is currently anticipated that future evaluation activities associated with specific actions in the Action Plan will include:

• Fair Work conditionality (Action 2.1);
• Real Living Wage and Hours Accreditation (Action 3.3); and,
• Employability Support (Actions 4.2 and 4.3).

2.29 Existing commitments to evaluation activity related to the Action Plan are included in the Evidence Plan. These include evaluations of the Workplace Equality Fund, Disability Public Social Partnership activity, support to encourage flexible working and the Minority Ethnic Recruitment Toolkit.

2.30 It is anticipated that the different scope, scale and focus of the activities listed above will mean that different approaches will be required for each evaluation. The evidence gathered through management information and narrative reporting will feed into evaluation scoping. It is also currently anticipated that evaluations will be taken forward over a period of several years, to allow sufficient time for policies to be implemented and impacts observed, and recognising the limitations on resources in Scottish Government and the wider research community.

2.31 Table 1 gives an overview of actions contained in the Action Plan and the anticipated approach to analysing and reporting on their impact.

Table 1: Evidence to report on impact of Fair Work Action Plan actions^[19]

Impact reporting approach: Management Information and narrative reporting

Action Plan activities (numbered)

1.1 SG Pay Audit

1.2 Ethnicity pay gap

1.3 SFC Equality Outcomes

1.4 Senior leader network

1.5 Stats dissemination

2.2 Update FWF criteria

3.1 Central FW resource

3.2 Comms strategy

3.4 Learning from WE Fund

3.5 Positive action good practice

3.6 Flexible working guidance

3.7 Anti-racist training framework

3.8 Fair Work agreements

3.9 Strengthen effective voice

4.1 BG review

4.4 SDS/SFC action on learning disabilities

4.5 SDS Career Review

4.6 PHS health offer

Annex B – Fair Work priorities being taken forward across government

Impact reporting approach: Progress indicators and evaluation evidence

Action Plan activities (numbered)

2.1 Fair Work conditionality

3.3 Real Living Wage and Hours Accreditation

4.2 & 4.3 Employability Support

Investigating impact for different groups

2.32 Fair Work is underpinned by the principles of equity and equality of opportunity and outcome for all individual or group characteristics. To understand the impact of the activities set out in the Action Plan, progress reporting will be explored through their impact on different groups.

2.33 This will take different forms, depending on the action concerned and method of reporting. Where data permits, reporting of progress against actions and outcomes will set out data by protected characteristics, in particular for disabled people, women, people from racialised minority backgrounds, and where possible the intersections of these groups. Process and impact evaluations will explicitly consider the experiences of those with protected characteristics. In principle, evaluation evidence will draw on both quantitative and qualitative approaches to develop a broader understanding of the impact of Fair Work activities on different groups in Scotland’s labour market.

2.34 Where possible, monitoring and evaluation activities for the Action Plan will take an intersectional approach. This will follow the existing Scottish Government guidance on intersectionality in policymaking and analysis.^[20] Aligning to available resources, we will: consider intersectionality during project scoping, including our approaches to data collection; produce intersectional analysis where this is possible; supplement data gaps with qualitative research; and consider how to best capture intersectional experiences in our evaluation work.