Flexible workforce development fund: first year evaluation

6. Conclusions and Recommendations

This final Chapter presents our overall conclusions and recommendations, and is based on all of the evidence presented in earlier Chapters.

6.1 Conclusions

We present the evaluation conclusions in line with the research questions identified by the Scottish Government at the outset of the study.

6.1.1 Fund Management and Administration

Is The FWDF Well Managed

The FWDF is a brand new programme that supports workforce development within levy-paying employers in Scotland. The evaluation findings (and conclusions) need, however, to be considered within the context that the Fund is still at a relatively early stage in its delivery. The Fund was formally launched in September of last year, and its first year of delivery is due to come to an end on 30 ^th June 2018 (albeit training contracted by this date will continue after this point).

Like any other new grant-giving project or programme, the first year has been rightly pitched by partners as a pilot year, and it has been a learning curve for the main partners’ involved. In particular, for those most closely involved in the Fund. This includes the SFC (responsible for the day-to-day management and administration of the Fund) and the college network (delivery vehicle).

Our overall assessment is that based on employer and stakeholder interviews, the Fund appears to be well managed, although there is some scope for improvement in certain areas.

The SFC has long-established relationships with the college network in Scotland, and has allocated the £10 million across the colleges/regions using an appropriate rationale and approach. In year one, the Fund has not been over-subscribed (as had perhaps been expected), and it seems entirely practical that there is flexibility in the management and administration of the Fund so that any unallocated funds can be reallocated to other colleges/regions where there is a clear unmet demand. Indeed, this point has almost been reached for Borders College and Fife College.

Deadlines for the submission of applications and for the contracting and delivery of training have changed several times in the first year, and it is apt that the SFC has listened and responded to feedback on this issue.

That being said, this posed certain challenges for the college network and employers (employers also found it confusing and frustrating), and suggests a need to think through appropriate timescales as these have not always been workable or practical. There needs to be sufficient time for initial enquires/contacts to be dealt with, for colleges to undertake a training needs analysis, and for applications to be fully worked up and agreed. Making timely (and consistent) communications messages about any planned changes to the process will also be important.

Guidance documents for the Fund have been developed for colleges and for employers, an application process adopted (covered below), and an Online Portal and Reporting Templates designed to enable access to real-time data on the Fund. Some points worth noting including:

• guidance documents – there appears to be some signs of different levels of knowledge and understanding of different aspects of the Fund (among both colleges and employers). This suggests a need for the guidance documents to be reviewed in the first instance to ensure clarity for those using it; and
• online portal and reporting templates – there is scope for improvement both in the monitoring process and the range of information and data collected. The main issues with the online portal/template are incomplete data fields, different interpretations of the guidance, and consistency and regularity of input. In the main it collects the right information, however, there is scope to add a few more indicators. A balance will, however, require to be struck between capturing the right data and not over-burdening the college network.

The final point on Fund management centres on the lack of a national promotional campaign to raise awareness of the Fund at the outset. Responsibility has rested with each college to promote the Fund – good efforts have been made and the Fund is beginning to achieve some traction. However, the inherent risk is that this leads to inconsistencies of approach, language, messages and communications.

Ease of finding out about the Fund was one the main areas of dissatisfaction (Employers Survey). Marketing and promotional efforts will therefore continue to be required to make further inroads, and for the college network to reach more levy-paying employers.

How Well Does The Application Process Work

An application process (and associated forms) have been designed by the SFC, and responsibility then rests with the college network to identify, engage, and support employers through the application and approval process.

The application process appears to work well in the main. This is evidenced through feedback from the college network, wider stakeholders and employers. The process itself is considered to be relatively straightforward and simple, and the majority of employers were satisfied with initial contact and support from colleges, the ease of completing the form, and the time taken to receive a decision. It is worth nothing, however, that there has been a considerable amount of hand-holding through the process by college staff, including support to complete application forms.

One aspect worthy of further comment is the training needs analysis. A relatively high proportion of employers did not recognise that this had been undertaken, and possibly reflects a number of factors. In part this might relate to the informal nature of the training needs analysis (and the guidance specifies that it should be proportionate), however, it might also relate to the tight timescales as discussed earlier. The training needs analysis is a crucial element in the process, and one that should be undertaken.

What Lessons Can Be Learned From The First Year of Fund Implementation

Lessons learned have been identified in the sections above.

One of the main lessons relates to the collation of monitoring data on the Fund. Our review found various issues, including significant data gaps and inconsistency of input, which made analysis difficult and at times time-consuming. Good monitoring and availability of data is a pre-requisite for future evaluations, and is hugely important as a useful management tool for organisations.

Our conclusion is that in order to be truly effective and of practical use, the data must be accurate, complete and up to date, and needs to feed directly into policy-making through regular management reporting.

6.1.2 Colleges

How Well Is The FWDF Working for Colleges

The FWDF has provided the college network with an additional (and valued) mechanism in its “tool box” through which colleges can actively engage and support not only some of their existing employer contacts but also larger employers that it might not have engaged with previously. A broad range of approaches to employer engagement have been undertaken, and are what we might expect for a programme of this nature. Good progress is being made to raise awareness of the Fund – but there is more do to reach to and engage a wider audience.

Despite the Fund getting off to a slow start, some traction and momentum is beginning to be made in its delivery. Uptake of the Fund is, however, varied across the college network – although it is important to also consider any waiting lists.

Some colleges are, however, doing better than others, and a number of factors are likely to be at play. While the Fund generally works well for colleges, there have been some issues, namely:

• the lack of a levy-employers contact list;
• tight lead-in times and changing deadlines (see above); and
• resource intensive nature of some of the activity (e.g. marketing and promotion, time taken to raise awareness and stimulate demand, hand-holding employers through the process, including support with application form completion, etc).

The college network do not receive any funding from the FWDF to support administration and delivery, and given that the Fund is only part of the college network’s commercial activity, for some this has impacted on other work commitments (e.g. engagement with SMEs). This is not a sustainable position, and colleges will therefore require to consider how commercial work, of which the FWDF is part, is adequately resourced in the future.

This process is happening now that Year 2 of the Fund has been announced, however, short-term year-to-year project funding could make this difficult for colleges to appropriately plan for (i.e. short-term funding does not address the longer term revenue implications associated with employing additional staff to support project delivery).

In order to make the Fund work as flexibility and efficiently as possible, most colleges are drawing on bank staff to support delivery of training and/or backfill timetabled lecturers’ posts who also deliver FWDF training. This seems a sensible approach, and there is some evidence that colleges are bringing in bank staff with particular specialisms to meet the training needs of employers.

Do Colleges Have Adequately Staffed and Skilled Business Development Departments To Provide Employers With The Right Support

The college network in Scotland generally has well-established structures and processes in place to support employer engagement, and are well placed to support delivery of the FWDF. The Further Education sector has an existing close working relationship with employers – this includes the development and delivery of work-based skills and producing a pipeline of skilled employees across different sectors - and ensuring that courses remain relevant. Of course, much of this pre-existing engagement and support is aimed at SMEs (some of whom will be, but the vast majority are unlikely to be levy-payers).

Typically this activity has been led through each college’s Business Development teams (or similar), albeit some of the smaller colleges do not have this function, and it therefore rests with various members of college staff. As might be expected, those colleges with larger Business Development teams have had greater capacity to undertake employer engagement activity, and without a levy-paying employer contact list it has been resource intensive. As such, an issue might be more about capacity rather than not having the appropriate skills.

As highlighted above, a broad and appropriate mix of approaches have been adopted by the college network to identify and engage employers, and feedback from the Employers Survey is that the majority of employers were satisfied with initial contact and help and understanding of the business’s needs.

A good example is North Lanarkshire College who put some staff through Training Needs Analysis training to enable its team to provide employers with the right support.

Impact On College Employer Engagement

Evidence on impact on college-employer engagement comes from the consultations with the college network and employers survey, and there are a number of positive messages.

The main feedback from the college sector is that the Fund has provided a new opportunity to not only engage with some of its existing clients but also that it has brought the colleges closer to larger employers. Strengthening existing and developing new employer relationships was unanimous feedback.

Some wider points to note from the Employers Survey were that:

• 25% had not previously engaged with the college sector prior to the Fund;
• 29% have accessed follow-on support from the college since the FWDF, and 59% report that they are likely to access additional support in the future (e.g. Modern Apprenticeships, workforce development training);
• 48% reported strengthened relationships with colleges, and 22% expect this to be achieved in the future; and
• 42% reported increased engagement with college(s).

6.2 Employers

How Well Is The FWDF Working For Employers

Overall, a fair assessment is that the FWDF is working well for employers. This is evidenced by the relatively high levels of satisfaction with initial contact and help, the application process, with the training delivered, etc (see also later sections for wider evidence on how the Fund works well for employers).

Wider evidence from the Employers’ Survey that helps show that it is working well includes the following:

• 79% reported that the training was value for money;
• almost all would recommend the Fund to others (96%); and
• all employers expressed an interest in accessing the Fund in future years.

These are all positive messages and suggest the potential for a strong level of repeat business for the FWDF in Year 2.

Where the Fund is perhaps not working as well for employers – and some of the points below were raised by employers, colleges, and wider stakeholders - include:

• ease of finding out above the Fund and timescales (see above);
• the scale of the funding available - £10,000 may be too small to be of interest to some levy-paying employers;
• in some cases employers appear to be offered a prescriptive “menu” of training; and
• there seems to be some confusion among employers about which college they need to work with (i.e. in the region they are based) – this is not the case, but seems to be a strongly held view among employers.

In addition, there is some support among wider stakeholders for an extension of the current delivery model to include the college network and all other accredited training providers (e.g. private training providers). There is some support for this from the Employers Survey and being able to access training from more than one college. This would help the Fund work better for national and/or multi-site employers.

How Many Employers Accessed The Fund

The monitoring data shows that 508 employers have accessed the Fund in its first year, and there are a further 115 employers working through the application process (i.e. waiting lists). Taken together, this is 623 levy-paying employers. As no Fund level targets have been set, it is difficult to comment on whether this is good, bad or somewhere in between.

However, there are estimated to be 4,000 levy-paying employers in Scotland, and the figure of 623 employers therefore represents 16% reach. This is a good start, however, there is plenty of scope for this to be increased as awareness of the FWDF increases, and as the Fund moves into its second year of delivery.

What Are Employers Using The Fund For

Both the monitoring data and employers’ survey show that employers are using the Fund for a wide range of training. In the main this has been leadership and management, health and safety, IT and digital, and technical/specialist training. These were the main areas in which employers reported skills gaps.

Did Training Address Employers’ Skills Gaps

As highlighted above, the main skills gaps identified by employers match the main types of training undertaken. Wider feedback from the survey is that:

• 75% agreed/strongly agreed that the training the college can provide matches training needs; and
• 90% agreed/strongly agreed that the training has/will help address skills gaps.

To What Extent Are There Sectoral / Regional Differences In How Employers Access The Fund

Some 70% of employers that have accessed the Fund are private sector employers, which is broadly in line with the proportion of large businesses (250 employees+) in Scotland (72%) ^[11] .

The monitoring data does not currently collate data on the business sector (e.g. food and drink, construction, etc) the employers operate in. The information is, however, captured in the application forms, and could be captured via the Online Portal in future. The employers’ survey did, however, engage with employers operating across a diverse range of sectors which is a positive sign.

In terms of Fund uptake by region, as highlighted above this has varied. This is likely to reflect different levels of pro-activeness across the network, internal capacity within the college network, and wider factors including the size of the patch and FWDF budget.

To What Extent is Emerging Employer Demand Being Understood and Met

A mixed picture has emerged from the evaluation regarding the extent to which employer demand is being understood and met.

On the one hand, feedback from the Employers Survey was positive regarding the extent to which colleges understood their needs, and whether the training has/will meet their skills gaps. This suggests that in the main employer demand is largely being understood and met.

On the other hand, a relatively high proportion of employers did not recognise having had a training needs analysis undertaken as part of the application process, and a picture has emerged of a menu of training being offered to employers rather than the Fund being demand-led. The extent to which the latter is happening across the college network is hard to tell, and in part might reflect the nature of initial timescales for colleges to have employers’ training contractually agreed and delivered.

How Consistent Is Employer Support

The college sector appear to have adopted a broadly similar approach to its engagement with employers, but as highlighted above some have had more success in converting engagement to approved awards.

In many cases, as a starting point, colleges have pulled together their list of existing training provision as part of the conversations they have with employers – however, while this might not have been the intention, there is a sense from the Employers Survey (and some wider stakeholders) that this menu is viewed by employers as an exhaustive or prescriptive offer.

Did The FWDF Meet Employer Expectations

A positive message from the Employers’ Survey is that employers’ expectations are largely being met. Where training is underway or complete, 79% of employers reported that the training has either met or exceeded their expectations.

Can Employers Easily Access The Fund

As reported earlier, while there have been some issues raised on the ease or otherwise of employers finding out about the Fund, the main feedback from employers on the application process was largely positive.

Why Are Eligible (Levy Paying) Employers Choosing Not To Access The Fund

The approach that required to be adopted for the primary research with levy-paying employers resulted in a sample of employers that a) were all aware of the Fund, and b) the vast majority were engaged in some way (i.e. training underway/complete, working on an application, etc).

The survey therefore only consulted a handful of eligible (levy-paying) employers who have chosen not to access the Fund. Some of the factors identified by these employers in informing their decision not to apply were:

• tight (and changing) timescales for application form to be submitted and for training to be delivered; and
• the training the college is able to offer does not match up to needs.

Wider factors include scale of funding (too little) and would prefer to choose which college or more than one college to provide the training.

6.3 Employees

How Well Is The FWDF Working For Employees

Given the early stage at which the Fund is at, much of the primary research focussed on employers. That being said, employers were asked whether the training has/will benefit employees, and in what ways.

While some employee benefits achieved to date were reported, benefits were typically expected to be achieved over the next year (and in part reflects the fact that training was underway or not yet started in some cases).

The main benefit achieved to date for employees was reported as increased skills levels (53%), and the expected benefits reported include:

• increased skills levels – 63%;
• increased confidence – 52%; and
• gained increased levels of qualifications – 46%.

These findings are further supported through the Case Study interviews with employees ( Appendix F).

How Many Employees Were Trained

A review of the monitoring data shows that circa 15,000 employees were to be trained by the FWDF. However, what the data does not provide detail of is how many employees were actually trained and completed the training.

To What Extent Does Learning/Training Result In Qualifications/ Certification Gained

Given some of the challenges described earlier regarding the monitoring data for the Fund, it is not possible to report on qualifications/certification gained in any robust or meaningful manner (data gaps, etc). In addition, the data captures information on the training to be undertaken, and not on the final outcome of that training for employees.

As highlighted above, however, 70% of employers reported that training would lead to qualifications/certification.

6.4 Future Monitoring and Evaluation

How Should The Long-Term Impact Of The Fund Be Monitored and Evaluated

Data being gathered via the Online Portal currently captures some information and data on inputs (e.g. cost of training), activities (e.g. type of training), and some immediate outputs (e.g. number of employees to be trained, etc). It does, however, not move beyond this to capture any emerging outcome and impact data (i.e. what difference the Fund has made).

Good monitoring and availability of data is a pre-requisite for evaluation. There is scope for current monitoring arrangements to be adapted and strengthened to allow for the wider capture of outcome data (e.g. number of employees completing training, number/type of qualifications gained, etc).

We outlined earlier some wider challenges encountered in the analysis of monitoring data (e.g. data gaps, etc). A more structured process needs to be put in place to improve both the quality and consistency of data provided by the college network, and which can then feed into any future evaluations of the Fund.

Routine monitoring will only be able to provide so much information on impact. There will always be a requirement for wider independent evaluation work to be undertaken to assess the longer-term impact of the Fund.

The evaluation of the first year of Fund has sought to capture benefits and outcomes achieved (especially for employers), however, given the early stage of the Fund it was not appropriate to undertake an Economic Impact Assessment. However, this could be considered towards the end of Year 3.

See also Appendix G for logic model.

What Data Are Required To Monitor and Evaluate The Impact Of The Fund Going Forward

As highlighted above, there is scope to enhance current monitoring arrangements, and this is covered in more detail in the Recommendations section.

Are There Any Employer Trends Or Indicators To Help Decide How A Second Phase of the FWDF Is Best Delivered

We outlined above some of the ways in which the Fund does not currently work well for employers ( Section 6.2).

The Fund is, however, largely working well for employers, and anticipated future skills gaps appear to match current issues (e.g. leadership and management, technical, IT and digital, and succession planning). More of the same is therefore required.

An ageing workforce and technological change (as well as Brexit) were identified by many employers as the main drivers of future skills/labour problems – albeit the Fund on its own cannot address all of these issues.

Are There Any Recommendations Around Measuring Fund Impact On Productivity

Measuring Fund impact on productivity should be assessed as part of any future evaluation work (Economic Impact Assessment).

Potential approaches to measuring productivity include capturing both the gross and net additional effects on Gross Value Added ( GVA) and/or GVA per head. Essentially, increased productivity results in greater returns from the resource input – which can be driven by a decrease in resource inputs, an increase in outputs, or a combination of both.

GVA is the Scottish Government’s preferred measure of impact, and there are well established approaches to measuring the economic impact of company level interventions, see the Scottish Enterprise guidance ^[12] .

The guidance provides an example question set to interrogate both the gross and net impact of the intervention. Any survey questions will therefore require to seek employers’ views on issues such as the counterfactual position (what would have happened in the absence of FWDF support), displacement effects, and other factors necessary to calculate estimates of net additional economic impacts.

6.5 Recommendations

Overview

Overall, the Fund is welcomed and valued by all stakeholders, and is providing benefit to colleges, employers and employees. Many of the recommendations outlined below are process related, and based on learning from the first year of the Fund.

At this stage and based on the above conclusions, we would not recommend any major or wholesale changes, but instead an incremental approach which can show that we have listened and made some adjustments. However, it will still require a few years to understand how well the Fund works before any macro changes should be considered or progressed.

We have therefore highlighted some short term improvements to the Fund together with some longer term issues for further consideration, once the FWDF has run for a few years.

Short Term Improvements for Consideration

• continue to adopt a flexible approach to budget allocation
  • the inbuilt flexibility of the Fund to reallocate additional budgets to those colleges/regions with evidence of unmet demand should continue;
• college allocation
  • consideration should be given to allowing employers to choose which college they engage with in the delivery of training – this could be a college in their region, a college outwith their (main) region, and/or more than one college
  • review ways to reduce competition within regional networks to ensure that it is the most appropriate college who delivers the training;
• training needs analysis
  • while some colleges/employers are completing this step not all appear to be (albeit this might be being undertaken relatively informally). In many cases a training needs analysis might not required, but some form of justification/rationale should be provided. Consideration of alternatives should be considered;
• improve marketing and awareness
  • consideration should be given to a national marketing programme
  • at a minimum, marketing materials and communications issued by the college network need to convey consistent and common messages to avoid any unnecessary confusion or misinterpretation among employers;
• data monitoring and reporting
  • in order to provide meaningful and robust information about the performance of the Fund, there are some steps which could be taken to improve both the consistency and quality of data provided, and this should be undertaken in partnership between the SFC and college representatives:
    • guidance should be reviewed to ensure that it is clear what is being asked for – more detail is perhaps required in places, and some worked up examples could be provided
    • there is a role for the SFC to review the forms when they are submitted, and to go back to colleges regarding any points of clarification and/or to plug any gaps
    • there might be value in hosting a session on monitoring with key college personnel to ensure that there is a clear and shared understanding of how the form should be completed, and what data would be useful to record
    • consideration could be given to developing a “drop-down” list of options under each indicator based on experience from Year 1. This would aid consistency (and ease) of input;
• output and outcome indicators
  • there would also be merit in expanding the indicators used for monitoring purposes to capture the following data:
    • business sector
    • size of business
    • existing relationship with college or new employer contact
    • existing training course and/or new/bespoke training course
    • cost of training should be broken down by FWDF award
    • number of employees completing the training
    • number/type of qualification/certification gained
    • private leverage; and
• aiding evaluation
• o in light of the challenges encountered with the Employers Survey, consideration could be given to amending the application form - and that a condition of the award is that employer contact details might be passed to a third party organisation for the purposes of evaluation only. The same could hold true for employees.

Long Term Recommendations for Consideration

• level of grant
  • consideration should be given to increasing the scale of funding available per employer. For example, this could range from £10,000 to £25,000 and could be scaled on the amount of levy paid by employers;
• widen delivery of training
  • consideration of allowing employers to access non-college and commercial training providers to deliver training if they are more appropriate; and
• future evaluations
  • any future evaluation work should include an economic impact assessment, and include a greater emphasis on securing feedback from employees and from eligible (levy paying) employers who have chosen not to access the Fund (albeit there are challenges in doing this in the absence of a levy-paying employer contact list).

We would recommend that these should be reviewed as part of an interim impact evaluation which should be undertaken towards the end of Year 3.