Child Poverty Action Group - advice, information and training services: evaluation

Evaluation of Child Poverty Action Group (CPAG) in Scotland’s second tier advice, information and training services.


3. Secondary data – analysis and findings

This section reports on the key findings from the evaluation of the secondary data provided by CPAG in Scotland up to 2018/19.

CPAG in Scotland provided a range of data and information on their second tier advice, information and training services for review in the evaluation. These included:

  • Quarterly grant funding progress reports detailing the number of cases supported by the Advice Line, number of participants attending training courses and events per topic, number of handbooks/factsheets/e-bulletins produced and distributed per topic, availability of the Appeals Toolkit, number of times E-learning courses were accessed as well as the number of people enrolled and the number of participants attending the annual welfare rights conference
  • Annual conference evaluation data containing findings from evaluation forms completed by attendees of the annual Welfare Rights Conferences from 2014-2018
  • Advice Line Database data containing details for each inquiry made by frontline workers/organisations/individuals to the Advice Line including the date, level of inquiry, subject and organisation/type of organisation making the enquiry
  • Advice Line use survey data containing findings from the surveys launched annually, assessing the Advice Line Service users' views and ratings of the Advice Line
  • Evaluation forms for E-learning courses containing feedback for the courses from participants as well as suggestions for improvements

3.1 Advice Line data

In total, the data provided by CPAG covered 16,413 enquiries, from 2014/15 to 2018/19. This section will consider the available data on the Advice Line, before briefly reviewing available feedback on training and data about use of online resources.

Overall there has been a 42% increase in the use of CPAG's advice helpline in the last five years (see Figure 2).

Figure 2. Use of CPAG Advice Line, 2014/15-2018/2019
Graph: Call volume usage of CPAG’s Advice Line from 2014 to 2019

This increase in call volume in the last five years has been largely driven by the effect of increased enquiries about Universal Credit (UC), which has increased by a far greater extent than other types of enquiries (see Figures 3 and 4).

Figure 3. Enquiries to CPAG Advice Line, by enquiry type, 2014/15-2018/2019
Graph: Number of enquiries made to CPAG’s advice line by enquiry type from 2014 to 2019
Figure 4. Percentage Increase/Decrease in enquiries to the Advice Line by enquiry type, 2014/15-2018/19
Enquiry Type Percentage change
Benefits for Students 96%
DLA/AA -11%
ESA 38%
HB -2%
IS -41%
JSA -60%
PC -23%
PIP 74%
UC 5922%
WTC/CTC -12%
Total 71%

Universal Credit-related enquiries have increased from 32 in 2014/15 to 1,927 in 2018/19, a percentage increase of 5922%, far exceeding any other enquiry type and comprising 40% of all calls made to the advice line.

The most frequent users of the Advice Line between 2014/15 to 2018/19 were from Citizens Advice Bureaux (see Figures 5 and 6). Citizens Advice Bureaus (CABs) made more than 30% of the total number of enquiries to the Advice Line every year. From 2017/18 to 2018/19 the Advice Line saw an increased usage by organisations defining themselves as "Other voluntary sector" while, at the same time, organisations defining themselves as "Support providers" appear to have decreased by a roughly similar amount. The reason for this is unknown, however, it is likely that this is a data anomaly and "Support providers" may have been included in "Other voluntary sector" in this time period. "Local Authorities", "Housing Associations" and the remaining organisation types (Other/Not defined) appear to have made almost the same quantity of enquiries to the Advice Line over the period 2014/15 – 2018/19.

The number of enquiries made via the Advice Line did see an increase among all organisation types over the reference period, with the exception of those identified as 'Support providers' which, as mentioned above, could be an anomaly.

Figure 5. Enquiries to CPAG Advice Line, by Organisation Type, 2014/15-2018/2019
Graph: Number of enquiries made to CPAG’s advice line by organisation type from 2014 to 2019
Figure 6. Percentage of enquiries made by Organisation Type
Charts: Percentage of enquiries made by organisation type per year from 2014/15 to 2018/19
Figure 7. No. of Enquiries made per Organisation, 2014/15-2018/19
Graph: Number of enquiries made per organisation to CPAG’s advice line from 2014 to 2019

The majority of organisations make one enquiry to the Advice Line (58% averaged between 2014/15 and 2018/19), with a small number (8%) making 10 or more.

Call volume varies a little per quarter, but overall represents similar ranges across the year when averaged between 2014/15-2018/19 (see Figure 8).

Figure 8. Call volume per quarter, averaged across five years, 2014/15-2018/19
Chart: Call volume to CPAG’s advice line, per quarter, averaged between 2014 and 2019

From 2014/15 – 2018/19 most of the enquiries were made via the phone, however, the number of enquiries made by email has been steadily increasing (see Figure 9). There was a sharp decrease in the number of enquiries made by phone between 2014/15 and 2015/16 and, since then, a gradual increase at an approximately similar rate as the increase seen in the number of email enquiries.

Figure 9. Email and phone enquiries per year, 2014/15-2018/2019
Graph: Number of email and phone enquiries made per year from 2014 to 2019

3.2 Advice Line survey user data

CPAG in Scotland runs an annual survey to see the overall opinion of people with regards to its Advice Line. Survey results were consistent across the years. The findings, averaged from the data collected between 2014/15 and 2018/19 is presented below:

3.2.1 Advice Line

  • 100 % of respondents were satisfied with the Advice Line's overall level of service
  • An average of 20% (49 out of the 249 respondents between 2014/15 and 2018/19) reported that they had trouble contacting the Advice Line. Over the years the percentage has fluctuated but seen an overall increase from 10% of respondents in 2014 to 25% of respondents in 2019. Text responses across all years reported similar results, that the advice line is sometimes busy, even if they call multiple times
  • 100% of respondents reported that the staff explained the advice clearly
  • 100% of respondents reported that staff were knowledgeable
  • 100% of respondents reported that staff were approachable
  • 100% of respondents reported that the Advice Line helps them feel more confident that their advice is up-to-date and accurate
  • 100% of respondents reported that the Advice Line helps them resolve issues for clients more quickly
  • 100% of respondents reported that the Advice Line helps them achieve better outcomes for clients
  • 100% of respondents reported that they feel better equipped to deal with similar cases
  • 97% of respondents reported that the Advice Line helps them prepare information materials for clients
  • 100% of respondents reported that the CPAG Advice Line is the best place to get second-tier benefits advice
  • 96% of respondents reported that they shared the information they got from the Advice Line with colleagues

3.2.2 Outcomes for clients after using the Advice Line

  • 38% of respondents reported that their client received general advice and information about eligibility for benefits
  • 12% of respondents reported that their client was helped to make an application for benefits
  • 16% of respondents reported they achieved financial gain for the client
  • 9% of respondents reported that they achieved reconsideration/the appeal was resolved in the client's favour
  • 31% of respondents reported that they referred the client on to another agency for more specialist advice

The overall findings from the surveys conducted by CPAG in Scotland suggest that most users are overwhelmingly supportive with regards to the quality of service provided. The only reported negative was increased difficulty in accessing the advice line.

3.3 E-learning

From 2014/15 to 2017/18 E-Learning courses were available on the following topics:

  • Universal Credit
  • Surviving the Appeals Process
  • Scottish Welfare Fund
  • Benefits Overview
  • Making the most of the Handbook
  • Planning and Paying for Funerals
  • Supporting Hard-up Households

In this same time period E-Learning courses were accessed 72,419 times. 93% of respondents rated the E-Learning courses as either good or excellent.

Between 2014/15 and 2018/19 a total of 806 courses and events were held in-house by CPAG in Scotland. With the exception of 2016/17, where there was a small decrease, the number of training courses held per year has remained fairly consistent (see Figure 10., below).

Figure 10. Number of training courses/events held, 2014/15-2018/19
Graph: Number of training courses and events held per year from 2014 to 2019

Demand for training courses has been varied, depending on the subject, and the changes in the courses available in a given year make it challenging to generalise about overall trends. However, since its introduction in 2017/18, the training course on 'Universal Credit' has been the most widely attended. This can be seen in Figure 11. below.

Figure 11. No. and proportion of participants attending courses by category, 2014/15-2018/2019
Year Topics No. of participants No. of courses No. of events Total courses & events for the year
2014-2015 Families/early years 334 11 7 178
Students 226 7 7
Children in care/disabled children 268 6 6
Minority ethnic 333 16 2
Welfare reform 1160 43 16
Other 1046 51 6
2015-2016 Families/early years 432 13 9 161
Students 345 15 5
Children in care/disabled children 181 7 4
Minority ethnic 310 7 9
Welfare reform 743 26 10
Appeals representation 78 4 0
Other 957 47 5
2016-2017 Families/early years 284 12 4 141
Students 182 7 3
Children in care/disabled children 108 4 4
Minority ethnic 224 5 3
Welfare reform 1037 22 22
Appeals representation 50 4 0
Other 712 43 8
2017-2018 Families/early years 429 16 6 171
Students 289 12 5
Children in care 88 3 1
Disability and carers 541 16 7
Minority ethnic 115 3 4
Universal credit 987 50 2
Appeals representation 122 7 0
Other 601 28 11
2018-2019 Families/early years 359 14 9 155
Students 262 12 4
Children in care 198 4 3
Disability and carers 261 8 5
Minority ethnic 228 4 4
Universal credit 1002 48 6
Appeals representation 68 5 0
Other 530 21 8

With regards to the factsheets downloaded there has been an overall increase over this period of time. It is unclear exactly why there has been an increase, it is possible that this is due to increased use of computers and other internet-enabled devices over this period. This can be seen in Figure 12. below.

Figure 12. Downloads per CPAG factsheet, 2014/15-2018/19
Graph: Number of downloads per CPAG factsheet per year from 2014 to 2019

The use of e-bulletins was fairly consistent from 2014/15 to 2017/18 however there was a sharp decline in 2018/19 as a consequence of the introduction of General Data Protection Regulations (GDPR) which saw the mailing contact list reduced significantly. See Figure 13. below.

Figure 13. Use of e-bulletins, 2014/15-2018/19
Graph: Use of e-bulletins from 2014 to 2019

The number of handbook page views, over the five year period, initially declined and recently increased. It is unclear why this is the case. The data can be seen in Figure 14. below.

Figure 14. Handbook page views, 2014/15-2018/19
Graph: Number of handbook page views from 2014 to 2019

The number of appeal toolkit downloads increased in between 2015-2017, and have declined since. It is unclear why this is the case however it is possible that users are finding the appeal toolkit of less use or less necessary. This could be explored further by CPAG in Scotland. The data can be seen in Figure 15. below.

Figure 15. Appeals toolkit views and downloads, 2014/15-2018/19
Graph: Main groups that receive advice or support from respondents’ organisation

3.4 Annual Welfare Rights Conference

From 2014 – 2018 there were 1138 attendees at CPAG in Scotland's annual Welfare Rights Conferences. Figure 16. Below, shows the distribution for each year.

Figure 16. Attendees at annual Welfare Rights Conference, 2014/15-2018/19
Graph: Number of attendees at annual Welfare Rights Conference from 2014 to 2019

The table below shows the percentages of respondents to the post-conference evaluation survey that felt that conference was poor/fair or good/excellent.

Figure 17. Rating of annual Welfare Rights Conference, 2014-18
How would you rate the conference overall
Year Poor/Fair Good/Excellent
2014 8% 92%
2015 2% 98%
2016 2% 98%
2017 4% 96%
2018 2% 98%

Overall, between 2014 – 2018 the vast majority (96%) of respondents rated the annual Welfare Rights Conferences as either good or excellent.

3.5 Summary of Findings from Secondary Data Analysis

The principal findings from the analysis of the secondary data show that there has been an increase in the call volume to CPAG in Scotland's advice helpline, largely driven by enquiries about Universal Credit. This coincided with an increase in the number of people attending training courses on Universal Credit which, although only introduced in 2017/18, has been the most widely attended training in the years since. The most frequent users of the advice helpline have been Citizens Advice Bureaux (CABs). Most organisations accessing the advice line make just one enquiry. Call volume appears to stay steady across the year suggesting that demand remains consistent.

Users of the advice line are overwhelmingly supportive of it with the only reported negative being that there has been increasing difficulty, year by year, in accessing it. This could be due to demand increasing at a rate that outpaces resource allocation. Opinions of the E-learning courses on offer and the welfare rights conferences have also been particularly positive however there were concerns raised about the cost and limited geographical locations of the conferences.

There has been an increase in the overall number of factsheets downloaded, however, this may coincide with greater use of computers and electronic devices amongst frontline advisers since 2014/15. The use of e-bulletins remained fairly constant until the introduction of GDPR regulations, which resulted in a sharp decline. The number of Appeals Toolkit downloads increased between 2014/2015 and 2016/2017, but have declined since. It is unclear why this is the case, however, it is possible that users are finding the Appeals Toolkit less useful. The number of handbook page views saw a steady decrease from 2014/15 to 2016/17 and then an increase back to relatively similar numbers as 2014/15 without any clear cause.

Contact

Email: socialresearch@gov.scot

Back to top