Child poverty cumulative impact assessment: update

5. Calibration

The poverty rates implied by our model can differ from the official poverty statistics, which are used to measure the child poverty targets. These differences can occur for a number of reasons, including the well-known problem of survey respondents underreporting their benefit income. We therefore adjust (or ‘calibrate’) the model outputs so that they are consistent with the official statistics.

Our method for calibrating relative child poverty takes the percentage-point difference between the modelled poverty rate and the official poverty rate in the latest year of input data, and applies this difference to all scenarios. To calibrate absolute child poverty, we take the difference between the modelled relative and absolute poverty rates in the latest year of official statistics, and compare it to the difference between the official relative and absolute poverty rates to give an adjustment ratio. We then ensure this ratio is held constant in all other scenarios, using the calibrated relative poverty rate as the reference point. This method ensures that we preserve the relationship between relative and absolute poverty that is evident in the official statistics, while allowing this relationship to change over time based on the outputs of our model.

Table 5.1 shows how the calibration method has been impacted by the inclusion of new input data. The outputs of the model are closer to the official statistics in the latest year of input data, meaning that less adjustment is required. Note that figures presented in the table are rounded, whereas our calculations use unrounded figures.

Note: figures rounded to nearest percentage point.

Source: Scottish Government analysis using UKMOD.

The fact that updating the input data has closed the gap between the outputs and the statistics is positive, since it means that our results rely more on the model and less on the calibration. This could be driven in part by the switch from a three-year pool to a two-year pool, which entails that a higher proportion of the input data falls in the calibration year. However, it introduces a degree of volatility, with a notable shift in projected poverty rates since our last publication for reasons that do not necessarily reflect changes in policy or the wider economy.

Calibration methods that make use of multiple years would help reduce this volatility, but would introduce other complications. Since the model simulates a single year at a time, multi-year calibration raises the question of which simulations should be calibrated to which statistics using which input data, with numerous possible permutations. Furthermore, we may consider the latest year of data to be the most reliable guide to any future discrepancies between the model and the survey. This may be particularly true during the roll-out of significant policy interventions such as the Scottish Child Payment, which was introduced in 2021.

In light of these trade-offs, we have opted not to make changes to the calibration methodology at this time. However, we acknowledge that our method generates volatility. This is shown in Table 5.2, which presents estimates using alternative input data and calibration methods. Consulting the table row-wise, we can see that, with the change of input data, estimated poverty rates have shifted by around 4 percentage points when using our single-year method, but have stayed relatively constant when using a possible multi-year method.

We also acknowledge that, at present, this volatility is acting to produce lower estimated poverty rates than would likely be estimated by multi-year methods. Consulting the table column-wise, Table 5.2 shows that, using the same input data (2019-22), a possible multi-year calibration method results in estimated child poverty being two percentage points higher than our single-year method on both measures in 2023-24. Note, however, that the volatility of the single-year method can work in both directions. Indeed, Table 5.2 also shows that, using the previous input data (2017-20), the pattern is reversed: the poverty rates estimated by the single-year method are two percentage points higher than those estimated by the multi-year method.

Notes: for single-year calibration, simulation for latest year of input data is compared to latest official statistics; for multi-year calibration, the same simulation is compared to average of official statistics covering full period of input data. Single year 2017-20 estimates are higher than previous modelling due to model updates.

Source: Scottish Government analysis using UKMOD.

Note also that the latest update is irregular in that it involves switching from a two-year to a three-year pool. This particularly affects the calibration process, with a higher proportion of input data falling in the calibration year, but can also affect the modelled outputs directly. We plan to continue updating the two-year pool until FRS data is available for 2023-24, at which point we will revert back to a three-year pool.

Finally, it is worth bearing in mind that the uncertainty surrounding calibration, and the volatility resulting from our particular method, primarily affect the projections of outturn child poverty rates rather than the estimated impacts of the policy package. This is because the policy scenario and the counterfactual scenario are equally affected, leaving the difference between them relatively unchanged. More generally, we can be more confident when isolating the impacts of policies than when projecting outcomes such as child poverty rates.^[10]