Scotland NCM software validation DSM and FI-SBEM software validation: guidance

2. Software Submission Requirements

The software validation process involves three key parties: (1) The software vendor, (2) the Scottish Government, and the (3) validation scheme operator. The interactions of these parties and the processes leading to software approval are summarised in Figure 1.

Overview

The core part of the validation process requires the software packages to be tested for accuracy in implementing the National Calculation Methodology (NCM). A set of test cases for the two software types have been developed. It requires the software vendor to use their software to generate specified building models, populate with prescribed data, and calculate results.

Details of the DSM and FI-SBEM test cases are provided as follows in the validation information pack:

• The validation guidance documents specify the building models and data to implement in their software.
• iSBEM model files (*.nct) for the test cases. The software vendors need to run these files on the latest government iSBEM software to generate the Reference Outputs.

The vendor's own software test case outputs are compared to the corresponding sets of Reference Outputs, which should match within the stipulated tolerance levels (see Section 2.2 below).

It should be noted that whilst the validation process interrogates the entire DSM software, for FI-SBEM software class, the validation process only checks the software front-end interface to the SBEM engine, which third party vendors would have developed to interface with and run the SBEM engine. The SBEM engine is locked and is maintained by BRE.

Software vendor: Software vendors obtain the validation information pack from the Scottish Government. Software vendors are required to check their own software outputs agree with the Reference Outputs. Software vendors also need to meet other pre-requisites to ensure that their software is in a good commercial state and fit-for-purpose. Prior approval in England will also be required. When these validation pre-requisites are met, the software vendor can submit a validation application to the Scottish Government. The Scottish Government is the initial point-of-contact for any validation application via the email address provided above.

Validation scheme operator: The Scottish Governments requests the scheme operator to validate the software. Further checks and assessments to those previously carried out by software vendors are carried out by the validation scheme operator during the validation process. As necessary, the scheme operator will engage with the software vendors on any issues identified. If a software passes the validation process, the scheme operator will recommend that the software is approved by the Scottish Government. If the software does not pass the validation process, the software vendor needs to retest and resubmit when ready.

Scottish Government: Upon receiving validation application from the software vendors, the Scottish Government will instruct the validation scheme operator to carry out the validation process. The Scottish Government will provide final approval of the validated software based on the recommendation provided by the validation scheme operator and issue the Approval Letter to the software vendors. The Scottish Government will update the software details on the approved software list.

2.1 Validation Information Pack

A validation information pack will be made available to software vendors, which comprises this guidance document, the associated test case Reference (*.nct) files, the validation submission pro-forma and any other relevant information. Software vendors will need to contact the Scottish Government for the latest validation information pack. Any updates to the validation information pack will be disseminated by the Scottish Government in the first instance, which can be subsequently requested if required.

2.2 Assessment Procedure

Test cases

Test cases 1 to 14 have been developed for Scotland and are described in detail in Appendix A. Test cases 1 to 8, which are based on the England test cases, and Test cases 9 to 11 will test various aspects of the implementation of the Scottish NCM for Section 6 compliance and EPC calculations as described in the 2022 NCM Modelling Guide for Non-domestic Buildings in Scotland^[1]. Test cases 12 to 14 have been designed to test the implementation of the Section 63 Action Plan.

Software vendors should be familiar with the differences in application of the NCM for the purposes of demonstrating building regulations compliance (section 6) and for EPC/Action Plan production in Scotland

Software vendors will find it useful to interrogate the test cases *.nct files in detail, especially when developing and setting up the test cases using the vendor software, as well as checking various intermediate outputs and the final Section 6 calculations and EPC ratings. Software vendors will need to ensure the latest version of iSBEM is used to generate the relevant Reference Outputs.

If there is a discrepancy between the details in the test case *.nct files and the corresponding test case description detailed in Appendix A, software vendors should use the information provided in Appendix A. Please feedback any discrepancy identified to the validation scheme operator either directly or via the Scottish Government support email provided in Section 1 above.

Reference Output

The Reference Output iSBEM test case models (*.nct files) are included in the validation information package for SBEM v6.1.e. Updates to the Reference Outputs have to be generated when new versions of the government software are released. This could be carried out either by the validation scheme operator or the software vendors. Currently, the approach is for the vendors to produce the Reference Outputs.

Acceptability threshold

The DSM vendor software will be required to produce EPC ratings for the test cases that match the corresponding Reference Output EPC rating band (A – G) for the Acceptability Threshold criterion to be met.

It is recognised that the Actual building's Building Emission Rate (BER) can differ between DSM and SBEM due to differences in the calculation engines. Therefore, whilst there may be difference in BERs, the vendor DSM software will need to be in the same EPC A-G rating band as that of the Reference Output for the corresponding test cases.

It is envisaged that further review of the vendor software through live assessment will be carried out in the event that the vendor software has not fully met this criterion and the outcome of passing this criterion will be evaluated on a case-by-case basis. For example, differences may cut across the EPC rating bands. The secondary checks, as described in the next sub-section, will supplement the validation process to capture discrepancies in the accuracy of the implementation of the Scottish NCM by the vendor software.

The FI-SBEM software will be required to both match the Reference output BER, EPC rating (numerical) as well as the EPC A-G rating band for the corresponding test cases to satisfy the Acceptability Threshold criterion.

Secondary Checks

The validation scheme operator will also apply a series of secondary checks on the Section 6 and EPC calculations. This will assess differences between vendor software test cases outputs and the Reference Outputs for output parameters for the Actual and Notional buildings in the *brukl.inp and as well as the details in the *epc.inp and *s63.inp files of the test cases. The intention is to identify parameters that have demonstrated a 'significant' difference relative to the Reference Outputs to warrant further checks and discussion with the software vendors. There is no specific tolerance set that vendor software will need to meet for this assessment and issues arising will be evaluated on case-by-case basis.

Live Assessment

As there are a wide variety of calculation methodologies employed by DSM software, the energy demands calculated for Section 6 assessments can differ significantly between different software as well as the SBEM calculation methodology.

Therefore, when validating DSM software, differences in the test case outputs between the vendor software and the Reference Output, which is based on the SBEM calculation methodology, may require closer investigation through a live assessment session. This will enable the details of the different building physics calculation approach to be compared to understand the impact on output. It will also enable vendors to demonstrate how their software appropriately implements the Scottish NCM.

Live assessment applies to DSM validations, which will be requested as part of the validation process where required and conducted via a MS Teams meeting. Subsequent follow-up sessions may be required where necessary to support the validation process.

One of the main focuses of the live assessment is to ensure that the software correctly implements the Actual Building and the Notional Building sections of the Scottish NCM. Ahead of the live assessment, the validation scheme operator may provide the software vendor with a checklist of requirements to be tested to better facilitate the process.

During the live assessment, software vendors may be asked to create a simple model from scratch. This will demonstrate the software usability and help provide an indication on the quality and extent of software functionality detail provided in the user guide(s).

Typically, a simple model will be used during a live assessment session to examine various features of the Scottish NCM. For example, in Scotland, Section 6 compliance requires that the proposed building meets both the Target Emission Rate (TER) and the Target Delivered Energy Rate (TDER). The vendor will need to demonstrate how the TDER and the associated Building Delivered Energy Rate (BDER) are calculated and displayed to the user.

Software vendors will need to demonstrate the Section 63 Action Plan feature in their software and showcase how some of the triggers for Prescriptive Measures are implemented.

The live assessment may highlight discrepancies in the implementation of the Scottish NCM by the vendor software. If appropriate, these issues may be addressed by the software vendor as part of the validation process. If the issues encountered are substantive, a resubmission may be required.

2.3 Additional Requirements

The validation and approval in England require DSM and FI-SBEM software vendors to submit the relevant user guides (with the exception of bug-fix validations). These have been reviewed at a high-level to ensure they are sufficiently detailed in terms of software use and functionality.

An additional review is needed to check whether the relevant user guides are suitable for implementation in Scotland. Software vendors should provide the user guide and accompany this with details of where the user guide describes the difference between the building regulations (section 6) compliance and EPC calculations in England and Scotland. Details on the Section 63 Action Plan should also be highlighted. These details could be provided through appropriate annotations within the copy of the user guide and/or a separate note. This should be included in the submission for Full Validation and Major and Minor Revalidation.

2.4 Parameters Not Covered

Whilst every effort is made to cover all of the elements of the Scottish NCM during software validation, there will inevitably be certain elements that cannot be accommodated in this process. Where there are components in the candidate software that are not governed by the Scottish NCM methodology, they will not be included in the validation process.

As elsewhere in the UK, there are a number of cases where the Scottish NCM modelling guide does not define certain calculations employed in DSM software. These include modelling of daylight availability for use in predicting lighting energy and renewable/low carbon technologies (e.g. SBEM and DSMs have slightly different ways to predict wind energy). In addition, variations inevitably exist within the core calculation engines used in each DSM software. For example, modelling solar gain through glazing systems and the storage effects of thermal mass will differ between each DSM software. Hence, none of the above will be evaluated directly during the validation process.

Furthermore, default values not included in the Scottish NCM Modelling Guide will not be included in the validation process. This includes the following parameters:

• The inference of HVAC system default efficiency
• The use of the SBEM construction library
• The use of the SBEM construction inference procedure library
• The use of lighting type load inference
• Glazing properties: surface to area ratio
• Glazing properties: window aspect ratio
• The use of 'Transpired Solar Collector'
• The use of 'additional thermal bridges' input
• No checks on the use of local and global psi values for thermal bridging

The above list of exclusions is the same as in the English and Welsh validation process.