Shared equity schemes: evaluation reports

2. Methodology

This chapter summarises the main strands of the evaluation approach and provides an overview of the profile of research participants.

Research design

Quantitative survey research was conducted with buyers and developers, and this was followed up by qualitative interviews with these groups. Qualitative interviews were also conducted with lenders and Registered Social Landlords (RSLs), Research instruments were developed in collaboration with the Scottish Government, drawing on materials used in previous evaluations of UK and Welsh Government shared equity schemes. The buyer survey questionnaire was piloted with a small random sample of buyers to ensure questions were relevant to buyers and would produce findings which accurately reflected their views and experiences.

The buyer survey was issued via email and post on September 13^th 2019 and closed on 10^th October 2019. Emails with a direct link to the online survey were issued to buyers for whom an email address was held, and a letter providing a short link to the survey was issued to all others. Contact details for all buyers were provided by the Scottish Government from sales log forms. Options to respond via online survey, paper copy and telephone were offered to all buyers. Overall, 66% of surveys were issued via email and 34% by post. A postal reminder was also on issued 14^th October 2019 to a random sample of 10,000 non-respondents at the mid-point of survey fieldwork. The mix of email and postal issue varied across the three schemes; postal was used for 44% of HtB, 100% of NSSE and 1% of OMSE buyers. Full buyer response rates are available in the annex to this report.

The developer survey was issued to all developers by email on 5^th September 2019, supported by direct communication with developers to encourage response by the Scottish Government and Homes for Scotland. The survey was closed on 16^th October 2019.

Telephone interviews with lenders were conducted in parallel with the buyer and developer survey fieldwork, with invites issued to all lenders currently or previously involved in the shared equity schemes and a sample of those not involved in the schemes. Interviews with buyers and private developers were recruited from respondents who volunteered to take part in further research. Interview invites were also issued to all RSLs involved in the NSSE scheme. Most interviews were conducted via telephone, with a small number of developer interviews conducted in person where the participant preferred this option. Qualitative fieldwork began on 21^st October 2019 and completed on 10^th December 2019.

An overview of the main evaluation fieldwork strands is provided below in table 2.

Table 2: Overview of fieldwork strands

Participant group	Approach	Sample
Buyers	Survey: online-based with postal and telephone options, mix of email and postal drive online invites	All buyers using shared equity schemes where a valid email and/or postal address was available: 17,123 HtB 1,059 NSSE 7,106 OMSE
	Qualitative: pre-arranged telephone interviews	Recruited via survey
Developers	Survey: online-based with direct email invites, follow-up promotion from Scottish Government and Homes for Scotland	All developers currently or previously involved in HtB or NSSE
	Qualitative: telephone and in-person interviews	Recruited via survey, invites to survey non-respondents
RSLs	Qualitative: telephone interviews	RSLs currently or previously involved in NSSE.
Lenders	Qualitative: telephone interviews	Lenders currently or previously involved in schemes, sample of lenders not involved

A desk review was conducted alongside the survey and qualitative fieldwork. A wide range of data sources were collated to provide as comprehensive an account as possible of the operation of the three shared equity schemes and their influence on new build development, lending and the wider housing market. The desk review also explored the issue of 'additionality' and thus the extent to which:

• New homes delivered through the HtB and NSSE were over and above what would have been built anyway.
• Households would have been unable to buy without the support offered by the three shared equity products.

The approach to assessing additionality has adapted the approach used in the evaluation of HtB in England (Whitehead et al, 2016 and 2018). This included the triangulation of survey results and qualitative findings for buyers and developers, with available secondary data sources.

Sampling and response rates

The fieldwork approach with buyers, developers, RSLs and lenders was designed to maximise the number and range of participants able to contribute to the evaluation. As noted above, this process was supported by direct communication with developers, RSLs and lenders from the Scottish Government, Homes for Scotland and UK Finance. This approach was vital in ensuring the evaluation could draw on a wide range of stakeholder experience in considering the operation and impact of the shared equity schemes. Table 3 provides an overview of all those who took part in the research.

Table 3: Response by respondent and participant group

		HtB	NSSE	OMSE	ALL
Buyers	Survey of all shared equity buyers	2,473	139	1,451	4,063
	Telephone interviews	19	13	13	45
Developers	Survey of developers				61
	Telephone and in-person interviews				10
RSLs	Telephone interviews				9
Lenders	Telephone interviews				10

A total of 4,063 responses were received to the buyer survey, giving an overall response rate of 16%. Response varied somewhat across the three shared equity schemes; 14% for HtB, 13% for NSSE and 20% for OMSE. This was in part due to the larger proportion of HtB and NSSE buyers for which only postal address details were available - a lower response was achieved to the 'postal drive online' approach. The developer survey achieved a response rate of 17%.

Due to the lower than anticipated response rates achieved, there is likely to be nonresponse bias in the results from the two surveys, especially the developers' survey where weighting has not been applied. Whilst weighting has been applied to the buyers survey, it only accounts for two characteristics - geography and scheme type. It is unlikely that this would fully account for the non-response bias potentially caused by the lower response rate. Given these concerns, the findings from the survey should be treated with caution.

Respondent and participant profiles

The profile of buyer respondents is summarised overleaf. HtB and OMSE buyers were the largest groups, accounting for 61% and 36% of respondents respectively, with NSSE buyers accounting for 3% of respondents. This is broadly consistent with the overall mix of buyers using the three schemes to date, although OMSE buyers were somewhat over-represented (36% of respondents compared with 28% of the survey sample) and HtB under-represented (61% compared with 68% of the sample). NSSE accounted for a larger proportion of buyers in the early years of the schemes, but has accounted for a smaller proportion in recent years such that the 3% share of respondents compares to a 4% share of all buyers in the sample. Buyer survey responses were weighted to be representative of shared equity scheme and the Scottish Government 6-fold urban/rural geography.^[4]

The responses from HtB and OMSE buyers to this survey are similar to sales log form survey data collected and published by the Scottish Government.^[5] This comparison indicates that HtB and OMSE respondents are similar to all buyers in terms of household income and property type purchased. However, HtB and OMSE survey respondents include a larger proportion of existing owners (35% of respondents compared with 21% of all buyers for HtB, 4% compared with 1% for OMSE). HtB respondents included a larger proportion of those buying larger (4+ bedroom) properties (23% compared with 13% of all buyers).

Study limitations

This evaluation has made use of a range of primary and secondary data sources in considering the operation and impact of the Scottish Government's shared equity schemes. However there are limitations in this assessment, particularly when estimating additionality.

Due to data limitations, there was no robust counterfactual or baseline available for this evaluation. There were also difficulties in disentangling the diverse dynamics of the different local economic and housing markets that function across Scotland, and the influence of other measures to boost housing supply and how these have shaped consumer and develop confidence. The evaluation therefore has drawn extensively on primary research evidence, using the survey of buyers to calculate demand side additionality, and in-depth interviews with developers and lenders to explore supply side additionality.

As previously mentioned, the overall buyer survey response rate (16%) fell below the target of 20%. As a result, caution should be exercised when interpreting results.

Due to the low number of developer interviews achieved (61), findings from this survey have been expressed as numbers rather than percentages.

Unless otherwise stated, mentions of buyers, developers and lenders throughout the report refers to those included in the research, not the wider population.

Where possible, this primary evidence is triangulated (or supplemented) with secondary data to provide a more robust and fine-grained assessment of the schemes. Limitations in available data mean that this is largely based on published statistics derived from administrative data and 'log form' returns from buyers. As such, the assessment of additionality should be considered indicative rather than precise.

Interpreting the survey data

Results presented in this report are based on respondents to each question - i.e. they exclude non-respondents to individual questions unless stated otherwise.

Differences between buyer groups are only included if statistically significant at a 95 per cent level. Differences were examined across the following characteristics:

• Shared equity scheme used;
• First-time buyers;
• Household income;
• Household type;
• Year of purchase;
• Purchase price; and
• Size of Scottish Government equity share.

Applying weights to data, while tending to make the quoted figures more representative of the population of interest, also serves to reduce the statistical reliability of the data. As such the 'effective' base size, which is used in any statistical testing, is smaller than the unweighted base size. This effect has been taken into account in determining whether or not differences described throughout the report are statistically significant.

We refer to those taking part in the evaluation as 'respondents' where their participation was via the buyer or developer survey, and 'participants' where this was via semi-structured qualitative interview.

Consideration of qualitative findings is based primarily on evidence from qualitative participants, but also draws on free text written responses from survey respondents.

Direct quotes have been included from buyers and developers where relevant to illustrate key points emerging through the evaluation. These comments have been lightly edited for brevity.