Registration of Civil Partnerships, Same Sex Marriage: Consultation Analysis

2 The consultation process

Overview of the consultation process

2.1 As noted earlier, the Scottish Government launched a public consultation on the possibility of allowing religious ceremonies for civil partnerships and the possible introduction of same sex marriage in September 2011. The consultation paper was available from the Scottish Government's website, and was also sent to a range of organisations likely to have an interest in the proposals, including religious groups, equality groups and a number of voluntary sector organisations^[7].

2.2 In addition to the public consultation process, representatives of the Scottish Government met with a range of organisations with particular interest in the issues covered by the proposals. These organisations included a number of the religious bodies that currently undertake religious marriages (as set out at paragraphs 1.7 and 1.8) and a range of equality or voluntary sector organisations with a specific interest in the proposals. A full list of all groups the Scottish Government met is at Annex C to this report.

2.3 The views expressed at those meetings will be outlined by the Scottish Government, together with the analysis of responses received to the public consultation set out within this report, are part of the evidence base used to inform decision making on this issue.

Types of responses received

2.4 The standard Scottish Government consultation response form set out all of the 20 consultation questions and was made available to any interested parties through the Scottish Government's website. Hard copies of the consultation response form were also distributed to various religious and equality groups, and sent out on request to members of the public. A copy of the response form is included as Annex A to this report.

2.5 Following the launch of the consultation, a small number of organisations (including the Christian Institute and the Equality Network) approached the Scottish Government to discuss the possibility of developing their own amended versions of the consultation form. The Scottish Government agreed that responses submitted on amended versions of the consultation form would be included in the analysis. In addition to those that consulted the Scottish Government, some other groups also developed their own approach to the consultation. All responses received, in whatever format, have been included in the analysis.

2.6 The alternative consultation forms were generally abridged, in that they selected a small number of the 20 questions. In some cases, the original consultation questions were either merged or reworded. In addition to these abridged or amended consultation forms, a number of other organisations prepared materials in other formats through which members of the public could express their opinion. These included prepared letters, petitions and postcard campaigns. Details of the non-standard responses received, along with a summary of their content, are set out in Table 1 below. Although it was not possible to identify the organising group for all postcards or amended forms submitted, information contained within all these responses has been included within the analysis.

Table 1 Details of main non-standard responses and their content

Organising group or body	Type of response	Content
Christian Institute	Amended form	Questions 1, 5, 6, 7, 9, 10, 11
Equality Network	Amended form	Questions 1, 5/11 (combined question), 10, 13, 19, 20
Equality Network	Postcard	Standard Text
Lanarkshire Mosque, North Lanarkshire Muslims Alliance	Amended form	All 20 questions, no additional comments other than at Question 20
LGBT Youth Scotland	Amended form	Questions 1, 5/11 (combined question), 10, 13, 19, 20
Muslim Council of Scotland	Postcard	Standard text
NUS Scotland (various)	Amended form	Questions 1, 5/11 (combined question), 10, 13, 19, 20
Roman Catholic Church	Postcard	Standard text
Scottish Youth Parliament	Prepared letter	Suggested text, plus option for additional comment Contains answers to Questions 1, 5, 10, 11, 13
Scottish Youth Parliament	Postcard	Standard text
UNISON Scotland Equal Marriage	Amended form	Questions 1, 5/11 (combined question), 10, 13, 19, 20

2.7 The Scottish Government also received a number of petitions. All petitions received contained a statement which clearly equated to at least one of the consultation questions (which was generally Question 10), and the number of signatories has been included in the main quantitative analysis at the appropriate questions.

2.8 Including all of the above response types (and also signatories on petitions) a total of 77,508 responses were received. The breakdown of these responses by response type is set out Table 2 below.

Table 2 Number of responses received by response type

Response type	Number received
Standard form	10,441
Amended form	33,634
Prepared letter	1,593
Postcard	26,383[8]
Petition	5,457[9]
Total	77,508

2.9 The analysis set out within this report looks at responses received through the standard or amended forms, along with prepared letters^[10]; these responses were accompanied by the Respondent Information Form that is required before a response can be included within the main analysis. However, the views expressed through postcards or petitions have been included in counts against any question that was clearly answered within their submission^[11]. For example, many of the postcards submitted asked whether people agreed that the law in Scotland should be changed to allow same sex marriage (Question 10). Examples of text submitted through the prepared letters, postcards and petitions are included as Annex D to this report and the proportion of overall responses received according to their principal 'source' is set out in Figure 1 below.

Figure 1 Responses by organisation(s) promoting responses

*There were a number of other amended forms submitted for which the source could not be identified and the total number of respones summed to less than 1% of all responses.

2.10 Other correspondence received by the Scottish Government has been logged and considered separately. A brief summary of the other correspondence received is included as Annex G, but otherwise their content is not considered further within this report.

Approach to the analysis

2.11 The overall aim of the Craigforth contract was to undertake a robust analysis of the quantitative and qualitative responses and then report that analysis identifying key issues and ensuring the range and depth of views was presented.

2.12 Before the main qualitative analysis began, the Scottish Government organised a briefing session with a small number of groups with a particular interest in the issue, including some of those that had previously met with the Scottish Government (as set out at paragraph 2.2). The aim of the briefing session was to share detail on how the analysis of the consultation responses would be conducted and for attendees to raise any questions or comments they may have had on this process.

Project management and quality assurance

2.13 Project management arrangements were established at the outset and were followed throughout, including regular contact between the Scottish Government's Project Manager and the Project Lead for Craigforth. A Scottish Government Reference Group was established and was available to offer support and advice throughout. Reference Group members were all Scottish Government social researchers experienced either in undertaking consultation analysis themselves or in the management of large-scale, externally commissioned analysis.

2.14 In addition to the usual quality assurance processes being followed by both the Scottish Government and Craigforth, additional measures were also built in. A researcher from the Scottish Government, who was not otherwise involved in the project, assisted with the development of the coding framework^[12] and undertook an independent analysis of many of the group responses. A senior associate of Craigforth (a highly experienced consultation analyst with no other involvement in the contract), undertook a review of the draft report, and Reference Group members also conducted a peer review.

Receipt and logging of responses

2.15 The Family Law Team of the Scottish Government was responsible for the initial processing of all responses submitted by the consultation closing date of 9^th December 2011. All responses received on a standard form, an amended form, or as a prepared letter were assigned a unique reference number and details of the response (name and country of respondent, whether replying as a group or individual, and degree of confidentiality requested by the respondent) recorded on a spread sheet. Any hard copy responses received were also scanned after logging to create an electronic copy.

2.16 As part of the logging process, checks were carried out for multiple responses submitted by the same individual. When more than one response had been submitted against an identical name and address, only one of the responses was recorded as to be included within the final analysis. If the content of any multiple responses differed, the response with greatest content was logged and retained for analysis^[13].

Transfer of responses

2.17 Electronic copies of all responses and the spread sheet on which these were recorded were transferred into a secure area of the Scottish Government's server to which Craigforth was allowed temporary access to retrieve data. All responses were transferred to Craigforth through this route; no responses were emailed.

2.18 Once received by Craigforth, all electronic responses were held on Craigforth's own internal server. No electronic responses were transferred to or saved to any other location. All responses were printed, numbered and checked off against the logging spread sheet created by the Scottish Government. On publication of the final report, all electronic copies of responses will be deleted and all hard copies will be disposed of through Craigforth's long-standing arrangements for the disposal of confidential material.

Quantitative analysis

2.19 Each of the consultation questions (with the exception of Question 20) invited respondents to give a quantitative response - principally to select 'Yes', 'No' or 'Don't Know'. Two questions (Questions 8 and 17) gave respondents two options to choose between. Respondents were also given the opportunity to make a further, qualitative comment if they chose.

2.20 All quantitative responses were entered into the statistical analysis package used by Craigforth. The unique reference number was used as the primary identifier for each response in the quantitative analysis and then all subsequent qualitative analysis. When an additional comment was made against any question, this was also recorded.

2.21 The number of respondents answering varied considerably from question to question, most obviously depending on whether a question was included within the amended forms (see Table 1 above). The quantitative information within this report presents the counts against each question simply according to the number of people that answered that question, and does not sum all responses to a standard base by including figures for 'not answered'. This is primarily because of the possible and varied combination of 'not answered' and 'not given the opportunity to answer' options. However, the number of potential maximum respondents has been given at each question. This is essentially the greatest number of people that could have answered any question based on the type of response they chose to submit (see Table 1 and Figure 1 above).

2.22 Once all data entry had been completed, Craigforth's standard data verification and quality assurance processes were followed, including completeness and internal logic checks - for example that no more than the potential maximum number of respondents were recorded as having answered a question.

Qualitative analysis

2.23 All the comments made were read and analysed. Over 180,000 separate comments were made, although the proportion of respondents that commented at each question was very variable: for example, 45% of respondents made a comment at Question 1 but only 15% made comments at Question 8.

2.24 Given the scale of the task, a structured approach to analysing the qualitative data was adopted. An analytical framework (coding framework) was developed using a sample of around 10% of responses. A separate code was created to cover each broad theme being raised - a code was created if upwards of around 1 out of 50 responses in the sample raised a particular issue^[14]. Examples of codes at Question 10 (on the introduction of same sex marriage) might, for example, include 'everyone should be able to marry the person they love' or 'marriage is between a man and a woman'. This coding framework was amended as required during the qualitative analysis. These amendments tended to relate to relatively specific but relevant issues being raised by small numbers of respondents - an example might include existing celebrants commenting on how they would be affected by the proposals.

2.25 Each code within the framework was assigned a unique identifier (in this consultation analysis the letters of the alphabet were used). The coding of all additional comments was recorded on hard copies of the responses. Every time a comment included a point that corresponded to a code, the unique identifier for that code was added to the margin of the paper copy. If multiple themes appeared within a single comment, all the relevant codes were applied.

2.26 All coding was then entered into Craigforth's statistical analysis package against the unique identifier for each respondent (as for the quantitative analysis). Once all coding had been entered, standard data verification and quality assurance processes were again followed, before the number of occasions on which each code was recorded - in effect the number of occasions on which each theme was raised by respondents - was generated.

2.27 The main focus of the report is on qualitative data, reflecting the nature of the material received. As noted above, quantitative analysis has been undertaken for the parts of each question where respondents were asked to agree or disagree with proposals. It is less appropriate to focus on quantitative material for a number of reasons, including that:

• Not all of the respondents addressed the questions specifically, nor did they all do so using the standard form.
• Respondents covered similar issues at different points on the response form.
• Some responses were submitted on behalf of organisations and represented the views of a number of respondents. It would be impossible to identify the actual number of individuals represented by a response.
• The consultation process involved "opting in" and cannot be seen to be representative.

Reporting

2.28 In advance of the reporting, a structure for the final report was agreed with the Reference Group. As noted above, drafts of the main report were submitted for comment and any necessary revisions made. Given the scale of the response, and the importance of fair and balanced reporting of all views expressed, it was decided that no single respondent would be referred to during reporting and no quotes would be used.

2.29 The main analysis is presented question by question from Chapter 4 onwards, echoing the structure of the consultation document. The results of the quantitative analysis (essentially the response to the yes/no questions) are presented within a table at each question. The subsequent commentary sets out the main findings from the analysis of additional comments made and reflects the nature of qualitative analysis itself, in that it aims to cover the breadth and depth of views expressed, rather than attempt quantification. However, to ensure consistency within the reporting, standard definitions were used when referring to the frequency with which any particular code appeared at any particular question. To reflect the breadth of opinion, this approach was applied separately to comments made by those that broadly supported the proposals and those that broadly opposed the proposals. The common terms used were applied as follows:

• 'many' was used when 1 in 3 or more of those who were in favour and commented or who were opposed and commented, raised a particular issue;
• 'some' was used when 1 in 10 or more but fewer than 1 in 3 raised an issue;
• 'a number' was used when 1 in 25 or more but fewer than 1 in 10 raised an issue: and
• a 'small number' or 'a few' was used when fewer than 1 in 25 raised an issue but it was considered of particular relevance (see paragraph 2.24 above).

2.30 Overall, therefore, the report is designed to reflect the nature of the consultation process, in that it provides an overview of the opinions submitted by members of the public with a particular focus on both the balance and also the range of views expressed.

2.31 The content of responses has not been vetted for factual accuracy and the importance placed on hearing views from all perspectives means responses which may contain factually inaccurate information but nevertheless reflect strongly held views have been taken into account. Chapters 3 to 9 reflect the views of those who have responded to the consultation and not the authors of this report, who do not offer any comment or judgement on those opinions.