Programme for International Student Assessment (PISA) 2015: highlights from Scotland's results: collaborative problem solving
The results of the PISA 2015 assessment of collaborative problem solving, showing Scotland’s results and those of other participating states.
2. Collaborative Problem Solving and how it is assessed
11. PISA is designed to measure what students know, but also the application of knowledge in real-life situations. Below, we summarise key features of the OECD’s framework for collaborative problem solving literacy.
The PISA framework for collaborative problem solving
12. The OECD define collaborative problem solving as:
the capacity of an individual to effectively engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills and efforts to reach that solution.
13. There are a number of concepts that underlie this definition. Some of these draw on problem-solving as a discipline in itself. PISA has its own definition of problem solving, developed for the Creative Problem Solving innovative domain administered in the 2012 cycle [1] , as follows.
- exploring and understanding: exploring the problem situation by observing it, interacting with it, searching for information and finding limitations or obstacles; and demonstrating understanding of the information given and the information discovered while interacting with the problem situation
- representing and formulating: using tables, graphs, symbols or words to represent aspects of the problem situation; and formulating hypotheses about the relevant factors in a problem and the relationships between them to build a coherent mental representation of the problem situation
- planning and executing: devising a plan or strategy to solve the problem; executing the strategy; and perhaps clarifying the overall goal and setting sub-goals
- monitoring and reflecting: monitoring progress, reacting to feedback, and reflecting on the solution, the information provided with the problem, or the strategy adopted.
14. To develop the framework for collaborative problem solving, a further element of collaboration is added with the following aspects:
- establishing and maintaining shared understanding: identifying the knowledge and perspectives that other group members hold and establishing a shared vision of the problem states and activities
- taking appropriate action to solve the problem: identifying the type of collaborative problem solving-related activities that are needed to solve the problem and carrying out these activities to achieve the solution
- establishing and maintaining team organisation: understanding one’s own role and the roles of other agents, following the rules of engagement for one’s role, monitoring group organisation, and facilitating the changes required to optimise performance or to handle a breakdown in communication or other obstacles to solving the problem.
15. This creates an overall framework with 12 components, set out in Table 2.1 below
Table 2.1: Skills evaluated in the PISA 2015 collaborative problem solving assessment
| |
Collaborative problem-solving competencies | |||
| (1) Establishing and maintaining shared understanding | (2) Taking appropriate action to solve the problem | (3) Establishing and maintaining team organisation | ||
| Problem-solving processes | (A) Exploring and understanding | (A1) Discovering perspectives and abilities of team members | (A2) Discovering the type of collaborative interaction to solve the problem, along with goals | (A3) Understanding roles to solve the problem |
| (B) Representing and formulating | (B1) Building a shared representation and negotiating the meaning of the problem (common ground) | (B2) Identifying and describing tasks to be completed | (B3) Describing roles and team organisation (communication protocol/rules of engagement) | |
| (C) Planning and executing | (C1) Communicating with team members about the actions to be/being performed | (C2) Enacting plans | (C3) Following rules of engagement (e.g. prompting other team members to perform their tasks) | |
| (D) Monitoring and reflecting | (D1) Monitoring and repairing the shared understanding | (D2) Monitoring results of actions and evaluating success in solving the problem | (D3) Monitoring, providing feedback and adapting the team organisation and roles | |
Source: OECD
The assessment
16. Participants undertake tasks, with the computer providing a number of simulated partners who take on various roles, such as a holder of information, or leading on a sub-task necessary to solve the problem. The student interacts with their virtual partners by selecting the next step from a range of options presented. Box 2.1 gives a flavour of the type of task involved.
Box 2.1: Types of collaborative problem-solving tasks
Jigsaw or hidden-profile tasks, where each group member is given different information or skills. Groups need to pool each member’s information and skills together in order to solve the problem and hence collaboration among group members is required. Moreover, group members are dependent on one another to arrive at the solution: no single member can achieve the solution on his or her own, and a group member who chooses not to participate can jeopardise the achievement of the group’s goal
Consensus-building tasks, where a group must agree on a decision after considering the views, opinions and arguments of all group members. A successful solution will involve all group members contributing their ideas and the careful yet efficient consideration of all such ideas. However, some group members may dominate the conversation and not allow for all ideas to be aired, while other group members may not be willing to disagree with what has already been said, potentially leading to “group think”
Negotiation tasks, where not all group members share the same individual goals. They must negotiate in order to achieve, in the best-case scenario, a win-win situation that satisfies both their individual goals and overall group goals.
Source: OECD
It should be noted, that task can cover more than one of these categories, for example, beginning as a jig-saw task, and ending up with a need to build a consensus, or negotiate a final outcome
17. The role of the computer in taking the place of a human has been examined and validated by a study undertaken by the University of Luxembourg. This study, plus the full framework for collaborative problem solving and a description of the typical question items, is available on the OECD website at www.oecd.org/pisa.
18. Questions are constructed drawing upon the framework above, and at varying levels of difficulty, in order to identify a student’s ability. Their score corresponds to proficiency levels, which are summarised in Table 2.2 below.
Table 2.2: Proficiency levels in collaborative problem solving, and what they mean
| Level | Score range | What students can typically do |
|---|---|---|
| 4 | Equal to or higher than 640 score points | At Level 4, students can successfully carry out complicated problem-solving tasks with high collaboration complexity. They can solve complex problems with multiple constraints, keeping relevant background information in mind. These students maintain an awareness of group dynamics and take actions to ensure that team members act in accordance with their agreed-upon roles. At the same time, they can monitor progress towards a solution and identify obstacles to overcome or gaps to be bridged. Level 4 students take initiative and perform actions or make requests to overcome obstacles and to resolve disagreements and conflicts. They can balance the collaboration and problem-solving aspects of a presented task, identify efficient pathways to a solution, and take actions to solve the given problem. |
| 3 | 540 to less than 640 score points | At Level 3, students can complete tasks with either complex problem-solving requirements or complex collaboration demands. These students can perform multi-step tasks that require integrating multiple pieces of information, often in complex and dynamic problems. They orchestrate roles within the team and identify information needed by particular team members to solve the problem. Level 3 students can recognise the information needed to solve a problem, request it from the appropriate team member, and identify when the provided information is incorrect. When conflicts arise, they can help team members negotiate a solution. |
| 2 | 440 to less than 540 score points | At Level 2, students can contribute to a collaborative effort to solve a problem of medium difficulty. They can help solve a problem by communicating with team members about the actions to be performed. They can volunteer information not specifically requested by another team member. Level 2 students understand that not all team members have the same information and can consider differing perspectives in their interactions. They can help the team establish a shared understanding of the steps required to solve a problem. These students can request additional information required to solve a problem and solicit agreement or confirmation from team members about the approach to be taken. Students near the top of Level 2 can take the initiative to suggest a logical next step, or propose a new approach, to solve a problem. |
| 1 | 340 to less than 440 score points | At Level 1, students can complete tasks with low problem complexity and limited collaboration complexity. They can provide requested information and take actions to enact plans when prompted. Level 1 students can confirm actions or proposals made by others. They tend to focus on their individual role within the group. With support from team members, and when working on a simple problem, these students can help find a solution to the given problem. |
Source: OECD
Contact
There is a problem
Thanks for your feedback