Skip to main content

Group Member Evaluation | Detect outliers

Hannah

Quick Configuration Video

This feature intends to help teachers quickly identify out-of-the-ordinary situations in teamwork assignments. This is based on how students rated themselves and their peers. Such situations can range from positive (like someone who performed exceptionally well) to a possible concern (like an apparent conflict between two students).

We'll discuss 3 topics:

  1. How to enable this feature

  2. How to ethically use this

  3. Overview of indicators

How to enable this feature

The feature can be enabled under step 2, "Given reviews". Currently, this feature requires the following conditions to be met:

  • Use the Group Member Evaluation tool

  • Enable self-assessment setting

  • Configure to review in groups

  • Configure that students review at least 4 peers or 'all'

  • "New user-interface design" setting (enabled by default, but required)

We recommend using it for teamwork assignments only. We aim to provide a generalized version of this feature in Peer Review too.

How to ethically use this

One might rightfully wonder: "Does this label and judge students?"

The short answer is: no, unless you do so. These are low-confidence indicators to help distinguish situations and guide further investigation. To prevent unethical use, let's clear up what this feature can and can't do:

Appropriate use

  • Use to guide attention: These indicators are just that; mere indicators of likely areas of interest. Please don't jump to conclusions with this feature. To think of these as definitive would be over interpreting the certainty of the indicator.

  • Dive deeper: Look at the student’s overview of received reviews ratings, compare it to the team average and their self assessment. Browse through some of the comments that were left for context.

  • Get in touch with the student: it’s good to hear their part of the story. If an indicator turns out legitimate, such a situation can be a big learning opportunity; show them the data to provide evidence in the discussion and help them understand how the situation occurred. Also touch upon the limits of the data in a single assignment to paint a picture of reality. Helping them understand the "why" unlocks a powerful learning experience to recalibrate their self perception and motivate them to set self-development goals.

Inappropriate use

  • Don’t jump to conclusions. They are just indicators of likely areas of interest.

  • Caution on "fixed mindset" interpretations: Even if the indicator turns out legitimate right now, it does not necessarily reflect a student's potential to learn and outgrow them in the future.

  • Don’t use indicators for grading without investigating per individual. We will never support blind automatic grading based on these indicators.

Overview of indicators

High performer

This occurs when a student received an average rating higher than 70% and their rating was more than 10% percentage points higher than the overall average of the team as a whole.

Low performer

This occurs when a student did not contribute to the team's success. This condition is triggered when their overall average rating is less than 50%.

Underconfident

This occurs when the overall team rating for a student is greater than 60%, but the student rated themselves at least 20% percentage points lower than this. This could indicate that the student is "underconfident" or too critical of their own contributions.

Overconfident

This occurs when the overall rating for a student is less than 60%, but the student rated themselves at least 20% percentage points higher than their average rating.

Self-promoter

This occurs when a student appears to be trying to influence their ratings because their self-assessment was 80% or higher while they gave all of the other group members at least 40% points below this rating. You might want to look at this student from the “Given reviews” perspective, where this will be more obvious from the ratings.

Conflict-origin

This student rated a team-member at 40% or less while the median rating from the rest of the team is 60% or more. This generally indicates that there is a conflict between 2 team members.

Conflict-target

This occurs when a student got rated at 40% or less by one of their team-mates, while the median rating from the rest of the team is 60% or more. This generally indicates that there is a conflict between 2 team members.

Clique

This occurs when it appears the team might have split into two non-cooperating groups, where there’s protective insider rating. This happens when there is significant disagreement between the ratings from various team members (evidenced by standard deviation of the 0-to-1 normalized ratings given by peers being above 0.23).

Due credits:

This feature is heavily inspired by the "exceptional conditions" of Purdue University's excellent CATme assessment technique. Pedagogy wise, minor adaptations and reinterpretations were subsequently made by FeedbackFruits based on our experience. The most substantial differences are in the interaction design of our implementation, intended to bring the user-friendliness to the level expected of FeedbackFruits. Also of note is the BuddyCheck.io (by Shareworks) adaptation of the same CATme feature called "Labels".

Tags:

exceptional conditions, special cases, extreme voting, outlier ratings, buddycheck labels, indicators, catme

This concludes the Detect outliers article.
​If you have any questions or experience a technical issue, please contact our friendly support team by clicking on the blue chat button (Note: support is available 24h every weekday & unavailable on the weekend).

Related articles