Introduction

The proper distribution of student work among peers is essential to effective Peer Review. It ensures that everyone gets feedback on their work, and it minimizes the need for teacher intervention to guide the process. For our Peer Review and Group Member Evaluation tool, we've designed a sophisticated algorithm to distribute student submissions in the most didactically desirable way.

Peer Assignment Methods

1. Automatic - One by One

• The students will be assigned a review one by one, the second review will appear after starting the first one, and so on.

• This method ensures a balanced distribution of reviews, especially in cases where not all students submit their work at the same time.

• Allocations become visible individually as the student begins each review. A new allocation appears with every review the student starts.

2. Automatic - All at once

• Only available if the amount of required reviews is set higher than 1.
• Peers receive all their assigned reviews at once, allowing them to see all their peers' work immediately.

• Works well for larger classes where manual assignment would be impractical.

• All allocations will appear at once after the hand-in deadline has passed. The student must first click “Start” to begin the allocation.

3. Manually

• Instructors can manually assign reviewers to specific submissions.

• Ideal for scenarios where specific pairing is necessary, such as group work evaluations.

• Allocations are only visible once the teacher manually assigns them to the student.

4. Student choose who to review

• Students can choose which peer or group to review when they start the assignment.
• This is ideal for promoting student autonomy and enabling them to review familiar work, such as group projects or collaborative tasks.
• Allocations appear one by one as the student starts each review and selects a peer to review.

How Does the Algorithm Work?

Initial Allocation

Our allocation algorithm takes the following aspects into consideration, in order of importance:

1. Every student receives enough submissions to meet the required amount of reviews.

2. Every student or group receives an equal amount of reviews, when possible.

3. Every student or group receives reviews one by one, not all reviews at the same time.

4. Early participation in the assignment is rewarded.

5. To prevent students from gaming the algorithm, review allocation is unpredictable for students.

During Reviewing

When the reviewing phase of an assignment has started and a student opens the assignment, we allocate the best available candidate to the student at that moment. This has major advantages over pre-allocating students to each other.

With this approach, we avoid these common scenarios:

• Students dropping out, not participating in the review part of an assignment, leaving some peers without reviews.

• Top-performing students not receiving reviews until late in the review phase, because their reviewers are late to start.

• Students handing in or joining the course late not being able to complete the assignment.

In all these unfortunate scenarios, our algorithm will lead to optimal distribution of peer reviews: every student or group will get the same amount of reviews.

Note: The one scenario that requires us to compromise is when every student has finished reviewing, but one students hands in late/ joins afterwards. This student will be able to fulfil their review duties, but won't get reviews from their peers.

If the teacher feels that the reason for the student submitting the work so late is compelling enough, teachers can always give that student feedback or manually adjust their grade.

Below you see an example of an assignment where 4 students need to review 2 peers. They handed in their work in alphabetical order, which gives a sense of the effect handing in early has on the order in which work is distributed.

1. Student A reviews Student B

2. Student A reviews Student C

3. Student B reviews Student A

4. Student C reviews Student D  

(Every student has received one review: the algorithm now starts the second batch of reviews)
​

5. Student C reviews Student A
6. Student D reviews Student B
7. Student B reviews Student D
​
The algorithm notices that only Student D still needs to review someone, and prevents that the only candidate left to review is themselves, allocating Student B to review Student D.

8. Student D reviews Student C
​
At this point the assignment is finished - everyone completed 2 reviews and has received 2 reviews, meaning the work of reviewing is distributed equitably among all participants.

Other Factors Affecting Allocation

In our Peer Review tool, there are two extra options that can be enabled to adjust how work is assigned. These options are not available in Group Member Evaluation, as there is no hand-in step.

Late Hand Ins

This option lets students submit their work after the deadline while still being assigned reviewers. Without this setting, late submissions would normally be excluded from peer review.

When a student submits their work after the deadline, they are added to the pool of students waiting for peer allocation. However, reviews are only assigned once a student reopens the tool to begin reviewing. This means:

• Late submissions will still be reviewed, as long as other students remain active in the assignment.

• Even if the "Assign all available peers instantly" option is enabled, reviews are only allocated when students log in again.

• If all students have already completed their reviews and one student submits late, that student may not receive a review—this serves as a penalty for late submission.

Instructors can manually assign reviewers to late submissions if needed.

Allow Students Who Didn’t Hand In to Review

By default, students must submit their work before they can review others. However, enabling the Allow students who didn’t hand in to participate in reviewing option lets them review peers even without submitting their own work.