Introduction
The proper distribution of student work among peers is essential to effective Peer Review. It ensures that everyone gets feedback on their work, and it minimizes the need for teacher intervention to guide the process. For our Peer Review and Group Member Evaluation tool, we've designed a sophisticated algorithm to distribute student submissions in the most didactically desirable way.
How Does the Algorithm Work?
Initial Allocation
Our allocation algorithm takes the following aspects into consideration, in order of importance:
Every student receives enough submissions to meet the required amount of reviews.
Every student or group receives an equal amount of reviews, when possible.
Every student or group receives reviews one by one, not all reviews at the same time.
Early participation in the assignment is rewarded.
To prevent students from gaming the algorithm, review allocation is unpredictable for students.
During Reviewing
When the reviewing phase of an assignment has started and a student opens the assignment, we allocate the best available candidate to the student at that moment. This has major advantages over pre-allocating students to each other.
With this approach, we avoid these common scenarios:
Students dropping out, not participating in the review part of an assignment, leaving some peers without reviews.
Top-performing students not receiving reviews until late in the review phase, because their reviewers are late to start.
Students handing in or joining the course late not being able to complete the assignment.
In all these unfortunate scenarios, our algorithm will lead to optimal distribution of peer reviews: every student or group will get the same amount of reviews.
Note: The one scenario that requires us to compromise is when every student has finished reviewing, but one students hands in late/ joins afterwards. This student will be able to fulfil their review duties, but won't get reviews from their peers.
If the teacher feels that the reason for the student submitting the work so late is compelling enough, teachers can always give that student feedback or manually adjust their grade.
Below you see an example of an assignment where 4 students need to review 2 peers. They handed in their work in alphabetical order, which gives a sense of the effect handing in early has on the order in which work is distributed.
Student A reviews Student B
Student A reviews Student C
Student B reviews Student A
Student C reviews Student D
(Every student has received one review: the algorithm now starts the second batch of reviews)
5. Student C reviews Student A
6. Student D reviews Student B
7. Student B reviews Student D
The algorithm notices that only Student D still needs to review someone, and prevents that the only candidate left to review is themselves, allocating Student B to review Student D.
8. Student D reviews Student C
At this point the assignment is finished - everyone completed 2 reviews and has received 2 reviews, meaning the work of reviewing is distributed equitably among all participants.
Other Factors Affecting Allocation
For our Peer Review tool, there are two extra options that you can enable concerning the assignment of work.
These functionalities are not applicable to our Group Member Evaluation tool since there is no hand in step.
In case you want to allow students to hand in their work after the deadline has passed, you can enable the option allow late hand in. With this option enabled, students can still be allocated to peers, even though this would normally be prevented due to the passing of the deadline.
Late Hand Ins
When a student hands in after the deadline, they will be added to the pool of students awaiting allocation. The allocation of work only happens after a student opens the tool again to start with the reviewing step. This means that even when people have handed in late, they will still be allocated to be reviewed.
Even when using the assign all available peers instantly option within the assignment, the allocations only happen after students come online again to open the tool. In the unlikely case that all students have finished reviewing already and one student hands in late, this late student will not receive a review. This is seen as the penalty of being late. Teachers are able to manually allocate these students to still receive a review if this happens.
You also have the option to allow students who did not complete their hand in to participate in reviewing by enabling the option allow students who didn't hand in to participate in reviewing. Normally, students are only able to start reviewing when they have completed the hand in step.