The proper distribution of student work among peers is essential to effective Peer Feedback. It ensures that everyone gets feedback on their work, and it minimizes the need for teacher intervention to guide the process. For our Peer Feedback and Group Member Evaluation tool, we've designed a sophisticated algorithm to distribute student submissions in the most didactically desirable way.
Our allocation algorithm takes the following aspects into consideration, in order of importance:
1. Every student receives enough submissions to meet the required amount of reviews.
2. Every student or group receives an equal amount of reviews, when possible.
3. Every student or group receives reviews one by one, not all reviews at the same time.
4. Early participation in the assignment is rewarded.
5. To prevent students from gaming the algorithm, review allocation is unpredictable for students.
When the reviewing phase of an assignment has started and a student opens the assignment, we allocate the best available candidate to the student at that moment. This follows the 'just-in-time' principle, and has major advantages over pre-allocating students to each other.
With this approach, we avoid these common scenarios:
- Students dropping out, not participating in the review part of an assignment, leaving some peers without reviews.
- Top-performing students not receiving reviews until late in the review phase, because their reviewers are late to start.
- Students handing in or joining the course late not being able to complete the assignment.
In all these scenarios, our algorithm will lead to optimal distribution of peer reviews: every student or group will get the same amount of reviews. The one scenario that requires us to compromise is when every student has finished reviewing, but one students hands in late/ joins afterwards. This student will be able to Fulfill their review duties, but won't get reviews from their peers. If you feel that the reason for the student submitting the work so late is compelling enough, teachers can always give that student feedback.
Below you see an example of an assignment where 4 students need to review 2 peers. They handed in their work in alphabetical order, which gives a sense of the effect handing in early has on the order in which work is distributed.
- Student A reviews Student B
- Student A reviews Student C
- Student B reviews Student A
- Student C reviews Student D
(Every student has received one review: the algorithm now starts the second batch of reviews)
5. Student C reviews Student A
6. Student D reviews Student B
7. Student B reviews Student D
(The algorithm notices that only Student D still needs to review someone, and prevents that the only candidate left to review is themselves, thus ensures Student B reviews Student D)
8. Student D reviews Student C
(Assignment finished, everyone completed 2 reviews, and has received 2 reviews)