Creating and Using Good Rubrics

Each rubric will differ based on the assignment you ask your students to complete, but you should focus on the specific learning objectives you wish students to develop³. Think about the observable attributes you want to see from them, as well as those that you don’t.

If you are creating a rubric for an existing assignment that has been used in past offerings of the course, it may be useful to re-read some of those assignments to gain a perspective of the typical spectrum of answers that students provided; it is helpful to predict the kind of answers you expect to see when designing a rubric to cover as many of these as possible, as well as keeping in mind what students must typically show in their assignments to achieve low, middle and high grades.

When creating rubrics (particularly if they are holistic), it is helpful to divide each criterion into levels, such that you create categories that relate to different responses that reflect the progression from novice to expert-like writing (e.g. a score of 1 might represent emerging ability, whereas a score of 5 might represent the quality of an expert.

Defining these categories can be useful for students who wish to monitor their learning progress, especially because the scores/marks will not represent linear progression (e.g. a student who scores 4 out of 5 for logical development does not mean that student has twice as much ability as one who scores 2 out of 5).

If you are creating a rubric for a new assignment, you should keep the following tips in mind:

1. First divide the total marks for the assignment into different sections (e.g. five marks for the depth of content, five for the quality of sources used, five for integration of these sources, five for quality of argument, and five for paragraph structure and transitions). Note that most rubrics focus on six to seven criteria⁴, but the absolute number should depend on what the assignment⁵ asks of students; in most cases, limiting the number of criteria is more practical, but sufficient scope should exist within these criteria to distinguish the full range of likely student abilities.

2. Provide a detailed explanation of what an answer would need to show to be awarded any mark/score within the range available for each section (e.g. if you have allocated up to five marks for the depth of content, you should clearly state what a student must cover to gain one, two, three, four and five marks).

3. Do not use potentially ambiguous explanations. For example, do not propose a scale of 0 – 3 marks where weak, fair, good and very good are the descriptors used to differentiate between scores of 0, 1, 2, and 3 because different graders will likely differ on their interpretation of what is weak, fair, good or very good. Instead, try to provide objective definitions (e.g. less than one primary source = 0, one or two primary sources = 1…). If your rubric is holistic rather than analytic, you should provide detailed summaries (with examples) to clearly distinguish between marks/scores.

4. Add a qualitative description to each of these marks/scores if you plan to share the rubrics with students at any stage, so that they can assess their own learning development based on the grade they receive for the assignment.

5. Share the rubric with colleagues and TAs and ask for feedback. Expect to make changes based on this feedback.

6. If you plan to share the rubrics with students, ask these same students whether it is clear to them before the assignment begins. If they misinterpret any part of it, this may be a sign that you should make changes.

By following the above steps, you should design rubrics that are straightforward and objective to use. However, when more than one person will grade the assignments, it is important for all graders (e.g. instructors and TAs) to meet to troubleshoot any issues at an early stage. Ensuring your graders can use your rubric objectively is a crucial part of the design process, which is why the following steps are very important before you grade the assignments from an entire class:

7. Try to select a range of assignments from students of varying abilities and then take turns to grade each one using the rubric.

8. Compare the grades you each assigned the same assignments and calculate your inter-rater reliability (see below).

9. If you have a low reliability (e.g. you have awarded different grades to the same assignments), this indicates issues with the rubric or the way it is being interpreted.

10. In such a scenario, work through each section and pinpoint where – and why – graders have awarded different scores based on the rubric, and then rephrase the rubric to make it more objective.

11. Grade a new set of assignments in the same way before comparing grades until you are satisfied that your inter-rater reliability is sufficiently high, and you are confident that you are all using the rubric in the same way.

Calculating inter-rater reliability (IRR) provides an estimate of the degree of agreement between different graders using the same rubric. A well designed, objective rubric should result in a high IRR (approaching 1), whereas a poorly designed, ambiguous one will result in a low IRR (approaching 0 or –1, depending on your method).

All graders who will be grading assignments using your rubric should take part in IRR assessments; if they do not, the IRR estimate you obtain will not encompass data from everyone whose interpretations will provide the final grades. As a result, the rubric may not be effectively assessed before it is used to grade the assignments of a whole class.

There are various techniques for providing IRR estimates, and the best one to use depends on the situation⁶. When you obtain data from three or more coders, it is generally best to use an extension of Scott’s Pi statistic⁷, or compute the arithmetic mean of kappa⁸ (a statistic used in IRR analysis⁹). There are no cast-iron guidelines for an acceptable level of agreement, but popular benchmarks for high agreement using kappa are 0.75¹⁰ and 0.8¹¹. Hallgren⁶ provides a detailed overview of these procedures.

You can save time when providing assignment feedback to large classes by referring to the rubric when highlighting example student answers, and then explaining how/why they were scored/marked as they were. By showing a spectrum of answers, you can also indicate to students how they must approach similar questions in future to attain high scores/marks.

Depending on the situation, you may also wish to provide a marked rubric with each student’s assignment, so that they can see how they have been assessed on each question¹². This can be very useful for students, and can again save you time in providing feedback in large classes as it can reduce the need to write explanations to justify scores/marks throughout.

In addition, we encourage interested instructors to contact us to obtain access to some example rubrics created by the ScWRL team.  Please fill out the Access Request and Feedback Form and then, once you have been granted access, input your password at the suggested solutions password protected page.