technology tip of the month Pointer and Clicker Article
Virgil E. Varvel Jr.
Rubrics


Evaluating Rubrics

Once you've used a rubric, it is important to see if it works. Are the results from the rubric valid, reliable, consistent, objective, usable, etc.? The following questions should be asked in order to put your rubric to the test. When a rubric is not up to par, a redesign is probably advisable; however, the redesign would not necessarily require repetition of all of the steps of rubric creation.

  • Validity - Was the rubric too hard, or too easy when reporting final scores? Courses can use absolute scoring, where students have to get a certain number of points for an A. If a rubric is too hard or too easy, then final grades could be affected.

  • Validity - Are you experiencing a problem of central tendencies? Some courses use a relative scale to distinguish amongst individual students. If everyone receives similar grades when using a rubric, then it is difficult to apply relative grades. The rubric may be written in such a way that too many works fit into the same relative level of achievement. You may want to rewrite the rubric so that there are more delineations or so that the deliniations in scores are more applicable.

  • Validity - Was there a clear basis for assigning scores at the various levels within each criteria? After using the rubric, did it make sense not just when, but why certain assignments were getting a score higher or lower than others? If not, you will want to see what did not make sense and address the issue in your redesign of the rubric.

  • Validity - Did the rubric address something that the students were not expecting? Especially when the rubric is not known to the students beforehand, there may be something assessed by the rubric that was not expressed in the directions or expected by the students. You may have even applied something in the rubric that was scoring something that you did not actually address in your instruction. Either the instruction or the rubric probably needs to be changed in order to reflect what the students are learning and should know. The rubric should not be assessing extraneous materials.

  • Validity - Was something left out? Even if you use student examples when creating the rubric, in some subjects, the state of knowledge changes with time. For that reason, the assignments turned in by students may vary as well and something that should have been assessed by the rubric may have been left out as well. There are other reasons that something may be missing as well, such as simply missing the concept when brainstorming. Take the lesson learned and apply it to a redesign of the rubric for future use.

  • Validity - Was the rubric developmentally appropriate? Especially at earlier grades, it is a common mistake to design a rubric at a level of understanding above which the students probably are capable of attaining. It is important to consider the audience in a redesign and initial design of a rubric. If a certain criteria is consistently graded low, either the instruction is lacking or the wording of the scale may be done in such a way as to negatively bias the results.

  • Reliability - Did you experience difficulties assigning scores for all student work? Perhaps some assignments did not fit well into your scoring plan.

  • Consistency - As you were using the rubric, did you notice yourself changing your mind over the scale of the criteria? While it is difficult to redesign a rubric half way through grading, it important to verify scores whenever your attitude changes in the process of applying the rubric. Keep a list of the changes that you desire to make so that you can update the rubric before the next time it is used.

  • Consistency - When using multiple scorers, it is often a good idea to have everyone grade a few assignments in common. Afterward, the scores can be compared to look for consistency among scorers. Usually, such issues should be addressed before rubric use, but weighting of scores can be used to correct for differences post-scoring.

  • Objectivity - Was there internal bias in the rubric and possibly in the assignment? By such techniques as a t-test on the differences of certain criteria by different groups of gender, race, economics, etc., you can see if there was some bias built into the rubric that you were not anticipating. Rarely do instructors take the time to address such issues post instruction, but rather rely on instructional design or simply their own feelings to look for such biases, but equally rare would be ones ability to determine bias within themselves. Often statistical analysis is a more accurate measurement of bias.

  • Usability - Can the rubric be applied to multiple assignments? Although it may seem like there is a lot of work ahead of you, it is important to keep in mind that a new rubric does not need to be constructed for every assignment. Often, only a few may be needed within an entire course. Look for ways of combining, altering, refining, or other method in order to apply the rubric to multiple assessments. Once applied, evaluate the ability of the rubric to be applied in such a way. If the rubric is not working out as you expected, then you may need multiple rubrics for the various assignments.

  • Usability - Was the rubric practical? After you have finished all of the work and considering all of the possible assignments that you could use the rubric on, are you really helping yourself by using the rubric?

If any of the above questions are answered no, then you need to consider redesigning the rubric or perhaps even consider another method of scoring or assessing an assignment. It is unusual to get everything right the first time. Even when the rubric is piloted, you may miss something that needs addressed later. So be prepared to make some changes.

Back to Top

ION Home | Pointers and Clickers

Home | Online Courses | Institutes & Presentations | Educational Resources | Initiatives | Institutional Partners | About ION

uiuc uic uis