Issue link: http://ncumarketing.uberflip.com/i/1003100
Elaine Willerton, PhD | Assessment Director, SSBS We've begun to hear a lot about rubrics, both for assessing student learning and for grading. But how do we know our rubrics are any good? Just as we would question the reliability and validity of an instrument used in a research study, we should hold our rubrics to similar standards. An effective rubric begins with careful construction. The rubric criteria should align with the assignment being graded and/or learning outcome being assessed. To support accuracy, criteria should be comprehensive, mutually exclusive and written in such a way that raters understand the discriminating characteristics of each criterion and performance levels. To determine consistency, rubric scores should be consistent across raters. Inter-rater reliability for rubrics is evaluated via a calibration or norming process. The specific process of norming sessions is unique to each organization, but in general should proceed as follows: • A group of experts (e.g., faculty teaching the course using the rubric) gathers for a facilitated norming session • The facilitator provides an overview of materials – assignment prompts, student papers, rubric/score sheets • The experts all read the same example of student work and score using the rubric, then discuss their ratings for each criteria • The facilitator tracks the scores and encourages dialog around different ratings (the goal is not consensus) • The scoring and discussion process is repeated one to two more times • Experts leave the session with three to five additional assignments to score and submit independently • The results of all scored rubrics are reported • Experts discuss results and make recommendations for improvements • Improvements are implemented and tested I recently participated in a mock norming session at the WASC Assessment Leadership Academy. We practiced scoring examples of student work using a VALUE rubric. Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics were developed by the Association of American Colleges and Universities. I found the exercise very helpful. Listening to other perspectives and interpretations of the rubric and the student work allowed me to reflect more deeply on my own evaluation. Scoring multiple examples of student work was eye opening and improved my understanding of the rubric's utility. Rubric norming sessions are an important part of the assessment process and I hope to arrange some in the coming months. As you begin using grading and assessment rubrics in your courses, please report any concerns or suggestions via the SSBS Faculty Feedback Form. Rubric Norming