How can I objectively grade something as subjective as communication?

S. Ruff, June 2013

When I first began teaching communication in the MIT Department of Mathematics almost a decade ago, the most frequent question I received was “How can I objectively grade something as subjective as communication?”

The process of doing so begins with identifying priorities: what is it that characterizes “excellent” communication? In the blog post “What are the characteristics of effective communication?” I used the priorities identified by more than 30 mathematics educators to characterize effective student communication. The resulting characterization encompasses mathematical correctness, whether the constraints of the assignment are met, whether the communication is clear, and whether it is engaging. The characterization is worded generally enough to apply to both mathematics presentations and papers.

Because the characterization is general, it may be tempting to use it to write a grading rubric that can be applied to any communication assignment; but such a generic rubric would be unlikely to work well in practice. To be successful as a teaching and assessment tool, a rubric should be tailored to the assignment, to the students, to the priorities of the educator, etc. For example, in a first proof-based course, precision and rigor might be weighted more heavily, while in a capstone, the focus might instead be on insightfulness and understanding of the priorities and conventions of mathematical discourse. Within any one course, specific assignments may target different aspects of effective communication, and the rubrics should be adjusted to match.

Below is an example of how the general list of characteristics of effective communication can be used as a starting point for drafting a rubric tailored to a specific assignment. The assignment is the final presentation of a team-based laboratory course for senior math majors at MIT. The students are required to do a practice presentation, so the practice and revision process is included in the rubric. These majors typically have strong math and presentation skills, so the rubric instead places emphasis on vision and on helping the target audience to understand the most interesting aspects of the research.

Process (20)

20   The practice presentation was carefully crafted. The revision took into account but was not limited to the feedback of course staff and of teammates.

15   The practice was a work in progress, with a pre-prepared general plan and with details decided  extemporaneously; the revision was limited to provided feedback; or some feedback was ignored.

10   The practice was extemporaneous or closely followed the project paper; or the revision addressed only a few major aspects of the provided feedback.

5      The practice was extemporaneous and the revision addressed minor aspects of the provided feedback.

Mathematical Correctness and Vision (40)

40   Substantial progress is made within the available scope of the project and identified phenomena are convincingly explained (i.e., proofs are rigorous; conjectures are supported with convincing evidence).

30   Some notable progress is made within the available scope of the project and explanations are suggested for identified phenomena (i.e., claims are rigorously stated and support goes beyond a few specific examples).

20   A good expository description of the problem and of the most interesting aspects of how it appears to behave (e.g., conjectures are stated).

10   Description of the problem and of some immediately apparent aspects of its behavior.

Communicates Effectively to Target Audience (40)

40   The presentation is engaging and clear. For example, it is tightly focused around the most interesting results while less important points are de-emphasized or omitted, the delivery is engaging and insightful, with connections made both within the talk and to other disciplines, the presentation is carefully prepared and delivered and it is crafted to be understood by classmates (see below).

30   The presentation is crafted to be understood by classmates. For example, the level of difficulty and formality are appropriate to the audience and to the content; examples, figures, and explanations aid understanding; the presentation is carefully structured; sufficient detail is provided and the relevance of details is clear; the language is precise.

20   The presenter shows his or her research, using terminology and notation approximately correctly, and taking sufficient care with structure, clarity, delivery, and board work to enable an expert to discern what was intended.

10   The presenter says and writes what he or she researched, without connecting to the language, knowledge, or interest of those watching. For example, nonstandard notation and terminology may be used without introduction, writing may be illegible, delivery may be inaudible.

Notice that each category includes grade definitions that may mirror the development of a learner. For example, the descriptions in the category “Communicates Effectively” are based on the assumption that as a novice learns to present, he or she may initially write illegibly or speak quietly, but with practice the presenter may master the mechanics of presenting and begin to focus more on crafting content to help the target audience understand. Rubrics such as this are called developmental rubrics. Although difficult to create, developmental rubrics are appealing because they reduce the need to provide specific comments: the grade itself is associated with meaningful feedback about what the student does well and which developmental steps the student should focus on next. In short, the rubric is designed to teach.

Just as paper drafts need revision, so too do rubrics—classroom testing is a must. A helpful strategy is John Bean’s “right brain/ left brain” approach [1, pp. 280-281]. When faced with a communication assignment to grade, Bean suggests first assigning a holistic grade: is this an A paper? a B paper? Then apply the assessment tool as objectively as is feasible, and compare the resulting grade to the holistic grade. If the assessment tool is designed well, the two should agree reasonably closely. If they disagree, look more closely at the reasons for the difference. Perhaps the holistic grade was inappropriate because the first holistic impression ignored certain important aspects of the communication, or perhaps the assessment tool needs refinement.

For example, consider applying the rubric above to Karen’s presentation:

Karen’s presentation on generating functions gained and held the rapt attention of many of the students; she successfully conveyed the beauty and power of generating functions…to the front half of the class. The rest couldn’t hear her and couldn’t tell from the board what she was saying, because the board remained blank except for occasionally scrawled examples of functions.

When we try to apply the rubric, we see an immediate problem: although Karen’s presentation is clear and engaging (a “Communicates Effectively” grade of 40), it’s also almost inaudible and illegible (a 10). A rubric that may mirror the development of many students does not necessarily mirror the development of all students.

What follows is a revision of the rubric that can be applied to a more diverse range of students. The category for communicating effectively has been divided into three categories. The mechanics of board work and delivery are de-emphasized because these students are fairly comfortable presenting: the emphasis is placed instead on communicating clearly to the target audience. The grade for mathematical correctness and vision has been removed because this is a large part of the grade for the associated project paper—including the math grade in both is double jeopardy.

Process (20)

The practice presentation was carefully crafted. The revision took into account but was not limited to the feedback of course staff and of teammates.

Mechanics (20)

Carefully structured board work complements vocal delivery by emphasizing important points and helping the audience absorb subtle points, without distracting. Slides are used as appropriate and are integrated seamlessly into the presentation.  Delivery is carefully prepared. Teammates work well together.

Clarity (40)

The presentation is crafted to guide classmates to understanding. For example, the level of detail, difficulty, and formality are appropriate to the audience and to the content; the well chosen topic, structure, examples, figures, explanations, and use of formality help the audience to understand both the problem statement and its solution; the language is precise.

Audience Engagement (20)

The presentation is engaging and clear. For example, it is tightly focused around the most interesting results while less important points are de-emphasized or omitted, the delivery is engaging and insightful, with connections made both within the talk and to other disciplines, and the presenters adapt well to audience reactions.

Notice that this assessment tool is not a developmental rubric: it does not include specific grade descriptions that reflect a student’s development within each category. These descriptions have been omitted so the categories can fit on one page with ample space for writing individualized comments to the student. The tradeoff is that this assessment tool no longer works as well as a stand-alone teaching tool or for ensuring grading consistency.

Which is better—a developmental rubric or a simpler assessment tool? As with presentations, what ultimately makes an assessment tool successful is whether it achieves its purpose for its target audience. To what extent is the purpose assessment? teaching? communication of priorities? consistency of grading? What other means for achieving these objectives are also used? How precise must the grades be? These answers inform the structure and level of detail of the rubric, while the educators’ priorities inform its content and weighting.

It’s easy to get lost in the details of rubric creation, so keep sight of the big picture. Whether Karen receives oral feedback, written feedback, or a Mechanics grade of 0/20, a Clarity grade of 40/40, and an Audience Engagement grade of 15/20, she should learn that she has interesting things to say and says them well, so she should focus on saying them loudly and visibly enough for all to benefit.

Acknowledgements: Thanks go to Suzanne Lane for introducing me to the advantages and challenges of developmental rubrics and to Haynes Miller for providing the original project lab rubric on which these are loosely based.

References