Rubric grading with inline feedback
Grade submissions against rubrics with annotations.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
Previous class address: R10.05.01.03 · Original proposal IDs are retained.
- Typical applications
- SpeedGrader · Gradescope · Turnitin
- Class GUI-need label
G2· GUI-essential
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R10.05.01.03-T01 Score three short essays on a three-criterion rubric in SpeedGrader
Original instruction
Canvas SpeedGrader, course 'ENGL 101 (sandbox)', assignment 'Reflection 1' (3 submissions, synthetic students). Score each on the attached rubric (Thesis, Evidence, Mechanics; 4 levels each) using input/grading_notes.pdf, which gives the instructor's judgments, and add a one-sentence summary comment per student.
- Platforms
web· Any OS (web)- Applications
- Canvas SpeedGrader
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
rubric_assessment per submission: rating per criterion equals the notes; total equals sum; one comment each. Agent has no API token; submissions show grader_id of the UI session.
Proposed partial credit
75% criterion ratings; 25% comments.
R10.05.01.03-T02 Grade five essays with inline comments anchored on each thesis sentence
Original instruction
Canvas SpeedGrader, assignment 'Argument Essay' (5 PDF submissions, synthetic). Score each on the 4-criterion rubric in input/rubric_guide.pdf, then in the DocViewer highlight each essay's thesis sentence and attach a comment that names the strength or weakness of the thesis, and add a summary comment.
- Platforms
web· Any OS (web)- Applications
- Canvas SpeedGrader
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathclick· Point & clicktype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Per-criterion MAE vs expert gold <= 0.5; DocViewer annotations: one highlight per essay overlapping the gold thesis sentence bbox >= 70% with a comment; summary comment present. Canvas API token creation is disabled for the agent's role, and DocViewer annotations are not exposed by the Canvas API, so they can be made only in the DocViewer UI.
Proposed partial credit
45% rubric MAE; 40% thesis highlights with comments; 15% summaries.
R10.05.01.03-T03 Mark three induction proofs with negative rubric items in Gradescope
Original instruction
Gradescope (sandbox course), assignment 'HW4 Induction', question 2 (3 scanned proofs). Create rubric items 'Missing base case (-3)', 'Inductive hypothesis not used (-4)', 'Algebra error (-1)' and 'Correct (0)' under negative scoring, apply them per input/key.pdf, and leave a point comment on the line with each algebra error.
- Platforms
web· Any OS (web)- Applications
- Gradescope
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmenttype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Evaluations export: rubric items defined with exact points; applied item sets per submission equal gold; algebra-error comments anchored on the correct page region (within 40 px of gold line). The Gradescope sandbox issues no API token and accepts grading only from browser sessions (CSRF + session cookie), so grading must be done in the web UI.
Proposed partial credit
25% rubric items; 55% applied sets; 20% anchored comments.
R10.05.01.03-T04 Grade one exam question across twenty scanned submissions with answer groups
Original instruction
Gradescope sandbox, 'Midterm 1', question 2 (short derivation; 20 scanned exams). Build the rubric from input/answer_key.pdf, group identical final answers into answer groups, grade each group consistently while checking the work, adjust two rubric items' points after noticing a key error (input/key_erratum.txt), and confirm the change propagated.
- Platforms
web· Any OS (web)- Applications
- Gradescope
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Per-submission score equals gold (after erratum) for >= 19 of 20; rubric item points equal corrected key; answer groups present; scores for all submissions reflect the adjusted points. The Gradescope sandbox issues no API token and accepts grading only from browser sessions (CSRF + session cookie), so grading must be done in the web UI.
Proposed partial credit
60% scores; 20% rubric correction propagated; 20% answer groups.
R10.05.01.03-T05 Review a similarity report and mark unattributed passages in Turnitin
Original instruction
Turnitin Feedback Studio (sandbox, via Canvas LTI), assignment 'Policy Brief'. For student submission S-1188, exclude quotes and bibliography as the course policy in input/integrity_policy.pdf allows, then mark each remaining match that is unattributed copying with the QuickMark 'Missing citation' and a comment naming the source, and enter the rubric grade from input/notes.pdf.
- Platforms
web· Any OS (web)- Applications
- Turnitin Feedback Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickpath· Continuous pathvis· Visual judgment- Work product
WP10· Annotations- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
Report settings: quotes/bibliography excluded; QuickMarks anchored on the 3 gold unattributed passages (char overlap >= 60%) with source named; no QuickMark on properly cited matches; grade equals notes. Turnitin is reachable only through the Canvas LTI launch and no Turnitin API key is issued, so QuickMarks must be placed in Feedback Studio.
Proposed partial credit
15% filters; 55% passages marked; 15% no false marks; 15% grade.
R10.05.01.03-T06 Grade a forty-student four-question exam with consistent partial credit
Original instruction
Gradescope sandbox, 'Final Exam' (40 scanned exams, 4 questions). Using input/solutions.pdf and the department partial-credit guide, build rubrics for each question, grade every submission, apply consistent point adjustments for the 2 common alternative methods, then publish grades and export the evaluations to output/final_evals.csv.
- Platforms
web· Any OS (web)- Applications
- Gradescope
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Per-question scores vs expert gold: MAE <= 0.5 points and >= 90% exact; same rubric item applied to equivalent answers (consistency check on seeded duplicates); grades published; export matches the platform state. The Gradescope sandbox issues no API token and accepts grading only from browser sessions (CSRF + session cookie).
Proposed partial credit
55% score accuracy; 20% consistency; 15% published; 10% export.
R10.05.01.03-T07 Annotate a PDF submission and grade with a marking guide in Moodle
Original instruction
Moodle sandbox, assignment 'Lab Report 1', student Ana Ruiz (test). In the PDF annotation tool, add a comment at the mislabeled axis on page 2 and a 'tick' stamp next to the correct conclusion on page 4, then score the 3-criterion marking guide per input/notes.txt and save.
- Platforms
web· Any OS (web)- Applications
- Moodle assignment
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickgeom· Precise geometrytype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 1 / 5
Proposed verifier
assignfeedback_editpdf_cmnt/annot: comment on page 2 within 60 px of gold axis label, stamp on page 4 near conclusion; gradingform_guide_fillings scores equal notes. Web services are disabled for the role and the agent has no server shell; only web-origin log events are accepted.
Proposed partial credit
35% comment placement; 25% stamp; 40% marking-guide scores.
R10.05.01.03-T08 Grade with a Moodle marking guide using frequently used comments
Original instruction
Moodle sandbox, assignment 'Case Analysis' (6 submissions). The marking guide has 4 criteria with a frequently-used-comments bank. Grade each submission per input/grading_decisions.xlsx (scores and which bank comments apply), inserting bank comments from the guide rather than typing new text, and add one custom remark where the decisions sheet says so.
- Platforms
web· Any OS (web)- Applications
- Moodle assignment · Excel
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsmulti· Cross-app- Work product
WP10· Annotations- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Fillings: scores exact; criterion remarks equal the bank comment texts specified (exact string) plus the custom remarks; 6 grades saved. Web services are disabled for the role and the agent has no server shell; only web-origin log events are accepted.
Proposed partial credit
50% scores; 35% bank comments; 15% custom remarks.
R10.05.01.03-T09 Grade a group project individually with peer-evaluation adjustments
Original instruction
Canvas SpeedGrader, group assignment 'Capstone Report' (graded individually, 4 groups). Score each group's report on the rubric from input/rubric_scores.csv, then adjust individual members' scores using the peer-evaluation multipliers in input/peer_evals.xlsx (cap at 100%), and leave a private comment for each adjusted student explaining the factor.
- Platforms
web· Any OS (web)- Applications
- Canvas SpeedGrader · Excel
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsmulti· Cross-app- Work product
WP10· Annotations- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
16 student scores equal gold (group rubric x multiplier, capped); rubric assessments present for each; private comments on the 5 adjusted students with the factor value. Personal access token creation is disabled for the agent's role, so Canvas API calls are blocked and grading must be done in SpeedGrader.
Proposed partial credit
30% group rubric scores; 45% adjusted individual scores; 25% comments.
R10.05.01.03-T10 Circle errors on an image submission with the drawing tool
Original instruction
Canvas SpeedGrader, assignment 'Circuit Diagram' (student submissions are PNG images). For the 3 submissions listed in input/errors.txt, use the DocViewer freehand/shape tools to circle each wiring error described, add a point comment explaining it, and enter the score.
- Platforms
web· Any OS (web)- Applications
- Canvas SpeedGrader
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathvis· Visual judgmentclick· Point & click- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Annotations: each of 7 gold error regions enclosed by an ink/shape annotation (bbox IoU >= 0.4) with a nearby point comment; no circles on correct regions; scores exact. Canvas API token creation is disabled for the agent's role, and DocViewer annotations are not exposed by the Canvas API, so they can be drawn only in DocViewer.
Proposed partial credit
55% error regions circled; 25% comments; 20% scores.
R10.05.01.03-T11 Override flaky autograder results and add a style rubric in Gradescope
Original instruction
Gradescope sandbox programming assignment 'PA3'. Review autograder logs for 12 submissions; for the 4 where a test failed due to the known timeout flake documented in input/flake_note.md, override the test score to full credit with an explanation. Then add a manual 'Code style' question with rubric items from input/style_rubric.pdf and grade it for all 12 using the code viewer.
- Platforms
web· Any OS (web)- Applications
- Gradescope
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Overrides exactly on the 4 flaky submissions (other failures untouched) with explanation; style question exists with rubric items exact; style item sets vs gold F1 >= 0.85. The Gradescope sandbox issues no API token and accepts grading only from browser sessions (CSRF + session cookie), so overrides and grading must be done in the web UI.
Proposed partial credit
40% correct overrides; 15% style rubric; 45% style grading.
R10.05.01.03-T12 Grade twelve lab reports to calibration with anchored data-table comments
Original instruction
Canvas SpeedGrader, 'Lab Report 3' (12 PDFs). Grade with the department rubric (5 criteria) following the calibration examples in input/calibration/, and for every incorrect value in a student's data table place a point annotation on that cell with the correct value. Your scores will be compared with the department's expert grades.
- Platforms
web· Any OS (web)- Applications
- Canvas SpeedGrader
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentclick· Point & clicknav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Quadratic weighted kappa vs expert criterion scores >= 0.7; point annotations: recall of the 31 seeded wrong cells >= 0.85 (within the cell bbox) with the correct value in the comment; precision >= 0.8. Canvas API token creation is disabled for the agent's role, and DocViewer annotations are not exposed by the Canvas API, so they can be placed only in DocViewer.
Proposed partial credit
50% rubric agreement (QWK); 35% wrong-cell annotations; 15% precision.
R10.05.01.03-T13 Annotate and rubric-grade submissions in Brightspace and publish feedback
Original instruction
D2L Brightspace sandbox, course 'NURS 310 (sandbox)', assignment 'Care Plan'. For 4 submissions, use the annotation tool to highlight the nursing diagnosis statement and add a comment, score the analytic rubric per input/notes.xlsx, and publish the feedback so students can see it.
- Platforms
web· Any OS (web)- Applications
- Brightspace
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathclick· Point & clicktype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Annotation export: highlight overlaps gold diagnosis statement >= 60% with comment; rubric levels exact; feedback published flag true for all 4. Valence API app keys are withheld from the agent, so annotations and rubric scores must be entered in the Brightspace UI from the browser session.
Proposed partial credit
35% annotations; 45% rubric levels; 20% published.
R10.05.01.03-T14 Fill a Turnitin rubric scorecard and leave summary feedback
Original instruction
Turnitin Feedback Studio sandbox, assignment 'Policy Brief', submission S-1203. Open the rubric scorecard, select levels for its 4 criteria as in input/notes.txt, and add the summary text comment provided.
- Platforms
web· Any OS (web)- Applications
- Turnitin Feedback Studio
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP10· Annotations- Conditions
licensed· Licensed pro app- Proposed difficulty
- 2 / 5
Proposed verifier
Scorecard levels exact; computed grade equals rubric; summary comment text exact. Turnitin is reachable only through the LMS LTI launch and no Turnitin API key is issued, so the scorecard must be filled in Feedback Studio.
Proposed partial credit
70% rubric levels; 30% summary comment.
R10.05.01.03-T15 Box incorrect steps in scanned handwritten math homework
Original instruction
Canvas SpeedGrader, 'Problem Set 6' (scanned handwritten PDFs). For 6 submissions, draw a box around each incorrect step (not just the final answer), attach a point comment naming the error type from input/error_types.md, and score each problem per the partial-credit rules in input/key.pdf.
- Platforms
web· Any OS (web)- Applications
- Canvas SpeedGrader
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
geom· Precise geometryvis· Visual judgmenttype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Box annotations vs 18 gold error-step regions (IoU >= 0.3, one-to-one); error-type labels in comments exact; per-problem scores equal gold for >= 90%. Canvas API token creation is disabled for the agent's role, and DocViewer annotations are not exposed by the Canvas API, so boxes can be drawn only in DocViewer.
Proposed partial credit
45% boxes on wrong steps; 20% error types; 35% scores.
R10.05.01.03-T16 Build a comment library and apply it consistently to ten submissions
Original instruction
Canvas SpeedGrader, 'Lab Report 2' (10 submissions). Create the 6 reusable comments in input/comment_bank.txt in the Comment Library, then for each submission add exactly the library comments listed for it in input/assignments.csv and the score given there.
- Platforms
web· Any OS (web)- Applications
- Canvas SpeedGrader
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Comment library contains the 6 comments; each submission's comments equal the listed set (exact text); scores exact. Personal access token creation is disabled for the agent's role, so Canvas API calls are blocked and comments must be added in SpeedGrader.
Proposed partial credit
25% library; 50% comments applied; 25% scores.
R10.05.01.03-T17 Leave inline code-review feedback on a GitHub Classroom pull request
Original instruction
GitHub Classroom (sandbox org), assignment 'linked-list'. For 3 student repositories, open the Feedback pull request, add review comments on the exact lines with the defects listed in input/defects.csv (off-by-one, leaked node, missing null check), fill the rubric score in the PR description template, and submit each review as 'Request changes'.
- Platforms
web· Any OS (web)- Applications
- GitHub Classroom · GitHub
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader reads PR reviews: comments on the gold file/line for each defect (exact line, defect named); PR body rubric totals exact; review state CHANGES_REQUESTED. Agent's token lacks API scope; reviews created via web UI.
Proposed partial credit
55% comments on correct lines; 25% rubric; 20% review state.
R10.05.01.03-T18 Investigate five high-similarity submissions and score with integrity deductions
Original instruction
Canvas with Turnitin Feedback Studio (sandbox), assignment 'Research Essay'. Five submissions exceed 35% similarity. For each, separate properly quoted and cited matches from copying, mark copied passages with QuickMarks naming the source, apply the rubric with the integrity deduction from input/policy.pdf, and file the integrity report form in the sandbox portal for the cases that meet the referral threshold.
- Platforms
web· Any OS (web)- Applications
- Turnitin Feedback Studio · Canvas SpeedGrader · integrity portal (sandbox)
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentpath· Continuous pathmulti· Cross-app- Work product
WP10· Annotations- Conditions
licensed· Licensed pro appmultiapp· Multi-applong· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Per submission: QuickMarks on gold copied passages (F1 >= 0.8), none on cited ones; rubric score with deduction exact; integrity reports filed for exactly the 2 cases over threshold with source list. No Turnitin or portal API keys are issued and Canvas token creation is disabled, so marks, scores and reports must be entered in the web UIs.
Proposed partial credit
40% copied-passage marking; 25% scores with deductions; 25% correct referrals; 10% no false marks.
R10.05.01.03-T19 Submit the same five rubric assessments through the Canvas API (control)
Original instruction
Control pair of T02. With a token in env CANVAS_TOKEN, post rubric assessments and summary comments for the 5 'Argument Essay' submissions via the submissions API, using the same rubric guide; inline thesis annotations are out of scope.
- Platforms
lin· Linux- Applications
- Canvas API · Python
- Interfaces
api· API/SDKterm· Terminal- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
code· Code & commands- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Same rubric MAE metric as T02; comments present. Isolates scoring judgment from DocViewer annotation work.
Proposed partial credit
75% rubric MAE; 25% comments.
R10.05.01.03-T20 Finalize moderated grades from two TAs' provisional marks
Original instruction
Canvas sandbox, moderated assignment 'Thesis Proposal' (8 submissions, two TA provisional graders). As final grader, compare each pair of provisional rubric assessments; where they differ by more than one level on any criterion, read the submission and enter your own final rubric; otherwise select the provisional grade from the TA named in input/policy.txt. Then post grades.
- Platforms
web· Any OS (web)- Applications
- Canvas SpeedGrader
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Moderation export: selected provisional grade source per submission equals gold rule; the 3 high-disagreement submissions have final-grader rubrics with criterion MAE <= 0.5 vs expert; grades posted. Personal access token creation is disabled for the agent's role, so Canvas API calls are blocked and moderation must be done in the web UI.
Proposed partial credit
40% correct selections; 40% own rubrics on disputed items; 20% posted.
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.