Media review & approval
Leave frame-accurate notes and approvals on video and design reviews.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Frame.io · Figma comments · Filestage
- Class GUI-need label
G3· GUI-only
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R11.04.01.02-T01 Leave frame-accurate review comments from a notes list
Original instruction
In Frame.io (test account), open asset promo_v2.mp4 (23.976 fps) in project 'Spring Promo'. Leave the 8 comments in director_notes.csv, each at its exact timecode (HH:MM:SS:FF), with the comment text as given.
- Platforms
web· Any OS (web)- Applications
- Frame.io
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrolltype· Text & formsav· Audio/video over time- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Verifier Frame.io API: 8 comments on the asset with frame numbers equal to the timecodes (exact frame), texts equal. Compared with control pair T17. The agent's account has no developer token or OAuth app (the verifier key is grader-only), so comments must come from the Frame.io web player session.
Proposed partial credit
60% frame-exact placement (7.5 points each); 40% text correct
R11.04.01.02-T02 Draw an annotation around the logo on a specific frame
Original instruction
In Frame.io asset promo_v2.mp4, at 00:00:12:05 draw a freehand or shape annotation enclosing the brand logo in the top-right corner and comment 'Logo too small - scale to 120%'. Then at 00:00:20:00 draw an arrow pointing to the price text with comment 'Price needs disclaimer'.
- Platforms
web· Any OS (web)- Applications
- Frame.io
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathnav· Pan/zoom/scrolltype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
API: two comments at exact frames with annotation JSON; first annotation polygon/bbox covers the logo GT box (IoU ≥0.5, contains centre); second arrow tip within 30 px of price text GT box; texts match. No Frame.io developer token or OAuth app is issued to the agent (the verifier API key is grader-only), so comments must come from the web player session.
Proposed partial credit
15% frames exact; 35% logo annotation covers logo; 30% arrow points to price; 20% texts
R11.04.01.02-T03 Verify previous notes are fixed in the new version and set status
Original instruction
In Frame.io, promo_v3.mp4 is stacked on v2, which has 6 notes. Check each note in v3 at the same moment. Reply 'Fixed' or 'Not fixed: <what>' on each v2 comment, then set v3's status to Approved only if all are fixed, otherwise Needs Review.
- Platforms
web· Any OS (web)- Applications
- Frame.io
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
API: replies on all 6 v2 comments with fixed/not-fixed matching gold (2 not fixed); v3 status Needs Review. Partial if statuses right but replies wrong. No Frame.io developer token or OAuth app is issued to the agent (the verifier API key is grader-only), so comments must come from the web player session.
Proposed partial credit
60% per-note verdicts; 20% replies explain unfixed; 20% correct asset status
R11.04.01.02-T04 Pin design-review comments on the right Figma layers
Original instruction
In Figma file 'Checkout v2' (test team), review frames Cart, Shipping and Pay against review_checklist.md (contrast ≥4.5:1, 8-pt spacing, button labels sentence case). Pin a comment on each violating layer stating the rule broken. There are 7 violations.
- Platforms
web· Any OS (web)mac· macOSwin· Windows- Applications
- Figma
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentclick· Point & clicktype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Figma REST comments: client_meta node_id/offset inside each violating layer's bounds; rule keyword in text; precision and recall vs 7 gold violations; no edits to the design file (version unchanged). The agent gets no Figma personal access token or OAuth app and plugins are disabled for the test team, so comments must be pinned from the Figma client session.
Proposed partial credit
50% recall; 25% precision; 15% rule named; 10% design untouched
R11.04.01.02-T05 Resolve Figma comments that the latest version addresses
Original instruction
In Figma file 'Landing page', 10 open comments were left on the previous version. Inspect the current design; resolve comments whose request is now implemented and reply 'Still open: <reason>' to the others without resolving them.
- Platforms
web· Any OS (web)mac· macOSwin· Windows- Applications
- Figma
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentclick· Point & clicknav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Figma REST: resolved_at set exactly on the 6 gold-implemented comments; 4 others unresolved with 'Still open' replies naming the missing element. The agent gets no Figma personal access token or OAuth app and plugins are disabled for the test team, so comments must be pinned from the Figma client session.
Proposed partial credit
60% resolve decisions; 25% replies on open; 15% reasons specific
R11.04.01.02-T06 Mark typos on a brochure proof and approve with changes
Original instruction
In Filestage (test workspace), open brochure_proof_v1.pdf in review step 'Copy check'. Find the 5 errors (misspellings or product names that differ from product_list.txt), add a text-highlight comment with the correction on each, then set your decision to 'Approved with changes'.
- Platforms
web· Any OS (web)- Applications
- Filestage
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentdrag· Drag & droptype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Filestage API (verifier): highlight comments whose page and text range overlap the 5 gold typos (≥1 char overlap), with correction text; decision state 'approved with changes'. No Filestage API key is issued to the agent (the verifier key is grader-only), so review actions must come from Filestage browser sessions.
Proposed partial credit
60% typos found (12 points each); 20% corrections stated; 20% decision
R11.04.01.02-T07 Mark music problem segments with range comments
Original instruction
In Frame.io asset interview_edit.mp4 (25 fps), mark the 3 segments where the music bed is louder than dialogue using range comments (in and out points) with text 'Duck music'. Segment boundaries must be within 0.5 s of where the problem starts and ends.
- Platforms
web· Any OS (web)- Applications
- Frame.io
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timenav· Pan/zoom/scrolldrag· Drag & drop- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
API: 3 comments with duration ranges; each range overlaps a gold segment with start/end within 12 frames; no extra ranges on clean sections. No Frame.io developer token or OAuth app is issued to the agent (the verifier API key is grader-only), so comments must come from the web player session.
Proposed partial credit
60% segments found with boundaries; 25% no false ranges; 15% text
R11.04.01.02-T08 Bring review comments into a Premiere timeline as markers
Original instruction
Premiere Pro project ad_cut.prproj is linked to Frame.io. Import all 11 Frame.io comments on ad_cut_v4 as sequence markers (via the Frame.io panel), fix the 3 notes that only require trimming clips by the stated frames, then mark those 3 comments complete in Frame.io.
- Platforms
mac· macOSwin· Windows- Applications
- Premiere Pro · Frame.io
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appav· Audio/video over timedrag· Drag & drop- Work product
WP05· Time-based media- Conditions
licensed· Licensed pro appmultiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Project XML: 11 markers at comment frames with text; 3 target clips trimmed by the exact frame counts; Frame.io API: those 3 comments completed, others open. Premiere ExtendScript/UXP scripting is disabled and no Frame.io developer token is issued, so markers and completions must come from the app panels.
Proposed partial credit
35% markers imported; 35% trims exact; 30% completion states
R11.04.01.02-T09 Check v4 notes against v5 in Vimeo review
Original instruction
In Vimeo (test account) review page for 'Explainer', version 4 has 5 timecoded notes. Open version 5 and reply to each v4 note with 'Fixed in v5' or 'Not fixed' after checking that moment, and add one new note on v5 at any remaining issue you find in the first 30 s (one gold issue).
- Platforms
web· Any OS (web)- Applications
- Vimeo Review
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmenttype· Text & forms- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Vimeo API (verifier): replies on 5 notes vs gold verdicts; new v5 note within ±1 s of the gold issue time with relevant keyword. No Vimeo API token or app is issued to the agent (the verifier token is grader-only), so replies must come from the review page session.
Proposed partial credit
60% verdicts; 40% new issue found
R11.04.01.02-T10 QC a cut for flash frames, black frames and audio dropouts
Original instruction
In Frame.io, QC the 3-minute asset final_cut_v7.mp4 (24 fps): leave a comment at the exact frame of every flash frame (single-frame cut), unintended black frame and audio dropout, labeled 'Flash frame', 'Black frame' or 'Audio dropout'. There are 9 defects.
- Platforms
web· Any OS (web)- Applications
- Frame.io
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
API comments vs gold defect list: match if frame within ±1 (±6 for audio) and label correct; precision/recall; agent cannot download the file (download disabled on share). No Frame.io developer token or OAuth app is issued to the agent (the verifier API key is grader-only), so comments must come from the web player session.
Proposed partial credit
60% recall; 25% precision; 15% labels correct
R11.04.01.02-T11 Mark frames ready for dev and flag spec mismatches
Original instruction
In Figma Dev Mode for file 'Settings app', mark frames Account and Privacy as 'Ready for development'. Compare their measurements with spec_tokens.json (spacing, radius, font sizes) and pin a comment on each mismatching layer stating expected vs actual (5 mismatches).
- Platforms
web· Any OS (web)mac· macOSwin· Windows- Applications
- Figma
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmenttype· Text & forms- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Figma REST: both frames have devStatus READY_FOR_DEV; comments on 5 gold layers with expected/actual numbers matching; no extra comments. The agent gets no Figma personal access token or OAuth app and plugins are disabled for the test team, so comments must be pinned from the Figma client session.
Proposed partial credit
25% dev status; 55% mismatches found; 20% values stated
R11.04.01.02-T12 Configure a two-step approval workflow for a project
Original instruction
In Filestage project 'Packaging Q1', set review step 1 'Brand' with reviewer group Brand (due 5 days, all must approve) and step 2 'Legal' (group Legal, due 3 days) that starts automatically once step 1 is approved. Upload carton_v1.pdf to step 1 and send review requests.
- Platforms
web· Any OS (web)- Applications
- Filestage
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsdrag· Drag & drop- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Filestage API: steps order/names, groups, due offsets, approval rule 'all', auto-start; file in step 1; review invitations sent to Brand group members (sandbox mail sink). No Filestage API key is issued to the agent (the verifier key is grader-only), so review actions must come from Filestage browser sessions.
Proposed partial credit
40% step config; 20% auto-start; 20% upload; 20% invitations
R11.04.01.02-T13 Sketch arc corrections over animation frames
Original instruction
In SyncSketch review 'Jump_v3' (24 fps), draw over frames 12 and 18 to show the corrected arc of the character's hand following the reference arc in notes/arc_ref.png, and leave a note on frame 15 'Ease out more before apex'.
- Platforms
web· Any OS (web)- Applications
- SyncSketch
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathav· Audio/video over timevis· Visual judgment- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
SyncSketch API (verifier): sketches exist on frames 12 and 18; stroke path Fréchet distance to reference arc (in frame coords) ≤40 px; note on frame 15 with text. No SyncSketch API key is issued to the agent (the verifier key is grader-only), so sketches must be drawn in the review page session.
Proposed partial credit
30% sketches on correct frames; 50% arc shape matches; 20% frame-15 note
R11.04.01.02-T14 Create a restricted review link for a client
Original instruction
In Frame.io, create a review link for the 3 assets in folder 'Client Round 2' with comments allowed, downloads disabled, visible watermark with viewer email, and expiry 2026-10-31. Send it to client@sandbox.test.
- Platforms
web· Any OS (web)- Applications
- Frame.io
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 2 / 5
Proposed verifier
API: review link with exactly the 3 assets; allow_comments true, allow_download false, watermark enabled with email, expires_at 2026-10-31; invitation in sandbox mail sink. No Frame.io developer token or OAuth app is issued to the agent (the verifier API key is grader-only), so comments must come from the web player session.
Proposed partial credit
25% assets; 50% permissions/watermark/expiry; 25% sent
R11.04.01.02-T15 Set version statuses and notes from a dailies session
Original instruction
In Flow Production Tracking (ShotGrid, test site) playlist 'Dailies 10-13', apply dailies_notes.txt: set each Version status (apr, rev, hld), create a Note linked to the Version and its Shot with the note text and addressed artist, for all 14 versions.
- Platforms
web· Any OS (web)- Applications
- Flow Production Tracking
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
ShotGrid API (verifier): 14 versions' sg_status_list equal notes; 14 Note entities linked to version and shot with correct addressings and text. No ShotGrid script key or personal access token is issued to the agent, so statuses and notes must be set in Flow Production Tracking browser sessions.
Proposed partial credit
45% statuses; 35% notes with text; 20% links/addressing
R11.04.01.02-T16 Spot visual regressions between two design versions
Original instruction
In Figma, file 'Marketing site' has version 'Release 12' and the current version. Using version history (or Compare), find the 6 unintended visual changes in the Home and Pricing frames and pin a comment on each changed layer describing the change. Ignore the 2 intended changes listed in release_notes.md.
- Platforms
web· Any OS (web)mac· macOSwin· Windows- Applications
- Figma
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentnav· Pan/zoom/scrollclick· Point & click- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Figma REST: comments on layers in gold changed set (precision/recall over 6); none on the 2 intended changes; text mentions property changed (color/position/size/text). The agent gets no Figma personal access token or OAuth app and plugins are disabled for the test team, so comments must be pinned from the Figma client session.
Proposed partial credit
55% recall; 25% precision; 20% change described
R11.04.01.02-T17 Post the director notes via the Frame.io API (control pair of T01)
Original instruction
Control pair of T01. With a Frame.io developer token, write post_notes.py that converts director_notes.csv timecodes to frame numbers at 23.976 fps and posts the 8 comments to promo_v2.mp4.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- Frame.io API
- Interfaces
api· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
code· Code & commands- Work product
WP10· Annotations- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Same frame and text checks as T01; API audit shows the script's token.
Proposed partial credit
60% frame-exact; 40% texts
R11.04.01.02-T18 QC burned-in subtitles against the script
Original instruction
In Frame.io asset spot_es_subs.mp4 (2 min, burned-in Spanish subtitles), compare each subtitle with script_es.docx. Comment at the first frame of every subtitle that has a wording error, missing line or timing more than 0.5 s off the dialogue. There are 7 issues.
- Platforms
web· Any OS (web)- Applications
- Frame.io · Word
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmentmulti· Cross-app- Work product
WP10· Annotations- Conditions
multiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
API comments vs gold: within ±6 frames of the subtitle's first frame and issue type keyword (wording/missing/timing); precision and recall. No Frame.io developer token or OAuth app is issued to the agent (the verifier API key is grader-only), so comments must come from the web player session.
Proposed partial credit
60% recall; 25% precision; 15% issue type
R11.04.01.02-T19 Consolidate four reviewers' feedback into one change list
Original instruction
In Filestage, brochure_v2.pdf has comments from 4 reviewers, some duplicating or contradicting each other. Produce change_list.xlsx (page, location, change, reviewers) merging duplicates, and reply in Filestage to each contradiction asking the brand lead to decide (@mention).
- Platforms
web· Any OS (web)mac· macOSwin· Windows- Applications
- Filestage · Excel
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentmulti· Cross-apptype· Text & forms- Work product
WP02· Records/table- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
change_list.xlsx rows match gold merged set (18 rows; duplicates merged with all reviewer names); Filestage API: replies with @brand-lead on the 3 contradiction threads. No Filestage API key is issued to the agent (the verifier key is grader-only), so review actions must come from Filestage browser sessions.
Proposed partial credit
45% merged rows; 20% reviewer attribution; 35% contradiction replies
R11.04.01.02-T20 Run a full campaign review round across video and statics
Original instruction
Run review round 2 for campaign 'Glow': in Frame.io review 3 videos and in Figma 5 static banners against brief_glow.pdf (mandatories: logo safe zone, legal line, CTA text, end card ≥2 s). Leave a note on every violation at the exact frame or layer, set each asset's status, and post a summary comment in the Frame.io project.
- Platforms
web· Any OS (web)- Applications
- Frame.io · Figma
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmentmulti· Cross-appnav· Pan/zoom/scroll- Work product
WP10· Annotations- Conditions
multiapp· Multi-applong· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
APIs: violation notes vs gold (12 across 8 assets) with frame ±6 / correct layer; statuses (Approved/Needs Review) per gold; summary lists counts per asset. No Frame.io or Figma API tokens are issued to the agent and Figma plugins are disabled, so notes and statuses must come from browser sessions.
Proposed partial credit
50% violation recall; 15% precision; 20% statuses; 15% summary
Mapped research references
No benchmark reference is assigned to this class in the current mapping.