Screenshots, recordings & clipboard
Capture exact regions or windows, record the screen, pick colors and move rich content through the clipboard.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Screenshot.app · Snipping Tool · OBS
- Class GUI-need label
G3· GUI-only
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R01.01.02.02-T01 Window-only screenshot of Calculator without shadow
Original instruction
Take a screenshot of only the Calculator window (no desktop, no window shadow) and save it to the Desktop as calc.png. Do not crop afterwards in another app.
- Platforms
mac· macOS- Applications
- Screenshot.app · Calculator
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
key· Shortcutsclick· Point & click- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
calc.png dims equal Calculator window bounds × backing scale (±2 px), no transparent shadow margin (alpha=255 at edges), SSIM ≥0.95 vs grader capture. Terminal withheld; eslogger detects screencapture or Python capture libs -> 0.
Proposed partial credit
40% file saved with correct name/location; 30% window-only dims; 30% no shadow
R01.01.02.02-T02 Snip an Excel chart region to PNG
Original instruction
fixtures\q3_sales.xlsx is open with a column chart on sheet 'Summary'. Use Snipping Tool rectangle mode to capture just the chart (chart border included, no cells outside) and save it as C:\out\chart.png.
- Platforms
win· Windows- Applications
- Snipping Tool · Excel
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
drag· Drag & dropgeom· Precise geometry- Work product
WP09· Files & folders- Conditions
licensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
chart.png bbox vs chart object screen rect (UIA): IoU ≥0.92; SSIM ≥0.9 vs reference crop; file written by SnippingTool.exe (Sysmon EID11 image path). PowerShell/Python blocked by AppLocker; ImageGrab or PowerShell capture detected -> 0.
Proposed partial credit
30% file saved; 50% crop IoU ≥0.92; 20% SSIM
R01.01.02.02-T03 Change screenshot destination and capture options
Original instruction
Using the Screenshot toolbar Options, save screenshots to ~/Pictures/Shots (create it), turn off 'Show Floating Thumbnail', turn on 'Show Mouse Pointer' and set a 5-second timer default. Then take one full-screen shot.
- Platforms
mac· macOS- Applications
- Screenshot.app
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsclick· Point & click- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
com.apple.screencapture location=~/Pictures/Shots, show-thumbnail=0, showsCursor=1, delay 5 (timer pref); one new PNG in Shots with screen dims. Terminal withheld; eslogger detects defaults write com.apple.screencapture -> 0.
Proposed partial credit
25% location; 20% thumbnail off; 35% pointer on and 5 s timer; 20% capture landed in Shots
R01.01.02.02-T04 Measure a logo color and record it in Notes
Original instruction
Open fixtures/slide.key in Keynote, use Digital Color Meter (Display native values) to measure the solid orange of the logo on slide 1, and create a Note titled 'Logo color' with the decimal RGB as 'R,G,B'.
- Platforms
mac· macOS- Applications
- Digital Color Meter · Keynote · Notes
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
hover· Hover & contextvis· Visual judgmentmulti· Cross-app- Work product
WP12· Answer/decision- Conditions
multiapp· Multi-applicensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
Notes DB note 'Logo color' body parses to R,G,B each within ±2 of the fixture's rendered value (display-native). Values typed from the file's XML/hex (sRGB not display) mismatch by design. Terminal withheld; eslogger detects screencapture/Python pixel sampling -> 0.
Proposed partial credit
30% note created with title; 70% RGB within ±2 (per-channel credit)
R01.01.02.02-T05 Region screen recording of an animation at 1280×720
Original instruction
Open fixtures/anim.html in Safari. Record a 10-second video of only the 1280×720 animation canvas region with mouse clicks shown, saving to ~/Movies/anim_rec.mov via the Screenshot toolbar. Click the canvas once during recording.
- Platforms
mac· macOS- Applications
- Screenshot.app · Safari
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometryav· Audio/video over time- Work product
WP05· Time-based media- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
ffprobe: duration 9–12 s, frame size 1280×720 ±8 (logical) or ×2; sampled frames SSIM ≥0.85 vs reference canvas frames; click indicator detected in ≥1 frame. Terminal withheld; eslogger detects screencapture -v/ffmpeg avfoundation -> 0.
Proposed partial credit
20% file exists; 30% duration; 30% region crop; 20% click shown
R01.01.02.02-T06 Snipping Tool video with system audio
Original instruction
Use Snipping Tool's video mode to record the region of the open media player window while it plays fixtures\tone_demo.mp4 for 8–12 s, with system audio on and mic off. Save as C:\out\demo.mp4.
- Platforms
win· Windows- Applications
- Snipping Tool · Media Player
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropgeom· Precise geometryav· Audio/video over time- Work product
WP05· Time-based media- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
ffprobe: one video stream with frame size within ±10 px of player window rect, duration 8–12 s, AAC audio stream whose spectrum has a 1 kHz peak and no mic noise floor >-50 dBFS. No ffmpeg on the VM and PowerShell AppLocker-blocked; Sysmon detects ffmpeg/gdigrab or a writer other than Snipping Tool -> 0.
Proposed partial credit
20% file saved; 30% region dims; 20% duration; 30% system audio present, mic off
R01.01.02.02-T07 OBS scene: cropped display plus picture-in-picture camera
Original instruction
In OBS create scene 'Tutorial' with a Display Capture cropped to the top-left quadrant and scaled to fill the 1920×1080 canvas, plus the virtual test-pattern camera at 25% size anchored bottom-right with a 20 px margin. Record 15 s to ~/Videos/tutorial.mkv.
- Platforms
mac· macOSwin· Windows- Applications
- OBS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
drag· Drag & dropgeom· Precise geometryvis· Visual judgment- Work product
WP05· Time-based media- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Scene collection JSON: crop values = half display dims, transform scale fills canvas; camera bounds 480×270 at (1420,790) ±6. Recording frames: test pattern template found bottom-right. obs-websocket disabled; collection edits outside OBS detected by mtime/OBS log.
Proposed partial credit
30% display crop/scale; 40% camera size/position; 30% recording with correct composition
R01.01.02.02-T08 OBS recording output for editing-friendly masters
Original instruction
Configure OBS Output (Advanced): recording format MKV, x264 CRF 18 preset 'veryfast', 1920×1080 at 30 fps, audio track 1 = Desktop Audio, track 2 = Mic/Aux, and turn on 'Automatically remux to MP4'. Record 10 s.
- Platforms
mac· macOSwin· Windows- Applications
- OBS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsnav· Pan/zoom/scroll- Work product
WP05· Time-based media- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
basic.ini RecFormat2=mkv, RecTracks=3, video base/output 1920x1080 FPS 30; recordEncoder.json crf=18 preset veryfast; per-source mixer tracks; ffprobe remuxed MP4: h264 1080p30 + 2 audio streams. obs-websocket disabled; profile/ini edits outside OBS detected by mtime/OBS log.
Proposed partial credit
30% format/encoder; 20% video settings; 30% audio track routing; 20% remuxed file correct
R01.01.02.02-T09 Paste a formatted Word table into an email draft
Original instruction
Copy the 4×3 table (bold header row, shaded column A) from fixtures\rates.docx and paste it into a new Outlook email draft to qa@example.test with subject 'Rates', keeping source formatting. Save the draft; do not send.
- Platforms
win· Windowsmac· macOS- Applications
- Word · Outlook
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
multi· Cross-appclick· Point & clickkey· Shortcuts- Work product
WP08· App/system state- Conditions
multiapp· Multi-applicensed· Licensed pro apprisk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Drafts folder (test mailbox via EWS/Graph read-only) has one draft: To and subject exact; body HTML contains a <table> with 4 rows × 3 cols, header cells bold, column A background color matches docx shading; Sent Items empty. Agent VM has no Graph/EWS credentials, so the draft must be saved by the Outlook client; scripted Word/Outlook automation (Sysmon/eslogger) -> 0.
Proposed partial credit
20% draft with recipient/subject; 40% table structure; 30% formatting kept; 10% not sent
R01.01.02.02-T10 Enable clipboard history and pin three snippets
Original instruction
Turn on clipboard history, copy the three lines in fixtures\snippets.txt one at a time, pin all three in the Win+V panel, then paste the second line into a new Notepad file saved as C:\out\paste.txt.
- Platforms
win· Windows- Applications
- Windows Settings · Clipboard history · Notepad
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsclick· Point & click- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
HKCU\Software\Microsoft\Clipboard EnableClipboardHistory=1; %LOCALAPPDATA%\Microsoft\Windows\Clipboard\Pinned contains exactly the 3 texts; paste.txt == line 2. No API pins history items; PowerShell blocked by AppLocker and Sysmon detects scripted clipboard writes -> 0.
Proposed partial credit
20% history enabled; 50% three pinned; 30% correct paste
R01.01.02.02-T11 Full-page screenshot of a long web page
Original instruction
Capture the entire scroll height of fixtures/longpage.html (about 9,000 px tall) as one PNG named page_full.png using Firefox's built-in Screenshot tool, on both test machines.
- Platforms
mac· macOSwin· Windowsweb· Any OS (web)- Applications
- Firefox
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicknav· Pan/zoom/scroll- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
page_full.png width equals page CSS width × DPR, height within ±2% of document.scrollHeight × DPR; bottom footer text found by OCR; SSIM vs reference render ≥0.9. Terminal/PowerShell withheld; headless --screenshot or devtools :screenshot commands detected via eslogger/Sysmon exec audit -> 0.
Proposed partial credit
30% file saved; 40% full height; 30% content matches reference
R01.01.02.02-T12 Annotate a settings screenshot with arrow and blur
Original instruction
Capture the Bluetooth settings window, then in Preview Markup (macOS) or Snipping Tool (Windows) draw a red arrow pointing at the Bluetooth toggle and blur or black out the device name 'Lab-Mac-07'. Save as annotated.png on the Desktop.
- Platforms
mac· macOSwin· Windows- Applications
- Preview · Snipping Tool
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
path· Continuous pathdrag· Drag & dropvis· Visual judgment- Work product
WP03· Visual layout- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Detect red (#FF0000±40) arrow blob; its tip within 40 px of toggle bbox (from grader reference capture); OCR on the name bbox must not read 'Lab-Mac-07'; rest of image SSIM ≥0.9 vs reference. Shells withheld; eslogger/Sysmon detect ImageMagick/PIL exec -> 0.
Proposed partial credit
20% screenshot saved; 40% arrow points to toggle; 40% device name obscured
R01.01.02.02-T13 Capture an open menu-bar dropdown
Original instruction
Open the Wi-Fi menu in the menu bar and capture just that dropdown menu (use a timer or menu capture) to ~/Desktop/wifi_menu.png. The menu must be fully visible, including the network list.
- Platforms
mac· macOS- Applications
- Screenshot.app · Control Center
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G3· GUI-only- Interaction primitives
key· Shortcutsclick· Point & clickvis· Visual judgment- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
OCR of wifi_menu.png contains 'Wi-Fi' and ≥2 seeded SSIDs; image dims within ±10% of menu window bounds; no desktop wallpaper pixels (>90% of image inside menu frame). Terminal withheld; eslogger detects screencapture CLI/Python capture -> 0.
Proposed partial credit
30% file saved; 40% menu visible with SSIDs; 30% cropped to menu
R01.01.02.02-T14 Capture an exact 640×360 region by coordinates
Original instruction
Capture exactly the screen region from (200,150) to (840,510) points on the main display as region.png, using the screenshot crosshair's coordinate readout. Retina output must be 1280×720 px.
- Platforms
mac· macOS- Applications
- Screenshot.app
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
drag· Drag & dropgeom· Precise geometry- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
region.png dims 1280×720 ±4 px; content SSIM ≥0.95 vs grader capture of the same rect from a frozen fixture wallpaper with a calibration grid (origin offset ≤2 pt). Terminal withheld; eslogger detects screencapture -R -> 0.
Proposed partial credit
50% exact dims; 50% correct origin (content match)
R01.01.02.02-T15 GIF bug repro with ShareX
Original instruction
Use ShareX screen recording (GIF) to capture a ≤8 s clip of clicking 'Submit' in fixtures\bug.html until the red error toast appears. Region: the browser content area only. Save as C:\out\bug.gif under 5 MB.
- Platforms
win· Windows- Applications
- ShareX · Edge
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropav· Audio/video over timeclick· Point & click- Work product
WP05· Time-based media- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
GIF: frame-count/duration ≤8 s, size <5 MB; last frames template-match the red toast; first frame lacks it; frame dims within ±10 px of content area. No ffmpeg on the VM and PowerShell AppLocker-blocked; Sysmon detects scripted capture -> 0.
Proposed partial credit
20% GIF saved; 40% shows before→after toast; 20% region; 20% size/duration
R01.01.02.02-T16 Live stream with timed scene switches
Original instruction
Stream from OBS to rtmp://127.0.0.1/live/test (local server) using scene 'Main'. Exactly 20 s after going live switch to scene 'BRB' (solid blue + text), hold 10 s, switch back to 'Main', and stop the stream at 45 s.
- Platforms
mac· macOSwin· Windows- Applications
- OBS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
click· Point & clickrt· Real-time control- Work product
WP05· Time-based media- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Server-side recording: frame classifier finds BRB (blue) from 20±1 s to 30±1 s, Main otherwise; total length 45±2 s. obs-websocket disabled; OBS log scene-switch events must come from UI.
Proposed partial credit
20% stream received; 40% BRB timing within ±1 s; 20% back to Main; 20% total length
R01.01.02.02-T17 Copy text out of a scanned image into a form
Original instruction
fixtures/scan.png is a scanned invoice. Use Live Text (Preview) or Snipping Tool Text actions to copy the invoice number and IBAN, and paste them into the matching fields of the open local form http://localhost:8080/ap-entry; save the form.
- Platforms
mac· macOSwin· Windows- Applications
- Preview · Snipping Tool · Browser
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropmulti· Cross-apptype· Text & forms- Work product
WP02· Records/table- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Form backend DB row: invoice_no and iban exactly equal ground truth (IBAN without spaces normalized); values must arrive via paste events logged in the page session; Terminal/PowerShell withheld and external OCR tools detected by eslogger/Sysmon -> 0.
Proposed partial credit
40% invoice number; 50% IBAN; 10% saved
R01.01.02.02-T18 Move an image and styled text between apps correctly
Original instruction
Copy photo 'Lighthouse' from Photos and paste it into slide 2 of fixtures/deck.key replacing the image placeholder; copy the paragraph from fixtures/article.html in Safari into fixtures/memo.pages with Paste and Match Style. Save both.
- Platforms
mac· macOS- Applications
- Photos · Keynote · Pages · Safari
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
multi· Cross-appkey· Shortcutsclick· Point & click- Work product
WP03· Visual layout- Conditions
multiapp· Multi-applicensed· Licensed pro app- Proposed difficulty
- 3 / 5
Proposed verifier
deck.key slide 2 placeholder replaced (no placeholder tag) by image with perceptual hash ≤6 from source; memo.pages new paragraph text equals source and uses the document Body style (no foreign font). Terminal and Script Editor withheld; eslogger detects osascript or writes to deck.key/memo.pages by non-iWork processes -> 0.
Proposed partial credit
50% image in placeholder; 40% text with matched style; 10% both files saved
R01.01.02.02-T19 Five-step tutorial screenshot set
Original instruction
Produce window-only screenshots step1.png–step5.png for 'Turn on Night light schedule' in Windows Settings: Settings home, System, Display, Night light page, schedule set to 21:00–07:00. Each image must show only the Settings window. Restore the original Night light state afterwards.
- Platforms
win· Windows- Applications
- Snipping Tool · Windows Settings
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentkey· Shortcuts- Work product
WP09· Files & folders- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Each PNG: dims ≈ Settings window rect (±10 px), OCR page header matches expected step; step5 OCR shows 21:00 and 07:00; final Night light settings equal seed. PowerShell/Python blocked by AppLocker; Sysmon detects scripted capture -> 0.
Proposed partial credit
50% five correct steps (equal share per step); 20% window-only crops; 30% original state restored
R01.01.02.02-T20 Save an instant replay containing a cued event
Original instruction
Enable OBS Replay Buffer (20 s) with hotkey F9 for 'Save Replay'. Play fixtures/cue_clip.mp4 full-screen in a Media Source; within 5 s after the red flash appears, press F9. The saved replay must contain the flash.
- Platforms
mac· macOSwin· Windows- Applications
- OBS
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
key· Shortcutsrt· Real-time controlav· Audio/video over time- Work product
WP05· Time-based media- Conditions
interrupt· Interruptions- Proposed difficulty
- 5 / 5
Proposed verifier
Replay file exists; duration ≤21 s; red-flash frame located within its last 5 s by color classifier; profile ini ReplayBuffer enabled 20 s; hotkey binding F9 in scene collection. obs-websocket disabled; profile/ini edits outside OBS detected by mtime/OBS log.
Proposed partial credit
30% buffer + hotkey configured; 30% replay saved; 40% flash within last 5 s
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.