Image, video & map evidence
Extract facts visible only in images, videos, street views or map canvases.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- YouTube · Google Maps · image sites
- Class GUI-need label
G3· GUI-only
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R04.01.02.03-T01 Find a classifieds listing by photo content
Original instruction
On the classifieds replica snapshot, among listings posted in the last 7 days, find the one whose photo shows a red bicycle with a child seat attached. Write the listing ID to answer.txt. The detail is only visible in the image, not the text.
- Platforms
web· Any OS (web)- Applications
- image sites
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
answer.txt listing ID equals the one whose photo matches (lab-fixed). The attribute is not in listing text/alt-text, so visual inspection is required; snapshot deterministic. The agent VM has no terminal, OCR or vision tools, and the snapshot serves images only inside the page with no EXIF, alt text or API carrying the answer.
Proposed partial credit
40% listings scanned; 30% correct photo identified; 30% exact listing ID
R04.01.02.03-T02 Timestamp and label of a chart in a video
Original instruction
In the embedded lecture video snapshot in task.txt, find the timestamp when the first bar chart appears on screen and read the label of its tallest bar. Write mm:ss and the label to answer.json.
- Platforms
web· Any OS (web)- Applications
- YouTube
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentav· Audio/video over time- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
answer.json timestamp within +/-3 s of the lab-fixed appearance and the tallest-bar label matches exactly. The chart is on-screen only (not in the transcript), so visual/temporal inspection is required. No YouTube Data API key is issued, captions/transcripts are disabled, and the video streams only in the embedded player (no file download; no terminal or ffmpeg on the agent VM).
Proposed partial credit
40% chart appearance found; 30% timestamp within tolerance; 30% tallest-bar label
R04.01.02.03-T03 Painting location in a 3D tour
Original instruction
In the museum 3D virtual-tour snapshot, navigate to Room 2 and report the title of the painting hanging immediately to the left of the room's entrance (read it from the placard in the view). Write the title to answer.txt.
- Platforms
web· Any OS (web)- Applications
- image sites
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
answer.txt title matches the placard text for that painting (lab-fixed). Only visible by navigating the tour; snapshot deterministic. The tour snapshot exposes no API or panorama download and the agent VM has no terminal or vision tools, so the view is reachable only by navigating the tour in the browser.
Proposed partial credit
40% reached Room 2; 30% correct painting located; 30% exact title
R04.01.02.03-T04 Product variant with a white sole
Original instruction
On the product gallery snapshot in task.txt, inspect the color-variant images and determine which variant has a white sole. Write that variant's SKU to answer.txt.
- Platforms
web· Any OS (web)- Applications
- image sites
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
answer.txt SKU equals the variant whose gallery image shows a white sole (lab-fixed). The sole color is only in the image; visual inspection required. The agent VM has no terminal, OCR or vision tools, and the snapshot serves images only inside the page with no EXIF, alt text or API carrying the answer.
Proposed partial credit
50% variant images inspected; 50% exact SKU
R04.01.02.03-T05 Identify a building at a corner on a map
Original instruction
On the map snapshot in task.txt, locate the corner of the two named streets and report the name of the business occupying the corner building (read the label/marker). Write the name to answer.txt.
- Platforms
web· Any OS (web)- Applications
- Google Maps
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
answer.txt business name matches the map marker at that corner (lab-fixed snapshot). Requires locating it on the map canvas; deterministic. No Maps API key is issued and the snapshot's Places/Geocoding lookups are disabled, so the business label is visible only on the map canvas; the agent VM has no terminal or vision tools.
Proposed partial credit
40% corner located; 30% building identified; 30% exact name
R04.01.02.03-T06 Read a shop name from street view
Original instruction
On the street-view snapshot in task.txt, read the name on the storefront sign at the indicated position (facing the given direction) and write it to answer.txt exactly as shown.
- Platforms
web· Any OS (web)- Applications
- Google Maps
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentdrag· Drag & drop- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
answer.txt matches the storefront sign text in the snapshot (lab-fixed). Only readable from the panorama; visual inspection required; deterministic. No Maps API key is issued and Places lookups are disabled in the snapshot, so the sign is readable only in the street-view panorama; the agent VM has no terminal or OCR tools.
Proposed partial credit
40% correct viewpoint/direction; 60% exact sign text
R04.01.02.03-T07 Count event occurrences in a video
Original instruction
In the video snapshot in task.txt, count how many times the specified on-screen event occurs during the first 2 minutes (e.g. the presenter clicks to a new slide, or a car passes). Write the integer count to answer.txt.
- Platforms
web· Any OS (web)- Applications
- YouTube
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentav· Audio/video over time- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
answer.txt integer equals the lab-fixed count of the event in the first 2 minutes. Requires watching the video; the count is not in any transcript; deterministic. No YouTube Data API key is issued, captions/transcripts are disabled, and the video streams only in the embedded player (no file download; no terminal or ffmpeg on the agent VM).
Proposed partial credit
50% most occurrences counted; 50% exact count
R04.01.02.03-T08 Which photo shows a specified object
Original instruction
On the image set snapshot in task.txt (8 photos), identify which single photo contains a specified object (e.g. a lighthouse). Write the photo's filename/ID to answer.txt.
- Platforms
web· Any OS (web)- Applications
- image sites
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
answer.txt equals the photo containing the object (lab-fixed). Visual inspection required; deterministic. The agent VM has no terminal, OCR or vision tools, and the snapshot serves images only inside the page with no EXIF, alt text or API carrying the answer.
Proposed partial credit
20% answer.txt holds a valid photo ID from the set; 80% exact correct photo
R04.01.02.03-T09 Landmark passed on a map route
Original instruction
On the map snapshot in task.txt, with the driving route between the two points shown, name the notable landmark the route passes directly by (visible on the route). Write it to answer.txt.
- Platforms
web· Any OS (web)- Applications
- Google Maps
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
answer.txt landmark matches the one on the route in the snapshot (lab-fixed). Requires tracing the route on the map; deterministic. No Maps API key is issued and the snapshot's Places/Geocoding lookups are disabled, so the landmark is visible only on the map canvas; the agent VM has no terminal or vision tools.
Proposed partial credit
40% route displayed; 30% landmark located; 30% exact name
R04.01.02.03-T10 Color of an object at a timestamp
Original instruction
In the video snapshot in task.txt, seek to the given timestamp and report the color of the specified object on screen (e.g. the car in the foreground). Write the color to answer.txt.
- Platforms
web· Any OS (web)- Applications
- YouTube
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentav· Audio/video over time- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
answer.txt color matches the object's color at that frame (lab-fixed). Requires seeking and viewing; deterministic. No YouTube Data API key is issued, captions/transcripts are disabled, and the video streams only in the embedded player (no file download; no terminal or ffmpeg on the agent VM).
Proposed partial credit
50% correct frame located; 50% correct color
R04.01.02.03-T11 Count objects in a 3D tour room
Original instruction
In the 3D-tour snapshot in task.txt, enter the named room and count the number of a specified item visible (e.g. dining chairs). Write the integer to answer.txt.
- Platforms
web· Any OS (web)- Applications
- image sites
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentdrag· Drag & drop- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
answer.txt equals the lab-fixed count visible in that room's panorama. Requires navigating/looking around the tour; deterministic. The tour snapshot exposes no API or panorama download and the agent VM has no terminal or vision tools, so the view is reachable only by navigating the tour in the browser.
Proposed partial credit
40% room reached; 30% items scanned; 30% exact count
R04.01.02.03-T12 Read a value off a chart image
Original instruction
On the image snapshot in task.txt (a bar chart saved as a picture, no data table), read the value of the specified bar by eye against the axis. Write the value to answer.txt.
- Platforms
web· Any OS (web)- Applications
- image sites
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
answer.txt value is within +/-5% of the lab-fixed bar value. The chart is a static image with no underlying data, so visual estimation against the axis is required. The agent VM has no terminal, OCR or vision tools, and the snapshot serves images only inside the page with no EXIF, alt text or API carrying the answer.
Proposed partial credit
50% correct bar identified; 50% value within tolerance
R04.01.02.03-T13 Transcribe whiteboard text in a video
Original instruction
In the video snapshot in task.txt, find the first moment the presenter writes on the whiteboard and transcribe exactly what is written at that point. Write mm:ss and the text to answer.json.
- Platforms
web· Any OS (web)- Applications
- YouTube
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentav· Audio/video over time- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
answer.json timestamp within +/-5 s and the transcribed text matches the whiteboard (lab-fixed). The text is on-screen only, not in the transcript; visual inspection required. No YouTube Data API key is issued, captions/transcripts are disabled, and the video streams only in the embedded player (no file download; no terminal or ffmpeg on the agent VM).
Proposed partial credit
40% moment found; 30% timestamp; 30% transcription
R04.01.02.03-T14 Count features on a satellite/map view
Original instruction
On the map snapshot in task.txt, switch to satellite view over the named park and count how many of a specified feature are present (e.g. tennis courts). Write the integer to answer.txt.
- Platforms
web· Any OS (web)- Applications
- Google Maps
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentdrag· Drag & drop- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
answer.txt equals the lab-fixed count of the feature visible in the satellite snapshot. Requires visual inspection of the imagery; deterministic. No Maps API key is issued and the snapshot's Places/Geocoding lookups are disabled, so the feature is visible only on the map canvas; the agent VM has no terminal or vision tools.
Proposed partial credit
40% correct area in satellite view; 30% features scanned; 30% exact count
R04.01.02.03-T15 Pick the night-time photo
Original instruction
On the image set snapshot in task.txt (6 photos of the same place), identify which photo was taken at night. Write its filename/ID to answer.txt.
- Platforms
web· Any OS (web)- Applications
- image sites
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
answer.txt equals the night-time photo (lab-fixed). Visual inspection required; deterministic. The agent VM has no terminal, OCR or vision tools, and the snapshot serves images only inside the page with no EXIF, alt text or API carrying the answer.
Proposed partial credit
20% answer.txt holds a valid photo ID from the set; 80% exact correct photo
R04.01.02.03-T16 Slide title when a keyword is spoken
Original instruction
In the video snapshot in task.txt, find the moment the presenter says a specified keyword and report the title of the slide shown on screen at that moment. Write mm:ss and the slide title to answer.json.
- Platforms
web· Any OS (web)- Applications
- YouTube
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentav· Audio/video over time- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
answer.json timestamp within +/-5 s of the spoken keyword and the slide title matches what is on screen then (lab-fixed). Requires combining audio timing with the visible slide. No YouTube Data API key is issued, captions/transcripts are disabled, and the video streams only in the embedded player (no file download; no terminal or ffmpeg on the agent VM).
Proposed partial credit
40% keyword moment found; 30% timestamp; 30% slide title
R04.01.02.03-T17 Measure a distance with the map tool
Original instruction
On the map snapshot in task.txt, use the measure tool to measure the straight-line distance between the two indicated points. Write the distance in meters to answer.txt.
- Platforms
web· Any OS (web)- Applications
- Google Maps
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollclick· Point & clickvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
answer.txt distance is within +/-5% of the lab-fixed measurement. Requires using the map's measure tool on the canvas; deterministic. No Maps API key is issued and the two points appear only as markers with no exposed coordinates, so the distance must come from the measure tool; the agent VM has no terminal.
Proposed partial credit
40% both points placed; 30% measure tool used; 30% distance within tolerance
R04.01.02.03-T18 Read a gauge/handwritten value in an image
Original instruction
On the image snapshot in task.txt, read the value shown on a dial gauge (or a handwritten number on a label). Write the value to answer.txt.
- Platforms
web· Any OS (web)- Applications
- image sites
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
answer.txt value matches the gauge/handwritten reading within the lab tolerance. The value is only in the image; visual reading required; deterministic. The agent VM has no terminal, OCR or vision tools, and the snapshot serves images only inside the page with no EXIF, alt text or API carrying the answer.
Proposed partial credit
50% region located; 50% value correct
R04.01.02.03-T19 Find when two video conditions co-occur
Original instruction
In the video snapshot in task.txt, find the first timestamp where two specified on-screen conditions hold simultaneously (e.g. the chart shows three bars AND the caption banner is visible). Write mm:ss and a one-line description to answer.json.
- Platforms
web· Any OS (web)- Applications
- YouTube
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentav· Audio/video over time- Work product
WP12· Answer/decision- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
answer.json timestamp within +/-3 s of the lab-fixed first co-occurrence and the description matches both conditions. Requires scrubbing and visual judgement across the clip; deterministic. No YouTube Data API key is issued, captions/transcripts are disabled, and the video streams only in the embedded player (no file download; no terminal or ffmpeg on the agent VM).
Proposed partial credit
30% first condition located; 30% second condition located; 40% correct co-occurrence timestamp
R04.01.02.03-T20 Combine map and image evidence
Original instruction
Using the map snapshot and the photo snapshot in task.txt, identify the place shown in the photo by matching its visible features to the map (the photo has no caption), then report the place name and a specified visible attribute (e.g. number of floors of the building). Write both to answer.json.
- Platforms
web· Any OS (web)- Applications
- Google Maps · image sites
- Interfaces
brow· Browser- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollvis· Visual judgmentclick· Point & click- Work product
WP12· Answer/decision- Conditions
long· Long-horizonmultiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
answer.json place name matches the lab-fixed location and the visible attribute is correct. Requires cross-referencing the uncaptioned photo against the map; deterministic. No Maps API key or reverse-image/geolocation service is reachable, the photo carries no EXIF/GPS data, and the agent VM has no terminal or vision tools.
Proposed partial credit
40% place identified from photo+map; 30% exact name; 30% visible attribute
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.