Music, podcast & video libraries
Build playlists, manage libraries, set subtitles and playback.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Music · Spotify · VLC · podcasts
- Class GUI-need label
G2· GUI-essential
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R12.07.01.01-T01 Create an ordered playlist from three named songs in Music
Original instruction
In the Music app on macOS (local test library, no Apple Music account), create a playlist 'Run' containing 'Electric Feel', 'Midnight City' and 'Dog Days Are Over' in that order. Two library songs are named 'Midnight City' (original and remix); use the original album version.
- Platforms
mac· macOS- Applications
- Music
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickdrag· Drag & drop- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 1 / 5
Proposed verifier
Music library XML export: playlist 'Run' with exactly 3 track persistent IDs in order; Midnight City id = original (album 'Hurry Up, We're Dreaming'). AppleScript access blocked via TCC automation denial.
Proposed partial credit
40% correct tracks; 30% order; 30% correct version.
R12.07.01.01-T02 Set VLC playback speed and seek to an exact timestamp
Original instruction
In VLC (Windows), open fixtures/talk.mp4, set playback speed to 1.5x, seek to 02:30, enable the English subtitle track, and pause. Leave VLC open.
- Platforms
win· Windowsmac· macOS- Applications
- VLC
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickdrag· Drag & dropav· Audio/video over time- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
VLC HTTP status (enabled by harness, password unknown to agent): rate=1.5, time=150 ±3 s, state=paused, active SPU track = English. The agent has no terminal, so VLC command-line flags and the rc/Lua interfaces are not reachable; state must be set in the running VLC window.
Proposed partial credit
30% rate; 40% position; 20% subtitle; 10% paused.
R12.07.01.01-T03 Remove all songs by one artist from a large playlist
Original instruction
In the Navidrome web player, remove every track by 'The Weekend Band' (not 'The Weeknd') from the playlist 'Favorites' (420 tracks), keeping everything else and the original order.
- Platforms
web· Any OS (web)- Applications
- Navidrome
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollvis· Visual judgment- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Navidrome DB: playlist tracks = original minus that artist's 14 tracks, order of the rest unchanged; The Weeknd tracks still present. The agent has no terminal and browser devtools are disabled, so the Subsonic/native API is not reachable outside the player; the server log must attribute edits to the web client (c=NavidromeUI).
Proposed partial credit
60% all target tracks removed; 25% no wrong removals; 15% order preserved.
R12.07.01.01-T04 Build a playlist from tagged tracks over four minutes long
Original instruction
In the Jellyfin web client music library, create playlist 'Calm' with all tracks tagged genre 'Ambient' longer than 4:00, sorted by duration descending.
- Platforms
web· Any OS (web)- Applications
- Jellyfin
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickdrag· Drag & dropnav· Pan/zoom/scroll- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Jellyfin DB: playlist items equal the filtered set (genre Ambient, runtime > 240 s), ordered by duration desc (reorder by drag in playlist view). API key creation is disabled for the user.
Proposed partial credit
50% correct set; 40% order; 10% name.
R12.07.01.01-T05 Subscribe to podcasts, set per-show playback speed and auto-download
Original instruction
In AntennaPod on the Android emulator, subscribe to the 3 feeds in fixtures/feeds.opml (import OPML), set playback speed 1.3x for 'Tech Weekly' only, enable auto-download for 'Daily News' keeping only the latest 3 episodes, and mark all 'Tech Weekly' episodes before 2026-09-01 as played.
- Platforms
and· Android- Applications
- AntennaPod
- Interfaces
mob· Mobile touch- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
touch· Touch gesturesclick· Point & click- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
AntennaPod DB: 3 feeds; feed preferences: Tech Weekly speed 1.3, Daily News auto-download on with episode cleanup 3; played flags on episodes before 09-01 for Tech Weekly only. The agent has no adb shell or push, so the AntennaPod DB is not reachable; settings must be made in the app and the OPML imported through its file picker (harness input log).
Proposed partial credit
20% import; 25% speed; 25% auto-download cleanup; 30% played flags.
R12.07.01.01-T06 Find the timestamp where a slide title first appears in a lecture video
Original instruction
In VLC on macOS, open fixtures/lecture.mp4 (58 min) and find the first time the slide titled 'Backpropagation' appears. Create a VLC bookmark at that time named 'Backprop' and write the timestamp (mm:ss) into answer.txt.
- Platforms
mac· macOSwin· Windows- Applications
- VLC
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
drag· Drag & dropav· Audio/video over timevis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
answer.txt within ±5 s of ground truth 12:41; VLC bookmark (saved in playlist .xspf on quit or via status) at that time ±5 s with name. The agent has no Terminal and no ffmpeg or OCR tools, so the video cannot be scanned by script; the bookmark must be created in VLC's Bookmarks window.
Proposed partial credit
70% timestamp; 30% bookmark.
R12.07.01.01-T07 Fix album artwork and metadata for a ripped album in Music
Original instruction
In the Music app on macOS, the album 'Blue Train' was imported with track numbers missing, disc split into two albums, and no artwork. Merge it into one album (set album artist), set track numbers 1-5 per fixtures/tracklist.txt, and add fixtures/cover.jpg as artwork for all tracks.
- Platforms
mac· macOS- Applications
- Music
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & formsdrag· Drag & drop- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Track files' ID3/MP4 tags: album, album artist, track numbers exact; embedded artwork hash equals cover.jpg (re-encoded allowed: perceptual hash distance <= 4); library shows 1 album. The agent has no Terminal or Script Editor (osascript blocked), so Music's AppleScript dictionary and tag tools are not reachable; tags must be edited in Music's Info window.
Proposed partial credit
30% merged album; 40% track numbers; 30% artwork.
R12.07.01.01-T08 Sort an artist's songs by duration and save as a playlist in a mobile music app
Original instruction
In Retro Music on the Android emulator, sort Pink Floyd's songs by duration descending and save the top 10 as a playlist 'Long Floyd'.
- Platforms
and· Android- Applications
- Retro Music
- Interfaces
mob· Mobile touch- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
touch· Touch gesturesclick· Point & click- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Android MediaStore playlist 'Long Floyd' with 10 tracks = top-10 longest Pink Floyd tracks, order descending. The agent has no adb shell, so MediaStore cannot be written by content commands; the playlist must be made in Retro Music, as the harness input log confirms.
Proposed partial credit
60% correct set; 30% order; 10% name.
R12.07.01.01-T09 Clean up Watch Later by removing mostly watched videos
Original instruction
In the PeerTube web client (test account), remove from the 'Watch later' playlist every video already watched more than 90% (watch progress bar on thumbnails), and move the remaining videos so the shortest is first.
- Platforms
web· Any OS (web)- Applications
- PeerTube
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
vis· Visual judgmentdrag· Drag & dropclick· Point & click- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Playlist elements after: seeded 4 removed (>90% watched), remaining ordered by duration ascending (drag reorder in UI). The agent has no terminal and browser devtools are disabled, so PeerTube REST calls cannot be made outside the web client; edits must come from its session.
Proposed partial credit
50% correct removals; 40% order; 10% no extra removals.
R12.07.01.01-T10 Merge a roommate's texted songs into a shared road-trip playlist
Original instruction
On the Android emulator, the 'Roommate' SMS thread (Messages) contains song suggestions in free text over the past week (some duplicates and one retracted). In the Spotify-style app clone, update playlist 'Road Trip' to include all suggested songs not already in it, excluding the retracted one.
- Platforms
and· Android- Applications
- Messages · Spotify-style clone
- Interfaces
mob· Mobile touch- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G3· GUI-only- Interaction primitives
multi· Cross-apptouch· Touch gesturestype· Text & forms- Work product
WP09· Files & folders- Conditions
multiapp· Multi-apppersonal· Personal context- Proposed difficulty
- 3 / 5
Proposed verifier
Playlist tracks = original ∪ suggested − retracted, no duplicates; track ids exact (versions disambiguated by artist named in text). The agent gets screen input only (no adb shell, no backend API token), so the playlist must be edited in the app; the harness input log confirms UI taps.
Proposed partial credit
60% suggestions added; 20% retraction respected; 20% no duplicates.
R12.07.01.01-T11 Extract a frame at a timestamp and use it as a slide background
Original instruction
On Ubuntu, take a snapshot of fixtures/promo.mp4 at 00:08 using VLC's snapshot feature, then set it as the background image of slide 2 in deck.odp in LibreOffice Impress and save.
- Platforms
lin· Linuxwin· Windows- Applications
- VLC · LibreOffice Impress
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
av· Audio/video over timeclick· Point & clickmulti· Cross-app- Work product
WP03· Visual layout- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
deck.odp slide 2 background fill is a bitmap with SSIM >= 0.95 to the ground-truth frame at 8.0 s ±0.1; other slides unchanged. The agent has no terminal, ffmpeg or Python, and LibreOffice macros are disabled, so the frame must come from VLC's snapshot and the background must be set in Impress.
Proposed partial credit
50% correct frame; 40% set as slide 2 background; 10% others unchanged.
R12.07.01.01-T12 Configure a smart playlist with nested rules in Music
Original instruction
In the Music app on macOS, create a smart playlist 'Fresh Rock' matching: genre contains 'Rock' AND (date added in the last 30 days OR play count is 0), excluding tracks rated 1 star, limited to 50 items selected by most recently added, live updating on.
- Platforms
mac· macOS- Applications
- Music
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Library XML smart criteria blob decoded: nested rule structure equals spec; limit 50 by date added; live updating; resulting membership equals reference evaluation. The agent has no Terminal or osascript and the library XML is read-only to it, so the smart playlist must be built in Music's rule editor.
Proposed partial credit
50% rule logic incl. nesting; 30% limit and selection; 20% resulting membership.
R12.07.01.01-T13 Set subtitles delay and style to sync a foreign film
Original instruction
In VLC on Windows, open fixtures/film.mkv with external subtitle film.en.srt which is out of sync. Determine the offset by matching the first spoken line to its subtitle, set the subtitle delay so they align (within 100 ms), and set subtitle font size to 'Large' with a background box. Save the delay into a corrected film.en.synced.srt via the subtitle track sync tool.
- Platforms
win· Windowsmac· macOS- Applications
- VLC
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
av· Audio/video over timekey· Shortcutsclick· Point & click- Work product
WP05· Time-based media- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
film.en.synced.srt timestamps shifted by ground-truth offset ±100 ms; VLC prefs (vlcrc): freetype size large, background opacity > 0. The agent has no terminal or Python, and file writes by other processes are flagged by the file audit log, so the synced SRT must be saved from VLC.
Proposed partial credit
60% sync accuracy; 25% saved SRT; 15% style prefs.
R12.07.01.01-T14 Organize a podcast queue by priority and set a sleep timer
Original instruction
In Apple Podcasts on the iOS Simulator (local test feeds), build the Up Next queue in this order: newest 'Daily News', then the two oldest unplayed 'History Hour' episodes, then 'Tech Weekly' episode 212. Set a sleep timer for 'End of episode'.
- Platforms
ios· iOS/iPadOS- Applications
- Apple Podcasts
- Interfaces
mob· Mobile touch- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G3· GUI-only- Interaction primitives
touch· Touch gesturesdrag· Drag & drop- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Podcasts app DB: queue episode GUIDs in exact order; sleep timer mode=end_of_episode. The agent drives only the Simulator window: it has no shell on the host (simctl and the app container are not reachable), so the queue must be built in Podcasts.
Proposed partial credit
60% queue contents; 25% order; 15% sleep timer.
R12.07.01.01-T15 Deduplicate a music library while keeping the highest-quality copy
Original instruction
In the Music app on macOS, the library has 38 duplicate tracks (same title/artist/duration ±2 s) from two imports: AAC 256 kbps and MP3 128 kbps. Delete the lower-bitrate duplicates from the library (move files to Trash), (play counts need not be merged). Do not delete live/remix versions.
- Platforms
mac· macOS- Applications
- Music
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickvis· Visual judgmentnav· Pan/zoom/scroll- Work product
WP09· Files & folders- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Library and Trash: the 38 MP3 duplicates removed; all AAC copies present; 6 decoy live/remix tracks with same titles retained. The agent has no Terminal or osascript, so Music's AppleScript dictionary is not reachable for scripted deletes; removals must be made in Music.
Proposed partial credit
60% duplicates removed (proportional); 30% decoys kept; 10% high-quality copies intact.
R12.07.01.01-T16 Report a speaker's on-screen name at a timestamp at a given speed with captions
Original instruction
In the PeerTube web client, open the video 'Panel: Open Data 2026', set speed to 1.5x and turn on English captions, then report the name shown in the lower-third caption at 03:00 and the first caption line displayed after 03:00.
- Platforms
web· Any OS (web)- Applications
- PeerTube
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
av· Audio/video over timevis· Visual judgmentclick· Point & click- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Answer name exact ('Dr. Lee'); caption line exact; player settings recorded in local storage (speed 1.5, captions en). The agent has no terminal and browser devtools are disabled, so caption files and local storage cannot be read or set by script; values must be read from the player.
Proposed partial credit
40% name; 40% caption line; 20% player settings.
R12.07.01.01-T17 Create a video library collection with custom posters and order in Jellyfin
Original instruction
In Jellyfin web, create a collection 'Nolan' with the 6 Christopher Nolan films in the library (two have wrong metadata matches you must fix via Identify), set each film's poster to the one in posters/ with the same name, and set the collection's display order to release date.
- Platforms
web· Any OS (web)- Applications
- Jellyfin
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentdrag· Drag & drop- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Jellyfin DB: collection with 6 correct item ids (after re-identify: provider ids match TMDB ids list), primary images hash-match posters, display order PremiereDate. The agent has no Jellyfin API key, no terminal and browser devtools are disabled, so REST calls and NFO edits are unavailable; changes must come from the web client (activity log).
Proposed partial credit
30% collection membership; 30% fixed metadata; 30% posters; 10% order.
R12.07.01.01-T18 Build a DJ-style crossfaded party mix in Music and export the playlist
Original instruction
In the Music app on macOS, create playlist 'Party' of 20 tracks from the library with BPM 118-128 (BPM column must be shown), ordered so adjacent tracks differ by <= 3 BPM, enable crossfade at 6 s in settings, and export the playlist as party.m3u8 to ~/Desktop.
- Platforms
mac· macOS- Applications
- Music
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G3· GUI-only- Interaction primitives
drag· Drag & dropclick· Point & clickvis· Visual judgment- Work product
WP09· Files & folders- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
party.m3u8: 20 tracks all in BPM range (from tags); adjacent diffs <= 3; Music prefs crossfade on, 6 s. The agent has no Terminal or osascript, so scripted playlist building is not reachable; the m3u8 must be written by Music's Export Playlist command.
Proposed partial credit
30% BPM range; 40% adjacency constraint; 15% crossfade; 15% export.
R12.07.01.01-T19 Migrate playlists from one music server to another preserving order
Original instruction
Move the 5 playlists from the Navidrome instance to the Jellyfin music library. Track files are identical but IDs differ; several tracks have slightly different titles (e.g. '(Remastered)'). Recreate each playlist in Jellyfin with the same order and report any track that could not be matched.
- Platforms
web· Any OS (web)- Applications
- Navidrome · Jellyfin
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appdrag· Drag & dropnav· Pan/zoom/scroll- Work product
WP09· Files & folders- Conditions
multiapp· Multi-applong· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Jellyfin playlists: 5 with track sequences equal to mapped Navidrome sequences (by file path); unmatched report equals the seeded 2 missing files.
Proposed partial credit
70% playlist sequences (per playlist); 20% unmatched report; 10% names.
R12.07.01.01-T20 Set up a family media server with per-user libraries, ratings limits and subtitle defaults
Original instruction
In the Jellyfin web admin, create users 'Kids' and 'Guest'. Kids: access only to 'Family Movies' and 'Cartoons' libraries, max parental rating 'PG', subtitles default English forced-only; Guest: Movies library only, no downloads, no transcoding above 1080p. Then fix 3 films in Family Movies whose official rating is missing (set from fixtures/ratings.csv) so the Kids filter works.
- Platforms
web· Any OS (web)- Applications
- Jellyfin
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
long· Long-horizon- Proposed difficulty
- 4 / 5
Proposed verifier
Jellyfin DB: user policies (EnabledFolders, MaxParentalRating, EnableContentDownloading, RemoteClientBitrateLimit), Kids subtitle mode=OnlyForced language eng; 3 items' OfficialRating set; Kids browse API (harness) returns only PG-or-below items. The agent has no Jellyfin API key, no terminal and browser devtools are disabled, so REST calls and NFO edits are unavailable; changes must come from the web client (activity log).
Proposed partial credit
30% Kids policy; 25% Guest policy; 20% subtitle defaults; 25% ratings fixed (effective filter).
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.
- AndroidDaily
- MobileBench-OL
- MobileWorld-Real
- OLLA blind-user diary corpus
- OSWorld-Pro
- VenusBench-Mobile
- A3 (Android Agent Arena)
- AgentNetBench (OpenCUA)
- OS-MAP
- OSWorld-Human
- OSWorld-MCP
- OSWorld-Verified
- WindowsAgentArena-V2
- WorldGUI
- AndroidArena
- AndroidLab
- AndroidWorld
- AppWorld
- GUI Odyssey
- Mobile-Bench
- OSWorld
- Windows Agent Arena (WAA)