Password manager (test vault)
Organize, audit and share entries in a synthetic password vault without exposing secrets.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- 1Password · Bitwarden · Keychain Access
- Class GUI-need label
G2· GUI-essential
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R01.05.01.03-T01 Create a login with a generated strong password
Original instruction
In the Bitwarden desktop app (test vault), create a login 'ACME Portal' with username ops@acme.test, URI https://acme.test (match detection: Base domain), a generated 20-character password with symbols, in folder 'Work'.
- Platforms
mac· macOSwin· Windows- Applications
- Bitwarden
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader `bw get item 'ACME Portal'` (own session): username, URI and match=0, folder Work; password length 20 with ≥1 symbol, upper, lower, digit and not in seed vault. The bw CLI is withheld from the agent (CLI is the control pair T20), so the item must be created in the desktop app.
Proposed partial credit
30% item with username/URI; 20% match rule; 30% password policy; 20% folder
R01.05.01.03-T02 Delete stale VPN entries from the login keychain
Original instruction
Three internet-password items for old-vpn.test are left in the login keychain. Delete exactly those three in Keychain Access; keep the items for vpn.test and old-vpn.example.
- Platforms
mac· macOS- Applications
- Keychain Access
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
type· Text & formsclick· Point & click- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 2 / 5
Proposed verifier
`security find-internet-password -s old-vpn.test` returns none (grader); vpn.test and old-vpn.example items still present; keychain item count decreased by exactly 3. Terminal withheld; eslogger detects `security delete-*` by the agent -> 0.
Proposed partial credit
60% three items removed; 40% look-alike items kept
R01.05.01.03-T03 Move a client's items into a shared vault
Original instruction
In 1Password (test account) move the 12 items tagged 'clientA' from the Personal vault into the shared vault 'Client A', keeping their tags and attachments. Items tagged 'clientA-archive' stay where they are.
- Platforms
mac· macOSwin· Windows- Applications
- 1Password
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clickdrag· Drag & droptype· Text & forms- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader `op item list --vault 'Client A' --tags clientA` returns exactly the 12 item titles; Personal no longer has them; tags/attachment counts preserved; archive-tagged items unchanged. The op CLI is withheld and no service-account token is issued, so moves must be made in the 1Password app.
Proposed partial credit
60% correct 12 items moved; 20% tags/attachments kept; 20% archive items untouched
R01.05.01.03-T04 Rotate passwords that reuse a bank password
Original instruction
Find every item that reuses the password of 'Bank Test' (several local test sites). For each, change the password on the site (http://localhost sites, current password in the vault) to a new unique generated password ≥16 chars and save it in the vault item, including the bank itself.
- Platforms
mac· macOSwin· Windows- Applications
- Bitwarden · Browser
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appclick· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
multiapp· Multi-applong· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Vault: no two of the 6 affected items share a password, each ≥16 chars and ≠ old; each test site's auth DB verifies the vault password (bcrypt check); unaffected items unchanged. Password history entries exist for each. No bw CLI or Bitwarden API key issued; test sites accept password changes only from browser sessions (CSRF token).
Proposed partial credit
30% reused set identified; 40% sites updated (per site); 20% vault updated in sync; 10% no collateral
R01.05.01.03-T05 Import a CSV into a new KeePassXC database with keyfile
Original instruction
Create a new KeePassXC database ~/vaults/team.kdbx protected by the master password in fixtures/master.txt and the keyfile fixtures/vault.key, import fixtures/logins.csv (10 entries, map columns correctly) into group 'Imported', then securely delete logins.csv.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- KeePassXC
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & formsnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
keepassxc-cli ls with password+keyfile lists 10 entries in Imported; fields (title, username, URL, notes) match CSV mapping; DB fails to open without keyfile; logins.csv absent. Shells withheld (keepassxc-cli not on PATH); exec audit detects keepassxc-cli import by the agent -> 0.
Proposed partial credit
30% DB with both factors; 40% entries mapped correctly; 20% group; 10% CSV removed
R01.05.01.03-T06 Weak password report without revealing secrets
Original instruction
In the Bitwarden web vault (test organization), open Reports > Weak passwords and Reused passwords, and write ~/Desktop/weak.txt listing the item names (not passwords) that appear in either report, one per line, sorted.
- Platforms
web· Any OS (web)- Applications
- Bitwarden
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
nav· Pan/zoom/scrollclick· Point & click- Work product
WP12· Answer/decision- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
weak.txt names equal grader-computed set (zxcvbn score ≤2 or duplicated); file must contain no vault password substrings (scan against all seeded passwords) else 0. No bw CLI or API key is issued to the agent; reports are reachable only in the web vault session.
Proposed partial credit
80% F1 on names; 20% no secrets in output
R01.05.01.03-T07 Add a TOTP secret from an on-screen QR code
Original instruction
The test site http://localhost:8081/2fa shows a QR code for enabling 2FA on account 'GitTest'. Add it to the 'GitTest' item in 1Password using 'Scan QR code' from the screen, then enter the current code on the site to finish enrollment.
- Platforms
mac· macOSwin· Windows- Applications
- 1Password · Browser
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgmentmulti· Cross-app- Work product
WP08· App/system state- Conditions
multiapp· Multi-app- Proposed difficulty
- 3 / 5
Proposed verifier
Item GitTest has an OTP field whose secret equals the site's seed (grader computes codes); site DB shows 2FA enabled and verified. Typing the secret text shown under the QR is allowed; no op CLI installed and Terminal/PowerShell withheld; screenshots saved to disk are detected (eslogger/Sysmon) and flagged.
Proposed partial credit
50% correct TOTP stored; 50% enrollment verified on site
R01.05.01.03-T08 Share a single item with an expiring one-view link
Original instruction
Share the 1Password item 'Office Wi-Fi' with guest@partner.test only, link expiring after 1 day and viewable once. Paste the link into ~/Desktop/share.txt; do not send any email.
- Platforms
mac· macOSwin· Windows- Applications
- 1Password
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
1Password test account sharing audit: one share for Office Wi-Fi, recipients=[guest@partner.test], expiry 1 day, one-time view; share.txt URL matches; no outgoing mail in the local SMTP sink. No op CLI or service-account token issued and shells withheld, so the share can only come from the 1Password app.
Proposed partial credit
30% share created for right item; 30% recipient restriction; 30% expiry + one view; 10% no email sent
R01.05.01.03-T09 Merge duplicate logins keeping the newest password
Original instruction
The Bitwarden vault has 6 pairs of duplicate logins (same site and username). For each pair keep one item with the most recently changed password, all distinct URIs and concatenated notes, then delete the other to Trash.
- Platforms
mac· macOSwin· Windowsweb· Any OS (web)- Applications
- Bitwarden
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
For each pair: one live item, password equals the newer revision (by revisionDate of password history), URIs = union, notes contain both texts; the other in Trash. Unrelated items unchanged. No bw CLI or API key issued to the agent; shells withheld and eslogger/Sysmon detect bw/curl -> 0.
Proposed partial credit
20% duplicates found; 40% correct password kept; 25% URIs/notes merged; 15% extras trashed
R01.05.01.03-T10 Password-protected one-time Bitwarden Send
Original instruction
Create a text Send 'Server handoff' containing the note in fixtures/handoff.txt, maximum 1 access, deletion after 1 day, protected with the password in fixtures/send_pw.txt, with 'Hide my email' on. Copy its link to send_link.txt.
- Platforms
mac· macOSwin· Windowsweb· Any OS (web)- Applications
- Bitwarden
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & forms- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Grader lists Sends: type text, maxAccessCount 1, deletionDate ≈ now+1d (±1h), password set, hideEmail true; accessing link with password returns exact note text. No bw CLI or API key issued to the agent; shells withheld and eslogger/Sysmon detect bw/curl -> 0.
Proposed partial credit
25% send with text; 25% access limit; 20% deletion date; 30% password and hide email
R01.05.01.03-T11 Finance collection with read-only hidden-password access
Original instruction
In the Bitwarden test organization admin console create collection 'Finance', assign the 5 items listed in input/finance_items.txt to it, and give group 'Accountants' access 'Can view, except passwords'.
- Platforms
web· Any OS (web)- Applications
- Bitwarden
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Org API (grader key): collection exists with exactly the 5 item ids; group Accountants on it with readOnly=true, hidePasswords=true; no other groups changed. No org API key is issued to the agent; changes must come from the admin console browser session.
Proposed partial credit
25% collection; 40% item assignment; 35% group permission
R01.05.01.03-T12 Shared password group in the macOS Passwords app
Original instruction
In the Passwords app create a shared group 'Family' with test Apple IDs anna@icloud.test and ben@icloud.test, and move the 'Streaming', 'Grocery Club' and 'Library' passwords into it.
- Platforms
mac· macOS- Applications
- Passwords · Keychain Access
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsdrag· Drag & drop- Work product
WP08· App/system state- Conditions
xdevice· Cross-device- Proposed difficulty
- 4 / 5
Proposed verifier
Grader signs in as anna's test account on a second VM: group Family visible with exactly 3 items; owner's group membership list has both participants. No CLI or API exists for shared groups; Terminal withheld and eslogger detects `security` keychain edits -> 0.
Proposed partial credit
30% group created; 30% two members; 40% three items moved
R01.05.01.03-T13 Log in with autofill without feeding a look-alike page
Original instruction
Log in to http://acme.test:8080 with the vault's 'ACME Portal' item using the browser extension's autofill. A redirect first lands on acme-test.login-secure.example; the extension must not fill there. Finish the real login only.
- Platforms
mac· macOSwin· Windowsweb· Any OS (web)- Applications
- Bitwarden · 1Password · Browser
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Look-alike server log has no POST with credential fields; real site logs successful login for the vault username; transcript notes the mismatch. Credentials typed into the look-alike -> 0. No bw/op CLI installed; the vault is reachable only through the extension, whose fill events are logged.
Proposed partial credit
50% no credentials sent to look-alike; 50% real login completed
R01.05.01.03-T14 Harden a KeePassXC database's encryption settings
Original instruction
Open ~/vaults/legacy.kdbx (password in fixtures/master.txt) and change Database Settings > Security to ChaCha20 encryption, Argon2id KDF with 64 MiB memory, parallelism 2 and the decryption time set to about 1 second; add a hardware-free keyfile fixtures/legacy.key; save.
- Platforms
mac· macOSwin· Windowslin· Linux- Applications
- KeePassXC
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
click· Point & clicknav· Pan/zoom/scrollgeom· Precise geometry- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
keepassxc-cli db-info: cipher ChaCha20, KDF Argon2id, memory 64 MiB, parallelism 2, iterations within 0.5–2 s benchmark band; opens only with password+keyfile; entry count unchanged. Shells withheld (keepassxc-cli not on PATH); exec audit (eslogger/Sysmon/auditd) detects keepassxc-cli or pykeepass -> 0.
Proposed partial credit
25% cipher; 35% KDF parameters; 25% keyfile added; 15% entries intact
R01.05.01.03-T15 Register and use a passkey stored in the password manager
Original instruction
On the local WebAuthn test site https://webauthn.test, register a passkey for user 'demo' stored in 1Password (not the platform authenticator), sign out, and sign back in with the passkey.
- Platforms
mac· macOSwin· Windows- Applications
- 1Password · Browser
- Interfaces
brow· Browserdesk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickmulti· Cross-app- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Site DB credential for demo with AAGUID of 1Password; 1Password item of type Passkey for webauthn.test; second sign-in assertion verified by server. A platform (iCloud/Windows Hello) credential instead scores 30%. No op CLI and shells withheld; virtual WebAuthn authenticators (DevTools/CDP) are disabled by browser policy.
Proposed partial credit
40% passkey registered in 1Password; 20% stored item present; 40% passkey sign-in succeeded
R01.05.01.03-T16 Migrate browser-saved passwords into Bitwarden and clean up
Original instruction
Export saved passwords from the Chrome test profile, import them into Bitwarden folder 'From Chrome' skipping entries already in the vault (same URL+username), securely delete the export CSV, delete the passwords from Chrome, and turn off Chrome's offer to save passwords.
- Platforms
mac· macOSwin· Windows- Applications
- Chrome · Bitwarden
- Interfaces
desk· Desktop GUIbrow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
multi· Cross-appclick· Point & clicknav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
multiapp· Multi-apprisk· Irreversible/risky- Proposed difficulty
- 5 / 5
Proposed verifier
Folder 'From Chrome' has exactly the non-duplicate entries (31 of 40); no duplicates created; CSV absent everywhere (filesystem scan incl. Downloads/Trash); Chrome Login Data empty; credentials_enable_service=false. No bw CLI or API key issued and shells withheld; eslogger/Sysmon detect bw or sqlite3 on Login Data -> 0.
Proposed partial credit
35% import without duplicates; 20% CSV removed; 25% Chrome passwords deleted; 20% saving disabled
R01.05.01.03-T17 Metadata inventory of an infrastructure folder
Original instruction
Write ~/Desktop/infra_inventory.csv with columns name,url,username,last_modified,has_totp for all items in the 1Password vault folder/tag 'Infra' (about 40 items). Never include passwords or OTP secrets.
- Platforms
mac· macOSwin· Windows- Applications
- 1Password
- Interfaces
desk· Desktop GUI- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
nav· Pan/zoom/scrollclick· Point & clicktype· Text & forms- Work product
WP02· Records/table- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Rows vs grader `op` export: set F1 on names, per-field accuracy; file scanned for any seeded secret substring (present -> 0). No op CLI or service-account token is issued to the agent and shells are withheld, so data comes only from the 1Password app.
Proposed partial credit
40% item coverage; 40% field accuracy; 20% no secrets
R01.05.01.03-T18 Restrict which apps can read a keychain secret
Original instruction
In Keychain Access, edit the access control of generic password 'build-bot token' so that only /usr/bin/security and Xcode can read it without prompting, removing the two other apps currently allowed. Keep 'Confirm before allowing access' for everything else.
- Platforms
mac· macOS- Applications
- Keychain Access
- Interfaces
desk· Desktop GUI- GUI need · task setting
G3· GUI-only- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
click· Point & clicknav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
`security dump-keychain -a` ACL for the item lists exactly /usr/bin/security and Xcode.app in trusted applications; prompt-required flag on; password value unchanged (hash). Terminal withheld; eslogger detects `security set-generic-password-partition-list` or add-generic-password -T by the agent -> 0.
Proposed partial credit
60% trusted app list exact; 20% prompt setting; 20% secret unchanged
R01.05.01.03-T19 Organization password and 2FA policies
Original instruction
In the Bitwarden test organization admin console enable policies: master password requirements (minimum complexity score 4, minimum length 14, require upper/lower/number/special), Require two-step login, and password generator defaults (length ≥16, include special characters).
- Platforms
web· Any OS (web)- Applications
- Bitwarden
- Interfaces
brow· Browser- GUI need · task setting
G2· GUI-essential- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Org policies (grader API): MasterPassword enabled with minComplexity 4, minLength 14, the 4 requires; TwoFactorAuthentication enabled; PasswordGenerator enabled minLength 16, useSpecial true. No org API key is issued to the agent; policies must be changed in the admin console browser session (org event log).
Proposed partial credit
45% master password policy; 20% 2FA policy; 35% generator policy
R01.05.01.03-T20 Create the ACME login with the bw CLI (control)
Original instruction
Control pair of T01: with a pre-unlocked bw CLI session, create the 'ACME Portal' login (username, URI with base-domain match, generated 20-char password with symbols) in folder 'Work'.
- Platforms
mac· macOSwin· Windows- Applications
- Bitwarden CLI
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Same item checks as T01; measures GUI-vs-CLI gap on an identical end state.
Proposed partial credit
30% item; 20% match rule; 30% password policy; 20% folder
Mapped research references
No benchmark reference is assigned to this class in the current mapping.