Secrets hygiene & history rewriting
Find leaked secrets, rotate them in test systems and purge them from git history safely.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- git · git-filter-repo · gitleaks
- Class GUI-need label
CL· CLI-sufficient
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R03.05.01.06-T01 Triage a gitleaks scan and allowlist false positives
Original instruction
Run gitleaks over the full history of ~/repos/webshop. Of the findings, 3 are known test fixtures listed in docs/fixtures.md. Add a .gitleaks.toml that extends the default config and allowlists exactly those 3 (by path + regex, not by disabling rules), and write the remaining true findings to ~/out/true_leaks.json (commit, file, rule).
- Platforms
lin· Linuxmac· macOS- Applications
- gitleaks · git
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader runs gitleaks with the agent's config: exactly the 5 seeded true leaks remain; config keeps [extend] useDefault and no rule is disabled; true_leaks.json equals the gold set by (commit, file, RuleID).
Proposed partial credit
40% allowlist removes exactly the 3 fixtures; 20% no rules disabled; 40% true findings listed correctly
R03.05.01.06-T02 Purge an env file from all history and force-push
Original instruction
config/prod.env with live-looking credentials was committed 40 commits ago in ~/repos/api and later deleted. Remove it from every commit on all branches and tags with git-filter-repo, keep everything else byte-identical, and force-push all refs to the test remote 'origin' (Gitea sandbox).
- Platforms
lin· Linuxmac· macOS- Applications
- git · git-filter-repo
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Fresh mirror clone of origin: no blob with the env file's content hashes in any object (git cat-file --batch-all-objects); same branch and tag names; commit count per branch equals original minus commits that only touched that file; every other path's tree content identical per commit.
Proposed partial credit
45% file absent from all reachable history; 30% other content and refs preserved; 25% remote updated
R03.05.01.06-T03 Redact leaked AWS keys in place across history
Original instruction
Two AWS access-key/secret pairs (test values) appear in several source files across history of ~/repos/infra. Replace each occurrence with the literal ***REMOVED*** using git-filter-repo --replace-text, without removing files, and push to the sandbox remote.
- Platforms
lin· Linuxmac· macOS- Applications
- git · git-filter-repo
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Mirror clone: grep over all blobs finds 0 occurrences of the 4 secret strings and N occurrences of ***REMOVED*** equal to original occurrence count; all files still exist at the same paths in every commit; gitleaks over history reports 0 AWS findings.
Proposed partial credit
50% secrets gone everywhere; 30% replacement count exact and files intact; 20% remote updated
R03.05.01.06-T04 Rotate a leaked Postgres password across Vault and Kubernetes
Original instruction
The password of DB role 'orders_app' leaked. In the sandbox: set a new random 32-char password on the Postgres role, update Vault KV secret/orders/db (new version, keep history), update Kubernetes secret orders-db in namespace shop on the kind cluster, and roll the orders deployment so pods use it.
- Platforms
lin· Linux- Applications
- psql · vault · kubectl
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
risk· Irreversible/riskymultiapp· Multi-app- Proposed difficulty
- 4 / 5
Proposed verifier
Old password fails to authenticate; Vault KV metadata current_version incremented and data equals the working password; k8s secret matches; orders pods restarted after the secret change and /healthz returns 200; password length 32 and not present in shell history files.
Proposed partial credit
30% DB password rotated; 25% Vault updated with history; 25% k8s secret and rollout; 20% service healthy and no secret in history files
R03.05.01.06-T05 Remove a secret baked into Docker image layers
Original instruction
registry.test/shop/api:1.4 contains an .npmrc with an auth token in an intermediate layer (deleted in a later layer). Rebuild the image with a fixed Dockerfile using a BuildKit secret mount, tag it 1.4.1, push to the local registry, and write the leaked layer digest to ~/out/layer.txt.
- Platforms
lin· Linux- Applications
- docker · dive · skopeo
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
layer.txt equals the digest of the layer containing the token; 1.4.1 manifest layers contain no .npmrc and no token string (grader extracts every layer); Dockerfile uses RUN --mount=type=secret; container passes the app smoke test.
Proposed partial credit
25% leaked layer identified; 45% new image free of token in all layers; 30% secret mount used and app works
R03.05.01.06-T06 Block new secrets with pre-commit and CI
Original instruction
In ~/repos/webshop add a pre-commit config running gitleaks (protect --staged) and a GitLab CI job 'secrets-scan' that runs gitleaks on the MR diff and fails on findings. Prove both: a test commit with a fake Stripe test key must be blocked locally, and a pushed branch with one must fail the CI job.
- Platforms
lin· Linuxmac· macOS- Applications
- pre-commit · gitleaks · GitLab CI
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader installs hooks in a fresh clone and attempts a commit with a seeded key: rejected; clean commit accepted; CI job on a grader-pushed leaking branch fails, on a clean branch passes; .pre-commit-config.yaml pins a gitleaks rev.
Proposed partial credit
40% pre-commit blocks and allows correctly; 40% CI job fails and passes correctly; 20% pinned versions
R03.05.01.06-T07 Purge a secret from a submodule's history and update the superproject
Original instruction
Superproject ~/repos/platform pins submodule libs/billing-sdk, whose history contains keys/stripe_test.pem. Purge that file from the submodule's history and push it to its sandbox remote, then update the superproject tips of main, release/1.x and release/2.x so each points at the rewritten equivalent of its pinned submodule commit, and push.
- Platforms
lin· Linuxmac· macOS- Applications
- git · git-filter-repo
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
risk· Irreversible/risky- Proposed difficulty
- 5 / 5
Proposed verifier
Submodule remote has no blob of the .pem; for main and release/1.x, release/2.x tips, the gitlink equals the rewritten commit mapped from the original via filter-repo's commit-map; `git submodule update --init` succeeds on each; superproject other content unchanged.
Proposed partial credit
40% submodule history purged and pushed; 45% three superproject tips repointed via commit-map; 15% clean checkout works
R03.05.01.06-T08 Purge a blob from a GitLab-hosted repo including MR refs
Original instruction
A test API key was committed to platform/api on gitlab.test and is still reachable via merge-request refs. Rewrite history locally with git-filter-repo, push, then use GitLab's Repository cleanup (upload commit-map) and 'Remove blobs' settings to drop the old objects and refs/merge-requests copies; confirm the blob is gone server-side.
- Platforms
web· Any OS (web)lin· Linux- Applications
- git-filter-repo · GitLab
- Interfaces
term· Terminalbrow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
HY· Hybrid- Interaction primitives
code· Code & commandsclick· Point & click- Work product
WP06· Code/engineering- Conditions
risk· Irreversible/riskymultiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
Server-side (grader rails runner): repository has no object with the blob id; refs/merge-requests/*/head no longer reference old commits; GitLab housekeeping ran (prune) after the cleanup upload. Fresh clone + MR refs fetch contains no secret string.
Proposed partial credit
30% local rewrite pushed; 35% repository cleanup with commit-map; 35% blob unreachable server-side
R03.05.01.06-T09 Strip notebook outputs carrying tokens from history
Original instruction
Jupyter notebooks under analysis/ in ~/repos/research saved cell outputs that print a Hugging Face test token. Rewrite history so every committed .ipynb has outputs and execution counts stripped (code cells unchanged), and add nbstripout as a git filter for future commits.
- Platforms
lin· Linuxmac· macOS- Applications
- git-filter-repo · nbstripout · git
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
For every commit, every .ipynb parses as JSON with empty outputs and null execution_count, and code cell sources equal the original; token string absent from all blobs; .gitattributes + .git/config filter for nbstripout present and a test commit gets stripped.
Proposed partial credit
45% outputs stripped in all history; 25% sources unchanged; 30% future-commit filter works
R03.05.01.06-T10 Rewrite author emails that leak personal addresses
Original instruction
Commits in ~/repos/site were authored with personal emails. Using a mailmap with git-filter-repo, rewrite the 3 personal addresses in fixtures/emails.csv to the given noreply addresses (author and committer), leave other identities unchanged, and add a .mailmap for future logs.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- git · git-filter-repo
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
git log --format='%ae %ce' over all refs contains none of the 3 personal emails; mapped counts equal original counts; other identities and commit messages unchanged; trees identical to original per commit.
Proposed partial credit
60% emails rewritten everywhere; 25% others untouched; 15% .mailmap added
R03.05.01.06-T11 Leak timeline report for an incident ticket
Original instruction
For the incident ticket, produce ~/out/leak_timeline.csv for ~/repos/mobile listing each secret introduction: commit SHA, author date (ISO UTC), author email, file, gitleaks RuleID, and whether the secret is still present at HEAD of main. Sort by date.
- Platforms
lin· Linuxmac· macOS- Applications
- gitleaks · git
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader gold: 8 introductions (one secret moved between files counts once per introduction commit); CSV compared row by row; still_present boolean checked against HEAD.
Proposed partial credit
50% introduction set correct; 30% metadata columns exact; 20% still-present flags
R03.05.01.06-T12 Rotate a leaked Gitea token used by CI
Original instruction
The token 'ci-deploy' of bot user 'deployer' on git.test leaked. Revoke it, create a new token with only write:repository scope, update the Gitea Actions secret DEPLOY_TOKEN in org 'apps', and re-run the last failed 'deploy' workflow to prove it works.
- Platforms
web· Any OS (web)lin· Linux- Applications
- Gitea · Gitea Actions
- Interfaces
brow· Browserterm· Terminal- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
click· Point & clickcode· Code & commands- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Old token returns 401; deployer has one new token with scopes exactly write:repository; org secret updated (grader decrypts via admin); workflow re-run succeeds and its log shows authentication with the new token's id.
Proposed partial credit
30% old token revoked; 25% new token scoped correctly; 20% secret updated; 25% workflow passes
R03.05.01.06-T13 Remove secret remnants from reflog, stash and dangling objects
Original instruction
In ~/repos/tools a secret was committed, amended away, and also stashed. HEAD is clean, but the local repo still holds it. Make the local repository free of the secret in every form (reflog, stash, dangling and packed objects) without losing the 4 other stashes or any branch.
- Platforms
lin· Linuxmac· macOS- Applications
- git
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
git cat-file --batch-all-objects --batch | grep secret -> nothing; git fsck --unreachable shows no objects containing it; stash list has the 4 other stashes with identical content; all branch tips unchanged.
Proposed partial credit
50% secret unreachable in any object; 30% other stashes preserved; 20% branches unchanged
R03.05.01.06-T14 Leaked SSH deploy key: purge, revoke and redeploy
Original instruction
deploy/id_ed25519 was committed to ~/repos/ops. Purge it from history and push; remove its public key from authorized_keys on hosts web1, web2, web3 (test containers); generate a new ed25519 key, install it on all three, and store the private key at ~/.ssh/deploy_2026 (mode 600).
- Platforms
lin· Linux- Applications
- git-filter-repo · ssh · ssh-keygen
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Remote mirror has no blob of the old key; old key cannot log in to any host; new key logs in to all three; authorized_keys on each host otherwise unchanged; new key file mode 600 and not committed anywhere.
Proposed partial credit
30% history purged; 30% old key revoked on all hosts; 30% new key works everywhere; 10% file permissions
R03.05.01.06-T15 Classify findings as verified or unverified with trufflehog
Original instruction
Run trufflehog against ~/repos/payments with verification pointed at the local mock endpoints (fixtures/endpoints.env redirects detectors to mock.test). Write ~/out/verified.json listing only secrets the mocks confirm as live (detector, file, commit) and ~/out/unverified_count.txt.
- Platforms
lin· Linux- Applications
- trufflehog · git
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Mocks accept 3 of 9 seeded credentials; verified.json equals those 3 by (detector, file, commit); unverified_count equals 6; grader's mock logs show the agent's verification requests went only to mock.test.
Proposed partial credit
60% verified set exact; 20% unverified count; 20% no external verification calls
R03.05.01.06-T16 Rebase in-flight branches onto rewritten history
Original instruction
origin/main of ~/repos/app was already rewritten with filter-repo to remove secrets (commit-map in fixtures/commit-map). Two local feature branches, feat/cart and feat/search, are still based on the old history. Move them onto the new history so their own commits are preserved and the removed file is not reintroduced, and push them.
- Platforms
lin· Linuxmac· macOS- Applications
- git
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
risk· Irreversible/risky- Proposed difficulty
- 5 / 5
Proposed verifier
Each pushed branch: merge-base with new main equals the commit-map image of its old base; patch-ids of its own commits equal the originals (5 and 3 commits); the purged file and secret string absent from all reachable objects of both branches.
Proposed partial credit
40% both branches rebased onto mapped bases; 30% own commits preserved by patch-id; 30% secret not reintroduced
R03.05.01.06-T17 Encrypt remaining config with sops and age
Original instruction
After the purge, ~/repos/api still needs config/prod.yaml in the repo. Encrypt it with sops using the age recipient in fixtures/age.pub (only values under 'secrets:' encrypted), add a .sops.yaml creation rule for config/*.yaml, commit the encrypted file and ensure the plaintext is git-ignored.
- Platforms
lin· Linuxmac· macOS- Applications
- sops · age · git
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Committed config/prod.yaml has sops metadata with that age recipient and encrypted_regex limiting to secrets; grader decrypts with the private key to the original; non-secret keys readable in plaintext; no plaintext version in any commit; .gitignore covers the plaintext path.
Proposed partial credit
40% decrypts to original with right recipient; 25% only secrets encrypted; 20% creation rule; 15% plaintext never committed
R03.05.01.06-T18 Rotate a payment-provider test key and update a GitHub secret
Original instruction
A Stripe-style test API key for the mock payment provider at pay.test leaked in a PR comment. Use the provider's sandbox API to roll the key (old key expires immediately), update the GitHub Actions secret PAY_KEY in sandbox repo acme-sbx/checkout with gh, and delete the PR comment.
- Platforms
lin· Linuxweb· Any OS (web)- Applications
- curl · gh · GitHub
- Interfaces
term· Terminalapi· API/SDK- GUI need · task setting
AP· API-sufficient- GUI need · ordinary access
AP· API-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 3 / 5
Proposed verifier
Mock provider: old key returns 401, exactly one active key; repo secret PAY_KEY updated_at after rotation and grader workflow using it gets 200 from pay.test; PR comment deleted (API 404).
Proposed partial credit
40% key rolled; 35% secret updated and working; 25% comment removed
R03.05.01.06-T19 Helm values leak: purge, rotate and move to sealed-secrets
Original instruction
charts/shop/values-prod.yaml in ~/repos/deploy held the Redis password for 30 commits. Purge that value from history (replace with a placeholder), rotate the Redis password on the kind cluster, and replace the plain value with a SealedSecret (kubeseal against the cluster's controller) referenced by the chart.
- Platforms
lin· Linux- Applications
- git-filter-repo · kubectl · kubeseal · helm
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
risk· Irreversible/riskymultiapp· Multi-app- Proposed difficulty
- 5 / 5
Proposed verifier
No blob contains the old password; Redis rejects old password and accepts the new one; SealedSecret in the chart decrypts in-cluster to a Secret used by the shop deployment; helm upgrade succeeds and pods connect (readiness true).
Proposed partial credit
30% history purged; 25% password rotated; 30% SealedSecret wired into the chart; 15% deployment healthy
R03.05.01.06-T20 Label 120 scanner findings as true or false positives
Original instruction
fixtures/findings.json has 120 gitleaks findings from the monorepo. Using docs/secrets_policy.md (test fixtures, revoked keys, example placeholders count as false positives), label each finding TP or FP with a one-line reason and write ~/out/labels.csv (id,label,reason).
- Platforms
lin· Linuxmac· macOS- Applications
- gitleaks · git · jq
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP02· Records/table- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Labels compared to gold: precision and recall on TP must both be >= 0.9 for full credit; grader spot-checks reasons only for format (non-empty).
Proposed partial credit
50% TP recall >= 0.9; 40% TP precision >= 0.9; 10% all 120 rows present
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.