Test authoring, smoke tests & CI pipelines
Write reproduction and smoke tests (including checks that tools and integrations work end to end) and fix CI workflows until the pipeline is green.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- pytest · GitHub Actions · GitLab CI
- Class GUI-need label
CL· CLI-sufficient
Digital-employee context
Project class assignments, not demonstrated employee capability or imported benchmark coverage. Acceptance criteria and value metrics are proposals; no ROI has been measured.
Related roles
- IT and software delivery · Separate deliverables cover service tickets, restored configuration, identity records, repository fixes, passing CI and release packages.
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R02.02.01.02-T01 Write a pytest that reproduces a reported bug before any fix exists
Original instruction
Repo `/work/invoicekit` (Python) at the buggy commit. Issue #88: credit notes with a negative discount produce a positive total. Write a test in `tests/test_issue_88.py` that fails on the current code for this reason and will pass once the bug is fixed. Do not change source files.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- pytest · git
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Test fails on the buggy commit with an assertion (not an error/import failure) and passes on the hidden gold patch; it also fails on 3 hidden plausible-but-wrong patches; src/ unchanged.
Proposed partial credit
40% fails on buggy; 40% passes on gold; 20% rejects wrong patches
R02.02.01.02-T02 Fix a failing GitHub Actions workflow until the pipeline is green
Original instruction
In the sandbox Gitea/Act-compatible runner for repo `/work/webapp`, the `ci.yml` workflow fails: wrong Node version for the lockfile, a cache key that never hits, and a test job that runs before the build artifact is uploaded. Fix the workflow (not the app) so it goes green and the cache hits on the second run.
- Platforms
lin· Linux- Applications
- GitHub Actions · act
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs the workflow twice with act: both runs succeed; second run logs a cache hit; job graph has test needs build with artifact download; only .github/ files changed.
Proposed partial credit
40% pipeline green; 25% cache hit on rerun; 25% job dependencies correct; 10% scope of change
R02.02.01.02-T03 Author a GitLab CI pipeline with stages, artifacts and rules
Original instruction
Write `.gitlab-ci.yml` for `/work/geo-service` (Go) with stages lint, test, build, package: test produces JUnit and coverage reports, build produces a binary artifact consumed by package (Docker image to the sandbox registry), package runs only on tags, and MR pipelines skip package.
- Platforms
lin· Linux- Applications
- GitLab CI · gitlab-runner
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Sandbox GitLab: pipelines for an MR and a tag both succeed; MR pipeline has no package job; tag pipeline pushes image tag = git tag; JUnit report and coverage % visible via API; `gitlab-ci-lint` clean.
Proposed partial credit
25% stages/lint; 25% test reports; 25% artifacts flow; 25% rules for MR vs tag
R02.02.01.02-T04 Raise a module's mutation score to at least 80%
Original instruction
In `/work/pricing-engine` (Python), mutmut reports a 41% mutation score for `pricing/discounts.py`. Add tests (do not change source) until the mutation score for that module is at least 80%, with tests that are deterministic and run under 30 s total.
- Platforms
lin· Linuxmac· macOS- Applications
- pytest · mutmut
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Grader reruns mutmut on the module: score >= 80% (partial credit linear from 41%); tests pass 5 times in random order; runtime < 30 s; source unchanged (hash).
Proposed partial credit
80% mutation score (linear credit from 0.41 to 0.80); 20% determinism + runtime
R02.02.01.02-T05 Find a seeded bug with Hypothesis property-based tests
Original instruction
Library `/work/intervalset` (Python) has a subtle bug in `union` for adjacent half-open intervals. Without reading the git history, write Hypothesis property tests (invariants such as idempotence, commutativity, membership agreement with a naive model) that find it, and save the minimal failing example in `FOUND.md`.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- pytest · Hypothesis
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Property tests fail on the buggy version within 200 examples (fixed seed) and pass on the gold fix; FOUND.md example reproduces the bug; also fail on 2 other hidden mutants.
Proposed partial credit
40% detects seeded bug; 30% passes on gold; 20% catches extra mutants; 10% minimal example
R02.02.01.02-T06 Write Playwright end-to-end tests for login and checkout journeys
Original instruction
For the sandbox shop at `/work/shop` (runs via docker compose), write Playwright tests covering: login with valid/invalid credentials, adding items to cart, coupon application, and checkout with the test card, using page objects and no fixed sleeps. Tests must be stable across 5 runs.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Playwright · Docker
- Interfaces
term· Terminalbrow· Browser- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Suite passes 5/5 runs headless; no waitForTimeout calls; tests fail on 4 hidden seeded app bugs (one per journey); traces enabled on failure in config.
Proposed partial credit
30% stable passing suite; 50% catches seeded bugs (scaled); 20% no sleeps + page objects
R02.02.01.02-T07 Make a cross-OS CI matrix pass, including a Windows-only path bug
Original instruction
Repo `/work/fsync-tool` (Python) has a GitHub Actions matrix over ubuntu/macos/windows × Python 3.10-3.13. The Windows jobs fail on path separators and file locking in tests and code; macOS fails on a case-insensitive filesystem assumption. Fix code and tests so all 12 jobs pass.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- GitHub Actions · pytest
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Grader runs the suite on three real VMs (Linux, macOS, Windows) for 4 Python versions: all green; no tests skipped or xfailed that weren't before; hidden Windows-path tests pass.
Proposed partial credit
40% Windows green; 25% macOS green; 20% no new skips; 15% hidden path tests
R02.02.01.02-T08 Stabilise a flaky time-dependent test suite
Original instruction
In `/work/subscriptions` (TypeScript, Jest), 9 tests fail intermittently around midnight UTC, month ends and DST. Make the suite deterministic by controlling time (fake timers / injected clock) without weakening assertions or changing production behavior.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Jest · Node
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader runs the suite 50 times with libfaketime at 12 adversarial wall-clock instants: all pass; assertion count not reduced (AST count); production code changes limited to clock injection; hidden behavior tests pass.
Proposed partial credit
60% deterministic across instants; 25% assertions preserved; 15% minimal prod changes
R02.02.01.02-T09 Add a coverage gate and write tests to reach it
Original instruction
Add a CI coverage gate of 85% line and 75% branch coverage for `/work/csvtools` (Python) in `ci.yml` and `pyproject.toml`, then write tests until the gate passes. Current coverage is 58%/40%. Exclusions are only allowed for the `__main__` block.
- Platforms
lin· Linuxmac· macOS- Applications
- pytest-cov · GitHub Actions
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader measures coverage: line >= 85%, branch >= 75%; fail_under configured; no new `pragma: no cover` beyond __main__; tests kill >= 60% of a hidden mutant set (prevents assertion-free tests).
Proposed partial credit
40% coverage targets; 20% gate configured; 40% mutant kill threshold
R02.02.01.02-T10 Add consumer-driven contract tests between two services
Original instruction
Services `/work/orders` (consumer, TypeScript) and `/work/inventory` (provider, Java Spring) talk over REST. Add Pact consumer tests for the 4 interactions the consumer uses, publish pacts to the sandbox Pact Broker, and add provider verification in the provider's build with provider states.
- Platforms
lin· Linux- Applications
- Pact · Jest · JUnit
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Broker shows a pact with 4 interactions and a successful verification; a hidden breaking change to the provider (renamed field) makes verification fail; can-i-deploy returns true for current versions.
Proposed partial credit
30% consumer pacts; 30% provider verification; 25% catches breaking change; 15% broker integration
R02.02.01.02-T11 Write JUnit 5 tests with mocks reaching 90% branch coverage
Original instruction
Class `RefundService` in `/work/payments-java` has complex branching and depends on a gateway client and a clock. Write JUnit 5 + Mockito tests that reach >= 90% branch coverage for that class (JaCoCo), using parameterized tests where appropriate, without modifying the class.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Maven · JUnit 5 · Mockito · JaCoCo
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
JaCoCo branch coverage for RefundService >= 90%; PIT mutation score for the class >= 70%; tests pass; class source unchanged.
Proposed partial credit
50% branch coverage; 40% mutation score; 10% unchanged source
R02.02.01.02-T12 Build a release workflow that publishes with OIDC instead of stored secrets
Original instruction
In the sandbox GitHub-compatible environment, write a release workflow for `/work/cli-tool` that, on a version tag, builds artifacts for 3 OSes, signs them with keyless signing, and pushes a container image to the sandbox registry using OIDC token federation (no long-lived secrets in the repo).
- Platforms
lin· Linux- Applications
- GitHub Actions · cosign · Docker
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Tag push triggers the workflow successfully; registry has image with tag; cosign verify with the expected OIDC identity passes; repo secrets list empty; permissions block minimal (id-token: write only where needed).
Proposed partial credit
30% multi-OS build; 30% OIDC push; 25% signatures verify; 15% least-privilege permissions
R02.02.01.02-T13 Convert a Jenkins pipeline to GitHub Actions with equivalent behavior
Original instruction
Repo `/work/legacy-api` has a declarative Jenkinsfile with parallel stages, a matrix over JDK 17/21, stash/unstash, a manual approval before deploy, and post-failure notifications. Write equivalent GitHub Actions workflows (environments for approval) and remove the Jenkinsfile.
- Platforms
lin· Linux- Applications
- GitHub Actions · Jenkins
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
act/sandbox runner executes the workflow: matrix jobs green; artifacts passed between jobs; deploy job gated by environment with required reviewer; failure notification step present with if: failure(); Jenkinsfile deleted.
Proposed partial credit
30% matrix + parallel; 25% artifacts; 25% approval gate; 20% notifications + cleanup
R02.02.01.02-T14 Add a libFuzzer harness to CI that finds a crash in a C parser
Original instruction
Library `/work/libini` (C) has no fuzzing. Write a libFuzzer harness and seed corpus for `ini_parse`, add a CI job that runs it for 5 minutes with ASan/UBSan, and keep the job green by fixing any bugs the fuzzer finds (at least one is seeded).
- Platforms
lin· Linux- Applications
- clang · libFuzzer · GitHub Actions
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Harness builds with -fsanitize=fuzzer,address,undefined; grader's 10-minute run on fixed code finds no crash; on the original code the harness finds the seeded crash within 5 min; CI job defined with corpus caching.
Proposed partial credit
30% harness finds seeded bug; 30% bug fixed; 25% clean 10-min run; 15% CI job
R02.02.01.02-T15 Add golden snapshot tests for a CLI's output
Original instruction
Add snapshot tests for `/work/tablefmt` CLI (Rust) using insta: cover all 6 output formats, Unicode width cases and error messages, with a review workflow documented in `CONTRIBUTING.md`. Snapshots must not include machine-specific paths or timestamps.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- cargo · insta
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
cargo test passes on Linux, macOS, Windows; >= 25 snapshots across 6 formats; snapshots free of absolute paths/timestamps (regex); a hidden output change in one format fails the suite.
Proposed partial credit
40% snapshots cover formats; 25% cross-OS stable; 25% detects change; 10% docs
R02.02.01.02-T16 Write table-driven Go tests and enable the race detector in CI
Original instruction
Package `/work/ratelimit-go` has two ad hoc tests. Write table-driven tests with subtests covering the limiter's documented behaviors, a concurrency test exercised under `-race`, a benchmark, and a CI job running `go test -race -count=3 ./...` plus `go vet`.
- Platforms
lin· Linuxmac· macOS- Applications
- go · GitHub Actions
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Tests pass with -race -count=3; hidden mutants (6) killed >= 5; t.Run subtests present; benchmark compiles and runs; CI job config valid (actionlint).
Proposed partial credit
50% mutants killed; 20% race test; 15% benchmark; 15% CI job
R02.02.01.02-T17 Set up pre-commit hooks so linters and type checks pass
Original instruction
Add a `.pre-commit-config.yaml` to `/work/datatools` (Python) running ruff (lint + format), mypy and a secrets scanner, pinned to specific versions, then fix the codebase so `pre-commit run --all-files` passes without disabling rules globally.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- pre-commit · ruff · mypy
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
`pre-commit run --all-files` exits 0; hooks pinned (rev tags); no global ignore lists added beyond 3 inline noqa; tests still pass; a hidden commit with a fake secret is blocked by the hook.
Proposed partial credit
30% config; 40% codebase passes; 15% minimal suppressions; 15% secret detection
R02.02.01.02-T18 Configure branch protection and required checks in a sandbox Git forge UI
Original instruction
In the sandbox GitHub Enterprise-like forge web UI, configure branch protection for `main` of repo `acme/api`: require PRs with 2 approvals including a CODEOWNERS review, require status checks `ci/test` and `ci/lint` to pass on up-to-date branches, block force-push and deletion, and require signed commits.
- Platforms
web· Any OS (web)- Applications
- Git forge web UI
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clicktype· Text & formsnav· Pan/zoom/scroll- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Forge API branch protection for main: required_approving_review_count 2, require_code_owner_reviews, required checks exactly the two contexts with strict true, allow_force_pushes false, allow_deletions false, required_signatures true.
Proposed partial credit
30% reviews; 30% status checks strict; 20% force-push/deletion; 20% signed commits
R02.02.01.02-T19 Cut a 40-minute CI test stage to under 12 minutes without dropping tests
Original instruction
CI for `/work/bigapp` (Python + Postgres) runs 6,800 tests serially in 40 min. Reduce wall time below 12 minutes on the grader's 8-core runner: parallelise safely (per-worker DB schemas), shard across jobs with timing-based balancing, and fix the order-dependent tests that break when parallelised. No test may be skipped or deleted.
- Platforms
lin· Linux- Applications
- pytest-xdist · GitHub Actions · Postgres
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Grader runs CI 3 times: wall time < 12 min each, all 6,800 tests collected and passing (count check), no new skip markers; order-dependence fixed (random-order run passes).
Proposed partial credit
40% time target (linear 40->12 min); 30% all tests run and pass; 30% order-independence
R02.02.01.02-T20 Fix a test failure that only happens in CI
Original instruction
`/work/reporting` passes locally but fails in CI with assertion diffs in currency formatting and file ordering. The CI container differs in locale, timezone, filesystem ordering and available fonts. Reproduce the CI environment locally with Docker, find each cause, and fix code/tests so results are environment-independent.
- Platforms
lin· Linux- Applications
- Docker · pytest · GitHub Actions
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
hidden· Hidden/stale state- Proposed difficulty
- 4 / 5
Proposed verifier
Grader runs the suite under 6 environment variants (LANG, TZ, overlay vs ext4 ordering, missing fonts): all pass; no environment pinning hacks in tests (e.g., forcing TZ only in CI); hidden tests pass.
Proposed partial credit
Per-variant credit: 6 variants scaled to 80%; 20% no environment-pinning hacks
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.