Library or service from spec + tests
Scaffold and implement a library, CLI, extension or backend service, from generating the project skeleton to satisfying a spec and hidden tests.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- Python · Node · Docker
- Class GUI-need label
CL· CLI-sufficient
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R02.01.02.01-T01 Implement a SemVer 2.0 parsing and comparison library
Original instruction
In `/work/semver-py` (empty package skeleton with `SPEC.md` and 40 visible tests), implement `parse`, `compare`, `bump` and range matching (`^`, `~`, `>=`, hyphen ranges, `||`) per the SemVer 2.0.0 spec and the range grammar in SPEC.md. Pure Python 3.12, no third-party dependencies.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Python · pytest
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Hidden tests (420 cases incl. prerelease precedence and build metadata) pass; no dependencies in pyproject; mypy --strict clean.
Proposed partial credit
40% parse/compare; 20% bump; 30% ranges; 10% typing
R02.01.02.01-T02 Build a TOML 1.0 parser that passes the official test suite
Original instruction
Implement a TOML 1.0 decoder and encoder in `/work/toml-rs` (Rust crate skeleton) exposing `from_str` -> `Value` and `to_string`. It must accept every valid document and reject every invalid one in the toml-test suite, with errors reporting line and column.
- Platforms
lin· Linuxmac· macOS- Applications
- cargo · toml-test
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
toml-test decoder harness: valid cases pass (JSON-tagged equality) and invalid cases rejected; encoder round-trips all valid cases; error positions correct on 30 hidden invalid docs; no external TOML crates in Cargo.lock.
Proposed partial credit
45% valid decode (scaled); 25% invalid rejected; 20% encoder round-trip; 10% error positions
R02.01.02.01-T03 Implement a token-bucket rate-limiter HTTP service from an API spec
Original instruction
Implement the service described in `/work/ratelimiter/openapi.yaml`: `POST /v1/check` decides allow/deny per key with per-tenant bucket configs from `config.yaml`, `GET /v1/buckets/{key}` reports state, and limits are consistent across 2 replicas sharing Redis. Any language; ship a Dockerfile.
- Platforms
lin· Linux- Applications
- Docker · Redis
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden compose test: schemathesis conformance to openapi.yaml; refill-rate correctness within ±1 token over 60 s; two replicas never jointly exceed capacity under 2,000 rps burst; p99 latency < 5 ms.
Proposed partial credit
30% spec conformance; 40% bucket correctness; 20% cross-replica consistency; 10% latency
R02.01.02.01-T04 Build a gRPC key-value store from a protobuf service definition
Original instruction
Using `/work/kvgrpc/kv.proto` (Get, Put, Delete, Watch streaming, CompareAndSwap), implement a server in Go with persistence to a write-ahead log so data survives restart, and generate client stubs. Ship a Makefile with `make run`.
- Platforms
lin· Linuxmac· macOS- Applications
- go · protoc · grpcurl
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden Go client tests: CRUD and CAS semantics; Watch delivers ordered events; kill -9 then restart preserves acknowledged writes (500 writes); proto unchanged (hash).
Proposed partial credit
35% CRUD+CAS; 25% Watch; 30% durability after crash; 10% build target
R02.01.02.01-T05 Implement a URL shortener service with analytics per spec
Original instruction
Implement the service in `/work/shorty/SPEC.md`: create short links (custom alias optional, collision-safe), 301/302 redirect per link setting, expiring links, per-link click counts by day, and an admin API with token auth. Python or TypeScript; SQLite persistence; Docker image.
- Platforms
lin· Linux- Applications
- Docker · SQLite
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden HTTP test suite (60 cases): alias rules, redirect codes, expiry (clock injected via header in test mode), analytics counts after 1,000 clicks, auth on admin routes; data persists across container restart.
Proposed partial credit
30% create/alias; 20% redirects+expiry; 30% analytics; 20% auth + persistence
R02.01.02.01-T06 Implement a Redis-protocol server supporting a command subset
Original instruction
In `/work/minikv` (C++ or Rust skeleton), implement a TCP server speaking RESP2 that supports GET, SET (EX/PX/NX/XX), DEL, EXISTS, INCR, EXPIRE, TTL, LPUSH, RPOP, LRANGE, HSET, HGETALL with exact Redis reply semantics, handling pipelining and 1,000 concurrent clients.
- Platforms
lin· Linux- Applications
- redis-cli · cargo · CMake
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Hidden differential tests replay 5,000 random command sequences against real Redis 7 and compare replies byte-for-byte; wrong-type errors exact; redis-benchmark with 1,000 clients no errors; TTL expiry within 10 ms.
Proposed partial credit
50% differential reply match; 20% expiry semantics; 20% concurrency/pipelining; 10% error strings
R02.01.02.01-T07 Build a CommonMark-compliant Markdown renderer
Original instruction
Implement `render(markdown) -> html` in `/work/cmark-ts` (TypeScript skeleton) conforming to CommonMark 0.31, no Markdown libraries allowed. Include an extension flag for GFM tables only.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Node · vitest
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
CommonMark spec tests (652 examples) via normalized HTML comparison: score = pass fraction; GFM table examples pass when flag on and are literal text when off; package.json has no markdown deps; 1 MB doc renders < 1 s.
Proposed partial credit
70% spec examples (scaled); 20% tables; 10% performance
R02.01.02.01-T08 Implement RFC 6902 JSON Patch and RFC 7386 Merge Patch
Original instruction
In `/work/jsonpatch-ts`, implement `applyPatch`, `createPatch` (diff producing a valid minimal-ish patch) and `mergePatch` per RFC 6902/7386 with atomic application (all-or-nothing) and JSON Pointer escaping. TypeScript with exported types.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Node · vitest
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
json-patch-tests conformance suite passes; atomicity tests (failed test op leaves doc unchanged); createPatch output applied to source equals target on 1,000 random pairs; merge patch RFC appendix cases pass.
Proposed partial credit
40% conformance; 20% atomicity; 25% createPatch round-trip; 15% merge patch
R02.01.02.01-T09 Implement a cron expression parser and next-run calculator
Original instruction
Implement `/work/cronlib` (Python) supporting 5- and 6-field cron syntax with ranges, steps, lists, names (JAN, MON), `L`, `W`, `#` and `@hourly`-style macros, returning next/previous run times in any IANA time zone with correct DST handling as described in `SPEC.md`.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Python · pytest
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden tests (900 cases) including DST gaps/overlaps in 6 zones per SPEC rules; invalid expressions raise with position; previous/next symmetry property holds on 10k random expressions.
Proposed partial credit
40% basic syntax; 25% special chars L/W/#; 25% DST rules; 10% errors
R02.01.02.01-T10 Implement a concurrent LRU cache with TTL in Go
Original instruction
Implement package `/work/lrucache` (Go) per `SPEC.md`: generic `Cache[K, V]` with capacity, per-entry TTL, `GetOrLoad` with single-flight loading, eviction callbacks, and safe concurrent use. Visible tests cover basics; hidden tests stress concurrency.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- go
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Hidden tests with -race: LRU order, TTL expiry via injected clock, single-flight (loader called once under 100 concurrent callers), eviction callbacks exactly once; benchmark Get >= 5M ops/s on grader box.
Proposed partial credit
35% LRU+TTL; 30% single-flight; 20% callbacks; 15% race-free + perf
R02.01.02.01-T11 Implement a sandbox OAuth 2.1 authorization server with PKCE
Original instruction
Implement the authorization server in `/work/authz/SPEC.md`: authorization code flow with mandatory PKCE (S256), refresh-token rotation with reuse detection, client credentials, token introspection (RFC 7662) and revocation (RFC 7009), JWT access tokens signed with rotating keys exposed at JWKS. Test clients only.
- Platforms
lin· Linux- Applications
- Docker · OAuth test clients
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Hidden conformance suite (OpenID-style test plans adapted): 70 flow tests; security tests: code reuse rejected, PKCE downgrade rejected, refresh reuse revokes family, redirect URI exact match; JWKS rotation without breaking valid tokens.
Proposed partial credit
40% flows; 40% security tests; 20% introspection/revocation/JWKS
R02.01.02.01-T12 Stand up a mailing-list service with SMTP delivery from a spec
Original instruction
In a bare Debian container `/work/maillist`, set up a mailing-list service per `SPEC.md`: lists `announce` (moderated, only owner posts) and `team` (members post), subscription by email command with confirmation, unsubscribe, and archives as mbox, using Postfix plus Mailman 3 or your own implementation.
- Platforms
lin· Linux- Applications
- Postfix · Mailman 3 · Docker
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Hidden SMTP-driven tests: join/confirm/leave flows; posts from non-members held/rejected per list; delivery to all members (local mailboxes); archive mbox contains posted messages; service survives container restart.
Proposed partial credit
30% subscription flows; 30% posting policies; 25% delivery; 15% archives + restart
R02.01.02.01-T13 Implement a SQLite-backed todo CLI to an exact output spec
Original instruction
Implement `todo` in `/work/todo-cli` (any language, single binary or script) per `SPEC.md`: add/list/done/undo/edit/tag/search commands, due-date parsing ('tomorrow', 'next fri'), sort orders and exact output formats including colors only when stdout is a TTY. Data in SQLite at `$TODO_DB`.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- SQLite
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Hidden golden tests (120 command transcripts with a fixed clock via env) compare stdout/stderr/exit codes exactly; TTY tests via pty check ANSI codes; DB schema migrates from v1 fixture DB.
Proposed partial credit
60% transcripts (scaled); 15% TTY coloring; 15% date parsing; 10% DB migration
R02.01.02.01-T14 Implement a double-entry ledger service with concurrency invariants
Original instruction
Implement the service in `/work/ledger/SPEC.md`: accounts, multi-leg journal entries that must balance per currency, idempotency keys, balance queries as of a timestamp, and no negative balances on accounts flagged `no_overdraft`, correct under concurrent requests. Postgres; any language.
- Platforms
lin· Linux- Applications
- Postgres · Docker
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Hidden tests: 50k random concurrent transfers (32 workers) then invariants: per-currency sum zero, no overdraft violations, idempotent replays create no duplicates, point-in-time balances match an oracle.
Proposed partial credit
30% API conformance; 30% balance invariant under concurrency; 20% idempotency; 20% point-in-time queries
R02.01.02.01-T15 Implement a backtracking-free regular expression engine subset
Original instruction
In `/work/rx` (C or Rust skeleton), implement a regex engine supporting literals, classes, anchors, `* + ? {m,n}`, alternation, groups with captures and non-greedy quantifiers, guaranteeing linear time in input length (Thompson NFA / Pike VM). API per `rx.h` or `lib.rs`.
- Platforms
lin· Linuxmac· macOS- Applications
- gcc · cargo
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Hidden differential tests vs RE2 on 20k random pattern/input pairs (match + capture spans); pathological `(a*)*b` on 100k chars completes < 100 ms; ASan clean.
Proposed partial credit
50% match correctness; 25% captures; 25% linear-time guarantee
R02.01.02.01-T16 Implement a REST API from an OpenAPI spec with request validation
Original instruction
Implement every operation in `/work/petclinic-api/openapi.yaml` (22 operations, nested resources, enum and pattern constraints, pagination headers) using any framework, with Postgres storage, returning RFC 7807 problem details on validation errors.
- Platforms
lin· Linux- Applications
- Docker · Postgres · schemathesis
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Schemathesis stateful run (2,000 examples) reports no failures; hidden scenario tests for business rules; invalid requests return 400/422 with problem+json; spec file unchanged.
Proposed partial credit
40% schemathesis clean; 35% scenarios; 25% validation/problem details
R02.01.02.01-T17 Implement a streaming CSV-to-Parquet converter in Rust
Original instruction
Build CLI `csv2pq` in `/work/csv2pq` per `SPEC.md`: schema inference over the first N rows with override file, type coercion rules, null tokens, dictionary encoding for low-cardinality strings, row-group size option, streaming with bounded memory.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- cargo · arrow-rs
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden tests read outputs with pyarrow: schema and values equal expected on 25 fixtures; 5 GB CSV converts with peak RSS < 512 MB; dictionary encoding present per spec; row-group sizes honored.
Proposed partial credit
40% values/schema; 20% inference rules; 25% memory bound; 15% encoding options
R02.01.02.01-T18 Implement Raft log replication that survives a partition test harness
Original instruction
In `/work/raftkv` (Go skeleton with RPC layer and test harness), implement Raft leader election, log replication, persistence and snapshotting behind the provided interfaces, and a linearizable KV on top.
- Platforms
lin· Linux- Applications
- go
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Hidden harness: election, replication, persistence and snapshot tests, each run 50 times with -race; linearizability check (porcupine) on KV histories under partitions/crashes; any failure in a group zeroes that group.
Proposed partial credit
20% election; 30% replication; 20% persistence/snapshot; 30% linearizable KV
R02.01.02.01-T19 Implement a WebSocket chat server with rooms and history
Original instruction
Implement the protocol in `/work/chat/PROTOCOL.md`: WebSocket server with rooms, join/leave events, message history (last 100 per room) on join, typing indicators, rate limit 10 msgs/10 s per user, and graceful reconnect with missed-message replay by sequence number.
- Platforms
lin· Linux- Applications
- Node · Docker
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden multi-client tests (500 clients): message ordering per room by seq, history on join, rate-limit error frames, reconnect replay with no gaps/duplicates; protocol frames validated against JSON schemas.
Proposed partial credit
30% rooms/events; 20% history; 20% rate limit; 30% reconnect replay
R02.01.02.01-T20 Implement a base64/base32/base58 C library hardened by fuzzing
Original instruction
Implement `/work/libbaseN` (C11, header `baseN.h` provided): encode/decode for base64 (std and URL-safe, with/without padding), base32 (RFC 4648) and base58 (Bitcoin alphabet), with strict-mode rejection of non-canonical input and caller-provided buffers.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- gcc · clang · libFuzzer
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP06· Code/engineering- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Hidden RFC test vectors pass; differential fuzzing vs reference implementations for 10 min finds no mismatch; ASan/UBSan clean; strict mode rejects 40 non-canonical inputs; builds with MSVC and clang -Wall -Werror.
Proposed partial credit
40% vectors; 30% fuzz differential; 20% strict mode; 10% portable build
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.