Incident diagnosis & remediation
Localize root cause from metrics, logs and traces and mitigate the incident.
20 project-authored proposals · taxonomy v1.1 · 0 new benchmark runs. Verifiers and partial-credit rules below are designs, not executed results.
- Typical applications
- kubectl · Prometheus · Grafana
- Class GUI-need label
CL· CLI-sufficient
Task proposals · 20
Expand to inspect the complete original wording and proposed checks. Original proposal text remains English in both interfaces.
R03.04.01.02-T01 Localize an injected fault to the responsible microservice
Original instruction
The SocialNetwork microservice app in namespace social (kind cluster) has elevated errors on /compose-post. Using kubectl, Prometheus and the Jaeger traces, identify the faulty microservice and write output/answer.json {service, evidence}. Do not change anything.
- Platforms
lin· Linux- Applications
- kubectl · Prometheus · Jaeger
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Service name exact vs the injected fault. Grader checks no mutating kubectl calls were made (audit log). Evidence must reference a metric or trace span from the faulty service (string match on service and metric/span name).
Proposed partial credit
70% correct service; 20% valid evidence; 10% no mutations
R03.04.01.02-T02 Identify the fault type and layer of an injected failure
Original instruction
In the 'hotel' microservice app (namespace hotel), users see intermittent 5xx and timeouts. Determine the root cause as {component, layer: app|network|storage|config, fault_type} using metrics, logs and traces, and write output/rca.json.
- Platforms
lin· Linux- Applications
- kubectl · Prometheus · Loki · Jaeger
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Gold: injected network delay + packet loss on the geo service's pod via chaos tool. Component, layer and fault type each scored exactly; fault type accepts synonyms from a fixed list.
Proposed partial credit
40% component; 30% layer; 30% fault type
R03.04.01.02-T03 Mitigate a pod failure and network partition to restore health
Original instruction
Alerts show the 'cart' service unavailable in namespace shop: one replica is crash-looping and another is partitioned by a chaos experiment. Restore the service to healthy (all SLO probes green for 5 minutes) without deleting the namespace or the chaos operator.
- Platforms
lin· Linux- Applications
- kubectl · Prometheus · Chaos Mesh
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader checks synthetic probe success ≥99% over 5 min after the agent finishes, all cart pods Ready, chaos experiment paused/removed, and namespace and operator still present.
Proposed partial credit
50% SLO probes green; 25% chaos removed correctly; 25% pods healthy
R03.04.01.02-T04 Roll back a bad configuration rollout behind a checkout error spike
Original instruction
Error rate on checkout jumped from 0.2% to 14% at 10:42. Find the change responsible (deployment revisions, ConfigMap history in git repo deploy/, events) and roll it back so the error rate returns below 0.5%, then record the change id in output/answer.txt.
- Platforms
lin· Linux- Applications
- kubectl · Prometheus · git
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Seeded cause: ConfigMap change setting PAYMENT_TIMEOUT_MS=50. Grader: error rate <0.5% over 5 min; ConfigMap value restored in the cluster and in git; answer names the commit SHA.
Proposed partial credit
50% error rate recovered; 25% fixed in git and cluster; 25% correct change identified
R03.04.01.02-T05 Diagnose OOMKilled restarts from memory metrics
Original instruction
Pods of 'report-gen' restart every ~40 minutes with OOMKilled. Use Prometheus memory metrics and the service's /debug endpoint to determine whether it is a leak or under-provisioning, apply a mitigation, and write output/rca.md with the cause and the evidence.
- Platforms
lin· Linux- Applications
- kubectl · Prometheus
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Seeded: unbounded in-memory cache (leak) triggered by a cron payload. Grader checks rca.md classifies 'leak' and names the cache; mitigation keeps pods alive ≥60 min in the accelerated replay (cache size env var bound set, not just a bigger limit).
Proposed partial credit
35% correct classification and component; 45% pods stable for the window; 20% mitigation not just raising limits
R03.04.01.02-T06 Free disk space on a full node without losing data
Original instruction
Node worker-2 hit DiskPressure and evicted pods. Free disk space safely (rotated logs and dangling images only), fix the container log rotation settings in the kubelet config, and confirm the node is Ready with no DiskPressure. Do not delete PVC data under /var/lib/data.
- Platforms
lin· Linux- Applications
- kubectl · ssh · crictl
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 2 / 5
Proposed verifier
Grader: node Ready, DiskPressure False; kubelet containerLogMaxSize/MaxFiles set; checksum of /var/lib/data unchanged; free space ≥30%.
Proposed partial credit
35% node healthy; 25% log rotation configured; 30% protected data intact (gate); 10% free-space target
R03.04.01.02-T07 Find a latency regression's slow span in distributed traces
Original instruction
After deploy v2.14 the /orders p95 rose from 180 ms to 1.9 s. Use Jaeger traces and service metrics to find the span and code path responsible (repo in src/), fix it, redeploy, and report in output/rca.json.
- Platforms
lin· Linux- Applications
- Jaeger · Prometheus · kubectl
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Seeded: N+1 queries in order-items loader. Grader checks rca.json span name and file/function; after redeploy p95 ≤250 ms under the load script; response bodies unchanged (contract tests).
Proposed partial credit
35% correct span/code path; 45% latency recovered; 20% contracts intact
R03.04.01.02-T08 Resolve intermittent DNS lookup failures in a cluster
Original instruction
Roughly 3% of outbound requests fail with 'no such host' in namespace api. Investigate CoreDNS metrics and logs, node conntrack, and pod dnsConfig, fix the cause, and keep it fixed under the provided load test.
- Platforms
lin· Linux- Applications
- kubectl · Prometheus · CoreDNS
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Seeded: CoreDNS replicas=1 with low max_concurrent plus ndots:5 causing search-domain amplification. Grader: DNS error rate <0.05% during load; explanation in output/rca.md names both factors.
Proposed partial credit
50% DNS errors resolved under load; 30% both causes named; 20% no unrelated config churn
R03.04.01.02-T09 Renew an expired certificate breaking TLS traffic
Original instruction
Clients get TLS errors on api.shop.local. Find the expired certificate, fix the cert-manager Issuer that stopped renewing (local CA in the cluster), force reissue, and confirm the ingress serves a valid chain.
- Platforms
lin· Linux- Applications
- kubectl · cert-manager · openssl
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Grader: openssl s_client shows a certificate valid for ≥60 days chained to the local CA; Certificate resource Ready; Issuer error fixed (wrong secret name); renewBefore set.
Proposed partial credit
40% valid cert served; 30% issuer fixed; 20% Certificate Ready; 10% renewBefore
R03.04.01.02-T10 Fix database connection-pool exhaustion in production
Original instruction
The 'accounts' service shows 'too many clients' errors from Postgres during peak. Determine whether a connection leak or pool sizing is responsible using pg_stat_activity, app metrics and logs, fix the code or config, and prove the fix under the peak replay.
- Platforms
lin· Linux- Applications
- kubectl · PostgreSQL · Prometheus
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 4 / 5
Proposed verifier
Seeded: transactions not closed on an error path plus pool size × replicas > max_connections. Grader: replay shows 0 connection errors, idle-in-transaction count ≤2, both causes in rca.md.
Proposed partial credit
50% zero errors under replay; 30% leak fixed in code; 20% sizing fixed
R03.04.01.02-T11 Stop a retry storm causing a cascading failure
Original instruction
A brief outage in 'inventory' triggered a cascade: callers retry aggressively and now everything is down even though inventory is healthy. Restore service and prevent recurrence by configuring retry budgets, timeouts and circuit breaking in the Istio mesh, then pass the fault-injection test.
- Platforms
lin· Linux- Applications
- kubectl · Istio · Prometheus
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Grader: system recovers (SLO probes ≥99%) within 10 min; then injects a 60 s inventory outage: upstream error rate during outage ≤ spec and recovery ≤30 s after it ends; DestinationRule/VirtualService settings present.
Proposed partial credit
35% initial recovery; 40% fault-injection test; 25% resilience config
R03.04.01.02-T12 Clear growing consumer lag caused by a poison message
Original instruction
Consumer group 'ledger-writer' lag grows on topic payments (12 partitions), but only on one partition. Find the cause, move the bad record to the dead-letter topic per runbook docs/dlq.md, and get lag back under 1,000 without skipping valid messages.
- Platforms
lin· Linux- Applications
- Kafka · kubectl · Prometheus
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Grader: lag <1,000 on all partitions; exactly the 1 poison record (offset known) in payments.dlq; ledger DB contains all other messages (count and checksum); no offset reset beyond the poison record.
Proposed partial credit
30% cause identified; 30% lag recovered; 40% no valid messages lost (gate)
R03.04.01.02-T13 Triage five firing alerts and identify the real incident
Original instruction
Five alerts are firing in the Grafana sandbox Alerting UI. Determine which are symptoms of one real incident, which are noise (flapping thresholds, stale data), and which one is the root alert. Silence the noise for 2 h with a comment and write output/triage.json.
- Platforms
web· Any OS (web)- Applications
- Grafana
- Interfaces
brow· Browser- GUI need · task setting
HY· Hybrid- GUI need · ordinary access
G2· GUI-essential- Interaction primitives
click· Point & clickvis· Visual judgment- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Gold classification per alert; root alert exact. Grader reads Alertmanager silences: matchers cover exactly the noise alerts, duration 2 h ±5 min, comment present. Silencing the real incident fails.
Proposed partial credit
40% classification; 25% root alert; 25% correct silences; 10% comments
R03.04.01.02-T14 Write a postmortem timeline from logs and alerts
Original instruction
Using incident/ (Loki log export, Alertmanager history, deploy log, chat transcript), write output/postmortem.md following the template: timeline in UTC, detection time, impact window and affected requests, root cause, contributing factors, and 3 action items.
- Platforms
lin· Linuxmac· macOSwin· Windows- Applications
- Loki · Prometheus
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP01· Document- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Grader parses the timeline table: key events with timestamps ±1 min of gold; impact window and affected-request count exact (computed from logs); root cause category exact. LLM rubric (20%) for action-item quality.
Proposed partial credit
40% timeline; 20% impact metrics; 20% root cause; 20% action items rubric
R03.04.01.02-T15 Bring a NotReady node back into service
Original instruction
Node worker-3 is NotReady. Diagnose via kubectl describe, journalctl on the node (ssh), and kubelet logs; fix the cause, and confirm workloads reschedule there.
- Platforms
lin· Linux- Applications
- kubectl · ssh · systemd
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Seeded: kubelet client certificate expired after clock skew plus disabled chrony. Grader: node Ready for 10 min, time synced (chronyc tracking offset <100 ms), new pods schedulable on it; rca.md names both factors.
Proposed partial credit
50% node Ready; 25% time sync fixed; 25% causes named
R03.04.01.02-T16 Stop an HPA from thrashing replicas
Original instruction
Deployment 'thumbnails' oscillates between 2 and 20 replicas every few minutes, causing cold-start errors. Determine why from HPA events and metrics and fix requests, metrics and behavior settings so replicas stay stable under the steady-load replay.
- Platforms
lin· Linux- Applications
- kubectl · Prometheus
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 3 / 5
Proposed verifier
Seeded: CPU request 10m (tiny) and no scale-down stabilization. Grader: during 20-min replay replica count changes ≤4 times, error rate <0.5%, p95 within SLO.
Proposed partial credit
40% stability; 30% error rate; 30% root cause in rca.md
R03.04.01.02-T17 Find and stop a runaway job behind a cost spike
Original instruction
The cloud cost dashboard export (finops/daily_costs.csv) shows a spike starting 2025-10-11. Using the cost data and cluster state, find the workload responsible, stop it safely (suspend, do not delete its namespace), and estimate the excess cost in output/answer.json.
- Platforms
lin· Linux- Applications
- kubectl · pandas
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP12· Answer/decision- Conditions
- None declared
- Proposed difficulty
- 2 / 5
Proposed verifier
Workload exact (a CronJob with concurrencyPolicy Allow spawning parallel GPU jobs); CronJob suspended; namespace intact; excess cost estimate within ±5% of gold.
Proposed partial credit
35% correct workload; 30% safely suspended; 35% cost estimate
R03.04.01.02-T18 Contain a cryptomining pod and find how it got in
Original instruction
Prometheus shows sustained 100% CPU in namespace tools from an unknown process. Contain it (isolate with a NetworkPolicy, then remove the workload), find the entry point from audit logs and service exposure, close it, and preserve evidence in output/evidence/ (pod spec, logs, image digest).
- Platforms
lin· Linux- Applications
- kubectl · Prometheus
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
risk· Irreversible/risky- Proposed difficulty
- 4 / 5
Proposed verifier
Seeded: an exposed unauthenticated dashboard with a service account allowed to create pods. Grader: miner gone, egress isolated before deletion (audit ordering), evidence files present with correct digest, RBAC binding removed and service no longer exposed.
Proposed partial credit
25% containment order; 20% removal; 25% entry point closed; 30% evidence preserved
R03.04.01.02-T19 Diagnose a cross-region partial outage from routing data
Original instruction
Users in EU see 30% failures while US is fine. The app runs in two clusters behind a global load balancer (Envoy) with health checks. Find the misroute or health-check misconfiguration from metrics and configs, fix it, and verify EU success rate ≥99.5%.
- Platforms
lin· Linux- Applications
- kubectl · Envoy · Prometheus
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
- None declared
- Proposed difficulty
- 5 / 5
Proposed verifier
Seeded: EU health check targets the wrong port, so a dead backend stays in rotation, plus a stale locality weight. Grader: EU success ≥99.5% for 10 min under synthetic traffic; both config defects fixed.
Proposed partial credit
50% EU success restored; 25% health check fixed; 25% weights fixed
R03.04.01.02-T20 Untangle two simultaneous faults where one masks the other
Original instruction
During a game-day, the payments path shows elevated latency and occasional data mismatches. Two independent faults were injected; fixing the obvious one reveals the second. Identify and remediate both, document the sequence in output/rca.md, and pass the end-to-end consistency check.
- Platforms
lin· Linux- Applications
- kubectl · Prometheus · Jaeger · PostgreSQL
- Interfaces
term· Terminal- GUI need · task setting
CL· CLI-sufficient- GUI need · ordinary access
CL· CLI-sufficient- Interaction primitives
code· Code & commands- Work product
WP08· App/system state- Conditions
long· Long-horizon- Proposed difficulty
- 5 / 5
Proposed verifier
Seeded: (1) Redis eviction policy causing cache misses → latency; (2) replica lag making reads stale → mismatches. Grader: both named; latency SLO met; consistency checker shows 0 mismatches over 10 min; no data loss.
Proposed partial credit
25% fault 1 identified/fixed; 35% fault 2 identified/fixed; 30% consistency check; 10% rca sequence
Mapped research references
Existing benchmark-to-class links from the research catalog. They are not record-level annotations, executions or product endorsements.