故障诊断与恢复
Incident diagnosis & remediation
Localize root cause from metrics, logs and traces and mitigate the incident.
20 条本项目编写的提案 · 分类 v1.1 · 新增基准运行 0 次。下方验证方式与部分得分规则是设计，不是执行结果。
- 典型应用
- kubectl · Prometheus · Grafana
- 小类 GUI 需求标签
CL· CLI 足够
任务提案 · 20
展开查看完整原文与建议检查方式；两种界面均保留提案英文原文。
R03.04.01.02-T01 Localize an injected fault to the responsible microservice
原始指令
The SocialNetwork microservice app in namespace social (kind cluster) has elevated errors on /compose-post. Using kubectl, Prometheus and the Jaeger traces, identify the faulty microservice and write output/answer.json {service, evidence}. Do not change anything.
- 平台
lin· Linux- 应用
- kubectl · Prometheus · Jaeger
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP12· 答案 / 决策- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Service name exact vs the injected fault. Grader checks no mutating kubectl calls were made (audit log). Evidence must reference a metric or trace span from the faulty service (string match on service and metric/span name).
建议部分得分
70% correct service; 20% valid evidence; 10% no mutations
R03.04.01.02-T02 Identify the fault type and layer of an injected failure
原始指令
In the 'hotel' microservice app (namespace hotel), users see intermittent 5xx and timeouts. Determine the root cause as {component, layer: app|network|storage|config, fault_type} using metrics, logs and traces, and write output/rca.json.
- 平台
lin· Linux- 应用
- kubectl · Prometheus · Loki · Jaeger
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP12· 答案 / 决策- 条件
- 未声明
- 提案难度
- 4 / 5
建议验证方式
Gold: injected network delay + packet loss on the geo service's pod via chaos tool. Component, layer and fault type each scored exactly; fault type accepts synonyms from a fixed list.
建议部分得分
40% component; 30% layer; 30% fault type
R03.04.01.02-T03 Mitigate a pod failure and network partition to restore health
原始指令
Alerts show the 'cart' service unavailable in namespace shop: one replica is crash-looping and another is partitioned by a chaos experiment. Restore the service to healthy (all SLO probes green for 5 minutes) without deleting the namespace or the chaos operator.
- 平台
lin· Linux- 应用
- kubectl · Prometheus · Chaos Mesh
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Grader checks synthetic probe success ≥99% over 5 min after the agent finishes, all cart pods Ready, chaos experiment paused/removed, and namespace and operator still present.
建议部分得分
50% SLO probes green; 25% chaos removed correctly; 25% pods healthy
R03.04.01.02-T04 Roll back a bad configuration rollout behind a checkout error spike
原始指令
Error rate on checkout jumped from 0.2% to 14% at 10:42. Find the change responsible (deployment revisions, ConfigMap history in git repo deploy/, events) and roll it back so the error rate returns below 0.5%, then record the change id in output/answer.txt.
- 平台
lin· Linux- 应用
- kubectl · Prometheus · git
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Seeded cause: ConfigMap change setting PAYMENT_TIMEOUT_MS=50. Grader: error rate <0.5% over 5 min; ConfigMap value restored in the cluster and in git; answer names the commit SHA.
建议部分得分
50% error rate recovered; 25% fixed in git and cluster; 25% correct change identified
R03.04.01.02-T05 Diagnose OOMKilled restarts from memory metrics
原始指令
Pods of 'report-gen' restart every ~40 minutes with OOMKilled. Use Prometheus memory metrics and the service's /debug endpoint to determine whether it is a leak or under-provisioning, apply a mitigation, and write output/rca.md with the cause and the evidence.
- 平台
lin· Linux- 应用
- kubectl · Prometheus
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Seeded: unbounded in-memory cache (leak) triggered by a cron payload. Grader checks rca.md classifies 'leak' and names the cache; mitigation keeps pods alive ≥60 min in the accelerated replay (cache size env var bound set, not just a bigger limit).
建议部分得分
35% correct classification and component; 45% pods stable for the window; 20% mitigation not just raising limits
R03.04.01.02-T06 Free disk space on a full node without losing data
原始指令
Node worker-2 hit DiskPressure and evicted pods. Free disk space safely (rotated logs and dangling images only), fix the container log rotation settings in the kubelet config, and confirm the node is Ready with no DiskPressure. Do not delete PVC data under /var/lib/data.
- 平台
lin· Linux- 应用
- kubectl · ssh · crictl
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
risk· 高风险 / 不可逆- 提案难度
- 2 / 5
建议验证方式
Grader: node Ready, DiskPressure False; kubelet containerLogMaxSize/MaxFiles set; checksum of /var/lib/data unchanged; free space ≥30%.
建议部分得分
35% node healthy; 25% log rotation configured; 30% protected data intact (gate); 10% free-space target
R03.04.01.02-T07 Find a latency regression's slow span in distributed traces
原始指令
After deploy v2.14 the /orders p95 rose from 180 ms to 1.9 s. Use Jaeger traces and service metrics to find the span and code path responsible (repo in src/), fix it, redeploy, and report in output/rca.json.
- 平台
lin· Linux- 应用
- Jaeger · Prometheus · kubectl
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
- 未声明
- 提案难度
- 4 / 5
建议验证方式
Seeded: N+1 queries in order-items loader. Grader checks rca.json span name and file/function; after redeploy p95 ≤250 ms under the load script; response bodies unchanged (contract tests).
建议部分得分
35% correct span/code path; 45% latency recovered; 20% contracts intact
R03.04.01.02-T08 Resolve intermittent DNS lookup failures in a cluster
原始指令
Roughly 3% of outbound requests fail with 'no such host' in namespace api. Investigate CoreDNS metrics and logs, node conntrack, and pod dnsConfig, fix the cause, and keep it fixed under the provided load test.
- 平台
lin· Linux- 应用
- kubectl · Prometheus · CoreDNS
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
- 未声明
- 提案难度
- 4 / 5
建议验证方式
Seeded: CoreDNS replicas=1 with low max_concurrent plus ndots:5 causing search-domain amplification. Grader: DNS error rate <0.05% during load; explanation in output/rca.md names both factors.
建议部分得分
50% DNS errors resolved under load; 30% both causes named; 20% no unrelated config churn
R03.04.01.02-T09 Renew an expired certificate breaking TLS traffic
原始指令
Clients get TLS errors on api.shop.local. Find the expired certificate, fix the cert-manager Issuer that stopped renewing (local CA in the cluster), force reissue, and confirm the ingress serves a valid chain.
- 平台
lin· Linux- 应用
- kubectl · cert-manager · openssl
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
- 未声明
- 提案难度
- 2 / 5
建议验证方式
Grader: openssl s_client shows a certificate valid for ≥60 days chained to the local CA; Certificate resource Ready; Issuer error fixed (wrong secret name); renewBefore set.
建议部分得分
40% valid cert served; 30% issuer fixed; 20% Certificate Ready; 10% renewBefore
R03.04.01.02-T10 Fix database connection-pool exhaustion in production
原始指令
The 'accounts' service shows 'too many clients' errors from Postgres during peak. Determine whether a connection leak or pool sizing is responsible using pg_stat_activity, app metrics and logs, fix the code or config, and prove the fix under the peak replay.
- 平台
lin· Linux- 应用
- kubectl · PostgreSQL · Prometheus
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
- 未声明
- 提案难度
- 4 / 5
建议验证方式
Seeded: transactions not closed on an error path plus pool size × replicas > max_connections. Grader: replay shows 0 connection errors, idle-in-transaction count ≤2, both causes in rca.md.
建议部分得分
50% zero errors under replay; 30% leak fixed in code; 20% sizing fixed
R03.04.01.02-T11 Stop a retry storm causing a cascading failure
原始指令
A brief outage in 'inventory' triggered a cascade: callers retry aggressively and now everything is down even though inventory is healthy. Restore service and prevent recurrence by configuring retry budgets, timeouts and circuit breaking in the Istio mesh, then pass the fault-injection test.
- 平台
lin· Linux- 应用
- kubectl · Istio · Prometheus
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
long· 长流程- 提案难度
- 5 / 5
建议验证方式
Grader: system recovers (SLO probes ≥99%) within 10 min; then injects a 60 s inventory outage: upstream error rate during outage ≤ spec and recovery ≤30 s after it ends; DestinationRule/VirtualService settings present.
建议部分得分
35% initial recovery; 40% fault-injection test; 25% resilience config
R03.04.01.02-T12 Clear growing consumer lag caused by a poison message
原始指令
Consumer group 'ledger-writer' lag grows on topic payments (12 partitions), but only on one partition. Find the cause, move the bad record to the dead-letter topic per runbook docs/dlq.md, and get lag back under 1,000 without skipping valid messages.
- 平台
lin· Linux- 应用
- Kafka · kubectl · Prometheus
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
risk· 高风险 / 不可逆- 提案难度
- 4 / 5
建议验证方式
Grader: lag <1,000 on all partitions; exactly the 1 poison record (offset known) in payments.dlq; ledger DB contains all other messages (count and checksum); no offset reset beyond the poison record.
建议部分得分
30% cause identified; 30% lag recovered; 40% no valid messages lost (gate)
R03.04.01.02-T13 Triage five firing alerts and identify the real incident
原始指令
Five alerts are firing in the Grafana sandbox Alerting UI. Determine which are symptoms of one real incident, which are noise (flapping thresholds, stale data), and which one is the root alert. Silence the noise for 2 h with a comment and write output/triage.json.
- 平台
web· 任意系统（网页）- 应用
- Grafana
- 交互表面
brow· 浏览器- GUI 需求 · 任务设定
HY· 混合- GUI 需求 · 普通权限
G2· GUI 必要- 交互原语
click· 指向与点击vis· 视觉判断- 工作产物
WP12· 答案 / 决策- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Gold classification per alert; root alert exact. Grader reads Alertmanager silences: matchers cover exactly the noise alerts, duration 2 h ±5 min, comment present. Silencing the real incident fails.
建议部分得分
40% classification; 25% root alert; 25% correct silences; 10% comments
R03.04.01.02-T14 Write a postmortem timeline from logs and alerts
原始指令
Using incident/ (Loki log export, Alertmanager history, deploy log, chat transcript), write output/postmortem.md following the template: timeline in UTC, detection time, impact window and affected requests, root cause, contributing factors, and 3 action items.
- 平台
lin· Linuxmac· macOSwin· Windows- 应用
- Loki · Prometheus
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP01· 文档- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Grader parses the timeline table: key events with timestamps ±1 min of gold; impact window and affected-request count exact (computed from logs); root cause category exact. LLM rubric (20%) for action-item quality.
建议部分得分
40% timeline; 20% impact metrics; 20% root cause; 20% action items rubric
R03.04.01.02-T15 Bring a NotReady node back into service
原始指令
Node worker-3 is NotReady. Diagnose via kubectl describe, journalctl on the node (ssh), and kubelet logs; fix the cause, and confirm workloads reschedule there.
- 平台
lin· Linux- 应用
- kubectl · ssh · systemd
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Seeded: kubelet client certificate expired after clock skew plus disabled chrony. Grader: node Ready for 10 min, time synced (chronyc tracking offset <100 ms), new pods schedulable on it; rca.md names both factors.
建议部分得分
50% node Ready; 25% time sync fixed; 25% causes named
R03.04.01.02-T16 Stop an HPA from thrashing replicas
原始指令
Deployment 'thumbnails' oscillates between 2 and 20 replicas every few minutes, causing cold-start errors. Determine why from HPA events and metrics and fix requests, metrics and behavior settings so replicas stay stable under the steady-load replay.
- 平台
lin· Linux- 应用
- kubectl · Prometheus
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
- 未声明
- 提案难度
- 3 / 5
建议验证方式
Seeded: CPU request 10m (tiny) and no scale-down stabilization. Grader: during 20-min replay replica count changes ≤4 times, error rate <0.5%, p95 within SLO.
建议部分得分
40% stability; 30% error rate; 30% root cause in rca.md
R03.04.01.02-T17 Find and stop a runaway job behind a cost spike
原始指令
The cloud cost dashboard export (finops/daily_costs.csv) shows a spike starting 2025-10-11. Using the cost data and cluster state, find the workload responsible, stop it safely (suspend, do not delete its namespace), and estimate the excess cost in output/answer.json.
- 平台
lin· Linux- 应用
- kubectl · pandas
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP12· 答案 / 决策- 条件
- 未声明
- 提案难度
- 2 / 5
建议验证方式
Workload exact (a CronJob with concurrencyPolicy Allow spawning parallel GPU jobs); CronJob suspended; namespace intact; excess cost estimate within ±5% of gold.
建议部分得分
35% correct workload; 30% safely suspended; 35% cost estimate
R03.04.01.02-T18 Contain a cryptomining pod and find how it got in
原始指令
Prometheus shows sustained 100% CPU in namespace tools from an unknown process. Contain it (isolate with a NetworkPolicy, then remove the workload), find the entry point from audit logs and service exposure, close it, and preserve evidence in output/evidence/ (pod spec, logs, image digest).
- 平台
lin· Linux- 应用
- kubectl · Prometheus
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
risk· 高风险 / 不可逆- 提案难度
- 4 / 5
建议验证方式
Seeded: an exposed unauthenticated dashboard with a service account allowed to create pods. Grader: miner gone, egress isolated before deletion (audit ordering), evidence files present with correct digest, RBAC binding removed and service no longer exposed.
建议部分得分
25% containment order; 20% removal; 25% entry point closed; 30% evidence preserved
R03.04.01.02-T19 Diagnose a cross-region partial outage from routing data
原始指令
Users in EU see 30% failures while US is fine. The app runs in two clusters behind a global load balancer (Envoy) with health checks. Find the misroute or health-check misconfiguration from metrics and configs, fix it, and verify EU success rate ≥99.5%.
- 平台
lin· Linux- 应用
- kubectl · Envoy · Prometheus
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
- 未声明
- 提案难度
- 5 / 5
建议验证方式
Seeded: EU health check targets the wrong port, so a dead backend stays in rotation, plus a stale locality weight. Grader: EU success ≥99.5% for 10 min under synthetic traffic; both config defects fixed.
建议部分得分
50% EU success restored; 25% health check fixed; 25% weights fixed
R03.04.01.02-T20 Untangle two simultaneous faults where one masks the other
原始指令
During a game-day, the payments path shows elevated latency and occasional data mismatches. Two independent faults were injected; fixing the obvious one reveals the second. Identify and remediate both, document the sequence in output/rca.md, and pass the end-to-end consistency check.
- 平台
lin· Linux- 应用
- kubectl · Prometheus · Jaeger · PostgreSQL
- 交互表面
term· 终端- GUI 需求 · 任务设定
CL· CLI 足够- GUI 需求 · 普通权限
CL· CLI 足够- 交互原语
code· 代码与命令- 工作产物
WP08· 应用 / 系统状态- 条件
long· 长流程- 提案难度
- 5 / 5
建议验证方式
Seeded: (1) Redis eviction policy causing cache misses → latency; (2) replica lag making reads stale → mismatches. Grader: both named; latency SLO met; consistency checker shows 0 mismatches over 10 min; no data loss.
建议部分得分
25% fault 1 identified/fixed; 35% fault 2 identified/fixed; 30% consistency check; 10% rca sequence
已有研究映射
以下沿用研究目录中的基准与小类映射，不是逐题标注、执行记录或产品背书。