Incident postmortem · INC-2026-005
Bias audit query log gap
SEV-3·resolved
Bias audit telemetry dropped ~3% of grader decisions due to a misconfigured sampler; backfilled from upstream logs.
Started
Apr 21, 2026, 01:55 PM UTC
Resolved
Apr 21, 2026, 03:02 PM UTC
Duration
1h 7m
Root cause
A staging-grade sampling rate of 0.97 was inadvertently promoted to production; bias_audit_query_log was missing a corresponding fraction of rows.
Customer impact
No external customer impact. The quarterly bias-audit report would have been computed against an incomplete sample if the gap had not been detected.
Remediation
- Restored production sampling rate to 1.00.
- Backfilled missing query_log rows from upstream request logs (record-by-record verifiable).
- Added a config-drift check that fails CI if the staging sampler escapes into production.