UpSkillZone

Incident postmortem · INC-2026-002

Talent search returned stale rankings

SEV-3·resolved

Index refresh job skipped a cycle; rankings reflected scores up to 3h old. No incorrect verifications issued.

Started

Feb 3, 2026, 09:41 AM UTC

Resolved

Feb 3, 2026, 11:15 AM UTC

Duration

1h 34m

Root cause

Cron worker was OOM-killed mid-cycle; the next scheduled run started cleanly but the gap was not detected until a recruiter flagged a known-stale profile.

Customer impact

Two recruiter seats viewed top-25 lists for two distinct SkillQuery records that excluded a profile updated within the prior 90 minutes.

Remediation

  • Doubled the indexer pod memory limit and set GOMEMLIMIT to match.
  • Added freshness telemetry — alert if the last-completed-cycle age > 75 minutes.
  • Documented the recovery playbook step for catching up missed indexer cycles.