Incident Report — Configuration values lost during deployment

Incident: INC-2026-08-26
Date: 26 August 2026
Window: 15:58 – 16:26 (28 min)
Severity: Low / Medium
Status: Resolved — follow-ups complete

A zero-downtime deployment on 26 August overwrote 15 live configuration entries with empty
values for 28 minutes. Affected features fell back to their built-in defaults. No transactions,
balances or customer records were touched; all values were restored from the audit trail the
same afternoon, and both preventive guards are now in place.

---

What happened

During a routine zero-downtime deployment, a configuration step submitted a partial payload
to the configuration store. Because that endpoint replaces the stored record rather than merging
into it, every key absent from the payload was written back with an empty value instead of being
left untouched.

In total 15 configuration entries in the system were blanked. The services reading them
behaved exactly as designed for a missing value: they fell back to their compiled-in defaults.
Nothing crashed, nothing errored, and the platform stayed available throughout — which is also
why the change was not immediately visible.

---

Timeline

Time	Event
15:58	Configuration overwritten. The deployment's configuration step runs and submits the partial payload. 15 entries are written back empty. Services begin resolving those keys to their defaults.
≈16:06	Deviation detected. Altered behaviour is noticed and traced to the configuration store rather than to the deployed code. (Derived from the 20-minute time-to-resolution — confirm exact time.)
16:06 – 16:26	Values identified and restored. The affected entries are identified from the audit trail, their pre-deployment values recovered, and each one written back.
16:26	Verified and closed. The full configuration set is compared against the pre-deployment state and confirmed identical. Normal behaviour resumes for all new and refreshed sessions.

---

Impact

For roughly 25 minutes, a small number of features ran on default configuration instead of their
configured values — Sportsbook among them. Configuration is resolved when a session is
established and again on page load, so the effect was limited to customers who logged in during
the window, or who were already logged in and refreshed the site. Sessions that stayed open
kept the values they had already resolved and saw no change at all.

The practical consequence for those customers was restricted gameplay: features gated by the
blanked configuration were unavailable or reduced until the values were restored. No customer was
charged, credited, blocked or logged out as a result.

What customers experienced

• Features running on default configuration for ~25 minutes, including Sportsbook
• Limited gameplay for customers who logged in during the window
• Limited gameplay for logged-in customers who refreshed the site during the window
• No error pages, no failed requests, no forced logouts

Confirmed not affected

• ✓ No transactions affected — none failed, reversed or were lost
• ✓ No balances affected — no customer balance was altered
• ✓ No customer data affected — no record was modified or exposed
• ✓ No data loss — every configuration value was recoverable in full
• ✓ No downtime — the platform stayed available throughout

---

Root cause

The deployment step wrote configuration instead of preserving it. The payload it submitted during
rollout contained only a subset of the keys, and the write is replace-by-default rather than
merge-by-default — so the omitted keys were not skipped, they were overwritten with empty values.

Two properties turned a bad payload into a live incident: the write was accepted without any
comparison against the values already stored, and blanking a configuration value produced no
signal anywhere. An empty value is indistinguishable from a deliberately unset one, so the
services fell back silently and the platform reported itself healthy for the full 28 minutes.

---

Resolution

The affected entries were identified from the configuration audit trail, which retained the
complete pre-deployment values. Each one was restored and the resulting configuration set was
verified line by line against the pre-deployment state. Restoration took effect immediately for
new and refreshed sessions; no restart or redeployment was required.

Time to resolution: 20 minutes from detection to verified restore.

---

Follow-up actions

Both actions target the properties that made this incident possible: an unchecked write, and a
silent failure mode. Both are implemented and in place.

Action	What it prevents	Status
Validate and diff configuration before write	The deployment now compares the submitted payload against the stored configuration and refuses any write that would blank an existing value. A partial payload can no longer reach the store.	Done
Alert on configuration values becoming empty	Any transition from a set value to an empty one now raises an alert. A silent fallback becomes a detection within minutes, rather than a behavioural deviation someone has to notice.	Done