Incidents and maintenance reported on status page for QBet https://status.qbet.bg/ https://d1lppblt9t2x15.cloudfront.net/logos/6940777d334cf6981fdeec27f45585a8.png https://status.qbet.bg/ en https://status.qbet.bg/incident/1036040 Thu, 27 Aug 2026 14:50:00 -0000 https://status.qbet.bg/incident/1036040#be715f7b8cf71f65b58c5eb17aad1a38f71eb7323a5fa27a9962d46046052bfe Incident # Incident Report — Configuration values lost during deployment **Incident:** INC-2026-08-26 **Date:** 26 August 2026 **Window:** 15:58 – 16:26 (28 min) **Severity:** Low / Medium **Status:** Resolved — follow-ups complete A zero-downtime deployment on 26 August overwrote 15 live configuration entries with empty values for 28 minutes. Affected features fell back to their built-in defaults. No transactions, balances or customer records were touched; all values were restored from the audit trail the same afternoon, and both preventive guards are now in place. --- ## What happened During a routine zero-downtime deployment, a configuration step submitted a **partial payload** to the configuration store. Because that endpoint replaces the stored record rather than merging into it, every key absent from the payload was written back with an empty value instead of being left untouched. In total **15 configuration entries** in the system were blanked. The services reading them behaved exactly as designed for a missing value: they fell back to their compiled-in defaults. Nothing crashed, nothing errored, and the platform stayed available throughout — which is also why the change was not immediately visible. --- ## Timeline | Time | Event | |---|---| | **15:58** | **Configuration overwritten.** The deployment's configuration step runs and submits the partial payload. 15 entries are written back empty. Services begin resolving those keys to their defaults. | | **≈16:06** | **Deviation detected.** Altered behaviour is noticed and traced to the configuration store rather than to the deployed code. *(Derived from the 20-minute time-to-resolution — confirm exact time.)* | | **16:06 – 16:26** | **Values identified and restored.** The affected entries are identified from the audit trail, their pre-deployment values recovered, and each one written back. | | **16:26** | **Verified and closed.** The full configuration set is compared against the pre-deployment state and confirmed identical. Normal behaviour resumes for all new and refreshed sessions. | --- ## Impact For roughly 25 minutes, a small number of features ran on default configuration instead of their configured values — **Sportsbook among them**. Configuration is resolved when a session is established and again on page load, so the effect was limited to customers who **logged in during the window**, or who were **already logged in and refreshed the site**. Sessions that stayed open kept the values they had already resolved and saw no change at all. The practical consequence for those customers was **restricted gameplay**: features gated by the blanked configuration were unavailable or reduced until the values were restored. No customer was charged, credited, blocked or logged out as a result. ### What customers experienced - Features running on default configuration for ~25 minutes, including Sportsbook - Limited gameplay for customers who logged in during the window - Limited gameplay for logged-in customers who refreshed the site during the window - No error pages, no failed requests, no forced logouts ### Confirmed not affected - ✓ **No transactions affected** — none failed, reversed or were lost - ✓ **No balances affected** — no customer balance was altered - ✓ **No customer data affected** — no record was modified or exposed - ✓ **No data loss** — every configuration value was recoverable in full - ✓ **No downtime** — the platform stayed available throughout --- ## Root cause The deployment step wrote configuration instead of preserving it. The payload it submitted during rollout contained only a subset of the keys, and the write is **replace-by-default rather than merge-by-default** — so the omitted keys were not skipped, they were overwritten with empty values. Two properties turned a bad payload into a live incident: the write was accepted without any comparison against the values already stored, and blanking a configuration value produced no signal anywhere. An empty value is indistinguishable from a deliberately unset one, so the services fell back silently and the platform reported itself healthy for the full 28 minutes. --- ## Resolution The affected entries were identified from the configuration audit trail, which retained the complete pre-deployment values. Each one was restored and the resulting configuration set was verified line by line against the pre-deployment state. Restoration took effect immediately for new and refreshed sessions; no restart or redeployment was required. **Time to resolution: 20 minutes** from detection to verified restore. --- ## Follow-up actions Both actions target the properties that made this incident possible: an unchecked write, and a silent failure mode. **Both are implemented and in place.** | Action | What it prevents | Status | |---|---|---| | **Validate and diff configuration before write** | The deployment now compares the submitted payload against the stored configuration and refuses any write that would blank an existing value. A partial payload can no longer reach the store. | **Done** | | **Alert on configuration values becoming empty** | Any transition from a set value to an empty one now raises an alert. A silent fallback becomes a detection within minutes, rather than a behavioural deviation someone has to notice. | **Done** | https://status.qbet.bg/maintenance/1007994 Tue, 11 Aug 2026 03:00:00 -0000 https://status.qbet.bg/incident/1007994#1df42de104eedf3061ff1984b350f294f6fd7e51c2d7734d81b1769068e07bff Maintenance Scheduled Maintenance https://status.qbet.bg/maintenance/963243 Tue, 14 Jul 2026 04:00:47 -0000 https://status.qbet.bg/incident/963243#e811299a3a73e41f2bc90047be4f3fcdfd5c9b396d9af090ed97ed80c48c905f Maintenance Scheduled Maintenance