# Failure Architecture: How Outages Propagate

> How the AWS, Google, Azure Front Door, Cloudflare and CrowdStrike outages propagated, the five fail modes, a failure matrix and a blast-radius method.

- Web page: https://www.alesanalytics.com/architecture/failure
- Markdown: https://www.alesanalytics.com/architecture/failure.md
- Part of: [Enterprise Architecture Atlas](https://www.alesanalytics.com/architecture.md), section "Failure and design"
- Last verified: 13 October 2026
- Publisher: Ales Analytics

## The shape every outage takes

Large outages look different in the news and alike in the post-incident reports. Nearly all of them follow the same path:

1. A **trigger:** a change, a race, a malformed file.
2. A **fault:** one component misbehaves.
3. **Propagation** through a shared dependency: DNS, identity, configuration, a control plane, time.
4. **Amplification:** retries, thundering herds, caches expiring together, crash loops.
5. The **outage.**
6. **Recovery,** slowed by tooling that shares fate with what failed.

**Figure: How a small fault becomes a large outage.** 1, a trigger: a change, a race, a malformed file. 2, a fault: one component misbehaves. 3, a shared plane: DNS, identity, configuration, a control plane or time. 4, amplification: retries, herds, caches expiring together, crash loops. 5, the outage: many services, many regions. 6, slow recovery, because the tools to fix it share its fate. Steps 3 and 4 are where design makes the difference. Steps 3 and 4 are where design makes the difference. Three layers each retrying three times put up to 64 times the normal load on the bottom dependency at the moment it can least take it.

Two steps on that path are where design makes the biggest difference. Propagation is decided by what components share, which you can map in advance. Amplification is decided by retry and backoff behaviour, which you can set in advance. The rest of this page is about both.

## Five incidents, read closely

These five were chosen because each provider published enough to see the mechanism. The dates and times are in each provider's own time zone.

### AWS us-east-1, 19 and 20 October 2025

**What happened.** A latent race condition in DynamoDB's DNS automation, between the components AWS calls the Planner and the Enactor, left an empty DNS record for `dynamodb.us-east-1.amazonaws.com`. DynamoDB API errors ran from 11:48 PM to 2:40 AM PDT. Console sign-ins using an IAM user saw increased authentication failures from 11:51 PM to 1:25 AM. Dependent services failed, EC2 launches were impaired and throttled, and the event ended at 2:20 PM PDT on 20 October after three distinct periods of impact.

**A detail worth getting right.** Customers using DynamoDB global tables could still connect to their replica tables in other Regions; what they saw was prolonged replication lag. Earlier summaries said "IAM updates and global tables were affected", which is broader than the post-event summary supports.

**What it teaches.** The multi-region data plane worked. The regional control and identity paths didn't. Local credentials survived, which is the pattern called **fail-static identity**. Version-fence your DNS plan updates, and make sure recovery paths don't depend on the service that failed. AWS disabled the automation worldwide.

### Google Cloud, 12 June 2025

**What happened.** A new quota-checking code path in Service Control, added on 29 May, lacked error handling and wasn't behind a feature flag. At about 10:45 AM PDT a policy with blank fields was inserted, and its metadata "replicated globally within seconds". The blank fields hit the unprotected code, causing a null pointer and a crash loop in every region, from 10:51 to 18:18 PDT. As tasks restarted in us-central1, they overloaded the Spanner database underneath without randomised backoff, and that region took up to about 2 hours 40 minutes to recover fully.

**What it teaches.** Google's report notes that Service Control is a regional service with a regional datastore, yet the policy data replicated globally. That mismatch has a name in this atlas: **regional compute, global state**. It made a regional service a global failure domain. Google's remediation was to modularise the checks so that the functionality is isolated and fails *open*, which is right for a quota check and would be wrong for an access check. The next section explains the difference.

### Azure Front Door, 29 October 2025

**What happened.** An inadvertent tenant configuration change introduced an invalid or inconsistent state that caused a significant number of AFD nodes to fail to load properly, from 15:45 UTC to 00:05 UTC the next day. Microsoft 365 and the Azure portal were affected. The portal was failed away from AFD at 17:26, and configuration changes were blocked at 17:30 while a last-known-good configuration was rolled out.

**The deeper mechanism.** Microsoft's later engineering post explained that a different sequence of configuration changes across two control-plane versions produced incompatible metadata, and that the bad configuration passed the protection system before a delayed asynchronous processing task crashed on it. A separate AFD incident on 9 October involved the protection system being bypassed during a manual clean-up.

**What it teaches.** Control-plane *version skew* can make configuration incompatible, so configuration needs a schema version and a skew policy between producers and consumers, as Kubernetes has for its components. Validation has to exercise the real consumer, including its asynchronous stages. And management tooling must not depend on the edge it manages.

### Cloudflare, 18 November 2025

**What happened.** A change to permissions in a ClickHouse database caused a query to return duplicate rows. The Bot Management feature file generated from that query doubled in size, past a hard limit that Cloudflare's postmortem describes as set to 200, well above its normal use of about 60 features. The proxy panicked. Failures began at 11:20 UTC. Because the file was regenerated every few minutes, the network flapped between working and failing. Core traffic was largely flowing normally by 14:30, and all systems were normal by 17:06. An unrelated failure of Cloudflare's off-platform status page at the same time suggested to responders that they were under attack.

**What it teaches.** Generated configuration is configuration whose producer is a query, so a permission change upstream of it is a configuration change. Validate generated configuration for size and cardinality, not just schema. And a non-critical module must not be able to crash the critical path.

### CrowdStrike Channel File 291, 19 July 2024

**What happened.** A new IPC Template Type defined 21 input parameter fields, but the integration code supplied only 20 [1]. The Content Validator checked against 21. A content update with a non-wildcard criterion in the 21st field made the sensor read past the end of the input, an out-of-bounds memory read, crashing Windows hosts between 04:09 and 05:27 UTC. Microsoft estimated 8.5 million devices, under 1% of Windows machines; later reporting said more than 8 million. Each had to be recovered by hand.

**What it teaches.** Content needs staged rings with bake time, just as code does, and customers need control over when it rolls out. Add runtime bounds checks, and test with the non-wildcard values production will actually use.

### What the five have in common

All five were data or configuration that propagated faster than code would have been allowed to. Three of them also impaired the tools people needed to see or recover. Two (Google and Azure Front Door) involved a mismatch between where a component ran and how far its data travelled.

> **A correction to an earlier claim.**
>
> An earlier edition said all five incidents "bypassed the staged rollout applied to binaries". That was stronger than the reports support. Azure Front Door *had* a protection system, which a defect let the bad configuration through. Google's code *was* rolled out in stages; the problem was that it wasn't behind a flag. The accurate claim is the one above: configuration moved faster than code would have.

## The five ways a check can fail

Saying "authorization never fails open" sounds safe, and then you read Google's remediation, which made its checks fail open deliberately. Both are right, because they're talking about different kinds of check. Every check needs a declared failure mode, and there are five:

| Mode | Use it for | What it requires |
| --- | --- | --- |
| **Fail-closed:** deny when unsure | Access to protected data; granting privilege | A break-glass path |
| **Fail-static:** use the last known good decision or data, up to a declared age | Authorization within a staleness budget; configuration; DNS fallback | A declared maximum age |
| **Fail-open:** allow when unsure | Quota, rate limits, budgets; advisory enrichment | An alarm, a time limit and bounded exposure |
| **Fail-degraded:** serve a reduced function | AI generation falling back to search-only; read-only mode | A defined experience and SLO |
| **Fail-to-manual:** a human decides | Irreversible side effects: payments, deletions | A staffed approval path |

And checks fall into four classes:

- **Protective checks** (authentication, and authorization over protected data or side effects) fail closed, or fail static within a declared revocation window.
- **Economic and availability checks** (quota, rate limit, budget) may fail open, with alarms, because over-admission is bounded and reversible.
- **Enrichment checks** (telemetry, advisory risk scoring) fail open.
- **Safety interlocks on irreversible actions** fail to manual.

A fail-open path is itself security-relevant configuration. Version it, alarm on it, and put a time limit on it.

## The failure matrix

| Failure | How it spreads | How you detect it | How you prevent or contain it |
| --- | --- | --- | --- |
| Identity-provider outage | New sign-ins and token refreshes fail | Synthetic sign-ins | Design token lifetimes; break-glass outside Conditional Access |
| DNS failure | Every client; TTLs delay recovery | Resolution SLIs from several vantage points | Fenced automation; static fallbacks |
| Certificate expiry | Everything fails at the same moment | Expiry inventory and alerts | ACME or managed certificates |
| Database failure | Connection storms | Lag and error rates | Tested failover; idempotent clients |
| Storage corruption | Replicas copy it | Integrity checks | Versioning, object lock, separate backups |
| Region or provider failure | Global features homed in that region | External monitoring | Static stability; pre-provisioned capacity |
| Model API outage | Every AI feature | Per-provider SLIs | Policy-equivalent fallback |
| GPU or quota exhaustion | Queues grow; 429s cascade | Queue time, KV utilisation, headroom | Reservations, admission control, a smaller fallback model |
| Network partition | Split brain | Path monitoring | Quorum with fencing |
| Corrupted data | Pipelines, indexes and answers | Quality checks | Data contracts, versioned tables, rewinding along lineage |
| Ransomware | Lateral movement through shared credentials | EDR, canaries | A separate backup identity plane; immutability |
| Credential theft | The attacker acts as the identity | Anomalous Graph use | Phishing-resistant MFA; no long-lived keys |
| Prompt injection or a malicious tool | Tools act for the attacker | Tool-call anomaly audit | Authorization inside tools; approvals; signed, pinned MCP servers |
| Logging outage | Blind operations; a compliance gap | Pipeline health | Local buffering; an independent failure domain |
| Backup or restore failure | Silent until you need it | Restore drills | Rebuild from IaC and restore data, tested together |
| Configuration drift | Inconsistent security | Drift detection | GitOps; block out-of-band changes |
| Clock skew | Tokens rejected, leases broken, logs misordered | An NTP-offset SLI | Redundant time sources; skew tolerance; monotonic clocks for timeouts |
| Signing-key rotation | Mass 401s | JWKS fetch errors | Overlap windows; publish keys before use |
| Admission-webhook outage | Every write blocked | API error rate | Scoped failure policies; a webhook path outside the cluster |

## Blast radius: a method

"What would this failure take with it?" is a question you can answer before the failure, if you map the dependencies.

1. Draw the dependency graph with typed edges: *depends on* (at runtime), *authenticates as*, *resolves via* (DNS), *keyed by* (KMS or PKI), *configured by*, *observed by*, *recovered by*.
2. For a failed node F, list its **direct dependents**, one edge away.
3. Add its **indirect dependents**: the transitive closure along runtime edges.
4. Add the **shared-plane dependents**: everything that shares F's control plane, configuration producer, identity issuer, DNS zone, telemetry pipeline or backup system.
5. Subtract what is **statically stable**: dependents whose edge to F is only needed for *change*, each with a declared survival time, such as the remaining token lifetime.
6. Flag **recovery coupling**: does restoring F need F, or anything in F's blast radius?

The blast radius is everything found in steps 2 to 4, less whatever step 5 removes. It changes with time, because cached tokens, DNS answers and certificates all expire.

| Dependency | Direct dependents | Indirect and shared | Statically stable for | Recovery coupling |
| --- | --- | --- | --- | --- |
| Identity provider | Sign-ins, token refresh, workload identity exchanges | All SaaS, CI/CD, agents, admin portals | The remaining token life: about an hour by default, up to 28 hours with CAE | Break-glass must not need the identity provider or Conditional Access |
| DNS | Every call by name | Private endpoints, the identity provider, certificate checks (OCSP, CRL) | The cached TTL | Recovery tools that resolve through the same zone |
| KMS | Decryption on read; envelope keys | Databases, storage, secrets, backups | Cached data keys, depending on the provider | Deleting a key or locking its policy makes backups unreadable |
| PKI | TLS and mTLS handshakes | Service mesh, identity-provider signing, device authentication | Until certificates expire | The offline procedures for the issuing CA |
| Network transit | Cross-network and on-premises flows | Hybrid DNS, Active Directory, backups | None: it's live | An out-of-band management path |
| Database | Application reads and writes | Caches (stampedes), indexes (CDC stalls) | Read caches | Connection storms on failover |
| Object storage | Artefacts, the data lake | Image pulls, model weights, backups | Local copies | Restores need storage in another domain |
| AI gateway | Every AI feature | Agents, metering, audit | None, unless clients bypass it, which they mustn't | A last-known-good gateway configuration |
| Model provider | Generation | Agents, evaluations | Only a policy-equivalent fallback | Quota in the fallback region |
| Kubernetes control plane | Scheduling, scaling, secret updates | Operators, admission | Existing pods keep running | Webhooks that fail closed block the fix |
| CI/CD | Deployments | IaC changes, policy rollout | Running systems | The pipeline identity is Tier-0; keep a manual emergency path |
| Observability | Detection, paging | Audit evidence, SLOs | Local buffers | Independent paging and status |
| Backup | Restore | Ransomware recovery, legal hold | Not applicable: it's latent | A separate identity, keys and control plane |

## The Tier-0 set

**Figure: The Tier-0 set: components whose failure or compromise takes everything with it.** The identity provider, and every identity that can grant permissions. DNS, including the private zones behind private endpoints. KMS and PKI: keys, signing certificates, issuing CAs. The edge and CDN, and the tooling that configures it. The AI gateway. CI/CD and IaC state, which can recreate or destroy everything. The backup control plane. The time source that leases, token expiry, certificates and logs all read. These are roles, not products, so they are drawn without product icons. Map your own product against each one first: most of the propagation paths in the failure matrix run through at least one of them.

A **Tier-0** component is one whose compromise or failure compromises or fails everything else. The revised set is: the identity provider and every **permission-granting identity**; DNS; KMS and PKI; the edge and CDN; the AI gateway; CI/CD and IaC state; the backup control plane; and the **time source** that leases, tokens, certificates and logs all read. The last one is the newest addition, and the one most estates have never mapped.

Map the Tier-0 set against the failure matrix first. Most propagation paths run through it.

## Verified claims

Each numbered statement above, tied to the page that says it, the date it was last checked and the conditions it holds under.

1. CrowdStrike Channel File 291 defined 21 input fields while the integration code supplied 20, and reading the 21st caused an out-of-bounds read. Source: https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf. Checked 13 October 2026, tier T1. Holds when: Falcon sensor for Windows, version 7.11 and above, on 19 July 2024.

Should you have any questions regarding this page, please contact us at solutions@alesanalytics.com.

## Sources

1. [Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/101925/). Amazon Web Services, 2025. AWS's post-event summary for 19 and 20 October 2025: a race in DynamoDB's DNS automation left an empty record, with impact in three phases ending at 2:20 PM PDT on 20 October.
2. [Google Cloud Service Health incident report, 12 June 2025](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW). Google Cloud, 2025. A policy with blank fields, replicated globally, crashed an unflagged code path in Service Control in every region. Remediation: modularise, and fail open.
3. [Azure status history: Azure Front Door, tracking ID YKYN-BWZ](https://azure.status.microsoft/en-us/status/history/). Microsoft, 2025. Microsoft's post-incident review for 29 October 2025, 15:45 to 00:05 UTC: an invalid configuration state caused AFD nodes to fail to load, and changes were blocked while a last-known-good configuration rolled out.
4. [Cloudflare outage on November 18, 2025](https://blog.cloudflare.com/18-november-2025-outage/). Cloudflare, 2025. Cloudflare's postmortem. A database permission change doubled the Bot Management feature file past a limit of 200 features against normal use of about 60.
5. [Channel File 291 Incident: Root Cause Analysis](https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf). CrowdStrike, 2024. The external technical root cause analysis of 6 August 2024: a template type defined 21 input fields, the integration code supplied 20, and reading the 21st was an out-of-bounds memory read.

## Where to go next

- Previous: [Permission-Aware Retrieval, MCP and AI Agents](https://www.alesanalytics.com/architecture/ai-retrieval-and-agents.md)
- Next: [Twenty-Two Architecture Principles](https://www.alesanalytics.com/architecture/principles.md)
- Overview and every page: [Enterprise Architecture Atlas](https://www.alesanalytics.com/architecture.md)
- The whole site in one file: https://www.alesanalytics.com/llms-full.txt