Architecture Exercises and Design Reviews

The gateway's own configuration pipeline, which is now a global configuration blast radius. Its certificates and PKI. The identity provider's JWKS, for validating tokens. Its DNS name. Its rate-limit store, which is state. Its logging pipeline. And its scaling control plane. It becomes Tier-1 or Tier-0, and needs bulkheads, more than one region and a last-known-good configuration.

Concurrent writes and lost updates. Conflict resolution, where last-writer-wins needs clocks, so skew matters. Uniqueness constraints across regions. Read-your-writes across regions. Sagas that span regions. Replication lag as your RPO. Split brain under a partition. And inter-region egress cost.

Capturing ACLs from broken inheritance and sharing links. Group expansion and nesting. Sync lag, which is your revocation window. Gaps in label coverage for Decision 2. Deleted or moved items lingering in the index. Ranking before authorization. Summaries that mix items with different labels. And the identity that grants permissions becoming Tier-0.

GPU capacity and quotas. Patching drivers, CUDA and the runtime. DRA and queue scheduling. KV-cache and batch tuning. Weight provenance and licensing. Safety filters. Evaluation for every version. On-call for Xid errors and out-of-memory. Isolation between tenants, with no shared prefix cache. And GPU capacity in the disaster-recovery region.

Private DNS zones, and who owns them. Resolvers and conditional forwarders from on-premises. Limits on zone links. An endpoint per region per service, each billed by the hour. DNS as a single point of failure. Much harder troubleshooting. And exfiltration through other tenants' endpoints unless endpoint policies exist.

New token exchanges fail. Workloads holding valid cached tokens carry on until they expire, so token lifetime sets how long they survive. Deployments from CI to the cloud stop. Recovery tooling that uses the same federation fails too. You need break-glass workload credentials stored offline, monitored, and scoped to recovery.

The API server and etcd, with its quorum. Admission webhooks, which are write-path dependencies. Controllers and operators, whose reconciliation bugs act on the whole cluster. The CNI and CoreDNS, which are a network plane. The registry, on the pull path. Cloud integration controllers for load balancers and volumes. And upgrade skew. Each cluster is a failure domain.

IAM models and policies. Network constructs. KMS and key policies. Logging and SIEM pipelines. IaC providers. Landing zones. Guardrails. Cost data, which FOCUS helps with. Skills and runbooks. And incident processes.

Windows of eventual consistency. Duplicates. Reordering across partitions. Late data. Poison messages, which need a dead-letter queue and replay tooling. Schema evolution. Dual writes without an outbox. Sagas that need compensations. And users who don't see their own write.

Revocation delay, which is the revocation window. Cache-key errors, such as caching by user but not by resource or context. Cross-tenant cache poisoning. Stale group membership. And failing open by serving stale entries during an outage, unless that is bounded. The mitigations: event-driven invalidation, a short TTL scaled to sensitivity, and never caching an allow for an irreversible action.

• The single database primary is a single point of failure.
• Snapshots in the same account and key domain aren't backups.
• No RPO or RTO is stated, which is a missing requirement.
• Two zones need double capacity to be statically stable.
• The dependency on the regional control plane hasn't been examined.
• Fix: a synchronous standby in the second zone, immutable cross-account backups under separate keys, and restore drills.

• A year-long SAS is a bearer credential embedded in a binary anyone can download, and it can't be revoked without rotating the account key.
• It bypasses identity entirely.
• Fix: a short-lived user-delegation SAS, minted by a backend after the user has signed in, scoped to a single blob path, with minimal permissions and a short expiry.

• Application-level Sites.Read.All gives the indexer tenant-wide access, which makes the index a Tier-0 data store.
• Filtering by email domain isn't authorization. It ignores ACLs, broken inheritance, groups and labels.
• There's no revocation window, no deletion propagation and no Decision 2.
• Fix: Sites.Selected or delegated retrieval, ACL capture with late verification, label-aware processing, and storage classification set to the maximum of the sources.

• If the webhook's pods fail, for instance in a node-pool outage, all pod creation is blocked, including the webhook's own replacement pods. That's recovery coupling.
• Fix: exempt the webhook's namespace, spread replicas across pools and zones, consider an out-of-cluster or managed policy service, and alert on webhook latency.

• Last-writer-wins on balances loses updates. Money needs single-writer ownership or consensus.
• Last-writer-wins depends on clocks, and default NTP skew can reorder writes.
• Three layers of three retries amplify load up to 64 times, and without idempotency they produce duplicate debits.
• Fix: partition ownership by account, use idempotency keys, retry at one layer with a budget, and either design conflict-free or accept a globally consistent store and its commit latency.

• A single blueprint secret is one credential that can mint tokens for every agent in every domain.
• Microsoft's guidance is not to use secrets in production.
• The blast radius spans HR, finance and IT.
• Stateless MCP plus re-issue on a broken stream makes duplicate payments likely.
• Fix: a blueprint per agent type and data domain, each with a managed identity's federated credential; an idempotent payment tool; fail-to-manual approvals; and anomaly detection on token issuance.

• Prompts in US telemetry are derived data that violate residency.
• Failing over to the nearest region globally violates residency, so failover has to be constrained to the EU, or degrade.
• The global SaaS identity provider is a dependency outside the sovereign boundary. Decide whether to accept it with documented terms for support access, or add a fail-static path.
• Also check where the KMS keys live, who has support access, where evaluation datasets are kept, and where backups go.
• Principle: residency binds every hop, including telemetry, fallback, backups and administrators.

• Remapping object IDs breaks ACLs, Graph grants, agent identities and labels.
• RSS settings don't migrate, and RSS is retiring, so the acquired tenant's restrictions may simply vanish.
• Copilot over the merged tenant exposes the acquired tenant's permission debt.
• Coexistence spans two identity providers (cross-tenant access, B2B), and the two DLP estates have to be reconciled.
• Order: permission remediation and labelling, then RCD and DLP in the target tenant, then identity and content migration, then index rebuild, then Copilot enablement. Measure the revocation window throughout the coexistence period.