# Observability, Operations and Governance

> Why observability must be independent of what it observes, error budgets, progressive delivery for configuration, governance, and FOCUS cost data.

- Web page: https://www.alesanalytics.com/architecture/observability-and-operations
- Markdown: https://www.alesanalytics.com/architecture/observability-and-operations.md
- Part of: [Enterprise Architecture Atlas](https://www.alesanalytics.com/architecture.md), section "Cloud platforms"
- Last verified: 13 October 2026
- Publisher: Ales Analytics

## See it from outside

You can only fix what you can see, and you have to see it from outside the thing that's broken. That's the whole of this page in a sentence, and the incidents in this atlas show why: in June 2025 Google's first incident report reached Cloud Service Health about an hour after the crashes began, because the Cloud Service Health infrastructure was itself down in the same outage. In November 2025 Cloudflare's status page, hosted off its own platform, failed at the same moment for unrelated reasons, and for a while suggested to responders that they were under attack.

The question to ask of any operational design is: **how do we know it's healthy, what changed, and how do we undo it?**

## Observability

### The vocabulary

Logs, metrics, traces and profiles. Service-level indicators (SLIs), objectives (SLOs) and error budgets. Runbooks and postmortems.

### Mechanisms worth understanding

- **An error budget is (1 − SLO) × the window.** At 99.9% over 30 days you have 43.2 minutes. The [numbers page](https://www.alesanalytics.com/architecture/numbers.md) has the table.
- **Sampling and redaction happen before storage, not after.** Telemetry that has been stored with a token or a prompt in it is already a data-protection problem.
- **Cardinality is the number of distinct label combinations,** so cost grows multiplicatively. Add a user ID as a metric label and you have one time series per user.

### The pipeline

OpenTelemetry SDKs in the services, a collector, a pipeline that samples and redacts, backends for each signal, and alerting on top. Plus external synthetic checks, independent paging, and progressive delivery so that a bad change is caught on a small slice of traffic.

### Principle: observability is independent of the observed

[Principle P13](https://www.alesanalytics.com/architecture/principles.md) spells out what that takes:

- a telemetry failure domain separate from the systems it watches, with external probes, and status and paging hosted outside the provider
- local buffering and local diagnostics, so a pipeline outage doesn't lose the evidence
- **time synchronisation,** because correlating logs across machines with skewed clocks produces a timeline that never happened
- audit continuity, because gaps in an audit log are compliance failures
- telemetry residency, privacy and redaction before storage
- access control on telemetry, because logs contain tokens, prompts and personal data
- cardinality and cost management

And one classification to get right: the telemetry pipeline is a Tier-0 dependency for *incident response*, not for *serving*. Keep it out of the request path, so that losing it makes you blind rather than down.

### How it fails

Telemetry goes down during the incident. The status page shares fate with the product. Alerts storm, so the one that matters is lost in a thousand. Clock skew scrambles the timeline.

### Trade-offs

A vendor SaaS backend is independent of your estate, and brings residency and egress questions with it. A self-hosted backend gives you control, and shares fate with whatever it runs on.

## Operations

Infrastructure as code as the source of truth. GitOps reconciliation. CI/CD. Patching. Backup and restore. Disaster recovery. Capacity, change and asset management.

The key pattern is **progressive delivery for code and for configuration**: canary first, then rings, a bake period at each stage, and automatic rollback when the metrics go wrong. Most organisations do this for binaries and skip it for configuration, which is exactly the gap every one of the five incidents in this atlas came through. See [Failure architecture](https://www.alesanalytics.com/architecture/failure.md).

Two consequences of treating infrastructure as code are easy to miss. **The IaC state files and pipelines become Tier-0,** because they can recreate or destroy everything. And **the CI/CD identity becomes the most privileged non-human principal in the estate.**

## Governance

Policy as code, attached at the organisation, management group or OU level. Controls mapped to frameworks such as NIST SP 800-53 and ISO 27001. Immutable audit. Data classification, retention and legal hold. An inventory of AI systems, risk tiering, and evaluation evidence. Budgets, and cost normalised to the FOCUS specification.

On FOCUS: version 1.4 was ratified on 4 June 2026, adding two datasets, 47 columns and six attributes, but no dedicated token columns. Version 1.5 proposes carrying token direction and cache action as properties of `SkuPriceDetails`, with ratification scheduled for 3 December 2026. Until then, metering AI tokens is a job for your model gateway.

## Enterprise SaaS

Microsoft 365, CRM, ERP, ITSM and HRIS systems (the last being the usual source of truth for the identity lifecycle), and an iPaaS to connect them. Every SaaS has its own control plane, authorization model, rate limits and API versioning. Bulk export is rate-limited, which constrains both how fast you can ingest data into an AI index and how fast you can get it out in a disaster.

## Questions to ask in a review

Where is our status page hosted? Can we page someone if our cloud is down? Do logs leave the jurisdiction? What's the error budget, and who stops releases when it's spent? Is configuration rolled out with the same canaries and bake times as code?

Should you have any questions regarding this page, please contact us at solutions@alesanalytics.com.

## Sources

1. [Google Cloud Service Health incident report, 12 June 2025](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW). Google Cloud, 2025. A policy with blank fields, replicated globally, crashed an unflagged code path in Service Control in every region. Remediation: modularise, and fail open.
2. [Cloudflare outage on November 18, 2025](https://blog.cloudflare.com/18-november-2025-outage/). Cloudflare, 2025. Cloudflare's postmortem. A database permission change doubled the Bot Management feature file past a limit of 200 features against normal use of about 60.
3. [FOCUS: the FinOps Open Cost and Usage Specification](https://focus.finops.org/). FinOps Foundation, 2026. Version 1.4, ratified 4 June 2026, adds two datasets, 47 columns and six attributes and no dedicated token columns. Token properties are proposed for 1.5.

## Where to go next

- Previous: [Security, Secrets and PKI](https://www.alesanalytics.com/architecture/security.md)
- Next: [Microsoft 365 as an Authorization Graph](https://www.alesanalytics.com/architecture/microsoft-365.md)
- Overview and every page: [Enterprise Architecture Atlas](https://www.alesanalytics.com/architecture.md)
- The whole site in one file: https://www.alesanalytics.com/llms-full.txt