One home, and copies with a delay
Data has one authoritative home. Everything else is a copy with a delay: a replica, a cache, a search index, a set of embeddings, a BI extract, a log line, an AI summary. That sentence is the most useful thing on this page, and the atlas makes it a principle, P0: every piece of state has a declared source of truth, and everything else is derived.
Once you accept it, a lot follows. Every derived copy needs a freshness target and a way to be invalidated. A deletion, an ACL change or a correction has to travel along the lineage from the source to every copy, or it hasn't really happened. And "where is this person's data?" becomes a question you can answer with a query rather than a project.
The question to ask of any data design is: where is the source of truth, what is derived from it, and how does a change, including a deletion, reach every copy?
Storage: four shapes
- Block storage is a disk: one writer, low latency. Managed Disks, EBS, Persistent Disk.
- File storage is a shared file system over POSIX or SMB.
- Object storage is HTTP access to immutable blobs with metadata, built for very high durability. S3, Blob Storage, Cloud Storage.
- Archive storage is cheap, durable and slow to retrieve.
The patterns that recur are tiering, lifecycle rules, object lock (write once, read many), and cross-region replication.
Object storage across the providers
Object storage is the atlas's worked example of how to learn one capability across several providers: learn the common concept first, then each realisation, then exactly what differs.
The common concept. Durability comes from replicas or erasure coding across devices and zones. A namespace (a bucket, a container, an account) is a unit of naming, policy, limits and billing. Objects are immutable blobs plus metadata. Consistency defines what a read sees after a write or a list. Replication is synchronous within a region and usually asynchronous across regions. Lifecycle covers tiering and expiry; versioning and object lock protect against change. Access control comes from identity policy, resource policy, and signed URLs.
AWS S3
Strong read-after-write consistency for PUT and DELETE of objects in all AWS Regions, with strongly consistent reads of ACLs, tags and metadata[1]. AWS announced it in December 2020. Access comes from bucket policies and IAM, capped by RCPs and SCPs, with presigned URLs for temporary access and object lock for immutability.
Azure Blob Storage
Strongly consistent within a region. The storage account is the namespace and the security unit, which is an E2 difference: the account, not the container, is where keys, network rules and many limits live. Access is through account keys, shared access signatures (SAS) or Entra RBAC data roles. Note that ARM deny assignments block management-plane actions; data-plane access follows data roles and keys.
Google Cloud Storage
Strong global consistency for object read-after-write, update and delete. Uniform bucket-level IAM is the recommended access model.
On-premises
S3-compatible stores such as Ceph RGW. Consistency and feature parity vary by product and version, so look them up rather than assuming S3's behaviour.
What's the same: the API shape, the durability goal, versioning and lifecycle concepts, and asynchronous cross-region replication on all three clouds.
What differs: the authorization model, the namespace unit and its limits, whether keys or SAS tokens bypass identity entirely, and the defaults for replication. Consistency, which used to be the headline difference, no longer is. The atlas writes this as E1 with facet deltas on authorization and namespace.
How it fails: public exposure through a policy mistake; long-lived SAS tokens and presigned URLs acting as bearer tokens that can't be revoked without rotating the key; and replication faithfully copying deletes unless versioning and object lock are on. In a review, ask: can any credential bypass identity? Which deletes are replicated? Is object lock administered from a separate domain?
Replication is not backup: keep recovery on a separate plane
Production account
Production account
Amazon DynamoDB
The primary table
Replica in a second Region
Copies deletes too
AWS KMS key
Production
Separate recovery account
AWS Backup vault
Locked, with its own credentials
AWS KMS key
Separate; production can't delete it
Restore drill
Rebuild from IaC and restore data, on a schedule, with evidence
Databases: pick by what the data must do
| Class | Use it when | Watch for |
|---|---|---|
| Relational | Integrity matters and you need joins | Scale-up limits; migrations |
| Key-value | You know the access pattern and need massive scale | Hot partitions |
| Document | Aggregates are flexible | Weak integrity across documents |
| Graph | Identity, lineage, fraud | Traversals that don't scale |
| Time-series | Metrics, IoT | Cardinality |
| Search | Keyword and BM25 retrieval | Eventual consistency |
| Vector | Semantic similarity, through HNSW or IVF indexes | It's derived data, and inherits the classification of its sources |
| Columnar, MPP | Analytics | Not for transactions |
| Distributed SQL | Global consistency | Commit latency |
Mechanisms worth understanding
- MVCC lets readers see a snapshot without blocking writers. Most modern databases do this, and it's why long-running reads see a consistent view.
- Lower isolation levels allow anomalies, such as lost updates and write skew. Know which level your code assumes.
- Read replicas lag, so a user who writes and then reads may not see their own write unless you route the read to the primary.
- The partition key decides where the hot spots are. Choose it for the access pattern, not for the shape of the data.
- CDC reads the commit log, so it gives you order per key or partition, not globally.
- Events arrive late and out of order. Watermarks bound how late you are willing to wait.
- Deletes have to be tombstones to propagate. A row that simply vanishes tells downstream systems nothing.
- Schema evolution needs compatibility modes. Backward compatibility means new readers can read old data; forward means old readers can read new data.
Data platforms
OLTP against OLAP; ETL against ELT; log-based change data capture (CDC); batch against streaming. The components are connectors, brokers, a lake on open table formats, a warehouse, a catalogue, lineage, quality checks, and masking and row-level security. The patterns that make them hold together:
- The outbox: write the state change and the event that announces it in the same transaction.
- Sagas: a sequence of local transactions with compensating actions, instead of a distributed two-phase commit.
- Data contracts between producers and consumers.
- A lake on open table formats, which keeps the data readable by more than one engine.
- Catalogue and lineage, row-level and column-level security, masking, and a semantic layer.
How data fails
Connection storms when a database fails over. Hot keys. Migrations that lock tables. Duplicate or replayed events. Schema drift that breaks consumers. Deletions that never reach the vector index or the cache. Silent quality decay that nobody notices until a report is wrong. And lineage gaps, which are the reason a deletion can't be proven.
Trade-offs
Strong against eventual consistency. Normalised against denormalised. Batch against stream (streaming is an always-on cost; a batch job reruns easily). ETL against ELT. A global database, which costs commit latency on every write, against regional ownership of each record.
Questions to ask in a review
What's the partition key? What isolation level does this code rely on? What happens on replay? How does a deletion reach the index, the cache, the backup and the BI extract? Is there a declared source of truth for each piece of state, and is it written down?