Serverless data lake layout

Intent

This pattern assumes a platform that takes in data as files, and needs a flexible way to query that data, including questions nobody planned for when it arrived. It lays out a data lake in tiers on serverless storage, one tier for each stage of the data's life, from files exactly as they were delivered to data ready for analysis. Each tier has one purpose, and its own rules about what may write to it and what may read from it. The tiers cover the bronze and silver layers of the medallion architecture, and the gold layer, which holds the business-level views people report from, sits on top of them, in a data warehouse or a query layer. This enables three outcomes:

1. Every file is kept exactly as it arrived, so a faulty transform can never destroy the original.
2. Any tier can be rebuilt from the one below it whenever the processing changes.
3. Access, retention and cost are set per tier, so each consumer is given exactly the data it needs, and no more.

Motivation

When a platform keeps all its data together, with no tiers between files as they were delivered and data ready to query, it fails in three ways:

1. Nobody can tell an untouched file from a cleaned one, because they sit side by side under the same permissions.
2. A faulty transform can overwrite the only copy of a file, and the original is lost for good.
3. Permissions, retention and cost can't be set per stage, so every consumer is given more access than it needs.

This pattern separates the data by how far it has been processed, keeping each tier in storage of its own, such as a separate bucket in an object store like Amazon S3. Files arrive in a landing tier, the only one anything outside the platform may write to. An untouched copy of each file is kept in a raw tier, so whatever happens further on, the original survives, and every later tier can be rebuilt from it. A processed tier holds each file converted into a format built for analysis, such as Parquet, ready to query. Because each tier is stored separately, access, retention and cost can be set for each one, and a consumer is given only the tiers it needs. A catalogue such as the AWS Glue Data Catalog describes the processed tier, so that a query engine such as Amazon Athena can read it, as in the serverless query layer, and an event-driven data pipeline moves each file from one tier to the next.

Applicability

Use this pattern when:

1. Data arrives as files, from people or from other systems.
2. You must keep every file exactly as it arrived.
3. You need processed versions you can rebuild whenever the processing changes.
4. People need to query the data flexibly, including in ways nobody planned when it arrived.

Do not use this pattern when:

1. A handful of files come from one source, where folders in one storage location are enough.
2. The data needs fast reads and updates, which belong in a database.

Structure

Variant A: One set of tiers

When the data has a single owner, such as an organisation's own internal data, this variant works best.

1. There's one set of tiers, each stored separately, named by environment and tier, and declared with the rest of the platform's infrastructure.
2. One catalogue describes the processed tier.

Variant B: One set per isolation boundary

When data comes from owners who must be kept apart, such as a SaaS platform's clients, or business units whose data mustn't mix, this variant works best. Each owner is an isolation boundary, with storage of its own.

1. Each boundary has its own set of tiers, named from a reference that never changes, so renaming a client never means moving their data.
2. Services work out where a boundary's tiers are stored from the verified caller's boundary, such as their record in config as data, so no request can name another boundary's storage.
3. Each boundary has its own catalogue, so queries are kept apart in the same way as storage.
4. A new boundary's tiers are created from its record, as in dynamic Terraform on demand, or through automatic client provisioning where the boundaries are clients.

Participants

Constraints and trade-offs

1. Every tier is another copy. The raw tier duplicates landing, and the processed tier duplicates raw in another format, so storage grows with each tier, and older copies need expiring or moving to cheaper storage.
2. The processed tier keeps each source's own structure, so combining data from several sources is left to the queries, or to a layer built on top.

Variant A: One set of tiers trade-offs

1. The core benefit is simplicity: one set of tiers and one catalogue to run.
2. The core cost is that access is set per tier rather than per owner, so it doesn't suit data that must be kept apart.

Variant B: One set per isolation boundary trade-offs

1. The core benefit is isolation that holds in the storage itself, so each boundary's access, retention and cost are its own.
2. The core cost is scale: storage locations multiply as boundaries times tiers, and object stores limit how many an account can hold, as S3 does with buckets, so thousands of boundaries need a quota increase, or tiers shared by prefix, with isolation that rests on policy instead.

Part of how I deliver

This is the layout I give a platform's data before anything is built on top of it, because every later decision depends on it. It's one of many patterns I bring through contract delivery, and it suits one team's data as well as a platform with thousands of clients.

Related