Data-driven Terraform

Intent

This pattern assumes infrastructure managed with Terraform, where the same kinds of resources, which this pattern calls primitives, such as storage buckets, repositories or accounts, are provisioned again and again. It shifts infrastructure left, moving its provisioning from a central operations team to the teams that use it, so they can serve themselves. It does this by separating the definition of each primitive from where it's instantiated, and expressing the infrastructure you want as data that anyone can edit, the single source of truth for that infrastructure. This enables three outcomes:

1. Infrastructure is provisioned, changed and audited entirely by creating, editing and reviewing data.
2. The teams that use infrastructure provision and change it themselves, without waiting on a specialist.
3. Infrastructure specialists deliver faster, because they maintain the patterns and interfaces everyone else uses, rather than making every change themselves.

Motivation

Adopting Terraform speeds delivery up at first, but the gain stalls while infrastructure stays an operations task. Written imperatively, with each new resource copied from the last, Terraform fails in three ways:

1. Specialists become a bottleneck. Only the few people who can read the code can change it, so every project waits on them, like Brent in The Phoenix Project, the one person without whom work stops.
2. Nobody can find the interface. With no documented inputs, options or dependencies, using a module means tracing thousands of lines of code to rebuild what its author intended.
3. The code becomes a big ball of mud. Copies multiply, every change means editing configuration by hand, and a single source of truth becomes unattainable, until adding or changing one resource risks breaking others.

This pattern treats infrastructure as a first-class software deliverable, and applies three fundamentals of software engineering to it:

1. Multiplicity. Each primitive takes a collection of entries, so another bucket or repository is another entry, not another copy of the code.
2. Extensibility. Entries are plain data that anyone can add to or amend, so changing infrastructure means changing data.
3. Contract-driven development. Each primitive has a documented interface, giving its required and optional fields, constraints, dependencies and outputs, which the code behind it enforces.

It splits the Terraform into three tiers, each with one job:

1. The data tier holds the infrastructure you want, as one collection of entries for each primitive. It's the single source of truth, and the only tier end users edit.
2. The infrastructure tier holds one generic module for each primitive. Each module validates its inputs and enforces its own invariants, so nothing can be provisioned outside the approved patterns, and one entry can create several related resources behind a single primitive.
3. The instantiation tier connects the data and infrastructure tiers, and instantiates the infrastructure in each environment.

Terraform becomes a transformer that turns the data into resources. Infrastructure specialists become custodians of patterns, architectures and interfaces, and everyone else serves themselves. This is the shift Team Topologies describes, where platform teams provide services that others use without waiting on them, and operations stops being a blocker to become a force multiplier for development.

The data tier doesn't have to be edited by hand. Where what to build is decided outside the repository, such as by records people edit at runtime, or where people should manage infrastructure through a graphical interface rather than data structures, the data tier can be derived from that system instead, as in dynamic Terraform on demand, and the rest of the pattern stays the same.

The two variants below suit different teams, and one business can run both.

Applicability

Use this pattern when:

1. You need to deploy many copies of the same primitives, again and again.
2. Your development teams need to serve their own infrastructure needs.
3. Imperative Terraform has stalled your deployment velocity.
4. Knowledge of your infrastructure code sits with a select few people.
5. Your Terraform has become a big ball of mud.

Do not use this pattern when:

1. Your infrastructure needs are trivial, with no call for multiplicity or rapid deployment at scale.
2. Your infrastructure team lacks development discipline, because data structures need strict schema validation and documentation to be usable.

Structure

Variant A: Centralised infrastructure tier

When an organisation has a large infrastructure team, which must be the custodian of the platform for every other team, this variant works best.

1. Each primitive in the infrastructure tier is its own versioned code artefact, released with semantic versions and used by many repositories across the business.
2. Each primitive's interface is documented where its consumers will find it, such as on a website or in release notes.
3. Each repository's instantiation tier connects its own data tier to a pinned release of each primitive.

Variant B: Colocated infrastructure tier

When an organisation has few infrastructure specialists, uses AI coding agents heavily, or relies on people who aren't infrastructure specialists to build its infrastructure, this variant works best.

1. The infrastructure tier lives in the repository that uses it, with a folder for each primitive.
2. Each primitive's interface is documented beside the code, such as in docstrings or Markdown files.
3. The instantiation tier connects the repository's own data and infrastructure tiers.

Participants

Constraints and trade-offs

1. Strict schema discipline is required. Without validation in your pipelines, as in composable pipelines, invalid data reaches the plan and only shows up when it runs.
2. Mature state management is required. Remote state with locking is essential, so that concurrent changes can't corrupt it.
3. Teams have a learning curve. Infrastructure specialists must learn to think in data structures and algorithms, Variant A needs a working knowledge of semantic versioning, and end users must learn the conventions for the primitives they use.

Variant A: Centralised infrastructure tier trade-offs

1. The core benefit is strong governance: the platform team controls the patterns, so every implementation is consistent, standardised and of the same quality.
2. The core cost is versioning: semantic versions, release notes and the ongoing maintenance that keeps modules current can delay new features.

Variant B: Colocated infrastructure tier trade-offs

1. The core benefit is speed: with no module versions to manage, the whole stack can change quickly as the business's needs change.
2. The core cost is less reuse: two repositories can each hold their own data and infrastructure tiers for the same primitive.

Part of how I deliver

This is how I shift infrastructure left, so the teams that use it serve themselves while your specialists set the patterns. It's one of many patterns I bring through contract delivery, and it starts with one primitive for one team, growing from their feedback.

Related