Dynamic Terraform on demand
Data-driven Terraform whose data tier is derived from a system outside the repository, such as a database people manage through a graphical interface, so they manage infrastructure without editing code or data structures.
By: Benjamin Clark
Intent
This pattern is a child of data-driven Terraform, and the parent of automatic client provisioning. It extends data-driven Terraform to infrastructure whose desired state lives outside the repository, either because another system already owns it, such as a product database whose records people create at runtime, or because the aim is a total shift left, where people manage infrastructure through a graphical interface and an API rather than by editing data structures. Rather than holding a hand-edited copy of that state, the data tier becomes dynamic: it's derived from the owning system each time a deploy runs, so the infrastructure follows the system that owns its data. This enables three outcomes:
- The owning system is the single source of truth for what should exist, and Terraform remains the only means by which it's built.
- People manage infrastructure through whatever interface the owning system offers, such as a graphical interface with an API behind it, without editing code or data structures.
- Each deploy reconciles every entry with the owning system, bringing the infrastructure back in line with the data that defines it.
Motivation
Data-driven Terraform assumes its data tier lives in the repository, beside the code, and that the people who change it are comfortable editing data structures there. Neither assumption always holds. Sometimes the desired state is already owned elsewhere, such as by records that operators create in a product at runtime. Sometimes the aim is a total shift left, where the people who need infrastructure never edit a data structure at all, but use a graphical interface, backed by an API, that presents the same data in a friendlier form. In either case, keeping a hand-edited copy of that data in the repository fails in three ways:
- The infrastructure lags behind the data. Every change in the owning system waits for someone to copy it into the repository and deploy it, which brings back the bottleneck data-driven Terraform sets out to remove.
- The two copies drift apart. Once the same data is held in two places, they sooner or later disagree, and nobody can say which is right, a split-brain problem.
- Copying introduces errors. Each change is transcribed by hand, so a mistyped or missing field builds the wrong infrastructure, or none at all.
This pattern resolves that by changing only the data tier of data-driven Terraform. The infrastructure and instantiation tiers, and each primitive's contract, stay exactly as they are, while the single source of truth moves from the data tier to the owning system:
- The owning system decides which entries exist. People create, change and remove entries there, through whatever interface it offers, such as a graphical interface whose API validates each change, with no change to the repository.
- The data tier becomes a view of the owning system. Each time a deploy runs, it's derived from the owning system as one collection of entries for each primitive, in the same shape as a data tier edited by hand.
- The infrastructure tier is unchanged. Its one generic module per primitive validates each entry and enforces its own invariants, so nothing can be provisioned outside the approved patterns.
- The instantiation tier is unchanged. It connects the data and infrastructure tiers, and instantiates the infrastructure in each environment.
Two properties make this safe. Because entries no longer pass through a pull request, the deploy workflow validates each one against its primitive's contract as it fetches them, so malformed data is rejected before anything is planned. And each entry is keyed by an immutable property, such as an ID the owning system assigns when the entry is created, so a change to any other field updates that entry's infrastructure rather than replacing it.
The pattern doesn't dictate when a deploy runs, whether at each release, on a schedule, or as soon as an entry changes. For a single organisation's internal system, the owning system can simply start a deploy whenever an entry changes. On a multi-tenant SaaS platform, where each change belongs to one client, automatic client provisioning builds on this pattern to provision each client as soon as its record changes, one client at a time.
Applicability
Use this pattern when:
- Your infrastructure is already described as data, as in data-driven Terraform.
- The desired state of that infrastructure is owned by a system outside the repository, such as a product database, another system's API, or a list another team maintains.
- You want a total shift left, where the people who need infrastructure manage it through a graphical interface and an API, rather than by editing data structures.
- New entries should be provisioned without a change to code, by the people who manage that data.
Do not use this pattern when:
- The data can live in the repository, and the people who change it are comfortable editing data structures and deploying, in which case data-driven Terraform alone is simpler.
- Each change is risky enough that a person should review it in a pull request before it's built.
Structure
Participants
| Participant | Role |
|---|---|
| End user | Creates, changes and removes entries in the owning system, through the interface it offers, such as an operator using a graphical interface or another team updating its list, with no knowledge of Terraform needed. |
| Infrastructure specialist | Builds the infrastructure and instantiation tiers, as in data-driven Terraform, and the step in the deploy workflow that derives the data tier. |
| Owning system | Holds the desired state, outside the repositories that manage the Terraform, and is its single source of truth. It can be any system that can list its entries, such as a database behind a graphical interface and an API, another system's API or a file another team maintains. |
| Deploy workflow | Fetches every entry from the owning system before each plan or apply, validates each against its primitive's contract, and derives the data tier from them. |
| Data tier | Holds the infrastructure you want as collections, one for each primitive, as in data-driven Terraform, but derived from the owning system rather than edited by hand. |
| Infrastructure tier | Turns entries into one primitive per module, each with a well-known public interface and expected outputs, and enforces its own invariants, unchanged from data-driven Terraform. |
| Instantiation tier | Connects the data and infrastructure tiers, and instantiates the infrastructure in each environment, with a state of its own for each, unchanged from data-driven Terraform. |
| Destination platform | Provisions each primitive, such as a cloud provider or a source control platform. |
Constraints and trade-offs
- Every constraint of data-driven Terraform still applies, and strict schema discipline matters even more, because the data no longer passes through a pull request before it's planned.
- The code no longer tells the whole story. What a deploy builds depends on data that changes without a pull request, so changes in the owning system need validation and an audit trail of their own, as in config as data and immutable audit trail.
- Deploys depend on the owning system being available. A deploy that can't read every entry has to stop, because Terraform would plan to destroy the infrastructure of any entry it missed, so the owning system needs to be as reliable as the pipeline that depends on it.
- Critical resources can be protected in the infrastructure tier, such as with Terraform's prevent_destroy setting, which makes any plan that would destroy them fail instead. The protection can't tell a missed entry from a deleted one, so removing a protected resource always takes a deliberate step by a person.
- Changes reach the infrastructure only when a deploy runs. Between deploys, the owning system can be ahead of the infrastructure, unless it starts a deploy whenever an entry changes, as automatic client provisioning does for each client of a SaaS platform.
- A total shift left needs an interface worth using. The graphical interface and API that replace editing data structures have to be built and maintained like any other product, and kept in step with each primitive's contract.
Part of how I deliver
When the data that decides your infrastructure lives in your product rather than your code, this is how I let Terraform build from it directly. It's one of many patterns I bring through contract delivery, and it keeps Terraform as the only thing that builds your infrastructure.
Are your engineers waiting on infrastructure?
I build infrastructure and pipelines your engineers serve themselves from, with your standards enforced in one place. If releases are slow, risky or queued behind one specialist, book a call and we'll talk it through.