Data-Driven Terraform for Developers and Users
How to turn infrastructure from a bottleneck into a force multiplier: Terraform your engineers serve themselves from by editing data, while your infrastructure specialists look after the patterns instead of a queue.
By: Benjamin Clark
Watch the talk
The problem it answers
DevOps grew out of decades of ideas: the Agile Manifesto in 2001, Lean software development bringing Toyota's manufacturing lessons to software in 2003, Flickr's ten deployments a day in 2009, and Rugged Software in 2010. The Phoenix Project drew them together in 2013, and gave us Brent: the one person everything depends on, without whom work stops. The lesson running through all of them is that automating infrastructure is what makes delivery fast, and it has to be shared between development and operations.
Most organisations adopted the tools and not the change. Terraform and CI/CD bring a burst of speed, then progress stalls: only Brent can read the code, so every new project still waits on Brent. Estates grown by copy and paste become a big ball of mud, which at one enterprise meant a module with over 50 inputs and a 10,000-line policy file. DORA's 2023 research shows the cost: elite performers deploy 1,640 times a year, low performers 1.5.
The industry has since turned to platform engineering, where, in Team Topologies, platform teams provide internal services so product teams can deliver on their own. Terraform needs the same shift. Treat infrastructure as a software deliverable rather than an operations task, and the mud becomes data your engineers serve themselves from.
What you'll take away
By the end, you'll be able to:
- Identify the signs that infrastructure has become your bottleneck: monolithic modules with dozens of inputs, policy files thousands of lines long, no documented interface, and every change waiting on one specialist.
- Explain how expressing the infrastructure you want as data, rather than imperative code, lets it be provisioned, changed and audited by reviewing data alone.
- Design the contract between the people who build modules and the people who use them: a documented interface with its required and optional fields, constraints and outputs, collections for any number of resources, and invariants the module enforces, so nobody can provision outside the approved patterns.
- Choose between a centralised infrastructure tier, versioned and shared for the strongest governance, and a colocated one that iterates faster with no versioning to manage.
- Outline what self-service depends on: schema validation in CI, remote state with locking, documentation kept up to date, and specialists who think in data structures.
- Plan the rollout one primitive and one team at a time, refining from their feedback before expanding.
- Explain how your infrastructure specialists become custodians of patterns rather than a queue, and what that does for release cadence.
See it working
Each principle is shown working in a live demo: a module seen from its user's side, where a new resource is a change to data, a repository that manages a whole GitHub organisation from data, and a repository that delivers one use case's infrastructure into AWS.
Previous and upcoming events
Events where I've run this, or will, with links to each event and its recording where there is one.
Is infrastructure your bottleneck?
I've used this pattern to take an enterprise from one release every four months to twelve a day, and a startup with no infrastructure to revenue in 90 days. If your engineers are waiting on infrastructure, book a call and we'll talk it through.