CI/CD at Scale

Watch the talk

The problem it answers

Automated deployment is what made DevOps fast. Flickr's ten deployments a day showed what it could do in 2009, but that speed strained what has to hold at any speed, such as security and quality, and the Rugged Software movement followed in 2010 to answer it. Self-service infrastructure multiplies the strain: once every team provisions its own, every team needs pipelines to deploy it.

Without patterns, pipelines grow by hand, each one copied from the last, and the platform becomes a big ball of mud. Three things fail. Nobody can say who deployed what, when or how. Every new security or quality standard has to be copied into every pipeline by hand. And your specialists spend their time hand-crafting pipelines rather than building the platform. At one enterprise, infrastructure changes skipped quality gates altogether, engineers deployed from their laptops, and two changes in three failed.

With a few pipelines, none of that matters. Past ten pipelines per specialist, or twenty repositories deploying to several environments, it's why releases are slow and risky. The answer is to treat CI/CD as a platform rather than a collection of pipelines: your specialists build components everyone serves themselves from, and the business enforces its standards in one place.

What you'll take away

By the end, you'll be able to:

• Identify the pains of CI/CD at scale: lost traceability, vendor lock-in, the big ball of mud, and specialists hand-crafting every pipeline.
• Justify how much standardisation your platform needs from the essential complexity of your domain, so a handful of pipelines doesn't carry the weight of thousands.
• Design a naming convention that says what was deployed, where, by whom and why, with no magic names such as "dev" and "prod".
• Compare a pull model, a versioned library of tasks that teams pull in, with a push model, which bootstraps repos with standard workflows, and choose between them for your organisation.
• Explain how a stable task interface lets the platform team improve tasks underneath without breaking anyone, and enforce checks such as formatting, cost and security scans in one place.
• Identify where internal CLI tooling fills the gaps your CI/CD platform leaves, and why it moves with you if you change platform.
• Describe how CI/CD webhooks feed analytics that point you at the slowest pipelines and the tasks that fail most.

See it working

Each principle is shown working in a live demo in GitHub Actions, though all of it carries over to Azure DevOps, GitLab and CircleCI: a library of reusable actions, the documentation that makes it self-service, and a repo's CI and CD built from it.

Previous and upcoming events

Events where I've run this, or will, with links to each event and its recording where there is one.