GitHub estate as code

Intent

This pattern is a specialised implementation of data-driven Terraform. It realises data-driven Terraform's core aim (shifting infrastructure left to the people who use it) within the management of a GitHub organisation, managed with the GitHub provider for Terraform. It assumes an organisation with more than a handful of repositories, and treats it as two separate but related concerns, each in a repository of its own:

• Organisation management, where administrators set the organisation's settings, its teams and the rules every repository follows.
• The repository factory, where anyone can request a new repository by pull request, and get it set up the same way as every other.

Keeping them apart lets each be owned, reviewed and changed by the right people. This enables three outcomes:

1. Anyone can get a new repository by pull request, set up the same way as every other, without waiting for an administrator to create it.
2. Administrators deliver faster, because they maintain the organisation's settings, access and rules from one place, rather than creating every repository themselves.
3. Every repository and every change of access is reviewed and recorded, and changes made by hand are put back.

Motivation

When a GitHub organisation grows by hand, with people creating repositories and setting their protection in the browser, it fails in three ways:

1. Nobody can say who has access to what. Access and protection are granted repository by repository and person by person, so the organisation's policy exists only as the sum of its settings.
2. Repositories drift. Each one is created differently, with its own files, protection and access, and nobody can say which meet the organisation's standards.
3. Administrators become a bottleneck. Every new repository and every change of access waits on the few people who can make it, and routine requests queue alongside decisions about policy.

This pattern separates the two concerns, because they differ in who changes them, how often, and how much a change can affect. Each lives in its own repository, with its own primitives (data-driven Terraform's name for each kind of resource a repository manages). Both use data-driven Terraform's colocated variant, keeping each primitive's module in the same repository as its data, because GitHub is a single service with nothing to version or share.

Only administrators change organisation management, and rarely, but one change can affect every repository at once. Its primitives typically include:

• The organisation's settings, such as one that stops members creating repositories by hand.
• Teams, which can take their members from the organisation's identity provider, such as Microsoft Entra ID or Okta, through team synchronisation, so GitHub access follows the organisation's own record of who works where. Without one, the data lists each person and the teams they belong to, because administrators think about which teams a person is in, rather than who is in a team.
• Organisation-wide roles, which give a team access to every repository at once.
• Rulesets that every repository follows, including repositories created later.

Anyone can change the repository factory's data tier, by pull request, and often, to request a repository, but each entry affects only the repository it describes. Only administrators change its infrastructure and instantiation tiers, which decide how every repository is set up. Each repository is one entry, with defaults for anything it leaves out, and the factory creates it with the same standard files, such as a licence and a code owners file naming who must review changes, and the same team access as every other repository.

The two concerns are related in three ways:

1. Organisation management stops repositories being created by hand, so the factory is the only way to get one.
2. The factory grants access to the teams organisation management defines, so a team is defined once, and every repository that names it follows any change to it.
3. Organisation-wide rulesets cover every repository the factory creates, so the factory never sets branch protection repository by repository.

Administrators then become custodians of the organisation's settings, access and rules, and of how the factory sets up each repository, as infrastructure specialists do in data-driven Terraform, while everyone else serves themselves through the factory's data tier.

Each concern has a pipeline of its own, which plans every pull request and applies on merge. Each pipeline also applies again on a schedule, putting back any change made outside the code. That does two things:

1. It combats configuration drift, where the estate gradually stops matching its code as people change settings by hand.
2. It shortens how long an unauthorised change stays in place. A change of access or protection made by a malicious actor lasts only until the next scheduled apply, rather than until someone notices it.

Each pipeline signs in to GitHub as an application with an identity of its own, such as a GitHub App, holding only the permissions its concern needs, never with a person's token. The factory's application can create and configure repositories, but never change the organisation's settings, teams or rulesets, so a repository request can never change organisation policy.

The two variants below differ in how many organisations the estate holds.

Applicability

Use this pattern when:

1. A GitHub organisation has more than a handful of repositories, and will be around for a while.
2. Repositories, their protection and their access should be consistent, rather than set by hand.
3. You need to say who has access to what, with every change reviewed.

Do not use this pattern when:

1. The organisation is short-lived, with a handful of repositories, where setting up the factory takes more effort than it saves.

Structure

Variant A: One organisation

When the business works in a single GitHub organisation, this variant works best.

1. One organisation management repository sets the organisation's settings, teams and rulesets.
2. One repository factory creates every repository in it.

Variant B: An enterprise of organisations

When a GitHub Enterprise holds an organisation for each business unit, and a platform team must set the same baseline in all of them, this variant works best.

1. Organisation management also creates the organisations themselves, within the enterprise, before anything inside them is set.
2. A baseline applies in every organisation, such as rules that stop long-lived branches being deleted or overwritten, while fuller standards are opt-in for each organisation.
3. Enterprise-wide teams, such as platform and security, join every organisation, while each business unit keeps its own teams and its own identity provider.
4. The repository factory creates each repository in whichever organisation it belongs to.

Participants

Constraints and trade-offs

1. Every constraint of data-driven Terraform still applies, from strict schema discipline to mature state management.
2. Organisation management has a wide blast radius. One change can affect every repository, so its pipeline's application needs broad permissions, kept with the same care as an administrator's credentials, and each change deserves the review of a change to production.
3. Where teams take their members from the identity provider, membership leaves code review. Whoever manages a group there can grant GitHub access without a pull request, and team synchronisation is a feature of GitHub Enterprise Cloud that needs the identity provider's administrator (often in another team) to set up. Without it, membership stays in the data, and every change to it is reviewed.
4. The concerns change in order. A team has to exist in organisation management before the factory can grant it access, so a new team and its first repository take two changes.
5. A new repository waits for its pull request to be reviewed and applied, so it takes minutes rather than seconds. In return, it starts out set up to the organisation's standards, with its access reviewed and recorded, whereas a repository created in the browser in seconds still has to be set up by hand afterwards.

Variant A: One organisation trade-offs

1. The core benefit is simplicity: one place for policy and access, and one place to request a repository.
2. The core cost is that every team follows the same settings and rules, which suits one business but not several units that each need their own.

Variant B: An enterprise of organisations trade-offs

1. The core benefit is one baseline everywhere, while each business unit keeps its own teams, identity provider and pace.
2. The core cost is more moving parts: organisations have to exist before anything inside them can be set, and enterprise-wide settings reach every organisation at once.

Part of how I deliver

This is how I let teams get a properly set-up repository by pull request, while administrators keep control of access and policy. It's one of many patterns I bring through contract delivery, and it works for a single organisation or an enterprise of them.

Related