Reconcilers

A reconciler is an automated job that reads the catalog and makes one part of the real world match it: the cloud projects, the Google groups, the GitLab repositories or the dashboards. After every merge to the catalog, a chain of reconcilers runs in a fixed order, and between them they build or change everything the catalog describes. This is how one file becomes a working product with nobody creating anything by hand.

What a reconciler is

Each reconciler is a Terraform root: a directory of Terraform code that reads the catalog’s files and declares what should exist. Terraform compares that with what really exists and makes the difference: it creates what is missing, changes what differs and removes what the catalog no longer asks for.

Every root has its own pipeline identity, and that identity can change only its own part of the platform. The project factory for the shop/store domain can create projects in that domain’s folders, and nothing else. The root that manages GitLab memberships cannot touch a cloud project. If one of them were compromised, the damage would stay inside its own part.

The catalog’s own pipeline holds none of these identities. On the catalog’s main branch, it checks the files and starts the other repositories’ pipelines, in order; it cannot create a project itself.

The chain after a catalog merge

When a merge request to the catalog is merged, the catalog’s pipeline on main runs these steps, each waiting for the one before it:

1. The lifecycle runs. If the merge changed a product’s lifecycle status, for example to retiring, the product’s own pipeline applies that change first, with the product’s own identities. A product’s services must leave before anything downstream deletes what they use.
2. landing-zone. Its roots create the folder of a new context, the Google groups and their members, and, through the project factory of the product’s domain, the product’s two projects, its plan and apply identities, its places for Terraform state and its Ready output: a small document that tells every later root what was made. The network roots then attach the projects that are billed.
3. operations. Its roots give the new identities what they need to send telemetry, and create the product’s folder, dashboards and alert rules in Grafana.
4. control-plane. Its GitLab roots create the product’s repository, protect its main branch, give the team its memberships and merge rights, and write the first commit, generated from the golden path’s template with the catalog entry’s values.
5. landing-zone again. A second pass finishes what needed the others first. For a product that is leaving, it unlinks billing or deletes the projects, after control-plane and operations have dropped what they granted to the product’s identities.

A note on the merged catalog merge request follows these steps as they run, with their times and links. For a new product, it ends when the product answers at its hostname, or names the step that failed.

Why the order matters

Each step uses what the step before it made. Operations cannot grant anything to an identity that does not exist yet, so it runs after landing-zone has created the identities. Control-plane writes the repository’s first commit from the Ready output, so it runs after the project factory has published it. And a product leaving the platform must first stop running, then lose its grants, and only then lose its projects, which is why the lifecycle runs first and the second landing-zone pass runs last.

The chain is strict about failure for the same reason: every step waits for its pipeline to finish, and if one fails, the steps after it do not run. Nothing downstream acts on something that was not made.

What “nothing derived is edited by hand” means

Everything a reconciler creates is recorded in its Terraform state, and the reconciler is the only way it changes. A project’s identities, a repository’s protections, a team’s memberships and a dashboard are all changed by changing the catalog, never in the Google Cloud console or GitLab’s settings.

This is a rule with teeth. If someone changes a derived thing by hand, the next plan of that root sees the difference and would undo it, and a scheduled plan of every root reports it as drift in an issue. The way to change it for good is a merge request to the catalog.

When a reconciler fails

A failure stops the chain at that step, and the progress note on the catalog merge request names the job that failed and what to do next. The usual causes are a mistake in the catalog that its checks did not catch, a Google Cloud quota, or a bug in a root.

The fix is a merge request: to the catalog, if the entry was wrong, or to the root’s repository, if the root was. The merge runs the chain again, and because every root only makes the difference between the catalog and what exists, running it again is safe: what was already made is left alone. Each repository’s daily schedule plans its roots without applying them, so nothing changes until someone merges a fix.

Where it lives

In the catalog repository:

• .gitlab-ci.yml: the chain, in order

In the landing-zone repository:

• roots/project-factory/: one factory per domain
• roots/groups/: the Google groups
• docs/guides/product-lifecycle.md: the steps in detail

In the operations repository:

• roots/grafana/: dashboards and alert rules

In the control-plane repository:

• roots/gitlab-config-shop-store/: a domain's repositories