Grants and break-glass

Nobody on Airframe holds a standing right to change production. When a person truly needs one, for an incident or for one of the few roots still applied by hand, they ask for a grant that lasts an hour or two, another person approves it, and the request, the approval and the reason are written down. This page is the how-to: which grant to ask for, the commands, what to record, and the separate break-glass for when the gate cannot approve a merge.

Identities and access explains why it works this way. A grant here is a Privileged Access Manager grant: Google Cloud adds the roles to your own account for a set time, then removes them.

What never needs a grant

Most work does not. Your everyday account can read projects and their logs, and a product’s team can add a new version of one of its secrets. Every change to infrastructure goes through a merge request and a pipeline. Rerunning a product’s pipeline on main, fixing drift in code and rolling back with a revert merge request need no grant either. Ask for a grant only when a merge request cannot do the job.

Which grant fits

Airframe defines four grants, as four entitlements in landing-zone’s roots/pam:

You need to	Grant, approver, length
Read the logs, metrics or state of a non-production project to investigate	Level 1: no approver, 1 hour
Fix something by hand in the demo organization’s projects during an incident	Level 2: a second person, 2 hours
Apply a root that stays laptop-applied, or replace the trusted runner	Platform apply: a second person, 2 hours
Change something at the organization level	Organization: a second person, 2 hours

Level 1 exists today on the platform’s non-production runtime project only. The identity guide says each billed product’s non-production project gets one from its project factory; no factory creates it yet, so that part is designed. A laptop apply of the identity root needs both the platform apply and the organization grants, because it also changes bindings at the organization.

Members of the break-glass group, lk-breakglass@, may request the first three. The organization grant is requested by a member of the platform group, lk-platform@, and every approver comes from that group. Privileged Access Manager refuses to let a requester approve their own request (enforced).

1. Request the grant

Who: you, signed in to gcloud with your own daily account, never the administrator account.

Name the entitlement, where it lives, how long you need it and why. For a level 2 grant on the demo organization’s folder:

gcloud pam grants create \
  --entitlement=airframe-level2-admin \
  --folder=<folder-id> \
  --location=global \
  --requested-duration=7200s \
  --justification="<what you will change, and why>"

The other three differ only in the entitlement and its place: airframe-platform-apply on the same folder; airframe-organisation-admin with --organization=<organization-id>; airframe-level1-read with --project=<project-id> and at most 3600s. The Privileged Access Manager page of the Google Cloud console does the same. A level 1 grant activates at once; the others wait for an approver. While a grant for the same access is still open, Privileged Access Manager refuses a second one, so reuse the open grant.

2. Approve it

Who: another member of lk-platform@, never the requester.

The approver lists what waits for them, reads the justification, and approves with a reason of their own:

gcloud pam grants search \
  --entitlement=airframe-level2-admin \
  --folder=<folder-id> --location=global \
  --caller-relationship=can-approve

gcloud pam grants approve <grant-id> \
  --entitlement=airframe-level2-admin \
  --folder=<folder-id> --location=global \
  --reason="<why this is approved>"

Today one person runs the platform, and the second approver is the maintainer’s second account: the mechanism is enforced, but the separation between two people is not yet real.

3. Use it for one root, then let it expire

The grant does not hand you a separate key: for its duration, your own account holds the entitlement’s roles. No repository document gives the exact command sequence for a laptop apply; the outline below is assembled from the roots’ own files and the bootstrap log’s records.

Where: landing-zone for the organization, audit and identity roots (roots/organisation, roots/audit, roots/wif); control-plane’s roots/runner to replace the trusted runner.

In a clean clone, check out the approved commit, and use the Terraform version the repository’s .terraform-version names.
Let Terraform’s Google provider act as your own account: gcloud auth application-default login.
In the root’s directory, initialize it against its state. backend.tf holds an empty gcs block, so pass the platform state bucket, which the root’s state_bucket variable names, and the prefix in the “State” row of the root’s README:
Terminal window
```
terraform init \
  -backend-config="bucket=<platform state bucket>" \
  -backend-config="prefix=<the State row of the README>"
```
Plan to a file, read it, apply exactly that file, then plan again; the last plan should be empty:
Terminal window
```
terraform plan -out=<file>
terraform apply <file>
terraform plan
```
Stop. The roles leave your account when the grant expires.

The diagram and these steps follow a platform apply or organization grant. A level 1 grant has no approver and changes nothing: while it lasts, read the project’s logs, metrics and state. A level 2 grant is for a fix by hand during an incident, often a few gcloud commands rather than an apply, as the bootstrap log’s rows show. Such a fix is drift by definition: the daily plan reports it, and it ends when the fix is written into the code or the next apply reverts it. See Drift.

4. Record it

Where: landing-zone’s docs/bootstrap-log.md, one row by merge request, approved by the platform team.

Two records exist without anyone writing them: Privileged Access Manager keeps each grant with its requester, justification, approver and reason, and the audit log keeps every change made under it, and the reads of the services whose Data Access logs the audit root turns on: Secret Manager, Cloud SQL, Cloud Storage, IAM and Cloud KMS (enforced). The log row is a convention, written down and taught, but nothing refuses a grant without one. It names the grant ids, who requested and approved them, the root and the commit applied, what changed, and the empty plan after it. Never paste a credential, a secret or a token into it.

Break-glass when the gate cannot approve or cannot be reached

Every repository but one needs the gate’s approval before it can merge, and every pipeline but that repository’s own starts with gate:integrity, which fails closed when the gate does not answer. So a gate that is down blocks every fix, including a fix to the gate itself. control-plane’s ADR-0032 sets the way out in two stages. It applies when gate:integrity fails with gate unreachable: fail closed in more than one repository, or when the gate stops approving merge requests that its model accepts. A GitLab.com outage is not a gate outage, and nothing here helps with it.

The person who notices opens an issue in control-plane with the failing job’s link and keeps a timeline in it.

Stage 1: recover the gate

Read the symptom: https://gate.projectairframe.dev/health and the gate’s log lines in Grafana. Neither needs a grant.
If the latest revision is the cause, request a level 2 grant with the issue as the justification, have another member of the platform group approve it, and route Cloud Run’s traffic to the last revision that served. No new revision is created, so Binary Authorization is not asked again. In the rehearsal this took about 22 seconds each way.
If a credential is the cause, a maintainer adds a new version of the secret the gate reads, under the same grant. The value never goes into the issue.

The pipeline is not the way back while the gate is down: building the gate’s image needs the gate. The rollback is how it serves again, and the fix then follows as an ordinary merge request.

Stage 2: one repository out of both policies, for two hours

Only when stage 1 has not restored the gate within 30 minutes and a change in one repository cannot wait.

Where: security-policies, .gitlab/security-policies/policy.yml. One merge request, with nothing else in it, makes two changes for that repository: it leaves the scope of the required ownership gate approval policy, and it joins the exclusions of the group-wide pipeline execution policy, beside security-policies itself. security-policies is outside that policy so that this merge request does not wait on the gate it exists for.
Who approves: a named person from the catalog who is neither its author nor a committer. Today the named approvers are all the maintainer’s own accounts: the mechanism is enforced, but the separation between two people is not yet real.
The reverse merge request is opened at the same time and left waiting for its approval, so that ending the window is one approval and one merge.

In the repository, during the window, the urgent merge request merges on a named person’s approval and its own validation. The executor plans nothing for it, so the reviewer reads the change and the validation output, and its main pipeline applies only roots that take no image. Its pipelines carry no gate:integrity, so the workload identity bindings to the protected ref and the environment are what still stands between them and the clouds. Every other repository keeps both controls. A product repository cannot use the procedure, because its merge also waits for the “Airframe cost” status check, which only the gate sets.

Ending the window

The gate serves again: /health answers 200. If stage 1 pinned the traffic to an older revision, route it back to the latest.
Merge the reverse merge request. The policy file is then identical to what it was before the window.
Let the grant expire. The person who requested a grant cannot revoke it, because revoking needs a role nobody holds standing.
Close the issue with its timeline and the identifiers of GitLab’s audit events of the outage, read from the audit events pages of the group and of each project that changed. The audit receiver, a second streaming destination, keeps them in the locked audit bucket while the gate is down; if it was down too, GitLab does not send them again, and the types it only streams are not kept anywhere: say so in the issue. Then write a postmortem within five working days.

The two-hour bound is checked and not enforced. On control-plane’s scheduled pipeline, which runs once a day, breakglass:check compares security-policies’ policy file with the steady state, warns while a difference is under two hours old and fails after, naming the repositories. breakglass:report keeps one issue open until the policies match again. Only the reverse merge request ends a window.

What was rehearsed

On 2026-10-02 the procedure ran on a platform repository in two parts, recorded in landing-zone’s docs/rehearsals/2026-10-02-the-gate-unreachable.md. With the gate up, the approval rule left the repository 10 seconds after the first merge and returned 16 seconds after the reverse. Then the gate was made unreachable for two minutes: a pipeline in another repository failed closed 10 seconds after it started, security-policies’ pipeline was unaffected, and the grant’s requester could not revoke it, so it expired on its own. The rehearsal also showed that the gate cannot redeploy itself through its own pipeline while it is down. control-plane keeps a mirror of the policy file, and its policy:schema job fails while the two differ, until the reverse merge makes them equal again.

Where it lives

In the landing-zone repository:

In the security-policies repository:

.gitlab/security-policies/policy.yml: the gate policy's scope

In the control-plane repository: