Dashboards, alerts and drift

Every product on Airframe gets dashboards and alert rules from its catalog entry, and every repository gets an issue when the cloud stops matching its code. This page is the how-to for using them day to day: where a product’s dashboards are and how to read them, how to add a dashboard of your own, what to do when an alert arrives, and how to deal with a drift issue. Observability explains how the signals travel and what each product gets; this page does not repeat it.

1. Read a product’s dashboards

Who can see them: every person in the catalog who has a Grafana Cloud account and a grafana_username in their file under people/. A change to that field is a catalog merge request that a lead of the person’s team approves. The Grafana root then adds the person to the stack and to their teams.

1. Open the one Grafana Cloud stack, projectairframe, and browse its folders: one per journey, domain, context and product, nested in that order. The demo organization’s orders product is under shop, store, ordering, orders.
2. The product’s folder holds three dashboards, named after the product: <product> service, <product> SLO and <product> cost. The product’s team may edit the folder; everyone else may view it.
3. The service dashboard shows requests, the share of server errors (red above 2 percent), latency against the entry’s objective (orange above it, red at twice it), instances, CPU and memory, the Cloud SQL and Pub/Sub panels, and the logs.
4. The SLO dashboard shows availability and the error budget left over 30 days against the entry’s slo, and how fast the budget burns.
5. The cost dashboard is on Cost and billing.

Each dashboard has an env filter that opens on prod, and a service filter. The filter selects by the env label, which the platform sets: the OTLP gateway from the application’s deployment.environment when the resource has no env, the platform agent on every series it collects, from the project, and the logs gateway from the entry’s label or else its perimeter’s. At the OTLP gateway a signal whose label is not valid for its edition is marked unknown; at the logs gateway only audit entries are. No demo product is billed today, so no product’s dashboards have data yet.

2. Add a dashboard of your own

A team that wants more than the three dashboards writes them as code, in its own repository. Grafana’s interface is fine for drafting, but a dashboard changed only there is overwritten at the next reconcile.

1. Where: the product’s repository, infra/grafana/main.tf. The template already renders one call there, overview, as a starting point.
2. Add a call of the golden path’s grafana-dashboard module with:
   • ready, the product’s Ready output, as the starter reads it;
   • name, a short slug unique in the product, never service, slo or cost, which the platform uses;
   • title, the name Grafana shows;
   • dashboard, with at least panels. A dashboard exported from Grafana’s interface works as jsondecode(file(...)); its own id, uid, version, folder and title are dropped, and data sources are named by type.
3. Run airframe validate infra/grafana, then open a merge request.
4. Who approves: a member of the product’s team other than the author, then the gate. Where no team member has a GitLab account, the platform team stands in; today its approvers are the maintainer’s own accounts, so the separation between two people is not yet real.
5. Merge. The pipeline plans, checks and applies the root as plan:grafana, policy:grafana and apply:grafana. The apply holds no Grafana credential: it runs with the product’s production apply identity and writes the dashboard as a file under the product’s prefix in the platform’s bucket. Then grafana:reconcile asks the gate to run the operations pipeline, whose Grafana root places the dashboard in the product’s folder, tagged product-owned.

Removing a call removes the dashboard at the next reconcile. A product’s own alert rules and objectives, through grafana-alert and grafana-slo modules, are designed; the baseline covers alerts meanwhile.

3. Check the application’s side

A product needs no setup for its telemetry, but the application has to keep to three habits, which the golden path’s adapters already follow:

• write logs as JSON lines on standard output, with severity, trace_id and span_id;
• send traces and metrics with OTLP to the address the platform injects, and send nothing when none is set;
• never hold a Grafana credential.

The rest of the path is the platform’s, as Observability describes.

4. Act on an alert

An alert reaches the product’s team by email, from the team’s contact point, and only the members who have a Grafana account receive it. Its title names the rule, such as Latency p95 above twice the SLO, and its summary names the product, the service, the value and the target. A rule is either a page or a ticket, and a page repeats every four hours until it resolves.

1. Open the product’s service and SLO dashboards for the time of the alert, and read the logs panel.
2. Where: the product’s runbook, a file in its repository that the team and its on-call people own. Follow it.
3. Fix it with a merge request. A rollback is a revert merge request, planned and applied like any other; the previous Cloud Run revision stays available meanwhile.
4. If a fix by hand cannot wait, ask for a level 2 grant, as Grants and break-glass describes, and write the fix into the code afterwards.
5. Write what happened in postmortems/, one file per incident.

Today every team’s alerts reach the maintainer, because Grafana Cloud accepts only its members’ addresses and the maintainer’s accounts are its only members. On-call schedules and escalation are designed; today an alert is an email. The baseline judges production by the same env label.

5. Read and close a drift issue

Where: the issues of the repository that owns the root, labeled audit::drift and titled Drift: <root>. The daily schedules plan every root at 06:00 UTC in landing-zone, 06:20 in control-plane and 06:40 in operations; product repositories have schedules of their own. Drift explains why.

The issue lists the resources that differ and the action the plan would take on each, never their values: the plan of a merge request shows those. Find who changed what in the audit log, then act on what you find:

What you find	What to do
A person changed a resource by hand	Merge nothing: the root’s next apply from main puts it back
A provider default or an upstream change	A merge request that adopts or pins it; its plan shows no change
A value stored differently than written, such as a duration	Write it the way it is stored
A root that stays laptop-applied drifted	Apply it under a grant, and record the grant
The plan failed	Fix the root or its identity’s roles in a merge request
A check fails on an empty plan	Do what the check’s message says

Do not close a drift issue by hand. The scheduled run that finds the root matching again adds a note, Converged, and closes it. An issue closed while the drift lasts comes back at the next scheduled run as a new one, because the report looks only for open issues.

Where it lives

In the operations repository:

• roots/grafana/: folders, dashboards and alert rules from the catalog
• docs/telemetry.md: the telemetry path

In the golden-path repository:

• modules/grafana-dashboard/: a product's own dashboard

In the control-plane repository:

• templates/drift-report/: the drift issues

In the landing-zone repository:

• docs/guides/operations.md: drift and what to do with it