Observability

Observability means being able to see what a running product is doing, through its logs, its metrics and its traces. On Airframe, a team gets it without building anything: its product’s signals travel to one Grafana Cloud stack, and its dashboards and alert rules come from its catalog entry. This page follows a signal from an application to Grafana, then lists what every product gets.

One Grafana Cloud stack

Grafana Cloud is a hosted service for logs, metrics, traces, dashboards and alerts. Airframe uses one stack of it, called projectairframe, for the whole organization rather than one per domain, so that a request crossing two domains can be followed from end to end. The environment a signal came from is a label, not a separate stack.

Grafana Cloud’s Free tier allows exactly one stack, so it was created by hand in Grafana’s portal during the bootstrap. Everything inside it is code: the operations repository’s Grafana root creates the teams, the folders and their permissions, and adds every person in the catalog who has a Grafana user name to the stack and to their teams.

The path of a signal

An application knows nothing about where its telemetry goes. It writes its logs as JSON lines to its standard output, sends traces and metrics with OTLP, the OpenTelemetry protocol, to the address the platform gives it, and never holds a Grafana credential. Everything between the application and Grafana is the platform’s.

Traces and metrics go straight to the edition’s OTLP gateway. With every batch, the service presents a short-lived identity token that names it, and the gateway admits only callers it has been told to trust.
Logs take a longer road. Cloud Run captures them into Cloud Logging, a sink in the product’s project sends them to a Pub/Sub topic that serves as a buffer, and the buffer pushes them to the edition’s logs gateway.
Browser events, such as errors and page timings, go to /_telemetry on the product’s own hostname, behind single sign-on, and the service forwards them with its identity.
Cloud resource metrics, such as a Cloud SQL instance’s CPU, are read from Google’s Cloud Monitoring by one small collector instance per edition, which sends them through the OTLP gateway.

The gateways run Grafana Alloy, Grafana’s telemetry collector, on Cloud Run: one pair per edition, one for OTLP and one for logs, because a Cloud Run service exposes a single port. They scale to zero. They drop credentials and noise, check each signal’s labels, and forward to Grafana with the edition’s write-only token, the only Grafana credential on that side. A product needs no setup: its address for the gateway arrives in its project factory’s Ready output, and the golden path’s modules pass it to every service.

Labels that say where a signal came from

Two labels are checked on every signal. The env label is the environment: prod, np, sandbox, eph-<iid> for a merge request’s preview, or platform. The OTLP gateway keeps it only when it is valid for its own edition and replaces a missing or foreign value with unknown; the logs gateway keeps a valid one and otherwise gives the line its perimeter’s, so only audit entries stay unknown. Both stamp the edition as perimeter. Where it comes from: the OTLP gateway takes it from deployment.environment.name or deployment.environment when the resource has no env, and the golden path’s Cloud Run module sets the latter on each service. The platform agent sets it on every series it collects, from the project and, in non-production, from the Cloud Run service’s name (sandbox or eph-<iid>). The logs gateway narrows the perimeter’s the same way and gives a platform project’s lines platform. The platform’s own scheduled jobs declare platform on their resource, so their series carry env="platform"; the cost report keeps the environment of the project a cost belongs to in a separate label, project_env. The drift report’s series still arrive as unknown until the next platform release, which brings the component that declares it.

The other labels depend on the signal’s path. The golden path’s Cloud Run module gives a service’s traces and metrics its service name, its domain and its environment, as the OpenTelemetry attributes service.name, service.namespace and deployment.environment. The platform agent labels the Cloud resource metrics of each billed product project with product, the product’s path, and domain, its journey and domain. The logs gateway keeps each log line’s project, service, resource type and severity. An application’s signals carry no team or criticality label.

What every product gets

The Grafana root runs in the chain of reconcilers after each catalog merge. From each product’s entry, it creates:

a folder for the product, which its team can edit and every member can view;
three dashboards: the service (requests, errors, latency, instances, resources), the SLO, or service level objective, for its reliability targets, and the cost, with the month to date and the forecast against monthly_budget, orange at 80 percent and red at 100. The service and SLO dashboards open on prod, and their first filter, env, switches the environment;
baseline alert rules, with thresholds from the entry’s slo: the error budget burning fast or slowly, more than 2 percent of server errors, latency above the objective or twice it, Cloud SQL near its limits, Pub/Sub messages waiting more than ten minutes, and a dead-letter topic that is not empty. They judge production alone, and a missing series is not an alert.

The alerts route by the team’s label to the team’s contact point, which emails the team’s members who have a Grafana account, and a page repeats every four hours until it is resolved. Grafana Cloud accepts only its own members’ addresses, and today those members are the maintainer’s own accounts, so every team’s alerts reach the maintainer.

The cost dashboards read the billing export, which a daily job turns into metrics; the FinOps page follows that budget.

A product’s own dashboards

A team that wants more than the three dashboards writes them as code in its own repository: a call of the golden path’s grafana-dashboard module in the product’s infra/grafana root, with the panels in Terraform or a dashboard exported from Grafana’s interface.

That root holds no Grafana credential. It runs with the product’s production apply identity, in the GitLab environment grafana, and writes each dashboard as a file under the product’s own prefix in a bucket. The platform’s Grafana root reads the bucket and places each dashboard in the product’s folder, tagged product-owned. The repository is the version of record: a dashboard changed only in Grafana’s interface is overwritten at the next reconcile, so a team exports it into its root to keep it.

Previews and the sandbox are observable too

Previews and the sandbox send through the non-production gateway like everything else. They carry their own env, so that a reviewer sees a merge request’s preview on the same dashboards by choosing eph-<iid>: the golden path sets deployment.environment to that value on the preview’s service, the OTLP gateway takes it as env, and the platform agent and the logs gateway derive it from the service’s name. The alert rules select production alone, so previews and the sandbox never page anyone.

The platform’s own

The platform’s scheduled jobs, such as drift detection, the preview collector and the cost and coverage reports, push their own metrics. A platform folder holds the overview, the platform’s cost and the delivery measures, and its rules send an email to the platform’s contact point, for example when a scheduled pipeline has not reported in a day. The gate has an uptime check of its own: a Grafana Cloud Synthetic Monitoring HTTP check requests its /health address every 60 seconds from two probes, and a rule pages the platform when more than one probe fails for two minutes, or when the check reports nothing (enforced). A second rule group watches the audit entries a narrow sink at the folder sends to Loki, such as a key creation attempt, a policy refusal or a grant (trust boundaries). Today that address is the maintainer’s own account too; the platform team’s group address takes over once it is a member of the stack.

Designed, not built

On-call (designed). Grafana’s on-call and incident tool would hold a schedule and an escalation chain for each team, built from the team’s on-call block in the catalog. Today an alert is an email.
The operations agent and the audit agent (designed). The first would investigate an alert in the tier the entry’s agentic_ops allows; the second would look across the organization for waste and orphans. Neither exists today.
Team and criticality on an application’s signals (designed). Its traces, metrics and logs carry no such label.
The rest of the baseline (designed). The baseline also names Cloud Run at its maximum instances, restarts or out-of-memory, and a contract check blocked on main; the Grafana root does not create these rules yet.
Uptime checks for products (designed). Uptime checks through each product’s production hostname are the next version of the telemetry root. Only the gate has one today (above).
A product’s own alert rules and objectives (designed). The grafana-alert and grafana-slo modules would publish them as grafana-dashboard publishes dashboards; the baseline covers alerts meanwhile.

The always-on cost is about USD 7 a month for the non-production collector instance; the production one fits Google’s always-free allowance, the gateways scale to zero, and Grafana stays on its Free tier.

Where it lives

In the operations repository:

In the golden-path repository:

modules/grafana-dashboard/: a product's own dashboard as code

In the landing-zone repository:

docs/guides/observability.md: the observability guide