Grafana Cloud Deployment

Alloy exports all signals to Grafana Cloud (Tempo, Mimir, Loki) when credentials are configured. When credentials are absent, cloud exporters are no-ops — local backends are unaffected.

This page documents the deploy-local.sh collector/export path and Azure Key Vault credential model. It is not the application-release promotion path: CD takes collector endpoints and any observability-gate token from protected GitHub Environments without rebuilding application images. See Immutable CI/CD Promotion for that boundary. The collector itself — as opposed to its credentials — is promoted to real dev/qa/prod clusters by deploy-environment.sh’s apply-collector phase, never by this page’s deploy-local.sh flow; see ADR-004 for the full reconciliation + rollback contract.

Credential architecture

Grafana Cloud uses per-signal instance IDs as Basic Auth usernames — one for Tempo (traces), one for Mimir (metrics), one for Loki (logs). A single shared API key is the password for all three signals.

flowchart TD
    Token["Shared Access Policy token (glc_...)<br/>grafana-example-org-alloy-writer-example-org-token<br/>Scopes: metrics:write logs:write traces:write"]

    subgraph GC["Grafana Cloud Stack"]
        Tempo["Tempo (traces)<br/>instance ID: 1541184<br/>tempo-prod-xx.grafana.net"]
        Mimir["Mimir (metrics)<br/>instance ID: 3102416<br/>prometheus-us-central2.grafana.net"]
        Loki["Loki (logs)<br/>instance ID: 1546883<br/>logs-prod-037.grafana.net"]
    end

    Token -->|authenticates| Tempo
    Token -->|authenticates| Mimir
    Token -->|authenticates| Loki

Important: The Grafana Cloud API key (grafana-example-org-cloud-api-key, prefix glsa_) is a Grafana organisation service account token — it authenticates with the Grafana frontend (example-org.grafana.net) but is rejected by Mimir/Loki/Tempo data ingestion endpoints (HTTP 401). Always use grafana-example-org-alloy-writer-example-org-token (prefix glc_) for data plane writes.

Endpoint format requirements

The raw Grafana Cloud endpoint URLs require path/format adjustments before they work with Alloy:

Signal	Raw Grafana Cloud URL	Required format
Traces (OTLP gRPC)	`https://tempo-prod-29-....grafana.net`	`tempo-prod-29-....grafana.net:443` — strip `https://`, append `:443`
Metrics (Prometheus remote_write)	`https://prometheus-us-central2.grafana.net/api/prom`	`https://prometheus-us-central2.grafana.net/api/prom/push` — append `/push`
Logs (Loki push)	`https://logs-prod-037.grafana.net`	`https://logs-prod-037.grafana.net/loki/api/v1/push`

Why Prometheus remote_write, not OTLP HTTP, for metrics? The Helm chart’s Alloy destinations use type: prometheus, which speaks Prometheus remote_write. App OTLP metrics arriving at alloy-receiver are converted OTLP → Prometheus inside Alloy before shipping. This gives a single ingestion path for scraped infra metrics and converted app metrics, and matches Grafana Cloud’s dashboard/query UX.

Why the difference for Tempo? gRPC uses HTTP/2 transport. Alloy’s otelcol.exporter.otlp expects a host:port endpoint without a URL scheme.

scripts/fetch-grafana-cloud-conf-from-akv.sh applies these adjustments automatically — append /push to the Mimir URL if missing, append /loki/api/v1/push to Loki, append :443 to Tempo.

Azure Key Vault integration

Credentials are stored in Azure Key Vault (example-org-prd-kv) under the grafana-example-org-* prefix. The AKV coordinates (tenant/subscription/RG/vault name) live in the env file named by conf.yml’s monitoring.grafana_cloud.use_env — as ARM_TENANT_ID / ARM_SUBSCRIPTION_ID / Resource_Group / Azure_KeyVault, the same keys the legacy Makefile flow (make secrets-fetch-akv) already reads from there. Safe to track in git — these are IDs/names, not credentials.

AKV secret name	env file key (== K8s Secret key)	Notes
`grafana-example-org-alloy-writer-example-org-token`	`GRAFANA_CLOUD_API_KEY`	`glc_` access-policy token — required for data-plane writes
`grafana-example-org-cloud-tempo-endpoint`	`GRAFANA_CLOUD_TEMPO_ENDPOINT`	fetch script appends `:443`
`grafana-example-org-cloud-tempo-username`	`GRAFANA_CLOUD_TEMPO_USER`
`grafana-example-org-cloud-mimir-endpoint`	`GRAFANA_CLOUD_MIMIR_ENDPOINT`	fetch script appends `/push` if missing
`grafana-example-org-cloud-mimir-username`	`GRAFANA_CLOUD_MIMIR_USER`
`grafana-example-org-cloud-loki-endpoint`	`GRAFANA_CLOUD_LOKI_ENDPOINT`	fetch script appends `/loki/api/v1/push` if missing
`grafana-example-org-cloud-loki-username`	`GRAFANA_CLOUD_LOKI_USER`
`grafana-example-org-faro-api-endpoint`	`FARO_COLLECTOR_URL`	frontend runtime env
`grafana-example-org-faro-sourcemap-token`	`FARO_API_KEY`	webpack build arg

fetch-grafana-cloud-conf-from-akv.sh rewrites only the keys above. Two further keys are optional, hand-maintained in the same env file, and read only by scripts/push-dashboards-to-grafana-cloud.sh (see Dashboards as code):

env file key	Notes
`GRAFANA_CLOUD_STACK_URL`	Grafana control-plane URL, e.g. `https://example.grafana.net` (no trailing `/`)
`GRAFANA_CLOUD_DASHBOARD_TOKEN`	Grafana service-account token, scope `dashboards:write`. A `glsa_` token — a Grafana API call, not an Alloy data-plane write, so `glc_` does not apply

Dashboards as code

k8s/monitoring/dashboards/*.json is the single source for both modes. Every dashboard carries an environment template variable (and service/service_version where relevant) — panel queries scope on $environment the same way config/observability/slos.yaml does, and datasource references are type: datasource template variables (DS_PROMETHEUS/DS_LOKI/DS_TRACES) rather than a hardcoded uid:

local — mounted through the grafana-dashboards ConfigMap embedded in k8s/monitoring/local/grafana/dashboards-configmap.yaml. scripts/ci/validate_observability.py fails the build if that embedded copy drifts from k8s/monitoring/dashboards/, if any panel query names a metric the pipeline never emits, if a dashboard has no environment template variable, or if a panel query is not scoped by $environment.
cloud (manual) — ./scripts/push-dashboards-to-grafana-cloud.sh (--dry-run reports create/update per dashboard; no flag pushes, then verifies live content digest/version/datasource wiring; --check prints the local bundle digest with no network; --snapshot-dir/--restore-dir capture/restore live state). It resolves credentials the same way push-slo-rules-to-mimir.sh does — the monitoring.grafana_cloud.use_env env file — and needs the two keys above. The target stack must expose data sources under the UIDs each dashboard’s DS_* variables default to (prometheus, jaeger, loki). Its payload/digest logic lives in scripts/ci/lib/dashboard_payload.py; the network reconcile/snapshot/restore/verify logic lives in scripts/ci/lib/grafana_dashboards.sh (shared with the CD phase below). Both are regression-tested in the blocking CI job.
cloud (CD promotion) — scripts/ci/deploy-environment.sh’s apply-dashboards phase runs immediately after apply-observability-rules for every DEPLOY_ENABLED=true environment, using a dedicated GRAFANA_CLOUD_DASHBOARD_TOKEN (dashboards:write only — never the GRAFANA_CLOUD_MIMIR_RW_TOKEN used for rules). It reproduces the release manifest’s promoted dashboards.bundle_sha256 from the checked-out commit before pushing, snapshots live state into the same OBSERVABILITY_ROLLBACK_DIR apply-observability-rules already exports (so the existing encrypted rollback-evidence upload captures it automatically), pushes, and verifies live content digest/version/datasource wiring. A failed release restores the prior dashboard state via deploy-environment.sh rollback. See ADR-013.

Cardinality / FinOps: the dashboards add no series — they only read. The validator’s metric-reference check is the guard that a panel cannot silently depend on a metric that is not in slo-rules.yaml or the known-emitted set, which is where an unbudgeted query would otherwise creep in.

Setup

1. Azure auth

The fetch script first resolves the target env file from conf.yml’s monitoring.grafana_cloud.use_env, then sources it to load ARM_TENANT_ID / ARM_SUBSCRIPTION_ID / Resource_Group / Azure_KeyVault (vault coordinates) and ARM_CLIENT_ID / ARM_CLIENT_SECRET (service-principal credentials, if set).

# Interactive login (your user credentials) — used if ARM_CLIENT_ID/ARM_CLIENT_SECRET
# are empty in the env file:
az login

Or fill in ARM_CLIENT_ID / ARM_CLIENT_SECRET in the env file itself for service-principal auth — the script logs in as that principal automatically.

2. Fetch into the env file (in place)

# Preview the changes:
./scripts/fetch-grafana-cloud-conf-from-akv.sh --dry-run

# Apply in place (creates <env file>.bak):
./scripts/fetch-grafana-cloud-conf-from-akv.sh

The script updates only the nine GRAFANA_CLOUD_*/FARO_* keys in the env file named by use_env (appending any that don’t exist yet). Comments, ordering, and every other line (including the ARM_*/Resource_Group/Azure_KeyVault coordinates) are preserved — see smoke-test-conf-updater.sh for the regression test that enforces this.

3. Deploy (re-materialises the Secret)

./deploy-local.sh --skip-cluster --skip-build

deploy-local.sh (see Local Deployment) sources the same env file named by monitoring.grafana_cloud.use_env (required — it fails fast if empty or missing) and writes the grafana-cloud-secrets K8s Secret into both otel-lab (apps/FARO consumers) and monitoring (Helm chart’s Alloy). Before helm upgrade, a contract validator asserts every key referenced by the rendered values file is present in the Secret — rename on either side fails fast.

Verifying cloud export is working

# Check Alloy receiver logs for successful exports
kubectl -n monitoring logs daemonset/grafana-k8s-alloy-receiver --tail=100 \
  | grep -E "grafana_cloud|export|error" | head -30

# Generate a trace
curl -s http://localhost:8080/api/projects

# Then check Grafana Cloud:
# → Explore → Tempo → search by service.name=gateway-api
# → Explore → Mimir → query: traces_spanmetrics_calls_total
# → Explore → Loki → query: {namespace="otel-lab"}

Expected log output when working:

level=debug component=otelcol.exporter.otlp.grafana_cloud_traces msg="successfully exported"
level=debug component=otelcol.exporter.otlphttp.grafana_cloud_metrics msg="successfully exported"

Troubleshooting

“endpoint is empty” in Alloy logs

The Secret was not applied or Alloy was not restarted after applying it.

# Verify the secret exists
kubectl -n otel-lab get secret grafana-cloud-secrets -o json | jq '.data | keys'

# Verify Alloy reads the env var
kubectl -n monitoring exec daemonset/grafana-k8s-alloy-receiver -- env | grep GRAFANA

# If env var is missing, Alloy needs a restart to pick up the new secret
kubectl -n monitoring rollout restart daemonset/grafana-k8s-alloy-receiver

“401 Unauthorized”

Wrong API key or wrong instance ID for that signal type.

make secrets-show   # verify all 7 values are non-empty and correct

Each signal type has its own instance ID. Using the Tempo ID for Mimir (or vice versa) causes 401 errors on that signal only.

“connection refused” for Tempo

Tempo endpoint must be host:443 without https://. If it includes https:// the gRPC transport fails.

make secrets-show
# GRAFANA_CLOUD_TEMPO_ENDPOINT should be:  tempo-prod-xx....grafana.net:443
# NOT: https://tempo-prod-xx....grafana.net

Re-run make secrets-fetch-akv to re-apply the adjusted format.

AKV authentication failing

source .env
az login --service-principal \
  --username  "$ARM_CLIENT_ID" \
  --password  "$ARM_CLIENT_SECRET" \
  --tenant    "$ARM_TENANT_ID"

# Verify SP has Key Vault Secrets User role
az keyvault show --name example-org-prd-kv \
  --query "properties.accessPolicies[?objectId=='<SP_OBJECT_ID>']"

# List available secrets
az keyvault secret list --vault-name example-org-prd-kv \
  --query "[?starts_with(name,'grafana-example-org')].name" -o tsv

Graceful degradation

When cloud credentials are absent:

optional: true on every secretKeyRef means Alloy pods start normally
Cloud exporters log: level=error msg="failed to export" err="endpoint is empty"
Local backends (Jaeger, Prometheus, Loki if deployed) receive all signals normally
No reconfiguration needed to switch modes — just apply or remove the Secret

Credentials for the CD release gate and rule promotion

The credentials above are for the data plane (deploy-local.sh → Alloy writes). The application-release promotion flow (.github/workflows/cd.yml) needs two more sets, held as GitHub Environment secrets/variables, not in AKV or conf.yml:

Purpose	Names	Access policy scope
Observability gate reads Grafana Cloud (`scripts/ci/observability_gate.py`)	vars `OBS_GATE_MIMIR_QUERY_URL`, `OBS_GATE_LOKI_QUERY_URL`, `OBS_GATE_TEMPO_QUERY_URL`; secrets `OBS_GATE_MIMIR_USER`, `OBS_GATE_LOKI_USER`, `OBS_GATE_TEMPO_USER` (the per-signal instance IDs), `OBS_GATE_TOKEN`	`metrics:read logs:read traces:read`
`apply-observability-rules` loads SLO rules + Alertmanager routing	secrets `GRAFANA_CLOUD_MIMIR_TENANT`, `GRAFANA_CLOUD_MIMIR_RULER_URL`, `GRAFANA_CLOUD_ALERTMANAGER_URL`, `GRAFANA_CLOUD_MIMIR_RW_TOKEN`	`metrics:write rules:write alerts:write`
Alert receivers	secrets `SLACK_WEBHOOK_URL`, `PAGERDUTY_ROUTING_KEY` (PROD)	n/a

Query endpoint hostnames are region-specific; if DEV/QA/PROD are separate Grafana Cloud stacks each gets its own set, if they are one stack with X-Scope-OrgID tenants the URLs match and the tenant differs. mimirtool derives the Ruler/Alertmanager address from GRAFANA_CLOUD_MIMIR_RULER_URL (strip any /api/prom/push suffix). The manual counterpart is ./scripts/push-slo-rules-to-mimir.sh [--alertmanager <env>], which reads the same nine data-plane keys from conf.yml’s use_env file.

Credential rotation

When the Grafana Cloud API key is rotated:

Update the secret in AKV:

az keyvault secret set --vault-name example-org-prd-kv \
  --name grafana-example-org-cloud-api-key --value "glsa_newtoken..."

Re-fetch and apply:
```
make secrets-fetch-akv
```
Alloy picks up the new env vars automatically on the next pod restart (or force it):
```
kubectl -n monitoring rollout restart daemonset/grafana-k8s-alloy-receiver
```

The old key remains valid until explicitly revoked in the Grafana Cloud Access Policies UI.