Supply-chain security

What CI and CD verify before an image can reach an environment: source/repository policy, vulnerability scanning, SBOM generation, keyless signing/attestation, and immutable digest verification.

.github/workflows/ci.yml first runs a blocking quality gate (Gitleaks, repository policy, unit/build/protobuf validation, CodeQL, SCA, Trivy IaC scan, and observability-as-policy). Its build-images job then runs a matrix across the four services (otel-frontend, gateway-api, order-api, notification-svc) and, for each:

Build the image with Docker buildx (load: true so the following steps see the same artifact).
Trivy scan for HIGH/CRITICAL CVEs with an available fix (ignore-unfixed: true). Fails the job if any are found.
SBOM generation with Syft (CycloneDX JSON) — uploaded as an artifact.
Push the already scanned local image to GHCR (main branch only), then record the registry-reported digest.
Sign with cosign keyless OIDC (main branch only).
Attest the SBOM via cosign attest --type cyclonedx — binds the SBOM to the image digest.
Verify the signature and attestation just produced, on the same digest, in the same job (main branch only) — see §Signing below.
Publish an attempt-specific release-manifest.json only after all four service records are present, scanned, pushed, signed, attested, and verified.

PRs run steps 1–3 only; push/sign/verify require id-token: write + GHCR auth, which we restrict to main.

A separate manual-only workflow, scheduled-vuln-rescan.yml, re-scans the last-published :latest image for each service when dispatched — same Trivy severities, but without ignore-unfixed, so a CVE that had no fix at build time still gets caught once one appears upstream. Report-only (exit-code: "0"), visible in the Security tab; it doesn’t block anything since there’s no PR or release transaction to fail. Its schedule is commented until an owner adopts an operational cadence; latest is never a CD input.

Promotion verification

CD consumes a successful main CI run ID, not a tag or source ref. It validates the GitHub run and its attempt-specific manifest, then passes the unchanged four-image digest set through DEV, QA, and optional protected PROD. The reusable deployment workflow verifies every digest exists in GHCR and verifies its Cosign signature and CycloneDX attestation against the trusted CI workflow identity before it writes kubeconfig or applies a Kubernetes object.

This is supply-chain verification at the deployment boundary; it is not a second image build. The renderer changes only environment-specific ConfigMaps, ingress host/TLS configuration, labels, and secret references. It replaces local application image markers with repository@sha256:... values from the manifest and rejects a partial service set.

See Immutable CI/CD Promotion for environment prerequisites, release ordering, live gates, and rollback behavior.

Trivy policy

Severities: HIGH,CRITICAL. We don’t block on MEDIUM because it triples noise without proportional security value.
ignore-unfixed: true: a HIGH CVE with no upstream patch would fail every CI run forever — nothing we can do about it beyond pinning to an alternative base image. Unfixed findings are still surfaced in the SARIF upload (GitHub Security tab) so they’re visible, just not blocking.
Scan scope: os,library — OS packages (apt/apk) + language dependencies (nuget, npm, pip). Trivy picks these up from the image layers.

SARIF output is uploaded to GitHub Advanced Security. Historical trend, dismissals, and PR-view annotations live in the “Security” tab of the repo.

SBOM format

CycloneDX JSON (cyclonedx.org/specification). Attach-rather-than-embed — the SBOM is an OCI referrer on the image manifest, not inlined into the image. This keeps the image itself minimal and lets consumers download SBOMs without pulling the image:

cosign download sbom ghcr.io/OWNER/signal-forge/gateway-api@sha256:...

Signing: keyless OIDC

Every image gets a cosign signature using the GitHub Actions OIDC identity — no long-lived keys. The signature’s certificate embeds:

The repo (OWNER/signal-forge)
The workflow path (.github/workflows/ci.yml)
The ref that produced the signature (refs/heads/main)
The commit SHA

Verified automatically: the build-images job runs cosign verify and cosign verify-attestation --type cyclonedx against the exact digest it just signed, in the same job, before the workflow completes. CD repeats the same identity-bound checks immediately before an enabled deployment. The two boundaries catch issuance/attestation errors before release metadata is published and prevent an unverified manifest digest from reaching a cluster.

Verify downstream yourself, any time:

cosign verify ghcr.io/OWNER/signal-forge/gateway-api@sha256:... \
  --certificate-identity-regexp "https://github.com/OWNER/signal-forge/.github/workflows/ci.yml@.*" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Admission-time enforcement (refuse to schedule an unsigned image at deploy time) is still not wired up in this repo — that’s a cluster-scoped decision, and a materially bigger lift than verifying in CI (it needs an admission controller installed in the target cluster). See §“Admission enforcement” below. What changed here is narrower but real: “we sign our images” used to mean the signature was produced and never checked again by anything; now it’s checked, by CI, on every push.

Digest pinning in base images

The Dockerfiles pin every FROM line by @sha256:...:

FROM mcr.microsoft.com/dotnet/aspnet:8.0@sha256:f88c77644f4c480a62d3b46dc74db8d5472a24e282df8b1e56195c689d35a6db

Why this matters even when CI is fine:

The :8.0 tag is mutable — Microsoft publishes new :8.0 images every month for security updates. Without a digest pin, a rebuild tomorrow might pull a different image than CI verified today.
Reproducibility for incident forensics: if production had CVE-X in its image, you can find the exact bytes that shipped.
Zscaler / corporate TLS rewriting does not alter image content (only TLS certs), so pinning is safe behind corporate proxies.

Refreshing pinned digests

Monthly-ish cadence. Fetch fresh digests:

for img in mcr.microsoft.com/dotnet/sdk:8.0 mcr.microsoft.com/dotnet/aspnet:8.0 \
           python:3.12-slim node:20-alpine nginxinc/nginx-unprivileged:alpine; do
  digest="$(docker buildx imagetools inspect "$img" --format '{{.Manifest.Digest}}')"
  echo "$img → $digest"
done

Paste the new sha256:... into the corresponding FROM line in each Dockerfile. Commit as chore(ci): bump base image digests (YYYY-MM-DD).

If a new digest introduces a regression, revert the Dockerfile change — the image is cached in the previous digest for ~90 days on Docker Hub / MCR, so pinning back is safe.

Admission enforcement (not implemented)

To reject unsigned images at deploy time, install one of:

sigstore/policy-controller — Kubernetes admission controller that verifies cosign signatures. Configure with a ClusterImagePolicy that requires images from ghcr.io/OWNER/signal-forge/* to have a signature chaining back to OWNER/signal-forge’s workflow.
connaisseur — similar, supports multiple signature backends (cosign, notary v1).
kyverno with verifyImages rules — already a multi-purpose admission controller, can also verify SBOM attestations.

None of these are installed in the lab. CI and an enabled CD deployment both verify signatures and attestations before apply (see §Signing above), but no cluster admission policy independently refuses an unsigned image at scheduling time.

What this doesn’t cover

Dependency pinning in app code: dotnet restore, npm ci, pip install -r requirements.txt respect lockfiles but we don’t run npm audit fix / Dependabot proactively. PRs pass dependency-vulnerability scans (dotnet list package --vulnerable, pip-audit, npm audit --audit-level=critical in the test-frontend job); update cadence is ad-hoc.
Container image provenance (SLSA): our signing stops at “this image was built by this workflow”; it doesn’t assert SLSA L3 hermeticity. For that, use slsa-framework/slsa-github-generator to produce a SLSA provenance attestation alongside the SBOM.