refactor: Shorten ADR

Author: fubezz

docs/decisions/0006-cloud-run-otel-telemetry-export.md

@@ -51,29 +51,25 @@ enrichment.
 the fleet authenticates callers — the entire telemetry stack chain today trusts network
 location alone (reachable only from inside the VPC/cluster pod network). Adding
 authentication — e.g. Contour's `HTTPProxy.spec.virtualhost.jwtProviders` (Envoy's `jwt_authn`
-filter, which could verify Google-issued Cloud Run service-account identity tokens) or an
-OTel-side mechanism — is a valid improvement worth considering, but it would apply to the
+filter, which could verify Google-issued Cloud Run service-account identity tokens)
+is a valid improvement worth considering, but it would apply to the
 whole chain, not just this one gateway, and is out of scope here rather than solved as a
 side effect of this decision.
 
 ### Which collector instance Cloud Run talks to
 
 A new, vpc-only instance of the standard OTel push gateway collector: `collector-push-gateway`
-(same chart, unchanged architecture) deployed onto `shared-tools`, in loadbalancing mode, as
-`aignx-otel-gateway.aignostics.ai`, reachable only over the shared VPC via Direct VPC egress, never
-publicly. No existing instance serves this purpose. Every OTel collector running today serves
-GKE-internal traffic for a specific cluster/service set, none of them expose an ingress or
-accept traffic from outside a cluster's pod network.
+(same chart, unchanged architecture) deployed in `shared-tools`, in loadbalancing mode,
+reachable only over the shared VPC via Direct VPC egress, never
+publicly.
 
-### Network reachability — Direct VPC egress is not automatic
+### Reachability — Direct VPC connection
 
 Being in a GCP project attached to the shared VPC does not, by itself, give Cloud Run a path
 to reach anything on that VPC. Google's own docs confirm Shared VPC service-project membership
 only makes subnets *available* but Cloud Run must be explicitly configured with Direct VPC
 egress (a specific network + subnet) to actually route traffic there instead of the public
-internet. This matters because it's the actual gating factor for whether any Foundry Cloud Run
-service can reach `aignx-otel-gateway.aignostics.ai` at all, regardless of what this ADR
-decides about the gateway itself.
+internet.
 
 What's already in place: 44 Foundry service-projects have `enable_cloudrun_vpc_access = true`
 (`gcp-service-projects` Terraform module), each granting the Cloud Run service agent
@@ -83,20 +79,17 @@ permits Direct VPC egress, it doesn't configure it on any actual Cloud Run resou
 What's incomplete on the `foundry-python` template side:
 - The generated Cloud Run manifest (`service.template.yaml.jinja`) already carries the Direct
   VPC egress annotations (`run.googleapis.com/network-interfaces`,
-  `vpc-access-egress: private-ranges-only`) — but only rendered
-  `{% if service_cloudsql_enabled or service_memorystore_enabled %}`, since they were added for
-  the Cloud SQL proxy's private-IP connectivity, not for general reachability. A service using
-  neither gets no VPC egress configuration at all.
+  `vpc-access-egress: private-ranges-only`), but only rendered
+  if cloud sql or memorystore is enabled, not for general reachability.
 - Even where it does render, the subnet values feeding it (`vpc_network`/`vpc_subnetwork` in
   the service's own `infrastructure/<env>/project.hcl`) are manual placeholders
   (`PLACEHOLDER_VPC_NETWORK`/`PLACEHOLDER_VPC_SUBNETWORK`) filled in by hand after onboarding.
 
 Net effect: reachability isn't a solved problem for any Foundry Cloud Run service today,
 Cloud-SQL/Memorystore-enabled or not. This is the largest open risk for this ADR's rollout.
 
-Resolved direction: Direct VPC egress should be unconditional, every Cloud Run service gets
-it regardless of whether it uses Cloud SQL/Memorystore, not just the ones that happened to need
-it for a database connection.
+Resolved direction: Direct VPC egress will be unconditional, every Cloud Run service gets
+it regardless of whether it uses Cloud SQL/Memorystore.
 
 ### Alternatives considered — Sidecar collector vs. direct SDK export from Cloud Run
 
@@ -111,40 +104,30 @@ in the `foundry-python` Copier template, applied via the `deploy-cloudrun` GitHu
 Cloud Run's multi-container/sidecar support is exposed directly through that manifest —
 mechanically adding a second container is straightforward. The real benefit would be scaling.
 A sidecar scales automatically alongside each Cloud Run instance, so collector capacity always
-tracks the service's own traffic with no separate scaling decision to make. This is unlike a shared
-gateway, which has to be sized/scaled to handle the combined load of every Cloud Run service
-sending to it.
+tracks the service's own traffic with no separate scaling decision to make.
 Metrics are the hard blocker: our stack ingests metrics exclusively via
 Prometheus **scrape** (a `ServiceMonitor` targeting a stable in-cluster Service). There is no Thanos
 Receive enabled anywhere in the fleet, and we don't run a real Prometheus Pushgateway
 either. A sidecar bundled into an ephemeral, non-cluster Cloud Run instance has no stable,
-discoverable target for Prometheus to scrape, and nothing to push to instead. Its metrics
+discoverable target for Prometheus to scrape. Its metrics
 would simply never be, or with a static target configuration in Prometheus potentially too late,
 collected. This rules the option out.
 
 **Option B (chosen) — Direct SDK export to the shared instance**
-Cloud Run services export via the OTel SDK's own OTLP exporter straight to
-`aignx-otel-gateway.aignostics.ai`, no local hop. This also matches how this same repo already
-handles error tracking: Sentry is integrated as in-process SDK instrumentation
-(`sentry_initialize()` in `boot()`), not as a sidecar agent — there's no existing precedent
-anywhere in the Foundry stack for a sidecar-based telemetry agent, and introducing one just for
-OTel would be a new, one-off operational pattern rather than an extension of how the team
-already does this.
+Cloud Run services export via the OTel SDK's own OTLP exporter straight to the gateway, no local hop.
+This also matches how this same repo already handles error tracking: Sentry is integrated as in-process SDK instrumentation.
 
 Accepted risk: with no local buffer, telemetry the SDK hasn't flushed yet (spans queued in
 `BatchSpanProcessor`, metric points accumulated since the last periodic export) is lost if the
 Cloud Run instance is torn down first — scale-to-zero, revision replacement, or a crash. Cloud
 Run sends `SIGTERM` with a grace period (10s default, extendable) before `SIGKILL`; registering
 a shutdown hook that force-flushes the tracer/meter providers catches most of this, but it's a
 mitigation, not a guarantee (an abrupt kill or a grace period shorter than the flush can still
-drop data). Not different in kind from what a sidecar would risk on its own termination — a
-sidecar doesn't survive instance teardown any better than the SDK's in-process buffers do.
+drop data).
 
 Because every service points at this one gateway endpoint rather than a vendor-specific one,
 the export destination is switchable later (e.g. to Grafana Cloud or another vendor) by
-reconfiguring only the gateway's own exporters. No change to any Cloud Run service's
-instrumentation. A sidecar or GCP-native export would each hard-code a specific destination
-directly into every service instead.
+reconfiguring only the gateway's own exporters.
 
 ### Alternatives considered — Gateway Ingress
 
@@ -160,10 +143,6 @@ elegance for this one gateway.
 Contour's `HTTPProxy` has a native per-route `services[].protocol: tls` field — no
 backend-protocol annotation needed. Required adding `shared-tools` to the `contour-internal`
 ApplicationSet (envoy-internal IngressClass), which didn't run there before.
-This isn't just a local trade-off for `shared-tools`: Contour is the company's chosen
-long-term ingress controller, replacing ingress-nginx across the fleet, so this is consistent
-with (and a small step towards) where every cluster is headed, not a one-off choice for this
-gateway.
 
 
 ### Alternatives considered — k8s metadata enrichment for a non-k8s source
@@ -186,13 +165,12 @@ Wrap the Cloud Run service's startup command with the `opentelemetry-instrument`
 configure entirely through environment variables.
 Rejected: the actual Cloud Run startup command is generated per-service by the
 `foundry-python` Copier *template* (`deployment/cloudrun/service.template.yaml.jinja`), which
-only affects newly generated or explicitly re-templated (`copier update`) services. Unlike a
-library dependency, there's no single version bump that retroactively reaches every already
-generated service. Additionally this kind of instrumentation only provides us predefined options.
+only affects newly generated or explicitly re-templated (`copier update`) services.
+Additionally this kind of instrumentation only provides predefined instrumentation and helps in case of lack of codeownership.
 
 **Option B (chosen) — Programmatic init inside this repo's `boot()`**
 Add `otel_initialize()` (`aignostics_foundry_core.otel`), following the pattern already
-established for Sentry (`sentry_initialize()`): a pydantic-settings-driven `enabled` flag,
+established for Sentry: a pydantic-settings-driven `enabled` flag,
 called once from `boot()`, which every generated Foundry service already calls before
 instantiating FastAPI/uvicorn. One `aignostics-foundry-core` version bump reaches every service
 that upgrades its dependency — no per-service code or template changes for process-level
@@ -207,11 +185,7 @@ template wiring is a separate, small follow-up in that repo.
 **Option A — One `enabled` flag governing traces, metrics, and logs together**
 Simplest mental model, matches the original `sentry_initialize()`-style single switch.
 Rejected: log volume and cost characteristics differ meaningfully from traces/metrics (far
-higher per-request volume in most services), and there's no first-party loguru↔OTel
-integration — the logs bridge (a loguru sink converting into OTel's `LoggingHandler`) is
-custom code, not a vendor-supported path. Bundling all three risks surprising volume/cost for
-a service that only wanted tracing, and gives no way to turn off just one signal without
-losing the others.
+higher per-request volume in most services).
 
 **Option B (chosen) — Independent per-signal toggles under a master switch**
 `enabled` remains the overall kill switch; `traces_enabled` and `metrics_enabled` default to
@@ -221,23 +195,15 @@ affecting the others.
 
 ### GCP-native observability — not a considered alternative
 
-Whether to use GCP-native observability (Cloud Trace / Cloud Monitoring / Cloud Logging, which
-Cloud Run integrates with automatically) instead of the shared Tempo/Loki/Prometheus stack was
-never a live fork in this decision: [OP-3108](https://aignx.atlassian.net/browse/OP-3108)
-explicitly scoped this to sending Cloud Run telemetry into the existing shared stack, matching
-every other service in the portfolio.
-
-It's also worth noting this ADR doesn't actually preclude it either way: since instrumentation
-is configured entirely through the standard `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable
-(see below), any individual service remains free to point its SDK at Google's own OTLP-compatible
-endpoint instead of `aignx-otel-gateway.aignostics.ai` — that's an instrumentation-level
-configuration choice, not something requiring different infrastructure or a different decision
-here.
+It's also worth noting this ADR doesn't exclude it, since instrumentation
+is configured entirely through the standard `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable.
+any individual service remains free to point its SDK at Google's own OTLP-compatible
+endpoint instead of our solution.
 
 ## Decision
 
 Deploy `collector-push-gateway` (loadbalancing mode) on `shared-tools` as `aignx-otel-gateway`,
-exposed only internally via a Contour `HTTPProxy` at `aignx-otel-gateway.aignostics.ai`
+exposed only internally via a Contour `HTTPProxy`
 (reachable from Cloud Run over the shared VPC via Direct VPC egress, never publicly), backed by
 an explicit `cert-manager.io/v1 Certificate` and with `k8sAttributes.enabled: false` for this
 instance only.
@@ -279,23 +245,9 @@ individually. No sidecar collector, no CLI-wrapper instrumentation, no changes t
   automatic, only process-level tracing/metrics/manually-created spans are.
 - Reachability isn't guaranteed for any Foundry Cloud Run service today (see Network
   reachability above) — closing that gap is out of scope for this ADR, tracked as follow-up.
-- The gateway starts with conservative replica counts (1 push-gateway / 1 loadbalancer-backend)
+- The gateway starts with conservative replica counts (2 push-gateway / 2 loadbalancer-backend)
   since this is a new, currently-zero-traffic path — will need revisiting once real traffic
   lands.
 - Logs being opt-in means a service that only sets `{PROJECT}_OTEL_ENABLED=true` will not see
   its logs in Loki via this path — a support/documentation risk if not made clear in onboarding
   material (the companion runbook covers this).
-
-## Follow-ups
-
-- Make Direct VPC egress unconditional in the `foundry-python` template's Cloud Run manifest
-  (remove the Cloud SQL/Memorystore gating) — see Network reachability above. In review as
-  [foundry-python#385](https://github.com/aignostics/foundry-python/pull/385), with a companion
-  fix in [foundry-infrastructure#10](https://github.com/aignostics/foundry-infrastructure/pull/10)
-  (the `project.hcl` placeholders that workflow now unconditionally reads were under the same
-  gate).
-- Wire `instrument_fastapi(app)` into `Service.api()` in the `foundry-python` template — see
-  Harder/risks. Also in [foundry-python#385](https://github.com/aignostics/foundry-python/pull/385).
-- Revisit collector replica counts once real Cloud Run traffic is flowing — see Harder/risks.
-- Evaluate enabling `jwtProviders` on the gateway's HTTPProxy (or an OTel-side mechanism) — see
-  the authentication note in Context above.