A failed apply can leave a deployment status:"running" with a null endpoint — published app 404s forever, nothing logged

[Mcstabbin]

A deployment can end up status: "running" with a null endpoint, and then every request to it 404s forever with nothing logged anywhere.

Version: @yc-software/qm 0.1.4, target: aws, Postgres store, AWS deploy provider (Lambda MicroVMs).

What happened

The agent was asked to build and publish a small dashboard. It did, and reported success with a URL. Every request to that URL returned {"error":"not_found"} — by name and by id, and for the owner of the deployment, so it was not an ACL or auth failure.

The admin API reported the deployment as healthy:

{
  "id": "ea365971-…", "name": "open-issues", "ownerScopeId": "personal:U…",
  "status": "running", "currentVersion": 2, "versions": 2
}

The backing MicroVM was alive and had a perfectly good endpoint:

{ "state": "RUNNING", "endpoint": "….lambda-microvm.us-east-1.on.aws", "imageVersion": "2.0" }

Why it 404s

reachDeployment (src/deploy/deploy-service.ts:482-484):

const d = (await deps.deployStore.get(idOrName)) ?? (await deps.deployStore.getByName(idOrName));
if (!d || d.status !== "running" || d.endpoint == null) return { status: "not_found" };

Name resolution is fine (getByName fallback is present here and in getDeployment at :375), and I confirmed that by requesting the deployment id directly — same 404. d exists and status === "running", so by elimination d.endpoint is null.

markVersionRunning (src/deploy/deploy-service.ts:165-169) writes the endpoint before the status:

await deps.deployStore.setEndpoint(id, endpoint);
await deps.deployStore.setStatus(id, "running");
await deps.deployStore.setAppliedVersion(id, version);

so reaching this state seems to require applyVersion resolving to a null/undefined endpoint without throwing, rather than an ordering problem. I have not been able to prove which of the apply paths did it — see "what I could not determine" below.

The deployment had two versions and was on currentVersion: 2, so a second apply had run. My guess is that the failure is specific to the update path rather than first deploy, but that is a guess.

Why it is worse than a 404

Nothing surfaces. Specifically:

• The agent reported success and handed over a URL it had not verified.
• /ecs/<stack>-core logged zero events across the whole deploy window.
• The MicroVM log group was empty.
• The admin UI shows the deployment as Running, which is what the record says.

So the only signal that anything is wrong is a user clicking the link. There is no state anywhere that says "this deployment is unreachable".

Suggested fixes

1. Make the invariant explicit. status: "running" with a null endpoint is not a meaningful state — either refuse to set running without an endpoint, or add a distinct broken/degraded status so reachDeployment can return something more useful than not_found.
2. Distinguish "no such deployment" from "deployment exists but is unreachable" in the /d/ response. The current 404 sends you looking for a typo in the URL, which is where I spent my time.
3. Log it. A failed or partial apply currently leaves no trace in the service log group at all.
4. Surface it in the admin deployments list, so Running cannot mean "registered but unreachable".

What I could not determine

I could not read the deployment row directly (Postgres is private-IP only in this deployment), so d.endpoint == null is by elimination from the guard above rather than by observation — the other two conditions in that expression are independently confirmed. I also could not reproduce it on demand; it happened once, on a second apply, and I did not want to iterate on a live deployment to chase it. Happy to run any specific query or attempt a controlled reproduction if that would help.