Summary
Any error returned by a Kubernetes API call during a CHI reconcile is treated as fatal: the error propagates up unchanged, reconcileCR coerces it to ErrCRUDAbort, and the whole reconcile is marked Aborted. There is no distinction between terminal errors (NotFound, Forbidden, ...) and transient ones (connection refused/reset, timeout, apiserver 5xx), and no retry at the API-call layer.
On large installations this makes reconciles fragile out of proportion to the actual failure: a reconcile of a several-hundred-host CHI issues tens of thousands of API calls over a multi-hour window, so even a sub-second control-plane blip (apiserver rolling restart, LB flap) is nearly certain to land on one of them and abort an otherwise healthy rollout.
What we observed (operator 0.25.3, production)
On a 387-host CHI, the same mechanism aborted a healthy rollout multiple times in one evening:
Error ReconcileFailed: FAILED to reconcile CR <ns>/<chi>, err:
Get ".../configmaps/chi-<chi>-deploy-confd-data-a-87-0":
dial tcp 10.x.0.1:443: connect: connection refused
Warning ReconcileFailed: reconcile completed UNSUCCESSFULLY
One of these aborts landed at hostsCompleted: 385 of 387 — two hosts short of done, with nothing wrong in ClickHouse. Analysis of the surrounding logs showed the blips were ~1-3 seconds long; a single failed GET each time discarded the entire reconcile's remaining work. The same event also surfaced through other unretried single-shot Gets (StatefulSet Get inside the readiness poller, PVC Gets), aborting reconciles of several other CHIs in the same cluster simultaneously.
Root cause
- The kube drivers in
pkg/controller/chi/kube/ (ConfigMap, CR, Pod, PVC, PDB, Secret, Service, STS) issue single-shot Get calls with no retry; transport errors return as-is.
- The object reconcilers (e.g.
reconcileConfigMap in worker-config-map.go) return the raw error.
reconcileCR (worker-reconciler-chi.go) replaces any error with ErrCRUDAbort and marks the CR Aborted — the error's transient/terminal nature is discarded before anything could act on it.- Nothing re-queues an
Aborted CHI, so a sub-second blip becomes a stuck CR until an operator restart or spec edit.
This still holds on current master/0.27.2 — the drivers have no transient-error handling.
Why reconcile.recovery.from.aborted.onPodReady (0.27) doesn't cover this
The auto-recovery introduced in 0.27.0 re-enqueues an Aborted CHI when a pod transitions NotReady->Ready. In this failure mode all pods are already Ready (the incident above had 387/387 Ready throughout) — no transition ever occurs, so recovery never fires and the CR stays Aborted indefinitely.
Reproduction
- Deploy a CHI with enough hosts that a reconcile takes a few minutes; trigger a full reconcile (e.g. change
spec.taskID).
- Mid-reconcile, inject a ~5s apiserver outage scoped to the operator pod — e.g. an ephemeral container in the operator pod's network namespace running:
iptables -I OUTPUT 1 -d <apiserver-ip> -p tcp --dport 443 -j REJECT --reject-with tcp-reset; sleep 5; iptables -D OUTPUT ...
- The next driver Get fails with
connection refused; the reconcile aborts (FAILED to reconcile ConfigMap: chi-...-deploy-confd-... -> FAILED to reconcile CR), discarding all remaining hosts.
Summary
Any error returned by a Kubernetes API call during a CHI reconcile is treated as fatal: the error propagates up unchanged,
reconcileCRcoerces it toErrCRUDAbort, and the whole reconcile is markedAborted. There is no distinction between terminal errors (NotFound, Forbidden, ...) and transient ones (connection refused/reset, timeout, apiserver 5xx), and no retry at the API-call layer.On large installations this makes reconciles fragile out of proportion to the actual failure: a reconcile of a several-hundred-host CHI issues tens of thousands of API calls over a multi-hour window, so even a sub-second control-plane blip (apiserver rolling restart, LB flap) is nearly certain to land on one of them and abort an otherwise healthy rollout.
What we observed (operator 0.25.3, production)
On a 387-host CHI, the same mechanism aborted a healthy rollout multiple times in one evening:
One of these aborts landed at
hostsCompleted: 385 of 387— two hosts short of done, with nothing wrong in ClickHouse. Analysis of the surrounding logs showed the blips were ~1-3 seconds long; a single failed GET each time discarded the entire reconcile's remaining work. The same event also surfaced through other unretried single-shot Gets (StatefulSet Get inside the readiness poller, PVC Gets), aborting reconciles of several other CHIs in the same cluster simultaneously.Root cause
pkg/controller/chi/kube/(ConfigMap,CR,Pod,PVC,PDB,Secret,Service,STS) issue single-shotGetcalls with no retry; transport errors return as-is.reconcileConfigMapinworker-config-map.go) return the raw error.reconcileCR(worker-reconciler-chi.go) replaces any error withErrCRUDAbortand marks the CRAborted— the error's transient/terminal nature is discarded before anything could act on it.AbortedCHI, so a sub-second blip becomes a stuck CR until an operator restart or spec edit.This still holds on current
master/0.27.2— the drivers have no transient-error handling.Why
reconcile.recovery.from.aborted.onPodReady(0.27) doesn't cover thisThe auto-recovery introduced in 0.27.0 re-enqueues an Aborted CHI when a pod transitions NotReady->Ready. In this failure mode all pods are already Ready (the incident above had 387/387 Ready throughout) — no transition ever occurs, so recovery never fires and the CR stays
Abortedindefinitely.Reproduction
spec.taskID).iptables -I OUTPUT 1 -d <apiserver-ip> -p tcp --dport 443 -j REJECT --reject-with tcp-reset; sleep 5; iptables -D OUTPUT ...connection refused; the reconcile aborts (FAILED to reconcile ConfigMap: chi-...-deploy-confd-...->FAILED to reconcile CR), discarding all remaining hosts.