fix: prevent ACL kubelet sysext from starting before CSE - #9088
fix: prevent ACL kubelet sysext from starting before CSE#9088aadhar-agarwal wants to merge 1 commit into
Conversation
The kubelet sysext adds Upholds=kubelet.service to multi-user.target, which starts AgentBaker's kubelet unit before CSE creates /etc/default/kubelet. GPU provisioning extends this window enough for kubelet to hit systemd's start limit. Mask the vendor drop-in before activating the ACL kubelet sysext so CSE owns initial kubelet startup. Fail provisioning if the mask cannot be created. Add regression coverage verifying that the mask is installed before sysext activation. Signed-off-by: Aadhar Agarwal <aadagarwal@microsoft.com>
Windows Unit Test Results 3 files 11 suites 36s ⏱️ Results for commit 04476ce. |
There was a problem hiding this comment.
Pull request overview
Addresses a kubelet startup race on Azure Container Linux (ACL) where the kubelet sysext can trigger kubelet startup (via a vendor Upholds= drop-in) before CSE finishes writing kubelet configuration, potentially exhausting systemd start retries during long GPU setup.
Changes:
- Add a helper to mask the ACL kubelet sysext
Upholds=drop-in by installing an overriding drop-in in/etc/systemd/system/multi-user.target.d/. - Ensure the mask is applied before kubelet/kubectl sysext activation in the ACL install flow.
- Add ShellSpec regression coverage to verify ordering and failure behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| spec/parts/linux/cloud-init/artifacts/cse_install_acl_spec.sh | Adds regression tests asserting the mask is created before mergeSysexts runs and that failures abort installation. |
| parts/linux/cloud-init/artifacts/cse_helpers.sh | Introduces maskKubeletSysextUpholds() to create the drop-in directory and symlink the vendor drop-in to /dev/null. |
| parts/linux/cloud-init/artifacts/acl/cse_install_acl.sh | Calls maskKubeletSysextUpholds before activating kubelet/kubectl sysexts, failing fast on masking errors. |
AgentBaker Linux gate detectiveTL;DR: Build 174613967 partially succeeded because Ubuntu 22.04 Gen2 containerd CIS scan reported rule Signature: First failing step: Likely root cause: VHD/CIS compliance drift for Ubuntu 22.04 Gen2 containerd. The same rule appeared minutes later on unrelated PR #9089 on the ARM64 flavor, reducing confidence that this ACL script change caused it. Classification: VHD/CIS compliance regression signal. Infra/product/test-code/change-caused assessment: product/CIS baseline or image-generation drift possible; infra less likely than baseline drift because the scan completed cleanly; test-code unlikely; PR-change-caused unlikely because this PR changes ACL CSE helper scripts, not Ubuntu 22.04 containerd CIS policy. Confidence: Medium that this is not PR-caused; medium-low on exact CIS remediation until the full CIS report is reviewed. Strongest alternative: the ACL helper changes altered shared CSE behavior used by Ubuntu images; less likely because the failing job is non-ACL Ubuntu 22.04 containerd and the same rule appeared on another PR. Owner/action: Linux VHD/CIS owner should inspect rule Evidence: build 174613967; timeline task Wiki signature ID: |
What this PR does / why we need it:
ACL has a kubelet startup race between sysext activation and CSE configuration. The sysext's
Upholds=kubelet.servicestarts kubelet before CSE creates/etc/default/kubelet; longer GPU setup allows the retries to hit systemd's start limit.Mask the vendor
Upholds=drop-in before activating the ACL kubelet sysext, ensuring CSE owns the initial kubelet start after configuration is complete.Add regression coverage verifying that the mask is installed before sysext activation.
[TEST All VHDs] AKS Linux VHD Build - Msft Tenant runs to make sure the ACL GPU tests pass consistently:
Which issue(s) this PR fixes:
Fixes #