Skip to content

fix(serve/integ): delete endpoints by name so cleanup survives deploy failures#6072

Open
jam-jee wants to merge 1 commit into
aws:masterfrom
jam-jee:fix/serve-integ-endpoint-cleanup
Open

fix(serve/integ): delete endpoints by name so cleanup survives deploy failures#6072
jam-jee wants to merge 1 commit into
aws:masterfrom
jam-jee:fix/serve-integ-endpoint-cleanup

Conversation

@jam-jee

@jam-jee jam-jee commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Issue / motivation

The sagemaker-serve integration tests each build a model and deploy a real SageMaker endpoint, then delete it in a finally block. Cleanup was guarded on the handle returned by deploy():

core_model = core_endpoint = None
try:
    core_model, core_endpoint = build_and_deploy()   # may raise / be killed
    ...
finally:
    if core_model and core_endpoint:                 # skipped when deploy() failed
        cleanup_resources(core_model, core_endpoint)

When deploy() raises (endpoint goes to Failed) or the build is killed mid-call (e.g. the CodeBuild 3h timeout), core_endpoint is never assigned, the guard is falsy, and the endpoint SageMaker already created is never torn down. Deleting by the object handle also cannot clean up a resource whose creating call never returned.

This is the root cause of endpoint accumulation in the CI account (large numbers of leaked Failed and InService endpoints, the InService ones billing continuously).

Fix

Add tests/integ/cleanup_helpers.py with:

  • delete_quietly(delete_fn, label) — best-effort, per-resource delete; one failure never aborts the rest of the sequence.
  • cleanup_by_name(endpoint_name, endpoint_config_name, model_name) — deletes endpoint → endpoint config → model by name, so cleanup no longer depends on deploy() returning a handle.

Each affected test now generates its resource names up front and always calls cleanup_by_name() in finally, so a Failed or half-created endpoint is still deleted.

Applies to: jumpstart, optimize, tgi, tei, huggingface, triton, train_inference_e2e, and both ai_inference_recommender tests. The recommender tests keep their job / workload-config / model-package cleanup, now routed through the shared delete_quietly(); their local duplicate helpers are removed. in_process is unchanged (deploy_local creates no cloud endpoint).

Testing

  • This change only touches integration-test cleanup; no SDK source and no test assertions are modified.
  • All edited files byte-compile; lines are within the repo flake8 limit (120).
  • Integration tests run in CI per the contributing guide; no new local integ run is required for a cleanup-only change.

Merge Checklist

  • I have read the CONTRIBUTING doc
  • Used commit message / PR title following the Conventional Commits format
  • Changes are limited to test cleanup logic (no SDK source or public API changes)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

… failures

The sagemaker-serve integration tests each build a model and deploy a real
SageMaker endpoint, then delete it in a finally block. Cleanup was guarded on
the handle returned by deploy():

    core_model = core_endpoint = None
    try:
        core_model, core_endpoint = build_and_deploy()   # may fail / hang
        ...
    finally:
        if core_model and core_endpoint:                 # skipped on failure
            cleanup_resources(core_model, core_endpoint)

When deploy() raises (endpoint -> Failed) or the build is killed mid-call
(CodeBuild timeout), core_endpoint is never assigned, the guard is falsy, and
the endpoint SageMaker already created is never torn down. Deleting by handle
also can't clean up a resource whose creating call never returned. This is the
root cause of the endpoint accumulation in the CI account (thousands of Failed
and InService endpoints).

Fix: add tests/integ/cleanup_helpers.py with delete_quietly() (best-effort,
per-resource) and cleanup_by_name() (deletes endpoint -> endpoint config ->
model by name). Each test now generates its resource names up front and always
calls cleanup_by_name() in finally, so a Failed or half-created endpoint is
still deleted, and one failed delete no longer aborts the rest.

- Applies to: jumpstart, optimize, tgi, tei, huggingface, triton,
  train_inference_e2e, and both ai_inference_recommender tests.
- ai_inference_recommender keeps its job / workload-config / model-package
  cleanup, now via the shared delete_quietly(); its local duplicate helpers are
  removed.
- in_process is unchanged (deploy_local creates no cloud endpoint).

This only changes test cleanup; no SDK source or test assertions are affected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant