fix(serve/integ): delete endpoints by name so cleanup survives deploy failures#6072
Open
jam-jee wants to merge 1 commit into
Open
fix(serve/integ): delete endpoints by name so cleanup survives deploy failures#6072jam-jee wants to merge 1 commit into
jam-jee wants to merge 1 commit into
Conversation
… failures
The sagemaker-serve integration tests each build a model and deploy a real
SageMaker endpoint, then delete it in a finally block. Cleanup was guarded on
the handle returned by deploy():
core_model = core_endpoint = None
try:
core_model, core_endpoint = build_and_deploy() # may fail / hang
...
finally:
if core_model and core_endpoint: # skipped on failure
cleanup_resources(core_model, core_endpoint)
When deploy() raises (endpoint -> Failed) or the build is killed mid-call
(CodeBuild timeout), core_endpoint is never assigned, the guard is falsy, and
the endpoint SageMaker already created is never torn down. Deleting by handle
also can't clean up a resource whose creating call never returned. This is the
root cause of the endpoint accumulation in the CI account (thousands of Failed
and InService endpoints).
Fix: add tests/integ/cleanup_helpers.py with delete_quietly() (best-effort,
per-resource) and cleanup_by_name() (deletes endpoint -> endpoint config ->
model by name). Each test now generates its resource names up front and always
calls cleanup_by_name() in finally, so a Failed or half-created endpoint is
still deleted, and one failed delete no longer aborts the rest.
- Applies to: jumpstart, optimize, tgi, tei, huggingface, triton,
train_inference_e2e, and both ai_inference_recommender tests.
- ai_inference_recommender keeps its job / workload-config / model-package
cleanup, now via the shared delete_quietly(); its local duplicate helpers are
removed.
- in_process is unchanged (deploy_local creates no cloud endpoint).
This only changes test cleanup; no SDK source or test assertions are affected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue / motivation
The
sagemaker-serveintegration tests each build a model and deploy a real SageMaker endpoint, then delete it in afinallyblock. Cleanup was guarded on the handle returned bydeploy():When
deploy()raises (endpoint goes toFailed) or the build is killed mid-call (e.g. the CodeBuild 3h timeout),core_endpointis never assigned, the guard is falsy, and the endpoint SageMaker already created is never torn down. Deleting by the object handle also cannot clean up a resource whose creating call never returned.This is the root cause of endpoint accumulation in the CI account (large numbers of leaked
FailedandInServiceendpoints, theInServiceones billing continuously).Fix
Add
tests/integ/cleanup_helpers.pywith:delete_quietly(delete_fn, label)— best-effort, per-resource delete; one failure never aborts the rest of the sequence.cleanup_by_name(endpoint_name, endpoint_config_name, model_name)— deletes endpoint → endpoint config → model by name, so cleanup no longer depends ondeploy()returning a handle.Each affected test now generates its resource names up front and always calls
cleanup_by_name()infinally, so aFailedor half-created endpoint is still deleted.Applies to:
jumpstart,optimize,tgi,tei,huggingface,triton,train_inference_e2e, and bothai_inference_recommendertests. The recommender tests keep their job / workload-config / model-package cleanup, now routed through the shareddelete_quietly(); their local duplicate helpers are removed.in_processis unchanged (deploy_localcreates no cloud endpoint).Testing
Merge Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.