Skip to content

Chore: Include GPU Integ Tests in CI-Health #6078

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lucasjia-aws merged 1 commit into from
Jul 22, 2026
+248 −226
Merged

Chore: Include GPU Integ Tests in CI-Health #6078

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 145 additions & 0 deletions .github/workflows/ci-health-v3-master.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
name: CI Health V3 Master
on:
schedule:
# Once a day at 05:00 UTC (9pm PDT / 10pm PST).
- cron: "0 5 * * *"
workflow_dispatch:

permissions:
id-token: write # This is required for requesting the JWT

jobs:
unit-test-v3:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
submodule: [sagemaker-core, sagemaker-train, sagemaker-serve, sagemaker-mlops]
steps:
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
aws-region: us-west-2
role-duration-seconds: 10800
- name: Run Unit Tests V3 for ${{ matrix.submodule }}
uses: aws-actions/aws-codebuild-run-build@v1
with:
project-name: sagemaker-python-sdk-ci-health-unit-test-v3
source-version-override: refs/heads/master
env-vars-for-codebuild: |
SUBMODULE
env:
SUBMODULE: ${{ matrix.submodule }}
canaries-v3-master:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
submodule: [sagemaker-core, sagemaker-train, sagemaker-serve, sagemaker-mlops]
steps:
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
aws-region: us-west-2
role-duration-seconds: 10800
- name: Run Canaries V3 master for ${{ matrix.submodule }}
uses: aws-actions/aws-codebuild-run-build@v1
with:
project-name: sagemaker-python-sdk-ci-health-canaries-v3-master
source-version-override: refs/heads/master
env-vars-for-codebuild: |
SUBMODULE
env:
SUBMODULE: ${{ matrix.submodule }}
# GPU integ tests against the master source. Two region jobs trigger the same
# CodeBuild project name (deployed in both regions); only the region/creds and
# the region-override buildspec differ. Their results are AND-ed with
# import-model in report-result into a single per-run pass/fail metric.
gpu-integ-tests-master:
runs-on: ubuntu-latest
steps:
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
aws-region: us-west-2
role-duration-seconds: 10800
- name: Run GPU Integ Tests (master)
uses: aws-actions/aws-codebuild-run-build@v1
with:
project-name: sagemaker-python-sdk-ci-health-gpu-integ-tests-master
source-version: refs/heads/master
gpu-integ-tests-master-us-east-1:
runs-on: ubuntu-latest
steps:
- name: Configure AWS Credentials (us-east-1)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.CI_AWS_ROLE_US_EAST_1_ARN }}
aws-region: us-east-1
role-duration-seconds: 10800
- name: Run GPU Integ Tests (master, us-east-1)
uses: aws-actions/aws-codebuild-run-build@v1
with:
project-name: sagemaker-python-sdk-ci-health-gpu-integ-tests-master
source-version: refs/heads/master
# Bedrock model-import integ tests. Run serially (concurrency 1) in their own
# CodeBuild project because the "Concurrent model import jobs" Bedrock quota is
# fixed at 1 and not raisable; running them in parallel (as PR checks did)
# makes them collide and flake. us-west-2 only (no us_east_1-marked tests).
# Folded into the master run-level pass/fail metric alongside the GPU jobs.
import-model-integ-tests:
runs-on: ubuntu-latest
steps:
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
aws-region: us-west-2
role-duration-seconds: 10800
- name: Run Bedrock Model-Import Integ Tests
uses: aws-actions/aws-codebuild-run-build@v1
with:
project-name: sagemaker-python-sdk-ci-health-import-model-integ-tests
source-version: refs/heads/master
# Run-level result: the master GPU run is successful only if the us-west-2,
# us-east-1, and import-model jobs all succeeded. Emits GpuIntegMasterRunFailure
# = 1 (failed) / 0 (succeeded) to CloudWatch in us-west-2. The CDK alarm
# (GpuIntegMasterRunAlarm) sums this over a UTC day; it is folded into
# CIHealthCompositeAlarm-V3, which owns ticketing. Only scheduled runs emit the
# metric (manual workflow_dispatch runs are excluded).
report-result:
needs: [gpu-integ-tests-master, gpu-integ-tests-master-us-east-1, import-model-integ-tests]
if: always()
runs-on: ubuntu-latest
steps:
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.MONITORING_ROLE_ARN }}
aws-region: us-west-2
- name: Emit run-level pass/fail metric
run: |
# Manual (workflow_dispatch) runs must not contribute to the daily
# GpuIntegMasterRunFailure count that drives GpuIntegMasterRunAlarm;
# only scheduled runs count.
if [ "${{ github.event_name }}" != "schedule" ]; then
echo "Not a scheduled run (${{ github.event_name }}); skipping metric emission."
exit 0
fi
if [ "${{ needs.gpu-integ-tests-master.result }}" == "success" ] && \
[ "${{ needs.gpu-integ-tests-master-us-east-1.result }}" == "success" ] && \
[ "${{ needs.import-model-integ-tests.result }}" == "success" ]; then
value=0
echo "All master GPU/import jobs succeeded; emitting GpuIntegMasterRunFailure=0"
else
value=1
echo "At least one master GPU/import job did not succeed; emitting GpuIntegMasterRunFailure=1"
fi
aws cloudwatch put-metric-data \
--namespace GpuIntegRunMetrics \
--metric-name GpuIntegMasterRunFailure \
--value "$value" \
--unit Count
103 changes: 103 additions & 0 deletions .github/workflows/ci-health-v3-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
name: CI Health V3 Release
on:
schedule:
# Once a day at 05:00 UTC (9pm PDT / 10pm PST).
- cron: "0 5 * * *"
workflow_dispatch:

permissions:
id-token: write # This is required for requesting the JWT

jobs:
canaries-v3-release:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
submodule: [sagemaker-core, sagemaker-train, sagemaker-serve, sagemaker-mlops]
steps:
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
aws-region: us-west-2
role-duration-seconds: 10800
- name: Run Canaries V3 release for ${{ matrix.submodule }}
uses: aws-actions/aws-codebuild-run-build@v1
with:
project-name: sagemaker-python-sdk-ci-health-canaries-v3-release
source-version-override: refs/heads/master
env-vars-for-codebuild: |
SUBMODULE
env:
SUBMODULE: ${{ matrix.submodule }}
# GPU integ tests against the published PyPI release. Two region jobs trigger
# the same CodeBuild project name (deployed in both regions); only the
# region/creds and the region-override buildspec differ. Their results are
# AND-ed in report-result into a single per-run pass/fail metric.
gpu-integ-tests-release:
runs-on: ubuntu-latest
steps:
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
aws-region: us-west-2
role-duration-seconds: 10800
- name: Run GPU Integ Tests (release)
uses: aws-actions/aws-codebuild-run-build@v1
with:
project-name: sagemaker-python-sdk-ci-health-gpu-integ-tests-release
source-version: refs/heads/master
gpu-integ-tests-release-us-east-1:
runs-on: ubuntu-latest
steps:
- name: Configure AWS Credentials (us-east-1)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.CI_AWS_ROLE_US_EAST_1_ARN }}
aws-region: us-east-1
role-duration-seconds: 10800
- name: Run GPU Integ Tests (release, us-east-1)
uses: aws-actions/aws-codebuild-run-build@v1
with:
project-name: sagemaker-python-sdk-ci-health-gpu-integ-tests-release
source-version: refs/heads/master
# Run-level result: the release GPU run is successful only if both region jobs
# succeeded. Emits GpuIntegReleaseRunFailure = 1 (failed) / 0 (succeeded) to
# CloudWatch in us-west-2. The CDK alarm (GpuIntegReleaseRunAlarm) sums this
# over a UTC day; it is folded into CIHealthCompositeAlarm-V3, which owns
# ticketing. Only scheduled runs emit the metric (manual workflow_dispatch runs
# are excluded).
report-result:
needs: [gpu-integ-tests-release, gpu-integ-tests-release-us-east-1]
if: always()
runs-on: ubuntu-latest
steps:
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.MONITORING_ROLE_ARN }}
aws-region: us-west-2
- name: Emit run-level pass/fail metric
run: |
# Manual (workflow_dispatch) runs must not contribute to the daily
# GpuIntegReleaseRunFailure count that drives GpuIntegReleaseRunAlarm;
# only scheduled runs count.
if [ "${{ github.event_name }}" != "schedule" ]; then
echo "Not a scheduled run (${{ github.event_name }}); skipping metric emission."
exit 0
fi
if [ "${{ needs.gpu-integ-tests-release.result }}" == "success" ] && \
[ "${{ needs.gpu-integ-tests-release-us-east-1.result }}" == "success" ]; then
value=0
echo "All release GPU jobs succeeded; emitting GpuIntegReleaseRunFailure=0"
else
value=1
echo "At least one release GPU job did not succeed; emitting GpuIntegReleaseRunFailure=1"
fi
aws cloudwatch put-metric-data \
--namespace GpuIntegRunMetrics \
--metric-name GpuIntegReleaseRunFailure \
--value "$value" \
--unit Count
77 changes: 0 additions & 77 deletions .github/workflows/ci-health-v3.yml
View file
Open in desktop

This file was deleted.

Loading
Loading