Skip to content

Commit abe5b79

Browse files
committed
fix(kubernetes): reject explicit gpu devices
Signed-off-by: Evan Lezar <elezar@nvidia.com>
1 parent a7760c7 commit abe5b79

4 files changed

Lines changed: 60 additions & 6 deletions

File tree

architecture/compute-runtimes.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,8 @@ dependencies, but they should not need to include the gateway. GPU-capable
8484
images must include the user-space libraries required by the workload. The
8585
runtime still owns GPU device injection. GPU requests can include a driver-native
8686
device identifier or a requested count; the gateway validates the request shape
87-
and each runtime enforces the GPU allocation modes it supports.
87+
and each runtime enforces the GPU allocation modes it supports. Kubernetes uses
88+
counted `nvidia.com/gpu` resources and rejects driver-native device identifiers.
8889

8990
## Deployment Shape
9091

crates/openshell-driver-kubernetes/README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,9 @@ the supervisor's network namespace mount setup on AppArmor-enabled nodes.
6464
When a sandbox requests GPU support, the driver checks node allocatable capacity
6565
for `nvidia.com/gpu` and requests the configured GPU count in the workload spec.
6666
When no count is set, the driver requests one GPU resource. The sandbox image
67-
must provide the user-space libraries needed by the agent workload.
67+
must provide the user-space libraries needed by the agent workload. The driver
68+
does not support explicit GPU device identifiers; use the public `gpu` flag or
69+
`gpu_count`.
6870

6971
## Driver Config POC
7072

crates/openshell-driver-kubernetes/src/driver.rs

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,20 @@ impl std::fmt::Debug for KubernetesComputeDriver {
173173
}
174174
}
175175

176+
fn gpu_request_validation_error(sandbox: &Sandbox) -> Option<&'static str> {
177+
if sandbox
178+
.spec
179+
.as_ref()
180+
.is_some_and(|spec| !spec.gpu_device.is_empty())
181+
{
182+
return Some(
183+
"gpu_device is not supported by the kubernetes compute driver; use gpu or gpu_count",
184+
);
185+
}
186+
187+
None
188+
}
189+
176190
impl KubernetesComputeDriver {
177191
pub async fn new(config: KubernetesComputeConfig) -> Result<Self, KubeError> {
178192
let base_config = match kube::Config::incluster() {
@@ -245,6 +259,10 @@ impl KubernetesComputeDriver {
245259
}
246260

247261
pub async fn validate_sandbox_create(&self, sandbox: &Sandbox) -> Result<(), tonic::Status> {
262+
if let Some(message) = gpu_request_validation_error(sandbox) {
263+
return Err(tonic::Status::invalid_argument(message));
264+
}
265+
248266
let gpu_requested = sandbox.spec.as_ref().is_some_and(|spec| spec.gpu);
249267
if gpu_requested
250268
&& !self.has_gpu_capacity().await.map_err(|err| {
@@ -2288,6 +2306,39 @@ mod tests {
22882306
);
22892307
}
22902308

2309+
#[test]
2310+
fn gpu_request_validation_error_rejects_gpu_device() {
2311+
let sandbox = Sandbox {
2312+
spec: Some(SandboxSpec {
2313+
gpu: true,
2314+
gpu_device: "nvidia.com/gpu=0".to_string(),
2315+
..Default::default()
2316+
}),
2317+
..Default::default()
2318+
};
2319+
2320+
assert_eq!(
2321+
gpu_request_validation_error(&sandbox),
2322+
Some(
2323+
"gpu_device is not supported by the kubernetes compute driver; use gpu or gpu_count"
2324+
)
2325+
);
2326+
}
2327+
2328+
#[test]
2329+
fn gpu_request_validation_error_accepts_gpu_count() {
2330+
let sandbox = Sandbox {
2331+
spec: Some(SandboxSpec {
2332+
gpu: true,
2333+
gpu_count: Some(2),
2334+
..Default::default()
2335+
}),
2336+
..Default::default()
2337+
};
2338+
2339+
assert_eq!(gpu_request_validation_error(&sandbox), None);
2340+
}
2341+
22912342
#[test]
22922343
fn gpu_sandbox_uses_template_runtime_class_name_when_set() {
22932344
let template = SandboxTemplate {

docs/sandboxes/manage-sandboxes.mdx

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -84,10 +84,10 @@ openshell sandbox create --gpu-device nvidia.com/gpu=0 -- claude
8484
```
8585
8686
Support for count and device selection is driver-dependent. Kubernetes honors
87-
`--gpu-count` by setting the `nvidia.com/gpu` limit. Docker and Podman support
88-
explicit CDI device IDs through `--gpu-device` but reject count-based selection.
89-
VM gateways accept only one GPU, either through `--gpu`, `--gpu-count 1`, or
90-
`--gpu-device`.
87+
`--gpu-count` by setting the `nvidia.com/gpu` limit and rejects
88+
`--gpu-device`. Docker and Podman support explicit CDI device IDs through
89+
`--gpu-device` but reject count-based selection. VM gateways accept only one
90+
GPU, either through `--gpu`, `--gpu-count 1`, or `--gpu-device`.
9191
9292
For Docker-backed sandboxes, GPU injection uses Docker CDI. If you enable Docker
9393
CDI after the gateway starts, restart the gateway so OpenShell can detect the

0 commit comments

Comments
 (0)