How Dynamic Resource Allocation changes the GPU request contract

Under Dynamic Resource Allocation a pod stops counting GPUs in resources.requests and starts referencing a ResourceClaim, which moves device selection into the scheduler and out of the kubelet’s device plugin.

The device plugin API gave the cluster one lever: an integer count of an opaque extended resource named nvidia.com/gpu. Anything the scheduler needed to know beyond that count — card model, memory size, NVLink domain, MIG profile — had to be smuggled in as node labels applied by GPU Feature Discovery and matched with a nodeSelector. Dynamic Resource Allocation (DRA) replaces that indirection. The core DRA APIs in the resource.k8s.io group graduated to GA in Kubernetes 1.34, and the stable version is resource.k8s.io/v1 (per the Kubernetes v1.34 DRA release announcement).

Four kinds carry the mechanism. A driver publishes ResourceSlice objects describing the devices attached to each node, along with their attributes and capacities. A DeviceClass names a category of device that workloads may claim. A ResourceClaim is a request for one or more devices; a ResourceClaimTemplate produces a fresh claim per pod. The scheduler reads the ResourceSlices, picks devices that satisfy the claim, and records the allocation on the claim’s status before binding the pod (the sequence described in the upstream DRA concept documentation).

The claim path, end to end

A claim describes devices by selector, not by count of an opaque resource. The selectors are CEL expressions evaluated against the attributes and capacities that the driver published in its ResourceSlice.

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: example-resource-claim-template
spec:
  spec:
    devices:
      requests:
      - name: gpu-claim
        exactly:
          deviceClassName: example-device-class
          selectors:
            - cel:
                expression: |-
                  device.attributes["driver.example.com"].type == "gpu" &&
                  device.capacity["driver.example.com"].memory == quantity("64Gi")

The workload then wires the claim into a pod template in two places. spec.resourceClaims declares the claim and its source; each container that needs the device lists the claim name under resources.claims. A container that omits the entry does not get the device, even though the pod holds the claim.

spec:
  template:
    spec:
      containers:
      - name: container0
        image: ubuntu:24.04
        command: ["sleep", "9999"]
        resources:
          claims:
          - name: separate-gpu-claim
      - name: container1
        image: ubuntu:24.04
        command: ["sleep", "9999"]
        resources:
          claims:
          - name: shared-gpu-claim
      - name: container2
        image: ubuntu:24.04
        command: ["sleep", "9999"]
        resources:
          claims:
          - name: shared-gpu-claim
      resourceClaims:
      - name: separate-gpu-claim
        resourceClaimTemplateName: example-resource-claim-template
      - name: shared-gpu-claim
        resourceClaimName: example-resource-claim

That distinction is the sharing model. resourceClaimTemplateName mints a new ResourceClaim per pod, so every pod gets its own device. resourceClaimName points at one pre-existing claim, so every pod referencing it shares the same allocated device. In the manifest above, container1 and container2 share one GPU and container0 gets its own — expressed in the pod spec rather than in a device-plugin sharing config on the node.

What the NVIDIA driver puts on the cluster

DRA in the API is inert without a driver. The NVIDIA DRA driver for GPUs installs the DeviceClasses that claims name: gpu.nvidia.com and mig.nvidia.com for GPU and MIG allocation, vfio.gpu.nvidia.com for VFIO passthrough, and compute-domain-daemon.nvidia.com plus compute-domain-default-channel.nvidia.com when ComputeDomain support is enabled. It also adds its own CRD, ComputeDomain in resource.nvidia.com/v1beta1, which describes a multi-node NVLink domain rather than a single card. The driver requires Kubernetes 1.32 or later; resource.k8s.io/v1 manifests need 1.34, while 1.32 and 1.33 use resource.k8s.io/v1beta2 (per the driver’s install guide).

Three commands establish whether the plumbing is present before any workload is written.

kubectl get deviceclass
kubectl get resourceslice -o wide
kubectl get resourceclaims -A

The first shows which DeviceClasses exist and therefore which names a claim may use. The second shows whether GPU nodes have published their devices — a node with no ResourceSlice is invisible to DRA scheduling regardless of what its status.allocatable says. The third shows claims and whether the scheduler has allocated them.

Quota changes shape along with the request. A ResourceQuota on requests.nvidia.com/gpu counts extended resources and does not see devices allocated through claims. DRA quota is expressed per DeviceClass; for a class named examplegpu, the quota key is examplegpu.deviceclass.resource.k8s.io/devices (the form documented under quota for DRA resource claims).

Failure modes

The most common first failure is a claim that names a DeviceClass that does not exist. The symptom is a scheduling error citing the missing class rather than an out-of-capacity message, which is how the upstream driver repository ended up with issues titled device class gpu.nvidia.com does not exist (issue 338 in the kubernetes-sigs/nvidia-dra-driver-gpu repository). The class is created by the driver’s install, so the fix is in the driver’s Helm values, not the workload.

The second is a CEL selector that matches nothing. An expression comparing device.capacity[...].memory with quantity("64Gi") for equality matches only devices reporting exactly that value; a card reporting 80Gi fails silently and the claim stays unallocated. The pod is Pending with no node-level error to read, because no node was ever a candidate. Selectors written with an inequality against a floor are more durable than selectors written against an exact model or size.

The third is quota that no longer binds. A namespace capped at requests.nvidia.com/gpu: 8 and migrated to claims has no effective GPU cap until a DeviceClass quota is added, and nothing in the migration surfaces a warning. The same trap applies to any admission policy — Kyverno rules matching on nvidia.com/gpu in resources.requests stop matching when the request moves into spec.resourceClaims.

Decision frame

The question before migrating a GPU workload to DRA is not whether the API is stable — the core kinds are GA in resource.k8s.io/v1 since 1.34. It is whether the workload needs a property the extended-resource integer cannot express: a specific MIG profile, an attribute-matched card, a device shared between two containers of the same pod, or a multi-node NVLink domain described by a ComputeDomain. If the workload only ever asks for “one GPU, any GPU,” the device plugin path costs nothing to keep and DRA buys nothing. If it asks for a particular device, the choice is between another round of GFD-label selectors and moving the selection into the claim, and the cost of moving is a full rewrite of the namespace’s quotas and admission policies, not just the pod spec.