Sharing a GPU in Kubernetes sounds simple: run multiple jobs on one physical GPU.

In practice, it quickly turns into a set of very concrete questions:

  • How do you ask for “0.2 GPU” when Kubernetes only knows nvidia.com/gpu: 1?
  • How do you prevent scheduling conflicts with full-GPU pods?
  • And the real pain point: how do you prevent one container from consuming all VRAM and killing its neighbors?

KAI-Scheduler (open-sourced after NVIDIA acquired Run:ai) ships a clever GPU sharing design centered around the idea of a Reservation Pod: allocate a whole GPU to a lightweight placeholder pod, then do fractional accounting inside the scheduler.

HAMi represents a different direction: scheduling-level allocation plus CUDA API-level runtime limits. This is stronger than application self-limiting, but it is not MIG-style hardware isolation.

The useful split is simple: KAI mainly solves the scheduling expression problem, while HAMi pushes harder on runtime enforcement. Treat them as different layers, not as mutually exclusive camps.

Start with the layer you are missing

GPU sharing has at least two separate problems. If you do not split them, tool selection becomes noisy:

  • Scheduling expression: vanilla Kubernetes only understands integer GPU devices, but the platform wants to express half a GPU, 0.2 GPU, or queue-level fractional accounting.
  • Runtime isolation: workloads may land on the same physical card, but memory, performance interference, and failure blast radius still need control.
  • Tenant governance: teams with different priorities and SLAs cannot rely on everyone voluntarily using less GPU.
  • Rollback and stop-loss: if sharing creates latency or failures, the platform needs a fast path back to whole-card allocation or a more conservative policy.

KAI is mostly about scheduling expression and queue accounting. HAMi pushes further into runtime limits and isolation. Comparing them as one generic “GPU sharing solution” usually hides the real tradeoff.

Common misreads

  • Scheduling several workloads onto one card does not prove runtime isolation is safe.
  • Memory limits do not remove every interference source; bandwidth, I/O, and CPU feed can still collide.
  • A Reservation Pod is not magic. It is a scheduling ledger built around Kubernetes’ integer device model.
  • Stronger isolation usually raises implementation complexity, compatibility testing, and operational cost.

1) Why “fractional GPU” is hard in vanilla Kubernetes

Kubernetes device resources are modeled as extended resources.

For NVIDIA GPUs, that’s typically nvidia.com/gpu. The model is intentionally simple and works well for whole-device allocation, but it comes with hard constraints:

  • integer only: you can’t request 0.2 of nvidia.com/gpu
  • the default scheduler doesn’t understand fractional semantics
  • there’s no standard way to represent “this GPU is partially used” to the rest of the cluster

A common instinct is:

“Fine, I’ll report one physical GPU as multiple virtual devices via a device plugin.”

That’s a valid route (and it’s close to how hard-isolation stacks often start), but KAI takes a different, scheduler-centric approach.

2) KAI’s key trick: the Reservation Pod

2.1 Intuition: reserve the whole GPU, then split the bill internally

KAI’s approach is basically:

Don’t fight Kubernetes over fractional GPU semantics. Use a standard pod to reserve the whole GPU first.

A fractional user pod expresses intent via an annotation, for example:

metadata:
  annotations:
    gpu-fraction: "0.2"  # 20%

Then KAI-Scheduler does this:

  1. Detects gpu-fraction.
  2. Selects a node and a physical GPU that can be shared.
  3. If this GPU has no existing sharing group yet, it creates a Reservation Pod that requests nvidia.com/gpu: "1".
  4. Once the Reservation Pod is scheduled, Kubernetes sees the GPU as fully allocated, preventing other schedulers/pods from conflicting.
  5. KAI assigns the fractional pods into the same logical group (for example via a gpu-group label) and keeps a private accounting of usage.

2.2 What is it “tricking”?

It’s not tricking the GPU.

It’s “tricking” the resource model: Kubernetes only sees that the GPU is owned by the Reservation Pod, while KAI tracks fractional usage out-of-band.

3) How KAI GPU sharing works (engineering view)

3.1 Scheduling entry: parse annotations and pick a GPU

KAI Scheduler’s stable behavior is:

  • read gpu-fraction, gpu-memory, and related requests
  • fit capacity at node and physical-GPU level
  • create a Reservation Pod when a card needs to be reserved
  • bind sharing Pods into one group and maintain private accounting

Internal function names change with releases. The durable interface is the Pod annotations, schedulerName, Queue label, and Reservation Pod behavior.

3.2 The critical step: create the Reservation Pod

The Reservation Pod is pinned to a node and requests a whole GPU:

func (rsc *service) createResourceReservationPod(
  nodeName, gpuGroup, podName, appName string,
  resources v1.ResourceRequirements,
) (*v1.Pod, error) {

  podSpec := &v1.Pod{
    ObjectMeta: metav1.ObjectMeta{
      Name:      podName,
      Namespace: "runai-reservation",
      Labels: map[string]string{
        "gpu-group": gpuGroup,
      },
    },
    Spec: v1.PodSpec{
      NodeName: nodeName,
      Containers: []v1.Container{{
        Name:      "reservation",
        Image:     rsc.reservationPodImage,
        Resources: resources, // request nvidia.com/gpu: "1"
      }},
    },
  }

  return podSpec, rsc.kubeClient.Create(context.Background(), podSpec)
}

Key takeaways:

  • It’s node-pinned (not “waiting” for the default scheduler).
  • It uses labels to define a logical “sharing group”.

3.3 Internal accounting: tracking VRAM usage per sharing group

KAI tracks sharing state internally, for example:

type GpuSharingNodeInfo struct {
  ReleasingSharedGPUs       map[string]bool
  UsedSharedGPUsMemory      map[string]int64
  ReleasingSharedGPUsMemory map[string]int64
  AllocatedSharedGPUsMemory map[string]int64
}

This implies a fundamental property:

KAI’s GPU sharing is implemented at the scheduler level. Not via runtime enforcement.

3.4 Cleanup: last fractional pod exits -> delete Reservation Pod

When the sharing group no longer has active fractional pods, KAI deletes the Reservation Pod to return the GPU to the cluster:

func (rsc *service) syncForPods(ctx context.Context, pods []*v1.Pod, gpuGroupToSync string) error {
  reservationPods := map[string]*v1.Pod{}
  fractionPods := map[string][]*v1.Pod{}

  for _, pod := range pods {
    if pod.Namespace == "runai-reservation" {
      reservationPods[gpuGroupToSync] = pod
      continue
    }
    if pod.Status.Phase == v1.PodRunning || pod.Status.Phase == v1.PodPending {
      fractionPods[gpuGroupToSync] = append(fractionPods[gpuGroupToSync], pod)
    }
  }

  for gpuGroup, reservationPod := range reservationPods {
    if _, found := fractionPods[gpuGroup]; !found {
      return rsc.deleteReservationPod(ctx, reservationPod)
    }
  }
  return nil
}

4) KAI’s strengths—and its unavoidable constraint

4.1 Strength: elegant, Kubernetes-native integration

KAI’s design is concrete:

  • Uses standard K8s primitives (pods, labels, annotations)
  • Avoids direct conflict with the default scheduler by reserving the whole GPU
  • Allows flexible fractions (not limited to a few predefined profiles)

If your goal is to improve utilization quickly and you can coordinate with application owners, KAI can be a fast win.

4.2 The core limitation: soft isolation

The trade-off is also fundamental:

KAI allocates “fractions” logically, but it can’t enforce those fractions at runtime.

That leads to:

  • no hard cap on VRAM/compute
  • noisy neighbor risks
  • application-side tuning often required (e.g., setting per-process VRAM limits)

Think of it as “sharing a room with house rules” rather than “private rooms with locked doors”.

5) The HAMi path: enforce limits at the CUDA API layer

HAMi reduces reliance on application self-discipline.

A typical design combines:

  • device plugin logic to expose schedulable units
  • runtime enforcement through HAMi-Core CUDA API interception for memory allocation and compute throttling

Benefits:

  • stronger memory and compute boundaries than application self-limiting
  • less cross-workload memory overcommit
  • less need for application-level configuration

Costs:

  • more components to deploy
  • more constraints on driver/runtime compatibility

This remains user-space interception, not hardware partitioning. Paths that bypass the hook, driver compatibility, and shared PCIe or memory-bandwidth interference still require validation.

6) Which one should you use?

Choose a KAI-style approach if:

  • you need a lightweight path to better utilization
  • you can tolerate some performance variance
  • you can coordinate with application teams to set VRAM usage limits

Choose a HAMi-style runtime-enforcement approach if:

  • you run a multi-tenant GPU platform
  • you need enforceable memory and compute quotas beyond application self-limiting
  • you want sharing to be transparent to users

7) Compose scheduler policies with runtime enforcement

It’s tempting to treat this as a “winner vs loser” debate.

In practice, they map nicely to two layers:

  • scheduling policies and UX (KAI’s strength)
  • CUDA API-level memory and compute enforcement (HAMi’s strength)

If the ecosystem aligns interfaces between the two, you can get the best of both worlds:

  1. rich scheduling policies and a clean fractional request UX
  2. runtime limits underneath to reduce memory overcommit and compute contention

This still does not remove every noisy-neighbor source. PCIe, memory bandwidth, CPU and I/O supply, and paths that bypass interception can interfere. Use MIG or vGPU when hardware-level isolation is required.

8) Hands-on: run two workloads on one shared GPU

A very common situation: you’ve got a 24GB GPU and you want to run two workloads on it:

  • Workload A: an online inference service (needs predictable VRAM)
  • Workload B: an offline evaluation / small training job (can be slower, but must not crash A)

With scheduler-level sharing (soft isolation), the scheduler can co-locate A and B on the same physical GPU, but preventing conflicts often requires application-side guardrails.

7.5.1 Pod side: request a fraction via annotation

Assume KAI was installed with global.gpuSharing=true, default-queue exists, and the Reservation Pod RuntimeClass can access NVML. Example (0.5 + 0.5):

apiVersion: v1
kind: Pod
metadata:
  name: infer-a
  labels:
    kai.scheduler/queue: default-queue
  annotations:
    gpu-fraction: "0.5"
spec:
  schedulerName: kai-scheduler
  containers:
    - name: app
      image: your-registry/infer:<pinned-version>
---
apiVersion: v1
kind: Pod
metadata:
  name: eval-b
  labels:
    kai.scheduler/queue: default-queue
  annotations:
    gpu-fraction: "0.5"
spec:
  schedulerName: kai-scheduler
  containers:
    - name: app
      image: your-registry/eval:<pinned-version>

Note: this intentionally doesn’t use resources.limits: nvidia.com/gpu, because fractional GPU is not a native Kubernetes resource. The exact knobs depend on how KAI is integrated in your cluster.

7.5.2 App side: VRAM self-limiting (PyTorch example)

The simplest guardrail is to cap VRAM usage in-process before loading the model:

import torch

# Limit this process to ~50% of GPU 0 VRAM.
# This limits only the PyTorch CUDA Caching Allocator.
torch.cuda.memory.set_per_process_memory_fraction(0.5, device=0)

# load model *after* the cap

Practical gotchas:

  • call it early (before allocating tensors / loading weights)
  • in containers, device=0 means the first visible GPU (respecting CUDA_VISIBLE_DEVICES)
  • it does not constrain other processes or CUDA allocations that bypass the PyTorch Caching Allocator

7.5.3 How to tell if it’s a noisy-neighbor problem

Typical symptoms:

  • inference latency spikes without code changes
  • occasional CUDA out of memory, and a restart “fixes” it
  • VRAM usage of one pod oscillates and aligns with another pod’s peaks on the same GPU

Quick checks:

  • use nvidia-smi to confirm multiple processes share the same GPU and whether VRAM is saturated
  • align pod-level GPU VRAM metrics with the incident timeline (who ramped up first?)
  • if workloads cannot self-limit, evaluate HAMi; use MIG or vGPU when failure-domain isolation is required

Final judgment

KAI-Scheduler’s Reservation Pod idea is a very Kubernetes-savvy way to bring fractional GPU semantics into a system that only understands whole devices.

HAMi adds runtime enforcement that platform teams need, but it is not MIG-level hardware isolation.

A pragmatic approach is:

  • start with scheduler-level sharing to lift utilization
  • add HAMi when memory and compute boundaries need enforcement
  • retain MIG or vGPU pools for strong compliance or failure isolation

The operational follow-up is clear: measure who consumes VRAM and SM, separate shared and exclusive workloads, and keep a rollback path for workloads that do not behave well under sharing.

References