Sharing a GPU in Kubernetes sounds simple: run multiple jobs on one physical GPU.

In practice, it quickly turns into a set of very concrete questions:

  • How do you ask for “0.2 GPU” when Kubernetes only knows nvidia.com/gpu: 1?
  • How do you prevent scheduling conflicts with full-GPU pods?
  • And the real pain point: how do you prevent one container from consuming all VRAM and killing its neighbors?

KAI-Scheduler (open-sourced after NVIDIA acquired Run:ai) ships a clever GPU sharing design centered around the idea of a Reservation Pod: allocate a whole GPU to a lightweight placeholder pod, then do fractional accounting inside the scheduler.

HAMi represents a different direction: not just scheduling-level allocation, but runtime-enforced limits (hard isolation) so that “shared” GPUs behave predictably in multi-tenant environments.

The useful split is simple: KAI mainly solves the scheduling expression problem, while HAMi pushes harder on runtime enforcement. Treat them as different layers, not as mutually exclusive camps.

Start with the layer you are missing

GPU sharing has at least two separate problems. If you do not split them, tool selection becomes noisy:

  • Scheduling expression: vanilla Kubernetes only understands integer GPU devices, but the platform wants to express half a GPU, 0.2 GPU, or queue-level fractional accounting.
  • Runtime isolation: workloads may land on the same physical card, but memory, performance interference, and failure blast radius still need control.
  • Tenant governance: teams with different priorities and SLAs cannot rely on everyone voluntarily using less GPU.
  • Rollback and stop-loss: if sharing creates latency or failures, the platform needs a fast path back to whole-card allocation or a more conservative policy.

KAI is mostly about scheduling expression and queue accounting. HAMi pushes further into runtime limits and isolation. Comparing them as one generic “GPU sharing solution” usually hides the real tradeoff.

Common misreads

  • Scheduling several workloads onto one card does not prove runtime isolation is safe.
  • Memory limits do not remove every interference source; bandwidth, I/O, and CPU feed can still collide.
  • A Reservation Pod is not magic. It is a scheduling ledger built around Kubernetes’ integer device model.
  • Stronger isolation usually raises implementation complexity, compatibility testing, and operational cost.

1) Why “fractional GPU” is hard in vanilla Kubernetes

Kubernetes device resources are modeled as extended resources.

For NVIDIA GPUs, that’s typically nvidia.com/gpu. The model is intentionally simple and works well for whole-device allocation, but it comes with hard constraints:

  • integer only: you can’t request 0.2 of nvidia.com/gpu
  • the default scheduler doesn’t understand fractional semantics
  • there’s no standard way to represent “this GPU is partially used” to the rest of the cluster

A common instinct is:

“Fine, I’ll report one physical GPU as multiple virtual devices via a device plugin.”

That’s a valid route (and it’s close to how hard-isolation stacks often start), but KAI takes a different, scheduler-centric approach.

2) KAI’s key trick: the Reservation Pod

2.1 Intuition: reserve the whole GPU, then split the bill internally

KAI’s approach is basically:

Don’t fight Kubernetes over fractional GPU semantics. Use a standard pod to reserve the whole GPU first.

A fractional user pod expresses intent via an annotation, for example:

metadata:
  annotations:
    gpu-fraction: "0.2"

Then KAI-Scheduler does this:

  1. Detects gpu-fraction.
  2. Selects a node and a physical GPU that can be shared.
  3. If this GPU has no existing sharing group yet, it creates a Reservation Pod that requests nvidia.com/gpu: "1".
  4. Once the Reservation Pod is scheduled, Kubernetes sees the GPU as fully allocated, preventing other schedulers/pods from conflicting.
  5. KAI assigns the fractional pods into the same logical group (for example via a gpu-group label) and keeps a private accounting of usage.

2.2 What is it “tricking”?

It’s not tricking the GPU.

It’s “tricking” the resource model: Kubernetes only sees that the GPU is owned by the Reservation Pod, while KAI tracks fractional usage out-of-band.

3) How KAI GPU sharing works (engineering view)

3.1 Scheduling entry: parse annotations and pick a GPU

At a high level, the control flow looks like:

  • find GPUs on the node that can fit
  • pick a GPU that’s preferable for sharing
  • attempt to allocate the fractional task

Example-shaped pseudocode:

func AllocateFractionalGPUTaskToNode(ssn *framework.Session, stmt *framework.Statement,
  pod *pod_info.PodInfo, node *node_info.NodeInfo, isPipelineOnly bool) bool {

  fittingGPUs := ssn.FittingGPUs(node, pod)
  gpuForSharing := getNodePreferableGpuForSharing(fittingGPUs, node, pod, isPipelineOnly)
  if gpuForSharing == nil {
    return false
  }

  pod.GPUGroups = gpuForSharing.Groups
  success := allocateSharedGPUTask(ssn, stmt, node, pod, isPipelineOnly)
  if !success {
    pod.GPUGroups = nil
  }
  return success
}

3.2 The critical step: create the Reservation Pod

The Reservation Pod is pinned to a node and requests a whole GPU:

func (rsc *service) createResourceReservationPod(
  nodeName, gpuGroup, podName, appName string,
  resources v1.ResourceRequirements,
) (*v1.Pod, error) {

  podSpec := &v1.Pod{
    ObjectMeta: metav1.ObjectMeta{
      Name:      podName,
      Namespace: "runai-reservation",
      Labels: map[string]string{
        "gpu-group": gpuGroup,
      },
    },
    Spec: v1.PodSpec{
      NodeName: nodeName,
      Containers: []v1.Container{{
        Name:      "reservation",
        Image:     rsc.reservationPodImage,
        Resources: resources, // request nvidia.com/gpu: "1"
      }},
    },
  }

  return podSpec, rsc.kubeClient.Create(context.Background(), podSpec)
}

Key takeaways:

  • It’s node-pinned (not “waiting” for the default scheduler).
  • It uses labels to define a logical “sharing group”.

3.3 Internal accounting: tracking VRAM usage per sharing group

KAI tracks sharing state internally, for example:

type GpuSharingNodeInfo struct {
  ReleasingSharedGPUs       map[string]bool
  UsedSharedGPUsMemory      map[string]int64
  ReleasingSharedGPUsMemory map[string]int64
  AllocatedSharedGPUsMemory map[string]int64
}

This implies a fundamental property:

KAI’s GPU sharing is implemented at the scheduler level. Not via runtime enforcement.

3.4 Cleanup: last fractional pod exits -> delete Reservation Pod

When the sharing group no longer has active fractional pods, KAI deletes the Reservation Pod to return the GPU to the cluster:

func (rsc *service) syncForPods(ctx context.Context, pods []*v1.Pod, gpuGroupToSync string) error {
  reservationPods := map[string]*v1.Pod{}
  fractionPods := map[string][]*v1.Pod{}

  for _, pod := range pods {
    if pod.Namespace == "runai-reservation" {
      reservationPods[gpuGroupToSync] = pod
      continue
    }
    if pod.Status.Phase == v1.PodRunning || pod.Status.Phase == v1.PodPending {
      fractionPods[gpuGroupToSync] = append(fractionPods[gpuGroupToSync], pod)
    }
  }

  for gpuGroup, reservationPod := range reservationPods {
    if _, found := fractionPods[gpuGroup]; !found {
      return rsc.deleteReservationPod(ctx, reservationPod)
    }
  }
  return nil
}

4) KAI’s strengths—and its unavoidable constraint

4.1 Strength: elegant, Kubernetes-native integration

KAI’s design is concrete:

  • Uses standard K8s primitives (pods, labels, annotations)
  • Avoids direct conflict with the default scheduler by reserving the whole GPU
  • Allows flexible fractions (not limited to a few predefined profiles)

If your goal is to improve utilization quickly and you can coordinate with application owners, KAI can be a fast win.

4.2 The core limitation: soft isolation

The trade-off is also fundamental:

KAI allocates “fractions” logically, but it can’t enforce those fractions at runtime.

That leads to:

  • no hard cap on VRAM/compute
  • noisy neighbor risks
  • application-side tuning often required (e.g., setting per-process VRAM limits)

Think of it as “sharing a room with house rules” rather than “private rooms with locked doors”.

5) The HAMi path: enforce isolation lower in the stack

Hard isolation solutions aim to avoid relying on application self-discipline.

A typical design combines:

  • device plugin logic to expose schedulable units
  • runtime/driver-level enforcement to cap VRAM (and sometimes other resources)

Benefits:

  • predictable limits per container
  • stronger multi-tenant QoS
  • less need for application-level configuration

Costs:

  • more components to deploy
  • more constraints on driver/runtime compatibility

6) Which one should you use?

Choose a KAI-style approach if:

  • you need a lightweight path to better utilization
  • you can tolerate some performance variance
  • you can coordinate with application teams to set VRAM usage limits

Choose a HAMi-style approach if:

  • you run a multi-tenant GPU platform
  • you need stable QoS and strict resource guarantees
  • you want sharing to be transparent to users

7) A realistic future: compose scheduler policies with hard isolation

It’s tempting to treat this as a “winner vs loser” debate.

In practice, they map nicely to two layers:

  • scheduling policies and UX (KAI’s strength)
  • enforcement and guarantees (HAMi’s strength)

If the ecosystem aligns interfaces between the two, you can get the best of both worlds:

  1. rich scheduling policies and a clean fractional request UX
  2. hard caps underneath to eliminate noisy-neighbor failure modes

7.5 Hands-on: running two workloads on one shared GPU (how to survive soft sharing)

A very common situation: you’ve got a 24GB GPU and you want to run two workloads on it:

  • Workload A: an online inference service (needs predictable VRAM)
  • Workload B: an offline evaluation / small training job (can be slower, but must not crash A)

With scheduler-level sharing (soft isolation), the scheduler can co-locate A and B on the same physical GPU, but preventing conflicts often requires application-side guardrails.

7.5.1 Pod side: request a fraction via annotation

Example (0.5 + 0.5):

apiVersion: v1
kind: Pod
metadata:
  name: infer-a
  annotations:
    gpu-fraction: "0.5"
spec:
  containers:
    - name: app
      image: your-registry/infer:latest
---
apiVersion: v1
kind: Pod
metadata:
  name: eval-b
  annotations:
    gpu-fraction: "0.5"
spec:
  containers:
    - name: app
      image: your-registry/eval:latest

Note: this intentionally doesn’t use resources.limits: nvidia.com/gpu, because fractional GPU is not a native Kubernetes resource. The exact knobs depend on how KAI is integrated in your cluster.

7.5.2 App side: VRAM self-limiting (PyTorch example)

The simplest guardrail is to cap VRAM usage in-process before loading the model:

import torch

# Limit this process to ~50% of GPU 0 VRAM.
# This is not hard isolation, but it reduces noisy-neighbor incidents in many real deployments.
torch.cuda.set_per_process_memory_fraction(0.5, device=0)

# load model *after* the cap

Practical gotchas:

  • call it early (before allocating tensors / loading weights)
  • in containers, device=0 means the first visible GPU (respecting CUDA_VISIBLE_DEVICES)

7.5.3 How to tell if it’s a noisy-neighbor problem

Typical symptoms:

  • inference latency spikes without code changes
  • occasional CUDA out of memory, and a restart “fixes” it
  • VRAM usage of one pod oscillates and aligns with another pod’s peaks on the same GPU

Quick checks:

  • use nvidia-smi to confirm multiple processes share the same GPU and whether VRAM is saturated
  • align pod-level GPU VRAM metrics with the incident timeline (who ramped up first?)
  • if you can’t require every workload to be “well-behaved”, you usually want hard isolation underneath

Final judgment

KAI-Scheduler’s Reservation Pod idea is a very Kubernetes-savvy way to bring fractional GPU semantics into a system that only understands whole devices.

HAMi’s direction is a platform-engineering answer to the uncomfortable truth: sharing without enforcement turns into operational risk.

A pragmatic approach is:

  • start with scheduler-level sharing to lift utilization
  • add hard isolation when you need production-grade multi-tenant guarantees

The operational follow-up is clear: measure who consumes VRAM and SM, separate shared and exclusive workloads, and keep a rollback path for workloads that do not behave well under sharing.