PD-Disaggregated Inference Deployment with sgl-project/rbg
Use RoleBasedGroup to run SGLang Prefill/Decode disaggregation as one operable unit: routing, startup dependencies, multi-node tensor parallelism, KV transfer, and coordinated rollouts.
Category archive
GPU
6 practical articles on GPU, covering setup, troubleshooting, and production decisions.
More topics
Other topics
Selected pieces
Featured in this category
Startup GPU Costs: Serverless, Dedicated, and Hybrid Capacity
Calculate Serverless versus dedicated GPU break-even using effective GPU hours, cold starts, idle capacity, operations, and SLOs, then plan hybrid capacity.
GPU Overprovisioning: Time-Slicing, MIG, and Backfill
Compare queue backfill, NVIDIA Time-Slicing, MIG, and runtime sharing with Kubernetes configuration, validation, isolation, and rollback steps.
KAI-Scheduler vs HAMi: Scheduling Accounting and Runtime Enforcement
Compare KAI-Scheduler Reservation Pod accounting with HAMi CUDA API-level memory and compute enforcement, including their boundary versus hardware isolation such as MIG.
hetGPU: What Cross-Vendor GPU Binary Portability Requires
A review of the hetGPU research prototype that separates paper design, conceptual pseudocode, and current cross-vendor GPU engineering capability.
gpu-manager Architecture: Device Plugin, vCUDA, and Migration
Review tkestack/gpu-manager device registration, vCUDA resources, library injection, topology allocation, CUDA limits, and migration choices.
Archive
All posts in this category
All posts in reverse chronological order.
PD-Disaggregated Inference Deployment with sgl-project/rbg
Use RoleBasedGroup to run SGLang Prefill/Decode disaggregation as one operable unit: routing, startup dependencies, multi-node tensor parallelism, KV transfer, and coordinated rollouts.
Startup GPU Costs: Serverless, Dedicated, and Hybrid Capacity
Calculate Serverless versus dedicated GPU break-even using effective GPU hours, cold starts, idle capacity, operations, and SLOs, then plan hybrid capacity.
GPU Overprovisioning: Time-Slicing, MIG, and Backfill
Compare queue backfill, NVIDIA Time-Slicing, MIG, and runtime sharing with Kubernetes configuration, validation, isolation, and rollback steps.
KAI-Scheduler vs HAMi: Scheduling Accounting and Runtime Enforcement
Compare KAI-Scheduler Reservation Pod accounting with HAMi CUDA API-level memory and compute enforcement, including their boundary versus hardware isolation such as MIG.
hetGPU: What Cross-Vendor GPU Binary Portability Requires
A review of the hetGPU research prototype that separates paper design, conceptual pseudocode, and current cross-vendor GPU engineering capability.
gpu-manager Architecture: Device Plugin, vCUDA, and Migration
Review tkestack/gpu-manager device registration, vCUDA resources, library injection, topology allocation, CUDA limits, and migration choices.