01huggingface.co
Impactful scheduling for GPU clusters
Ai2 replaced its priority-based GPU scheduler with GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract to prioritize impactful research while keeping clusters occupied. The system gives projects guaranteed capacity shares, charges protected workloads against budgets, and preempts/requeues jobs after declared minimum runtimes; unallocated time remains interruptible. Across a 30-day test, teams received 98% of owed GPU hours and occupancy stayed at 98%; debug-job p90 wait fell from two hours to 30 seconds, while human-required repairs dropped 74%.