Ai2's AI Infrastructure team replaced a priority-based GPU scheduler with GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract to prioritize
Ai2 manages thousands of NVIDIA H100, B200, and B300 GPUs arranged in clusters ranging in size from 88 to 1024 GPUs.
The team recently replaced a priority-based scheduler with a system including GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract.
At any moment in time we have outstanding requests for 2-3x more GPUs than are available.
We observed priority inflation, where eventually 100% of scheduled workloads used HIGH priority.
The AI Infrastructure team at Ai2, as reported in a Hugging Face blog post, recently replaced a priority-based scheduler with a system including GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract. The team manages thousands of NVIDIA H100, B200, and B300 GPUs arranged in clusters ranging in size from 88 to 1024 GPUs. The change shifted the debate about how much GPU time each research project deserves from a case-by-case operational task to a transparent administrative budgeting process. The post describes the team's responsibility for providing the institute's GPU compute capacity, specifically targeting large, distributed training workloads.
The previous priority-based scheduler allowed workloads to opt out of preemptability, leading to predictable pathologies. The source reports instances of GPU 'squatting' where users parked no-op workloads they could connect to when the need arose. Priority inflation occurred, with eventually 100% of scheduled workloads using HIGH priority, starving lower priority levels of GPU time. On-call engineers spent a majority of their ticket response time negotiating the organized shutdown of non-preemptable workloads running on hosts with known maintenance problems. The team observed a 'tragedy of the commons' where individuals competing over a scarce, shared resource achieved a non-optimal global result.
Editorial interpretation: The source describes a shift from manual, case-by-case priority decisions to a transparent budgeting process where leadership decides how to fund each research effort with GPU time before workloads exist. This approach addresses the problem of predicting demand, which the source states cannot be forecast with precision because it would require knowing the result of novel science experiments. The scheduler can use the funding information when prioritizing workloads. The source also notes that assigning teams monopolies over sets of GPUs caused GPUs to sit idle due to the seasonality of research, which the new time-based allocation aims to avoid while maintaining full occupancy.
Readers can verify the reported details in the Hugging Face blog post, including the metrics pyramid of availability, occupancy, impact, and utilization. The source describes outstanding requests for 2-3x more GPUs than are available at any moment. The supplied feed entry is a concise summary; consult the canonical article for full context. The source text is truncated at the point where the scheduler would use funding information when prioritizing, so details about simulations, results, challenges, and the future are not available in the supplied material. The post's full content would be needed to understand the outcomes of the new scheduling system.
Ai2's AI Infrastructure team replaced a priority-based scheduler with a system including GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract, as reported in a Hugging Face blog post. The change shifted GPU time allocation debates from operational tasks to a transparent administrative budgeting process.
The old scheduler led to GPU 'squatting' where users parked no-op workloads, priority inflation where 100% of scheduled workloads used HIGH priority, and on-call engineers spending a majority of ticket response time negotiating shutdowns of non-preemptable workloads on hosts with maintenance problems, according to the source.