As GPUA100 nodes are a highly sacred and in-demand resource, manual intervention by administrators has become unmanageable 🖥️. Effective today (Wednesday – 21st October 2026), we are implementing an automated job cancellation policy without any further consideration or communication of your job resource utilisation.
⚙️ Key Policy Details:
- A background scheduler now runs on GPU-A100 nodes (gpu06 & gpu07) every 5 minutes.
- The scheduler directly measures GPU usage using
nvidia-smirather than Grafana to ensure accurate, real-time readings and prevent false positives. - If your GPU usage drops below 40%, a timer will start ⏱️.
- Jobs utilizing less than 40% of the GPU for 1 hour straight will be automatically cancelled by the background scheduler ❌.
- If your GPU usage rises back above 40% at any point between checks, the timer resets to zero 🔄.
- Jobs that are “younger” than 5 minutes are skipped by the checker, but if your initial data loading phase takes longer than 1 hour, the job may still be cancelled.
- To ensure efficient resource allocation, jobs requiring under 32 GB of VRAM (40% of GPU-A100 VRAM) must be routed to other partitions, namely gpu-k40c, gpu-titan, and gpu-v100s 🔀.
For the complete official guidelines, please refer to the document “GPUA100 Utilisation Threshold &
Auto-Cancellation Policy“
💬 Support and Notifications:
- If you need any assistance or have questions regarding your workflows, you may submit a ticket on the DICC Service Desk .
- You may also get news notifications on Telegram; please join us via our official link 📱.