Skip to main content
Training tasks, LLM inference services, general services, VSCS, and JupyterLab follow the same task priority rules. Use these rules to assess preemption eligibility, configure resource pool policies, choose or adjust priorities, and identify actual preemption events.

Understand priority and preemption

Task priority determines queue order and whether a waiting workload can preempt another workload. Ordinary priorities are P1~P9: a higher number takes precedence. P99 is a special highest priority for administrators handling urgent tasks. Workload types have no inherent precedence. By default, tasks are ordered by descending priority, then by ascending original submission time within each priority. Distinguish three separate decisions:
  • Queue priority determines a workload’s position in the queue.
  • Preemption capability determines whether a waiting workload can preempt eligible lower-priority workloads.
  • Preemption eligibility determines whether a running workload can be preempted, based on its priority and the policies for its actual resource pool, instance type, and workload type.
Guaranteed priority is a scheduling category, not a reserved resource purchase or a promise of immediate startup. The previous resource guarantee feature is no longer supported. Queue order determines which tasks are considered first; each task must still meet resource, quota, and other scheduling requirements.

Determine whether a task can be preempted

A pool’s per-workload-type preemption switch controls whether running workloads of that type can become preemption targets. It does not change task priority, queue order, or the ability to preempt other workloads. Priority-based preemption requires all of the following:
  • The waiting task has priority P7~P9 or P99 and can preempt other workloads.
  • The target is a running P1~P6 task.
  • The policies for the target’s actual pool, instance type, and workload type permit preemption.
Use the following table to assess a task’s eligibility:
  • Protection from priority-based preemption does not prevent interruptions from other causes. Being preemptible does not mean that a task will be preempted or that enabling the switch immediately interrupts it.
  • Actual preemption also depends on the waiting task’s resource requirements and scheduling conditions.
For example, a P1 VSCS task cannot be preempted by P9 or P99 tasks while the pool’s VSCS preemption switch is off. Enabling that switch makes the task eligible. A P8 VSCS task remains protected from priority-based preemption regardless of the switch.

Configure resource pool preemption policies

Administrators with resource pool management permissions can configure preemption when creating or editing a reserved resource pool. Configure each permitted workload type for each instance type. The same workload type can have different policies in different pools. Defaults apply only to the initial configuration. Check the pool’s current policy. Before changing an existing pool, assess interruptions to running tasks and save required training checkpoints, service data, and development files to persistent storage. To configure a reserved resource pool:
  1. Create a pool or open an existing pool for editing from the pool list.
  2. In Workload Settings, select the Allowed workload types for each instance type. Allowed workload types and Preemptible switches in Workload Settings
  3. Set Preemptible for each workload type and save the configuration.
Permitted workload types determine which workloads can use the pool. Preemption switches determine which eligible running workloads can become preemption targets. Turning a switch off does not change queue priority or prevent P7~P9 or P99 workloads of that type from preempting other eligible tasks.
For pool management procedures, see Configure workload-type preemption policies.

Choose a priority when creating a task

Configure the priority

Administrators assign each user’s priority ceiling separately for each pool. Choose a priority within your authorized range when creating a workload.
  1. Create the workload and select a resource pool and instance specification.
  2. Select the task priority.
    • Your priority ceiling can differ between pools. Available priorities depend on your authorization in the selected pool.
    • Choose a priority that reflects the task’s urgency. For a single-specification task with an authorized ceiling of 8, you can select 1~8, but not 9.
  3. Before submitting, check resource requirements and the pool’s preemption policy for the workload type.
For workload creation procedures, see:

Default priority

When you submit a workload through the API or SDK, the priority field is optional for all workload types. It no longer defaults to your highest authorized priority. If you omit the priority, the task defaults to P3, or to your applicable authorized ceiling if it is below 3. If you explicitly specify a priority within your authorized range, the task uses that value; it is not automatically raised to your authorized ceiling. For example, with a ceiling of 8, an unspecified priority defaults to P3. With a ceiling of 2, it defaults to P2. The default never exceeds your authorized ceiling.

Queue across multiple specifications

When queuing across multiple instance specifications, calculate the selectable priority ceiling in this order:
  1. For each specification, find your highest authorized priority among the resource pools available for that specification.
  2. Take the lowest of those per-specification values as the task’s selectable priority ceiling.
For example, if the two specifications have maximum authorized priorities of 6 and 8, the selectable ceiling is P6. This does not set the task’s actual priority to P6. When submitting through the API or SDK, omitting the priority gives P3; explicitly specifying P2 gives P2. Each pool still applies its own workload-type preemption policy. Disabling preemption in one candidate pool does not disable it in the others.
In training tasks and development environments, Resource Priority orders candidate instance specifications. It is separate from the task priority that determines workload queue order. Reordering specifications does not change task priority.

Understand existing tasks

When workloads join the priority system, existing tasks without a priority are assigned P1. Tasks that already have a priority retain their value. The P1 migration rule for existing tasks is separate from the P3 default for new tasks. An existing P1 task’s eligibility still depends on its pool policy. Reserved pools initially disable preemption for LLM inference services, general services, VSCS, and JupyterLab. Do not assume that a task can be preempted solely because its priority is P1.

View current eligibility and queue status

In workload lists, Priority shows the task’s current priority, and Preemptible shows whether the combined priority and pool policy allow preemption:
  • A positive eligibility result means that the task may be preempted by P7~P9 or P99 workloads after it starts.
  • A negative result means that its priority or pool policy does not permit preemption.
Hover over the question-mark icon next to Preemptible to inspect the current decision and distinguish priority protection from pool restrictions. The tooltip also shows each pool’s policy for that workload type. Check each candidate pool separately when several pools are selected. Priority, Preemptible status, and the current preemption decision tooltip A queued task is not yet occupying running resources and is not a running preemption target. Its eligibility describes what can happen after it starts, not a historical preemption event. To inspect waiting tasks, hover over the information icon next to the queued status. View the queue position, number of tasks ahead, and priority distribution for each instance specification. With several candidates, inspect each queue separately. Queue position does not guarantee a fixed waiting time. For LLM services with prefill/decode disaggregation, Prefill and Decode show separate queues. Both sides must meet their resource requirements before the service can leave the queue.

Adjust queued priorities and handle urgent tasks

Adjust a queued task’s priority

  • Submitters can adjust their own queued workloads within their authorized ceilings. Tasks are reordered using the new priority but retain their original submission time.
  • Pool administrators can adjust queued workloads within their management scope from the pool details.
Tenant administrators can also filter queues across pools, inspect queue metrics, and adjust queued priorities on the Queued Tasks tab of the Resources > Task Dashboard page. See View queued tasks.

Manage urgent tasks

Only administrators can assign P99 to a queued task. Ordinary users cannot assign this special highest priority.
Preemption can interrupt running workloads. Assess the impact on services and development environments before changing priorities. Ensure that training checkpoints and required files are saved to persistent storage.
P99 has these limits:
  • At most one queued P99 task can exist in a pool at a time.
  • Assigning P99 to another task in that pool requires confirmation to replace the existing queued P99 task. The previous task is demoted to P9.
  • Canceled or replaced P99 tasks retain their P99 marker and precede other P9 workloads.
  • Tasks displaced by P99 are placed at the front of the P9 queue in reverse displacement order.
  • P99 takes precedence over P1~P9, but can preempt only eligible running P1~P6 workloads. It cannot preempt P7~P9, another P99, or a task protected by pool policy.
  • P99 must still wait if resources are occupied by protected workloads. It does not increase capacity or guarantee immediate startup.
While a P99 task is queued in a pool:
  • P7~P9 tasks stop leaving the queue. Starting more protected workloads could extend the P99 task’s wait.
  • P1~P6 tasks can backfill only if their workload type permits preemption and idle resources meet scheduling requirements. They may then be preempted when the P99 task’s full requirements are met. Protected P1~P6 tasks cannot backfill.
  • P7~P9 tasks become eligible for scheduling again once P99 leaves the queue; they do not have to wait for P99 to finish. Their actual start still depends on priority, submission time, and resources.
These rules affect subsequent scheduling of queued tasks, not already-running P7~P9 tasks, and do not override pool protection. If P99 still cannot start, check quota, capacity, placement, and eligible preemption targets. See Troubleshoot insufficient resources or long queue times.

Confirm whether preemption occurred

If a task or service is interrupted, has unready instances, or returns to waiting for resources, inspect its details, events, and explicit reasons. Priority or queued status alone does not prove preemption.
  1. Check status details and events for an explicit priority-based preemption reason. VSCS status details can show this reason directly. Expand the details if the text does not fit on one line.
  2. Check the task priority and actual pool, instance type, and workload-type policy at the time of the event. The current configuration may differ.
  3. Follow the workload’s recovery procedure. For training tasks, see Resume training after an interruption.
For continuous services and long training runs, check preemption policies and recovery procedures in advance. Save checkpoints regularly and verify that training can resume. Keep critical files and progress on persistent storage; do not assume automatic or lossless recovery after preemption.

FAQ

Pool, instance type, and workload-type policies can differ. P1 meets the priority condition but must also meet the actual pool policy. Under a reserved pool’s defaults, a P1 training task is preemptible while a P1 VSCS task is not. Check the current decision in the workload list.
No. The workload policy must permit preemption and the target priority must be P1~P6. P8 is a guaranteed priority and cannot be preempted by P7~P9 or P99 tasks through priority-based preemption.
No. The switch controls eligibility as a preemption target. Queue order still depends on priority, original submission time, and resource conditions.
Neither is allowed. P1~P6 cannot preempt other tasks, and P7~P9 cannot preempt each other. A higher priority can improve queue order without granting unrestricted preemption.
No. P99 has the same target scope as P7~P9: running P1~P6 workloads whose actual pool permits preemption of that workload type.
Check the pool’s policy for the workload type first.
  • To protect all eligible workloads of a type in a pool, ask an administrator to disable that type’s preemption switch.
  • To distinguish tasks by business urgency, choose an appropriate priority within your authorized range.
These controls have different scopes. Use them according to your workload requirements.
Priority does not increase capacity. A task may still wait because resources are occupied by protected workloads, or because quota, single-node capacity, distributed placement, or topology requirements are unmet. See Troubleshoot insufficient resources or long queue times.