Training Advanced Config

In addition to the basic creation workflow, Training provides a set of Advanced Config options that can be enabled as needed to control costs, scheduling scope, RayJob behavior, failure recovery, collaborative reuse, and observability. This document focuses on the configurations required when creating a task; links to the relevant documentation are provided for task management and troubleshooting after creation. For the complete task creation workflow, see Create Training.

Low-Cost Resources

Spot Instances

Spot instances are suitable for non-critical tasks that can be interrupted and have checkpointing or retry mechanisms. When using Spot resources, tasks may be evicted when resources are constrained, and the task status may change to Failed. When creating a training task, go to the Compute Resources configuration, select the Spot resource type, and then select the Instance Type, instance specification, and Count Per Node as displayed on the page. Spot resource availability varies with cluster machine utilization. If no instances are available on the page, they are currently unavailable within the selected filter scope. When using Spot, it is recommended to pay attention to the following information:

Suitable for experimental tasks, batch jobs, or training that can be quickly resubmitted.
Not suitable for long-running critical training without checkpoints.
Spot is a non-guaranteed resource. Tasks may be evicted when resources on the host machine are constrained.
After eviction, the task may enter the failed state. We recommend resubmitting based on the task output and checkpoints; use reserved or pay-as-you-go resources for critical training.

Idle Tasks

Idle tasks run on the Cluster’s currently available resources. When submitting an idle task, you do not need to bind it to a purchased quota. Idle tasks are low-cost, but resources are not guaranteed and may be preempted by tasks with a higher Priority. When creating a training task, go to the Compute Resources configuration and select Idle Resource. Then, select the instance type, instance specification, and Count Per Node as displayed on the page, and configure the Idle Stability Level as needed. When using idle-time tasks, we recommend paying attention to the following information:

Suitable for interruptible, retryable tasks that do not require a fixed completion time.
The available capacity of idle resources changes dynamically, and tasks may be preempted by higher-priority tasks.
Idle Resource Fragments vary by stability level. Distributed training requires each replica to be placed on an available node; you cannot simply add up fragments across multiple rows to determine whether scheduling is possible.
Idle resources do not support setting Task Priority or Guarantee. Refer to the disabled state on the page.
After a task is preempted, it may enter the failed state, and the status information may include was preempted by another task.
Idle resources are billed according to the actual billing terms and should not be assumed to be free resources.
We recommend configuring a checkpoint or retry mechanism for training scripts to avoid relying entirely on non-guaranteed resources for critical training workflows.

Scheduling and Guarantee

Task Priority and Resource Priority

Training supports the configuration of Priority and Resource Priority. The former affects the scheduling order during resource contention, while the latter affects the order in which multiple candidate instance configurations within the same task are attempted. When creating a training task, configure the Priority as displayed on the page. Typically, you can select the task priority based on the task’s importance and timeliness:

Task Type	Recommendation
Critical training	Use a higher Priority together with Guarantee to reduce the risk of preemption or prolonged queuing.
Routine experiments	Use normal Priority to balance stability and resource fairness.
Interruptible experiments or batch processing	Use a lower Priority or Idle Resource to reduce the impact on critical tasks.

The normal task priority ranges from 1 to 9; a higher value indicates a higher priority. Task priority does not guarantee resource reservation. Whether a task can start still depends on the resource pool status, personal quota, instance specification, and scheduling conditions. Resource priority applies when multiple candidate instance configurations are available. The platform first attempts the candidate instances listed earlier. If a preceding candidate is temporarily unavailable, it attempts subsequent candidates. Before configuring multiple candidate instances, ensure that your training code is compatible with the corresponding GPU model, GPU memory, CPU/memory combination, and communication capabilities.

Guarantee

Resource Guarantee controls whether a task uses non-preemptible resources. When Resource Guarantee is enabled, tasks are more stable; otherwise, they may be preempted during resource contention. When creating a training task, if the page provides the Guarantee configuration, you can choose whether to enable it based on the task’s stability requirements. For critical training, long-running training, or tasks with high recovery costs, enabling the guarantee is recommended. For short-term experiments or tasks that can be quickly resubmitted, you can choose not to enable it based on cost and resource availability. Priority and Guarantee jointly affect scheduling outcomes during resource contention. Guarantee applies to scheduling and preemption; it does not address code exceptions, image pull failures, incorrect data paths, hardware failures, network failures, or stalled programs. These issues must be troubleshot using logs, events, monitoring, and fault-tolerance mechanisms.

Accept Idle Resource

When using a reserved instance, if Accept Idle Resource is available on the page, you can allow the task to use idle resources as an alternative when reserved resources are insufficient. When idle resources are actually used, the task’s priority, stability, and reclamation risk are handled according to the idle resource rules. When configuring this capability, consider the following information:

Configuration Item	Description
Accept Idle Resource	Allows the scheduler to attempt to use idle resources when reserved resources are insufficient. The resource pool actually used by the task is determined by the actual resources shown in the task details.
Prioritize Idle Resources	If this option is available on the page, the task can attempt to use idle resources first. Whether it can be started still depends on the availability of idle resources and the complete task scheduling conditions.
Idle Resource Stability Requirements	You can select a stability level, such as High, Medium, or Low. Higher stability requirements may result in fewer available idle resources. Lowering the stability requirement does not guarantee a faster startup or any specific runtime duration.

Enabling idle-resource fallback does not mean that the platform combines reserved resources and idle resources into a single distributed task. The task must still meet the scheduling requirements for the full replica count.

Node Schedule Policy

Node scheduling policies are centrally maintained under Config Management > Schedule Policy and can be applied to training tasks by workload type. You can exclude specified nodes through Block, or restrict tasks to run only on specified nodes through Schedule Only. Policies can apply to matching workloads by default, or take effect only after being explicitly selected when a workload is created. Node scheduling policies change the range of nodes available to tasks and may affect whether tasks can be scheduled. Before submitting a task, ensure that the policy’s Schedule Type, Scope, Effect Mode, Workload Type, Node Name, and Expire At meet your expectations. For information about creating, finding, and maintaining policies, see Manage Node Scheduling Policies. If a task cannot be scheduled due to a node blacklist policy, the event or error message may contain quota check failed due to node blacklist: XXX. In this case, check whether any Schedule Policy applies to the current task.

RayJob Enhancements

Shared Storage

RayJob can be configured with shared storage by mounting the same Volume to the Head and Worker Pods to share checkpoints, logs, intermediate results, or cached data. When creating a training task, set Task Type to RayJob. In the RayJob-specific configuration, enable Mount Shared Storage and select the Volume to mount. Choose whether to clear the shared storage data after the task exits, as needed. After submission, the platform verifies that the current account has permission to access the Volume and mounts the shared storage to each Pod in the RayJob. Task code can read from and write to the shared storage through /tmp/ray.

import os
import torch

with open("/tmp/ray/checkpoint.pt", "wb") as f:
    torch.save(model.state_dict(), f)

state_dict = torch.load("/tmp/ray/checkpoint.pt")
files = os.listdir("/tmp/ray")

Shared storage is suitable for the following tasks:

Task	Description
Checkpoint sharing	Multiple Ray Pods need to read from and write to the same checkpoint.
Log aggregation	Write the outputs of the Head and Workers to a unified directory.
Intermediate result transfer	A multi-stage Ray Pipeline needs to directly read the output of the preceding stage.
Data preprocessing cache	After the Head performs preprocessing, Workers reuse the cached results.
Debug data collection	Centrally store profiling, trace, or temporary troubleshooting files.

When using shared storage, pay attention to the following information:

Shared storage is available only for RayJob. PyTorchJob does not support this capability.
Shared storage is mounted to the /tmp/ray directory of each Pod in a RayJob.
During debugging, you can leave Clean Up After Exit unchecked to retain data for troubleshooting. For production tasks, cleanup is recommended to prevent storage leaks.
Shared storage is intended for sharing data across Pods. It does not replace task input datasets or long-term model archives.
Shared storage does not participate in scheduling decisions. Whether a task can be scheduled successfully still depends on the resource pool, instance specification, node policies, and quota.
Shared storage requires a Volume that supports read/write access by multiple Pods. The available Volumes are subject to what is displayed on the page and the administrator’s configuration.
Shared storage organizes directories by task. The Head and Worker Pods of the same task can share data under /tmp/ray. To reduce data confusion between tasks, use different task name prefixes and clearly defined subdirectories in your code.
Cleaning up shared storage data after a task exits does not affect the task’s final status. If cleanup fails, go to the corresponding Volume and manually delete the directory for the task.

Auto Scaling

RayJob can automatically adjust the number of Workers based on the workload. When configuring it, pay attention to the minimum scale-down limit, maximum scale-up limit, reclamation time, and scale-up mode. When creating a training task, select RayJob as the Task Type, enable this capability in the Ray autoscaling configuration, and enter the autoscaling parameters.

Configuration Item	Description
Minimum Scale-In Limit	The minimum number of Worker replicas, preventing scale-in from reducing the workload below the minimum size required for the task to run.
Maximum Scale-Out Limit	The maximum number of Worker replicas, used to limit resource consumption.
Reclamation Time	If a Worker remains idle longer than this period, the platform can reclaim the corresponding Worker.
Scale-Out Mode	Controls the scale-out pace. Conservative mode is more stable, while aggressive mode reaches the desired number of replicas faster.

Ray jobs with significant workload fluctuations are well suited for enabling auto scaling. For jobs with high requirements for resource stability, set the scaling limits carefully to avoid frequent scaling operations affecting the training or computation process.

Runtime Env

Runtime Env is used to configure runtime dependencies, environment variables, and other settings for Ray jobs. Before use, ensure that the image already includes a basic Ray environment and that the Runtime Env configuration format is correct. When creating a training task, select RayJob as the Task Type and enter the Runtime Env in the RayJob-specific configuration. Runtime Env uses the YAML format supported by Ray and can be used to declare runtime dependencies, environment variables, and other settings. Leave it blank if no additional runtime configuration is required. RayJob requires an image that includes the Ray runtime environment. If you use a custom image, ensure that it contains the Ray components and dependencies required for the job; otherwise, Ray processes may fail to start, and the job may be unable to enter the Running state.

Pod Composition and Logs

After a RayJob is submitted, it typically includes the following Pods:

Pod Type	Description	Consumes Quota
Head Pod	The Head node of a Ray cluster. There is one such node.	Yes
Worker Pod	Worker nodes of a Ray cluster. The number is determined by the number of worker nodes.	Yes
Submitter Pod	Executes Ray submission commands and aggregates user code logs.	Typically does not consume training resource quota.

The number of Pods ultimately used by a RayJob for training resources is typically number of Workers + 1. User code logs are typically viewed through the Submitter Pod, while Ray cluster logs can be viewed in the Head Pod or Worker Pods. To access the Ray Dashboard, ensure that the image includes the required dependencies, and use the entry point shown on the task details page. During RayJob execution, if Debug is not enabled, the Ray cluster is typically reclaimed after the Submitter Pod exits. If Debug is enabled, the Ray cluster is retained for the configured retention period, allowing you to access the Head Pod and view the cluster status. Stopping a RayJob that is not in the stopped state releases the associated Pod resources.

Debugging and Recovery

Resource Precheck

Resource precheck is used to check the health status of the selected resources before the training command is executed. When enabled, the platform first runs a preset diagnostic script to check NCCL process group initialization, GPU GEMM performance, and AllReduce communication performance. The platform executes the training command only after the checks pass. If a check fails, the training command will not be executed, and the task will be marked as failed. If you need to pass variables to the diagnostic items, you can add precheck environment variables when creating the task. Precheck environment variables are used only for resource prechecks and are not equivalent to training container environment variables.

Debug Retry

Debug resubmission is suitable when the cause of a task failure is unclear and you need to enter the container to inspect the environment, paths, dependencies, or node status. It resubmits a new task for troubleshooting based on the original task configuration, typically retaining the original resource requirements, but it does not directly run the original training command. Perform this action after the task is created. Before you begin, ensure that the checkpoints, logs, or model outputs to be retained have been written to a persistent Volume. For specific access points and troubleshooting steps, see Troubleshoot Training Tasks.

Automatic Failover

Automatic fault tolerance retries or resumes tasks according to the policy when exceptions occur. It is suitable for tasks that have written checkpoints to a persistent Volume and whose training programs can continue running from the checkpoints. When creating a training task, enable Fault Tolerance and Diagnostics in Advanced Options, then configure Maximum Task Resubmissions and other settings. Automatic fault tolerance cannot resolve deterministic errors such as an unavailable image, an incorrect startup command, an invalid data path, or insufficient GPU memory.

Collaboration and Reuse

Labels

Labels are used to categorize, filter, collect statistics on, and audit training tasks. When creating a task, you can select labels displayed on the page. If the administrator has enabled label validation, you must select from the maintained labels. After a task is created, you can modify its labels from the task list or details page. For more information, see Find, View, and Manage Training Tasks. Task owners can make tasks available to other users through Project Group sharing or personal sharing, allowing them to view task details and logs and clone tasks. Task visibility does not automatically grant write permissions for resource pools, volumes, directories, or operations such as stopping and deleting tasks. For information about the differences between Project Group visibility, personal sharing, cloning and reuse, and storage permissions, see Managing Training Task Sharing and Project Group Visibility.

Environment Variables

Temporary TaskEnv variables specified when creating a task apply only to the current task. Persistent environment variables in Config Management can be automatically injected into Training tasks. If the keys of the two types of environment variables conflict, the temporary TaskEnv variables specified for the current task take precedence. When submitting a task, the platform validates variable names, variable values, duplicate variable names, the number of variables, and byte size. For rules, see Create Training.

Custom Monitor

Training monitoring includes platform preset metrics and custom metrics. Platform preset metrics are collected by default. To collect custom metrics exposed by training code, enable Metrics Monitor when creating a task and configure metric endpoints. For end-to-end integration and multi-port collection examples, see Integrate Custom Metrics into Training. To view collected metrics, see View Training Task Monitoring.

Get Started

Resources

Development & Training

Models & Inference

Operations & Governance

Best Practices

Developer Access

FAQ

Training Advanced Config

Low-Cost Resources

Spot Instances

Idle Tasks

Scheduling and Guarantee

Task Priority and Resource Priority

Guarantee

Accept Idle Resource

Node Schedule Policy

RayJob Enhancements

Shared Storage

Auto Scaling

Runtime Env

Pod Composition and Logs

Debugging and Recovery

Resource Precheck

Debug Retry

Automatic Failover

Collaboration and Reuse

Labels

Environment Variables

Custom Monitor

​Low-Cost Resources

​Spot Instances

​Idle Tasks

​Scheduling and Guarantee

​Task Priority and Resource Priority

​Guarantee

​Accept Idle Resource

​Node Schedule Policy

​RayJob Enhancements

​Shared Storage

​Auto Scaling

​Runtime Env

​Pod Composition and Logs

​Debugging and Recovery

​Resource Precheck

​Debug Retry

​Automatic Failover

​Collaboration and Reuse

​Labels

​Share

​Environment Variables

​Custom Monitor

​Related Documentation

Low-Cost Resources

Spot Instances

Idle Tasks

Scheduling and Guarantee

Task Priority and Resource Priority

Guarantee

Accept Idle Resource

Node Schedule Policy

RayJob Enhancements

Shared Storage

Auto Scaling

Runtime Env

Pod Composition and Logs

Debugging and Recovery

Resource Precheck

Debug Retry

Automatic Failover

Collaboration and Reuse

Labels

Share

Environment Variables

Custom Monitor

Related Documentation