Create Training

After you create a training task, the platform schedules resources based on the selected Region, Cluster, resource pool, and instance specification, and runs the training command in a container. This document uses console-based form creation as an example to describe the complete configuration process. Before formally submitting a long-running training job, we recommend first using a small-scale task to validate the image, mount path, startup command, and checkpoint writing logic.

Create Training

1. Go to the creation page

Go to Training > Workloads.
On the Personal Tasks tab, click Create.

2. Configure General Settings

First, complete the shared basic settings for PyTorchJob and RayJob. The task type determines the specific settings that must be added later.

Basic information

Configuration Item	Description
Region and Cluster	Determines where the task runs and also affects the available resource pools, images, volumes, datasets, and models.
Project Group	The project group to which the task belongs, used to categorize resources and permissions.
Task Name Prefix	The prefix for the task name. The platform appends a unique suffix to the task name prefix to generate the final task name. We recommend using lowercase letters, numbers, and hyphens, with the name starting and ending with a letter or number.
Task Type	Select a task type. PyTorchJob: Suitable for running single-node or distributed training jobs based on PyTorch. RayJob: Suitable for running distributed training, scheduling, or parallel computing jobs based on Ray. Different task types affect subsequent resource configuration, startup methods, and dedicated settings.
Public within Project Group	Controls whether the task is visible within the project group. If it is not public, it is generally visible only to the task owner. For the differences between project group visibility and personal sharing, see Manage Training Task Sharing and Project Group Visibility.
Labels	Add labels to the task to facilitate categorization, filtering, and auditing. After an administrator enables label validation, you may need to select maintained labels when submitting a task.

Environment Information

The runtime environment determines which image the training container uses, which inputs it reads, where it writes outputs, and which cmd it executes after startup. Before creating it, ensure that the image, code path, data path, and output path are ready.

Configuration Item	Description
Image	You can select a preset image, a custom image, or enter an image address directly. The image must include the frameworks and dependencies required to run the training script. RayJob requires an image that includes the Ray runtime environment. To use the full Ray Dashboard functionality, the custom image must also include the corresponding dependencies.
Environment Variables	Click Add to add temporary environment variables for the current task. These can be used to pass paths, parameters, switches, or credentials. Note the following when configuring them: Variable names can contain only English letters, digits, and underscores, and cannot start with a digit. They cannot be duplicated or contain `=`, spaces, line breaks, Chinese characters, full-width symbols, zero-width characters, or similar characters. Variable values must be strings. They can be `0`, but cannot be empty or contain NUL characters. You can hide variable values as needed. If a variable has the same name as a persistent environment variable in TaskEnv Management, the configuration for the current task takes precedence.
Volume	Mount a Volume to store code, checkpoints, training outputs, and model files. You must select storage and enter the mount path in the Pod. The same storage volume can be mounted only once at the same mount path. Writable persistent storage is recommended for the output and checkpoint directories.
Dataset	Mount datasets for the training script to read input data. Datasets are typically used as read-only input, and you must specify the container mount path.
Model	Mount models for fine-tuning, continued training, or evaluation, and specify the container mount path.
cmd	The training command executed after the container starts. The platform runs it using `bash -c`. PyTorchJob can use the distributed training environment variables automatically injected by the platform in the command. A RayJob command is typically a Ray entrypoint, such as `python main.py`.

Use separate directories starting with /volume or /mnt as mount paths for storage, datasets, and models to avoid overwriting default image directories such as /opt, /workspace, and /usr. Path conflicts may render dependencies, code, or startup commands unavailable. Container local disks have limited capacity; writing large amounts of data may slow down the system, and local disk data may also be cleared after a restart or migration. Important files, such as training code, datasets, models, checkpoints, and output results, should be written to mounted volumes.

COMPUTE Resources

Compute resources determine the type of resources used by a task, the number of replicas to launch, and the CPU/GPU resources requested by each replica. We recommend completing the configuration in the following order: Resource Type -> Replicas -> Instance Spec -> Quota and Scheduling. 1. Select Resource Type Different resource types affect the available instances, quota validation, resource guarantees, and interruption risks. Select a resource type as needed.

Resource type	Suitable scenarios
Reserved	Run training tasks using purchased or allocated reserved quota.
Ondemand	Temporarily start training tasks without reserving quota in advance.
Spot	Cost-sensitive, interruptible, and retryable training tasks.
Idle Resource	Tasks that can tolerate unstable run durations and whose training programs support checkpointing or resubmission.

2. Configure Replicas

PyTorchJob PyTorchJob has a fixed master node count of 1, while the number of worker nodes can be set to 0. The master node and worker nodes use the same specifications. For distributed training, the number of worker nodes affects the number of replicas that must be scheduled simultaneously. For the total number of task replicas, PyTorchJob typically calculates the number of replicas that must be scheduled simultaneously as 1 + number of worker nodes.
RayJob RayJob configures the Head and Worker separately. The number of Heads and their resources are specified in the Ray form, while the number of Workers determines the scale of worker nodes. For the total number of task replicas, RayJob resources are calculated separately for the Head and Worker.

3. Select Instance Spec

Configuration Item	Description
Instance Type	Select GPU or CPU, then select a specific instance specification from the instance list. Specification displays the resources included in that specification, such as GPU, vCPU, and memory.
Count Per Node	Configure the number of selected instance specifications used by each replica. Resource requirements for a single replica = resources in one specification in Specification × Count Per Node.
Resource Priority	When multiple instance specifications are selected, the platform attempts scheduling in descending priority order, giving preference to instances listed first. Select only specifications compatible with the training code to prevent the task from being scheduled to incompatible GPU models or CPU/memory combinations.

Resource quantities are calculated based on the role configuration. The specification values on the page are for displaying the resources of a single selected instance specification only. Actual resource quantities depend on the selected specification and number of replicas.

Task Type	Compute Method
PyTorchJob	Total replicas = `1 + number of Workers`; total specification units = `total replicas × Count Per Node`; total GPUs = `total specification units × number of GPUs per specification`. CPU and memory are also calculated based on the total specification units.
RayJob	Resource requirements for the Head and Worker are calculated separately. Quotas are verified collectively for the same specifications and separately for different specifications.

4. Confirm Quota and Scheduling Before submitting, do not consider only whether an individual instance specification is available; also confirm that the entire task can be scheduled simultaneously.

Item to Check	How to Assess
Remaining Quota	The quota that the current account can continue to use for the selected resource type, resource pool, and instance specification. When creating a task, verify against the resource requirements of the entire task, rather than a single replica only.
Pool Available	A reference value for the capacity currently available from the resource pool. This value changes with other tasks and the status of the resource pool and must not be treated as a resource reservation.
Idle Resource Quantity / Idle Resource Fragmentation	When using the idle-resource supplementation capability of Reserved instances or directly selecting Idle Resource, check the displayed idle-resource quantity, stability level, and fragmentation. Fragmentation information is for availability reference only; distributed training must still meet overall scheduling requirements.
Guarantee	Based on the resource type and page capabilities, choose whether to enable a more stable guarantee policy. For critical training, long-running training, or tasks with high recovery costs, configurations that support Guarantee are recommended.

Distributed training uses gang scheduling. The replicas required by a job cannot be assessed solely by summing total resources; it is also necessary to ensure that each replica can be placed on an available node and that all replicas can start simultaneously. If resources are insufficient, the entire job enters the queue and is automatically scheduled once the resource requirements are met, without manual retries.

Priority affects task queuing and scheduling order and may also affect outcomes in resource contention. For more details about low-cost resources, Guarantee, node scheduling policies, and preemption, see Training Advanced Config.

Advanced Options

Configuration Item	Description
Resource Precheck	Supported only by PyTorchJob. When enabled, the platform runs a preset resource precheck before training starts. Note the following when configuring it: The checks include NCCL process group initialization, GPU GEMM performance, and AllReduce communication performance. The checks increase task startup time. The platform runs the training command only after the checks pass. If a check fails, the training command will not run and the task will be marked as failed. To pass variables to the checks, click Add to add precheck environment variables.
Fault Tolerance and Diagnostics	When enabled, the platform can retry or recover tasks that encounter exceptions according to the configured policy. Maximum Task Resubmissions: The maximum number of times the platform automatically resubmits a task after it fails. This excludes the initial submission, manual resubmissions, and restarts caused by resource replacement. Setting this value to 0 disables automatic resubmission. Automatic Node Replacement Upon Failure: When enabled, if a PyTorch task fails due to a machine failure such as a GPU or NIC failure, the platform automatically replaces the node and resumes task execution. Automatic retries cannot resolve issues that require changes to the configuration or scripts, such as code issues, data path issues, or insufficient GPU memory.
Exit Checkpoint Notification	Supported only by PyTorchJob. When enabled, the Pod notifies the training process to save a checkpoint before exiting. Resources are released only after the checkpoint is saved, with a maximum wait time of 600 seconds. Notifications are sent for preemption reclamation, manual stops, and task deletion, which can slow down task exit.
Metrics Monitor	When enabled, collects custom metrics exposed by the training code. Preset resource metrics are collected by default and do not need to be configured here. For each metrics collection endpoint, specify Metrics Port and Metrics Path. To collect multiple endpoints, click Add Metrics Endpoint to add them one by one. Within the same task, the combination of Metrics Port and Metrics Path cannot be duplicated. For detailed examples, see Integrate Custom Metrics into Training.
Image Pull Tolerance	Configure the maximum number of retries for failed image pulls to handle temporary pull fluctuations. The default value is subject to the current page display. A value of `0` indicates that the task is considered failed upon the first failure; `-1` disables image pull circuit breaking for this task; a positive integer uses a custom threshold for this task.
Passwordless Login Between Pods	When this setting is enabled, passwordless SSH login between multiple Pods is automatically enabled, and node information is generated and stored in the `/root/hostfile` and `/root/hostfile.hostname` files, respectively. This feature is available in common operating systems, such as Ubuntu, and common images, such as NGC images. However, it may not be available in certain special environments. If you encounter any issues, contact technical support.
Alarm Subscription	To configure alarms when creating a task, click Add Alarm Subscription. Refer to the Alarm Subscription page for the specific fields.

3. Complete the configuration based on the Task Type

PyTorchJob

PyTorchJob is suitable for running single-node or distributed training jobs based on PyTorch. After completing the general configuration, check carefully whether cmd can correctly start the training script. For PyTorchJob multi-node multi-GPU training, you can use the platform-injected distributed training environment variables in the startup command:

Environment Variable	Description
`MASTER_ADDR`	Address of the master node.
`MASTER_PORT`	Port of the master node.
`WORLD_SIZE`	Total number of machines participating in training.
`RANK`	ID of the current node.

The platform also injects task context-related TaskEnv, such as X_USER_NAME, X_ORG_NAME, X_TENANT_ID, X_REGION, X_CLUSTER, and TASK_UUID. If the training script needs to record the task source, region, cluster, or unique task identifier, it can read these variables. Example:

torchrun \
  --master_addr "$MASTER_ADDR" \
  --master_port "$MASTER_PORT" \
  --nnodes "$WORLD_SIZE" \
  --nproc_per_node 8 \
  --node_rank "$RANK" \
  train.py

Adjust --nproc_per_node based on the actual number of GPUs per node, and replace train.py with the actual training script.

RayJob

RayJob is suitable for running Ray-based distributed training, scheduling, or parallel computing tasks. After completing the general configuration, pay particular attention to the Head / Worker, Runtime Env, shared storage, and autoscaling settings.

Configuration Item	Description
Head Instance Specification	Configures the instance specification used by the Head node. If not configured separately, it typically uses the same specification as the Worker nodes.
Runtime Env	Enter the Ray Runtime Env in a YAML format supported by Ray. It can be used to declare runtime dependencies, environment variables, and more. Leave it blank if no additional runtime configuration is required.
Mount Shared Storage	Mounts the same Volume to the `/tmp/ray` directory of each RayJob Pod for sharing checkpoints, logs, intermediate results, or cached data across Pods. Shared storage is available only for RayJob, and you must select a Volume that supports read/write access by multiple Pods.
Clean Up Shared Storage Data After Task Exit	When selected, data generated by this task in the shared storage is deleted when the task ends. If not selected, the data is retained for debugging, but you need to monitor storage usage.
Debug Mode / Cluster Retention Time	When enabled, the Ray cluster is retained for a period after the RayJob ends, allowing you to access the Head Pod for troubleshooting. The default retention time is typically 600 seconds. The actual value depends on the configuration when creating the task.
Auto Scaling	When enabled, Ray can adjust the number of Workers based on the workload. When configuring this option, pay attention to the minimum scale-down limit, maximum scale-up limit, idle timeout, and scaling mode to prevent the scaling range from exceeding task expectations or the resource budget.

After selecting RayJob, pay attention to how Ray is run and its resource usage:

A RayJob consists of one Head Pod and multiple Worker Pods.
The number of Worker Pods is determined by the number of Workers specified in the creation form. The total number of resource-consuming Pods is typically number of Workers + 1.
User code logs for a RayJob are typically viewed through the Submitter Pod.
For Ray cluster logs, view the Head Pod or Worker Pods.
The startup command for a RayJob is typically a Ray entrypoint, such as python main.py.

For more configuration recommendations on shared storage, auto scaling, Runtime Env, or Debug mode, see Advanced Training Configuration.

4. Submit Creation

Click Create to submit a task. After submission, the task enters the queue and waits to be scheduled. You can view its status in the task list or go to the training task details page to view events, logs, and monitoring. After submitting, return to the task list and search or refresh to verify that the task has been created, then record the task ID. If the creation request has succeeded but the list has not been refreshed in time, search or refresh first to avoid submitting the same task repeatedly. After opening the training task details page, first verify the actual resources, the status of each replica, and the Event. Then review the logs to confirm that training initialization is complete and that training steps are continuing to progress, and check whether outputs and checkpoints are being written to the expected paths. If the task remains queued for an extended period, scheduling fails, or startup fails, first review Event on the training task details page, then check the resource quota, image, mount path, and startup command based on the event information.

Get Started

Resources

Development & Training

Models & Inference

Operations & Governance

Best Practices

Developer Access

FAQ

Create Training

Create Training

1. Go to the creation page

2. Configure General Settings

Basic information

Environment Information

COMPUTE Resources

Advanced Options

3. Complete the configuration based on the Task Type

PyTorchJob

RayJob

4. Submit Creation

Next steps

​Create Training

​1. Go to the creation page

​2. Configure General Settings

​Basic information

​Environment Information

​COMPUTE Resources

​Advanced Options

​3. Complete the configuration based on the Task Type

​PyTorchJob

​RayJob

​4. Submit Creation

​Next steps

Create Training

1. Go to the creation page

2. Configure General Settings

Basic information

Environment Information

COMPUTE Resources

Advanced Options

3. Complete the configuration based on the Task Type

PyTorchJob

RayJob

4. Submit Creation

Next steps