> ## Documentation Index > Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt > Use this file to discover all available pages before exploring further. # Create Training > Create a PyTorchJob or RayJob training job on Siverse, and configure the image, mounts, resources, schedule policy, startup command, and enhanced capabilities. After you create a training task, the platform schedules resources based on the selected Region, Cluster, resource pool, and instance specification, and runs the training command in a container. This document uses console-based form creation as an example to describe the complete configuration process. Before formally submitting a long-running training job, we recommend first using a small-scale task to validate the image, mount path, startup command, and checkpoint writing logic. ## Create Training ### 1. Go to the creation page 1. Go to **Training > Workloads**. 2. On the **Personal Tasks** tab, click **Create**. ### 2. Configure General Settings First, complete the shared basic settings for **PyTorchJob** and **RayJob**. The task type determines the specific settings that must be added later. #### Basic information | Configuration Item | Description | | - | - | | **Region and Cluster** | Determines where the task runs and also affects the available resource pools, images, volumes, datasets, and models. | | **Project Group** | The project group to which the task belongs, used to categorize resources and permissions. | | **Task Name Prefix** | The prefix for the task name. The platform appends a unique suffix to the task name prefix to generate the final task name. We recommend using lowercase letters, numbers, and hyphens, with the name starting and ending with a letter or number. | | **Task Type** | Select a task type.Different task types affect subsequent resource configuration, startup methods, and dedicated settings. | | **Public within Project Group** | Controls whether the task is visible within the project group. If it is not public, it is generally visible only to the task owner. For the differences between project group visibility and personal sharing, see [Manage Training Task Sharing and Project Group Visibility](/siverse/training/share-training-tasks-and-project-group-visibility.md). | | **Labels** | Add labels to the task to facilitate categorization, filtering, and auditing. After an administrator enables label validation, you may need to select maintained labels when submitting a task. | #### Environment Information The runtime environment determines which image the training container uses, which inputs it reads, where it writes outputs, and which cmd it executes after startup. Before creating it, ensure that the image, code path, data path, and output path are ready. | Configuration Item | Description | | - | - | | **Image** | You can select a preset image, a custom image, or enter an image address directly. The image must include the frameworks and dependencies required to run the training script. RayJob requires an image that includes the Ray runtime environment. To use the full Ray Dashboard functionality, the custom image must also include the corresponding dependencies. | | **Environment Variables** | Click **Add** to add temporary environment variables for the current task. These can be used to pass paths, parameters, switches, or credentials. Note the following when configuring them: | | **Volume** | Mount a Volume to store code, checkpoints, training outputs, and model files. You must select storage and enter the **mount path in the Pod**. The same storage volume can be mounted only once at the same mount path. Writable persistent storage is recommended for the output and checkpoint directories. | | **Dataset** | Mount datasets for the training script to read input data. Datasets are typically used as read-only input, and you must specify the container mount path. | | **Model** | Mount models for fine-tuning, continued training, or evaluation, and specify the container mount path. | | **cmd** | The training command executed after the container starts. The platform runs it using `bash -c`. PyTorchJob can use the distributed training environment variables automatically injected by the platform in the command. A RayJob command is typically a Ray entrypoint, such as `python main.py`. | > [!TIP] > > Use separate directories starting with `/volume` or `/mnt` as mount paths for storage, datasets, and models to avoid overwriting default image directories such as `/opt`, `/workspace`, and `/usr`. Path conflicts may render dependencies, code, or startup commands unavailable. Container local disks have limited capacity; writing large amounts of data may slow down the system, and local disk data may also be cleared after a restart or migration. Important files, such as training code, datasets, models, checkpoints, and output results, should be written to mounted volumes. #### COMPUTE Resources Compute resources determine the type of resources used by a task, the number of replicas to launch, and the CPU/GPU resources requested by each replica. We recommend completing the configuration in the following order: Resource Type -> Replicas -> Instance Spec -> Quota and Scheduling. **1. Select Resource Type** Different resource types affect the available instances, quota validation, resource guarantees, and interruption risks. Select a resource type as needed. | Resource type | Suitable scenarios | | - | - | | **Reserved** | Run training tasks using purchased or allocated reserved quota. | | **Ondemand** | Temporarily start training tasks without reserving quota in advance. | | **Spot** | Cost-sensitive, interruptible, and retryable training tasks. | | **Idle Resource** | Tasks that can tolerate unstable run durations and whose training programs support checkpointing or resubmission. | **2. Configure Replicas** * **PyTorchJob** PyTorchJob has a fixed master node count of `1`, while the number of worker nodes can be set to `0`. The master node and worker nodes use the same specifications. For distributed training, the number of worker nodes affects the number of replicas that must be scheduled simultaneously. For the total number of task replicas, PyTorchJob typically calculates the number of replicas that must be scheduled simultaneously as `1 + number of worker nodes`. * **RayJob** RayJob configures the Head and Worker separately. The number of Heads and their resources are specified in the Ray form, while the number of Workers determines the scale of worker nodes. For the total number of task replicas, RayJob resources are calculated separately for the Head and Worker. **3. Select Instance Spec** | Configuration Item | Description | | - | - | | **Instance Type** | Select **GPU** or **CPU**, then select a specific instance specification from the instance list. Specification displays the resources included in that specification, such as GPU, vCPU, and memory. | | **Count Per Node** | Configure the number of selected instance specifications used by each replica. Resource requirements for a single replica = resources in one specification in Specification × **Count Per Node**. | | **Resource Priority** | When multiple instance specifications are selected, the platform attempts scheduling in descending priority order, giving preference to instances listed first. Select only specifications compatible with the training code to prevent the task from being scheduled to incompatible GPU models or CPU/memory combinations. | Resource quantities are calculated based on the role configuration. The specification values on the page are for displaying the resources of a single selected instance specification only. Actual resource quantities depend on the selected specification and number of replicas. | Task Type | Compute Method | | - | - | | **PyTorchJob** | Total replicas = `1 + number of Workers`; total specification units = `total replicas × Count Per Node`; total GPUs = `total specification units × number of GPUs per specification`. CPU and memory are also calculated based on the total specification units. | | **RayJob** | Resource requirements for the Head and Worker are calculated separately. Quotas are verified collectively for the same specifications and separately for different specifications. | **4. Confirm Quota and Scheduling** Before submitting, do not consider only whether an individual instance specification is available; also confirm that the entire task can be scheduled simultaneously. | Item to Check | How to Assess | | - | - | | **Remaining Quota** | The quota that the current account can continue to use for the selected resource type, resource pool, and instance specification. When creating a task, verify against the resource requirements of the entire task, rather than a single replica only. | | **Pool Available** | A reference value for the capacity currently available from the resource pool. This value changes with other tasks and the status of the resource pool and must not be treated as a resource reservation. | | **Idle Resource Quantity / Idle Resource Fragmentation** | When using the idle-resource supplementation capability of Reserved instances or directly selecting Idle Resource, check the displayed idle-resource quantity, stability level, and fragmentation. Fragmentation information is for availability reference only; distributed training must still meet overall scheduling requirements. | | **Guarantee** | Based on the resource type and page capabilities, choose whether to enable a more stable guarantee policy. For critical training, long-running training, or tasks with high recovery costs, configurations that support Guarantee are recommended. | > [!TIP] > > Distributed training uses gang scheduling. The replicas required by a job cannot be assessed solely by summing total resources; it is also necessary to ensure that each replica can be placed on an available node and that all replicas can start simultaneously. If resources are insufficient, the entire job enters the queue and is automatically scheduled once the resource requirements are met, without manual retries. Priority affects task queuing and scheduling order and may also affect outcomes in resource contention. For more details about low-cost resources, Guarantee, node scheduling policies, and preemption, see [Training Advanced Config](/siverse/training/advanced-training-task-configuration.md). #### Advanced Options | Configuration Item | Description | | - | - | | **Resource Precheck** | Supported only by **PyTorchJob**. When enabled, the platform runs a preset resource precheck before training starts. Note the following when configuring it: | | **Fault Tolerance and Diagnostics** | When enabled, the platform can retry or recover tasks that encounter exceptions according to the configured policy.Automatic retries cannot resolve issues that require changes to the configuration or scripts, such as code issues, data path issues, or insufficient GPU memory. | | **Exit Checkpoint Notification** | Supported only by PyTorchJob. When enabled, the Pod notifies the training process to save a checkpoint before exiting. Resources are released only after the checkpoint is saved, with a maximum wait time of 600 seconds. Notifications are sent for preemption reclamation, manual stops, and task deletion, which can slow down task exit. | | **Metrics Monitor** | When enabled, collects custom metrics exposed by the training code.For detailed examples, see [Integrate Custom Metrics into Training](/siverse/training/collect-training-task-custom-metrics.md). | | **Image Pull Tolerance** | Configure the maximum number of retries for failed image pulls to handle temporary pull fluctuations. The default value is subject to the current page display. A value of `0` indicates that the task is considered failed upon the first failure; `-1` disables image pull circuit breaking for this task; a positive integer uses a custom threshold for this task. | | **Passwordless Login Between Pods** | When this setting is enabled, passwordless SSH login between multiple Pods is automatically enabled, and node information is generated and stored in the `/root/hostfile` and `/root/hostfile.hostname` files, respectively. This feature is available in common operating systems, such as Ubuntu, and common images, such as NGC images. However, it may not be available in certain special environments. If you encounter any issues, contact technical support. | | **Alarm Subscription** | To configure alarms when creating a task, click **Add Alarm Subscription**. Refer to the Alarm Subscription page for the specific fields. | ### 3. Complete the configuration based on the Task Type #### PyTorchJob PyTorchJob is suitable for running single-node or distributed training jobs based on PyTorch. After completing the general configuration, check carefully whether **cmd** can correctly start the training script. For PyTorchJob multi-node multi-GPU training, you can use the platform-injected distributed training environment variables in the startup command: | Environment Variable | Description | | - | - | | `MASTER_ADDR` | Address of the master node. | | `MASTER_PORT` | Port of the master node. | | `WORLD_SIZE` | Total number of machines participating in training. | | `RANK` | ID of the current node. | The platform also injects task context-related TaskEnv, such as `X_USER_NAME`, `X_ORG_NAME`, `X_TENANT_ID`, `X_REGION`, `X_CLUSTER`, and `TASK_UUID`. If the training script needs to record the task source, region, cluster, or unique task identifier, it can read these variables. Example: ```bash theme={null} torchrun \ --master_addr "$MASTER_ADDR" \ --master_port "$MASTER_PORT" \ --nnodes "$WORLD_SIZE" \ --nproc_per_node 8 \ --node_rank "$RANK" \ train.py ``` Adjust `--nproc_per_node` based on the actual number of GPUs per node, and replace `train.py` with the actual training script. #### RayJob RayJob is suitable for running Ray-based distributed training, scheduling, or parallel computing tasks. After completing the general configuration, pay particular attention to the Head / Worker, Runtime Env, shared storage, and autoscaling settings. | Configuration Item | Description | | - | - | | **Head Instance Specification** | Configures the instance specification used by the Head node. If not configured separately, it typically uses the same specification as the Worker nodes. | | **Runtime Env** | Enter the Ray Runtime Env in a YAML format supported by Ray. It can be used to declare runtime dependencies, environment variables, and more. Leave it blank if no additional runtime configuration is required. | | **Mount Shared Storage** | Mounts the same Volume to the `/tmp/ray` directory of each RayJob Pod for sharing checkpoints, logs, intermediate results, or cached data across Pods. Shared storage is available only for RayJob, and you must select a Volume that supports read/write access by multiple Pods. | | **Clean Up Shared Storage Data After Task Exit** | When selected, data generated by this task in the shared storage is deleted when the task ends. If not selected, the data is retained for debugging, but you need to monitor storage usage. | | **Debug Mode / Cluster Retention Time** | When enabled, the Ray cluster is retained for a period after the RayJob ends, allowing you to access the Head Pod for troubleshooting. The default retention time is typically 600 seconds. The actual value depends on the configuration when creating the task. | | **Auto Scaling** | When enabled, Ray can adjust the number of Workers based on the workload. When configuring this option, pay attention to the minimum scale-down limit, maximum scale-up limit, idle timeout, and scaling mode to prevent the scaling range from exceeding task expectations or the resource budget. | After selecting RayJob, pay attention to how Ray is run and its resource usage: * A RayJob consists of one Head Pod and multiple Worker Pods. * The number of Worker Pods is determined by the number of Workers specified in the creation form. The total number of resource-consuming Pods is typically `number of Workers + 1`. * User code logs for a RayJob are typically viewed through the Submitter Pod. * For Ray cluster logs, view the Head Pod or Worker Pods. * The startup command for a RayJob is typically a Ray entrypoint, such as `python main.py`. For more configuration recommendations on shared storage, auto scaling, Runtime Env, or Debug mode, see [Advanced Training Configuration](/siverse/training/advanced-training-task-configuration.md). ### 4. Submit Creation Click **Create** to submit a task. After submission, the task enters the queue and waits to be scheduled. You can view its status in the task list or go to the training task details page to view events, logs, and monitoring. After submitting, return to the task list and search or refresh to verify that the task has been created, then record the task ID. If the creation request has succeeded but the list has not been refreshed in time, search or refresh first to avoid submitting the same task repeatedly. After opening the training task details page, first verify the actual resources, the status of each replica, and the **Event**. Then review the logs to confirm that training initialization is complete and that training steps are continuing to progress, and check whether outputs and checkpoints are being written to the expected paths. If the task remains queued for an extended period, scheduling fails, or startup fails, first review **Event** on the training task details page, then check the resource quota, image, mount path, and startup command based on the event information. ## Next steps * [Find, View, and Manage Training Tasks](/siverse/training/view-and-manage-training-tasks.md) * [View Training Task Logs](/siverse/training/view-training-task-logs.md) * [View Training Task Monitoring](/siverse/training/view-training-task-monitoring.md) * [Integrate Custom Metrics into Training](/siverse/training/collect-training-task-custom-metrics.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.