> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Training Overview

> Learn about training jobs, task types, resource configurations, runtime lifecycles, and how Workloads relate to Workflows in Siverse.

Training jobs are used to submit, schedule, and run model training workloads in a GPU cluster. In the console, you can select an image, resource pool, instance specification, storage, dataset, model, and startup command to create a PyTorchJob or RayJob. The platform schedules resources, launches training containers, and provides status, event, log, monitoring, and fault recovery capabilities.

Training supports single-job submission as well as orchestration of multiple training or inference nodes through Workflows. After a task is completed, training artifacts are typically written to the mounted Volume and can then be registered with Models for inference services.

## Applicable Scenarios

* Submit PyTorch multi-node, multi-GPU training, model fine-tuning, or continued training tasks.
* Use Ray to run distributed computing tasks, and configure shared storage, Runtime Env, or auto scaling as needed.

## Task Type

| Task Type | Description | Applicable Scenarios |
| - | - | - |
| PyTorchJob | A task type for PyTorch distributed training. The platform injects distributed environment variables for multi-node training. | Multi-node, multi-GPU pre-training, fine-tuning, and continued training. |
| RayJob | A task type for Ray clusters, including Head, Worker, and Submitter Pods. | Ray distributed workloads and tasks requiring Runtime Env, shared storage, or automatic scaling. |

## Resource Types

Training tasks run on the platform's GPU clusters, with computing resources organized by region and cluster. When submitting a task, you need to request CPU/GPU resources through a resource pool and instance specification. The platform supports the following resource usage methods:

| Usage Type | Requires Advance Purchase | Resource Guarantee | Suitable Scenarios |
| - | - | - | - |
| reserved instance | Requires reserved quota. | Resources are typically guaranteed. | Critical training, long-running training, and tasks with high recovery costs. |
| pay-as-you-go instance | Requires pay-as-you-go quota. | Affected by priority and resource guarantee settings. | Routine training and phased experiments. |
| Spot instance | Does not require reserved or pay-as-you-go quota. | Unguaranteed resources. | Non-critical interruptible tasks with checkpoints. |
| Off-peak task | Not tied to purchased quota. | Unguaranteed resources and may be preempted by higher-priority tasks. | Experimental computing, batch processing, and retryable tasks. |

Both Spot and off-peak tasks are suitable for low-cost, interruptible workloads. Critical training tasks should prioritize reserved or pay-as-you-go resources, together with checkpoints, automatic fault tolerance, and monitoring capabilities.

## Relationships Between Resources, Images, and Data

* **Resource Pool and Instance Spec**: Determine the CPU/GPU computing power available to the task. When the task is submitted, the platform performs quota validation and scheduling based on the resource pool and instance specification.
* **Image**: Determines the task runtime environment, framework version, dependency libraries, and training tools.
* **Volume**: Used to store code, checkpoints, training outputs, and model files.
* **Dataset and Model**: Typically mounted in the training container as read-only inputs.
* **Startup Command**: Defines the training command to be executed after the container starts.

## Training Lifecycle

1. Prepare quotas, images, datasets, models, and Volumes. Spot and Idle Resource do not require quotas to be purchased in advance, but you must confirm that the task can be interrupted.
2. Create a training task in **Training > Workloads**, and configure the basic information, runtime environment, resources, scheduling, and advanced options.
3. The task enters the queue and waits to be scheduled to a resource pool.
4. The platform starts the training container and runs the startup command. PyTorchJob can use the platform-injected distributed training environment variables, while RayJob can use RayJob-specific capabilities.
5. On the training task details page, view the lifecycle timeline, Status details, events, logs, monitoring, and fault-tolerance details.
6. If a task is queued, fails, becomes stuck, or has abnormal resource utilization, troubleshoot the issue based on events, logs, and monitoring, then use automatic fault tolerance, Debug resubmission, or Bisect Diagnose as needed.
7. After a task succeeds, fails, is stopped, or is deleted, clone, resubmit, share, add labels, or register the training artifacts with Models as needed.

To submit training tasks through scripts or automated workflows, see the SDK documentation in the Developer Access module. The Training module primarily describes console navigation and training task operation management.

## Differences Between Workloads and Workflows

* **Workloads**: Designed for individual training jobs, suitable for directly submitting PyTorchJob or RayJob and managing the job lifecycle.
* **Workflows**: Designed for multi-step processes, suitable for orchestrating multiple SF job nodes or SF inference nodes into a DAG, and creating workflows through forms or code.

If you only need to run a training job once, use Workloads. If the training process includes multiple dependent steps or requires a DAG to organize the task execution order, you can use Workflows.

## Related Documentation

* [Create Training](/siverse/training/create-training-tasks.md)
* [Find, View, and Manage Training Tasks](/siverse/training/view-and-manage-training-tasks.md)
* [View Training Task Logs](/siverse/training/view-training-task-logs.md)
* [View Training Task Monitoring](/siverse/training/view-training-task-monitoring.md)
* [Troubleshoot Training Issues](/siverse/training/troubleshoot-training-tasks.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.