Skip to main content
Training jobs are used to submit, schedule, and run model training workloads in a GPU cluster. In the console, you can select an image, resource pool, instance specification, storage, dataset, model, and startup command to create a PyTorchJob or RayJob. The platform schedules resources, launches training containers, and provides status, event, log, monitoring, and fault recovery capabilities. Training supports single-job submission as well as orchestration of multiple training or inference nodes through Workflows. After a task is completed, training artifacts are typically written to the mounted Volume and can then be registered with Models for inference services.

​
Applicable Scenarios

  • Submit PyTorch multi-node, multi-GPU training, model fine-tuning, or continued training tasks.
  • Use Ray to run distributed computing tasks, and configure shared storage, Runtime Env, or auto scaling as needed.

​
Task Type

​
Resource Types

Training tasks run on the platform’s GPU clusters, with computing resources organized by region and cluster. When submitting a task, you need to request CPU/GPU resources through a resource pool and instance specification. The platform supports the following resource usage methods: Both Spot and off-peak tasks are suitable for low-cost, interruptible workloads. Critical training tasks should prioritize reserved or pay-as-you-go resources, together with checkpoints, automatic fault tolerance, and monitoring capabilities.

​
Relationships Between Resources, Images, and Data

  • Resource Pool and Instance Spec: Determine the CPU/GPU computing power available to the task. When the task is submitted, the platform performs quota validation and scheduling based on the resource pool and instance specification.
  • Image: Determines the task runtime environment, framework version, dependency libraries, and training tools.
  • Volume: Used to store code, checkpoints, training outputs, and model files.
  • Dataset and Model: Typically mounted in the training container as read-only inputs.
  • Startup Command: Defines the training command to be executed after the container starts.

​
Training Lifecycle

  1. Prepare quotas, images, datasets, models, and Volumes. Spot and Idle Resource do not require quotas to be purchased in advance, but you must confirm that the task can be interrupted.
  2. Create a training task in Training > Workloads, and configure the basic information, runtime environment, resources, scheduling, and advanced options.
  3. The task enters the queue and waits to be scheduled to a resource pool.
  4. The platform starts the training container and runs the startup command. PyTorchJob can use the platform-injected distributed training environment variables, while RayJob can use RayJob-specific capabilities.
  5. On the training task details page, view the lifecycle timeline, Status details, events, logs, monitoring, and fault-tolerance details.
  6. If a task is queued, fails, becomes stuck, or has abnormal resource utilization, troubleshoot the issue based on events, logs, and monitoring, then use automatic fault tolerance, Debug resubmission, or Bisect Diagnose as needed.
  7. After a task succeeds, fails, is stopped, or is deleted, clone, resubmit, share, add labels, or register the training artifacts with Models as needed.
To submit training tasks through scripts or automated workflows, see the SDK documentation in the Developer Access module. The Training module primarily describes console navigation and training task operation management.

​
Differences Between Workloads and Workflows

  • Workloads: Designed for individual training jobs, suitable for directly submitting PyTorchJob or RayJob and managing the job lifecycle.
  • Workflows: Designed for multi-step processes, suitable for orchestrating multiple SF job nodes or SF inference nodes into a DAG, and creating workflows through forms or code.
If you only need to run a training job once, use Workloads. If the training process includes multiple dependent steps or requires a DAG to organize the task execution order, you can use Workflows.