Skip to main content
Automatic fault tolerance restarts processes or resubmits jobs after qualifying training failures, reducing the need for manual supervision during long-running training. The platform detects failures and performs recovery actions; the training program must write checkpoints to a persistent Volume and correctly load the state in new attempts.

​
Understanding Automatic Fault Tolerance

​
Understanding Fault Tolerance

Training fault tolerance consists of the following capabilities. Different capabilities apply to different tasks and have different handling objectives. Fault Tolerance Policy and Task Hang Detection are linked: closing Fault Tolerance Policy also closes Task Hang Detection; enabling Task Hang Detection also enables Fault Tolerance Policy. Exit Checkpoint Notification is an independent capability and does not belong to the three fault tolerance modes below.

​
Select a Fault Tolerance Mode

After understanding each capability, choose the basic mode or combined mode based on the task type, training duration, and the faults to be handled. RayJob uses the basic mode and does not apply PyTorch-specific configurations.

​
Understanding Recovery Mechanisms and Capability Boundaries

After a qualifying fault occurs, the platform typically processes it through the following workflow:
  1. Detect faults: Monitor exceptions such as process exits, GPU or hardware errors, and communication hangs, and use logs and error messages to help determine the fault type. After relevant capabilities are enabled, the platform may also perform health checks before training officially begins.
  2. Select a recovery action: Based on the fault type and current configuration, restart the process in place, resubmit the job, avoid suspected faulty nodes, or trigger diagnostics.
  3. Continue training: If the training program has saved a checkpoint, the new process or job can continue training from the checkpoint.
Automatic fault tolerance may produce false positives or false negatives, and it cannot resolve deterministic issues such as code errors, incorrect data paths, or insufficient GPU memory. When a task is stopped manually, the retry budget is exhausted, or a fast-fail policy is triggered, the task may no longer be automatically resubmitted. When configuring Hang detection, account for normally long-running stages such as model loading, data preparation, evaluation, and communication initialization to avoid setting the detection window too short.

​
Preparing a checkpoint

Whether training progress can be resumed depends on whether the training program correctly saves and loads checkpoints. Before configuring automatic fault tolerance, complete the following preparations: The checkpoint save frequency should balance storage overhead against rerun costs. Saving only Model weights is generally suitable only for initialization or fine-tuning and may not support fully resuming training.

​
Configure Automatic Fault Tolerance

When creating a Training job, configure automatic fault tolerance in Advanced Options.

​
1. Configure the task retry budget

Enable Auto Failover and set Maximum Task Restart Attempts. This parameter controls the number of times a task is automatically resubmitted after failure; it is counted separately from in-place process restarts. When configured through the API, a task retry budget of 0 means that the task-level rerun process is not triggered; a negative value does not indicate unlimited retries.

​
2. Configure Advanced PyTorch Capabilities as Needed

For PyTorchJob, you can continue to configure Fault Tolerance Policy and Job Hang Detection. The page currently indicates that hang detection starts 30 minutes after the task is launched. The above value describes the current configuration and does not mean that different clusters and future versions will always use the same default value.

​
Integrate checkpoint exit notifications

Exit checkpoint notifications and automatic fault tolerance are independent. To notify the training program to save its state before the job exits, you must integrate the program interface and enable the corresponding configuration when creating a PyTorchJob.
  1. Implement /v1/preempt/notify in the training program to receive checkpoint save notifications.
  2. Implement /v1/preempt/status to return the save result.
  3. On the PyTorchJob creation page, enable Exit ckpt Notification and configure the program listening port. The page currently defaults to 9000, with a valid range of 9000 to 9999.
  4. Use a test job to verify notification receipt, save result reporting, and recovery logic.
After integration, the corresponding exit process waits for the save operation to complete, fail, or time out. The maximum wait time currently displayed on the page is 600 seconds. Preemption, manual stopping, or deletion may therefore be delayed; you can check the training program’s save logs to determine the progress.
Exit checkpoint notifications cannot replace periodic checkpoints. In scenarios such as node power loss or unexpected process termination, pre-exit notifications may not be completed. For long-running production training, the state required to resume training should still be written to persistent Volume storage at regular intervals.

​
View and Verify Recovery Results

​
1. View Fault Tolerance Records

If automatic fault tolerance is enabled for a task, or the task status shows a fault-tolerance-related substatus, you can open the Fault Tolerance Detail tab to view the recovery process.
  1. Click the task name to open the training task details page.
  2. Open the Fault Tolerance Detail tab.
  3. View the retry chain, fault description, and retry task status.
Fault Tolerance Detail helps determine whether the platform has triggered automatic recovery, whether the retry limit has been reached, and whether manual resubmission, Debug resubmission, or task configuration adjustments are required. If the platform has resubmitted an associated task, use the associated task ID to open the latest attempt and verify whether it has obtained resources, loaded the checkpoint, and continued writing new output or checkpoints. On the original training task details page, view the status, fault description, start and end times, and processing result of the original task and each retry task in Fault Tolerance Detail. The summary status of the original task may differ from the latest attempt. Use the associated task ID to locate the attempt that is currently running or being recovered.

​
2. Determine the Recovery Action

First confirm which type of recovery action the platform performed, then determine the next step. If a fault type is displayed in Fault Tolerance Detail, use the following table to understand the classification result. The fault type is derived by matching logs and error messages to help explain recovery records; it does not represent a confirmed root cause. For further troubleshooting, see Troubleshoot Training Issues. To identify potentially faulty nodes in multi-node training, use Bisect Diagnose.

​
3. Verify Training Progress

The platform restarting a process or resubmitting a task does not mean that training progress has been restored. Open the latest attempt and continue to verify:
  • The task has acquired resources, and the associated instances are ready.
  • The logs show that the training program loaded the expected checkpoint.
  • Training steps, loss, and business metrics can continue progressing seamlessly before and after recovery.
  • New checkpoints and output files are written successfully, with no abnormal duplicate output or data processing.
Retain the original task ID, latest attempt ID, recovery actions, checkpoint path, and key Logs to facilitate subsequent troubleshooting.

​
FAQ

First, check Fault Tolerance Detail to confirm whether the platform performed an in-place restart, whether the Maximum Task Restart Count was reached, and whether recovery was stopped due to manual termination or the fast-failure policy. An in-place restart does not create a new task record.
Not necessarily. Hang detection applies to PyTorch GPU tasks and relies on the detection and recovery components in the current cluster. Low GPU utilization alone cannot be used to determine a hang. Please assess it together with training steps, logs, stack traces, monitoring data, and normally time-consuming phases.
The new attempt still needs to acquire resources again. Check the associated task’s Event and verify the resource quota, resource pool availability, node avoidance policy, and scheduling constraints. After a fault-tolerance action avoids suspected faulty nodes, the range of available nodes may be reduced.
Check the recovery command, checkpoint mount path, file integrity, and read/write permissions. The platform can relaunch the task, but the training program must actively load the checkpoint and resume from the correct training step.
If the task encounters an OOM error, path error, or dependency failure at the same step every time, first correct the batch size, resource configuration, data path, image, or program issue. Increasing the number of retries usually cannot resolve deterministic errors.
Compare the fault categories and training phases in Fault Tolerance Detail, and check whether the Hang detection window covers normally long-running phases such as model loading, data preparation, and evaluation. A window that is too short may increase the risk of false positives.
Task-level resubmissions and in-place process restarts are counted separately, and diagnostic processes may also generate associated records. In Fault Tolerance Detail, distinguish between task reruns, in-place restarts, and diagnostic actions. Do not estimate the number of fault-tolerance actions solely based on the number of tasks in the task list.