> ## Documentation Index > Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt > Use this file to discover all available pages before exploring further. # Recovering Training with Automatic Fault Tolerance > Learn about the scope of automatic fault tolerance for Training, configure task retries, PyTorch fault tolerance policies, and checkpoints, and verify whether training resumes. Automatic fault tolerance restarts processes or resubmits jobs after qualifying training failures, reducing the need for manual supervision during long-running training. The platform detects failures and performs recovery actions; the training program must write checkpoints to a persistent Volume and correctly load the state in new attempts. ## Understanding Automatic Fault Tolerance ### Understanding Fault Tolerance Training fault tolerance consists of the following capabilities. Different capabilities apply to different tasks and have different handling objectives. | Capability | Function | Scope | | - | - | - | | **Automatic fault tolerance** | Automatically resubmits a task after it fails, up to the configured number of retries. Each resubmission requires the task to be queued again and resources to be requested again. | Applies to eligible PyTorchJob and RayJob workloads. | | **Fault tolerance policy** | Determines the failure type based on exit codes and error messages, then selects handling methods such as in-place restart, resubmission, or fast failure. | Applies to PyTorchJob workloads. | | **Task hang detection** | Proactively detects cases where processes have not exited but training has made no progress for an extended period, and attempts recovery according to the fault tolerance policy. | Applies to PyTorch GPU tasks and depends on cluster detection and recovery components. | | **Exit checkpoint notification** | Notifies the training program to save its state before the task exits. | Applies to PyTorchJob; the training program must implement the notification interface. | **Fault Tolerance Policy** and **Task Hang Detection** are linked: closing **Fault Tolerance Policy** also closes **Task Hang Detection**; enabling **Task Hang Detection** also enables **Fault Tolerance Policy**. **Exit Checkpoint Notification** is an independent capability and does not belong to the three fault tolerance modes below. ### Select a Fault Tolerance Mode After understanding each capability, choose the basic mode or combined mode based on the task type, training duration, and the faults to be handled. | Mode | Enabled Capabilities | Recommendation | | - | - | - | | Basic mode | **Automatic Fault Tolerance** | Automatically resubmits a task after it fails, but does not further select recovery actions based on failure type and cannot detect cases where a process has not exited but training has made no progress for an extended period. Suitable for PyTorchJob, RayJob, or tasks that require only basic retry capabilities. | | Advanced mode | **Automatic Fault Tolerance** + **Fault Tolerance Policy** | Allows recovery actions to be selected based on the failure type. If the communication library or training framework detects a hang and exits with an error, the platform can continue processing; however, this mode may still be unable to detect processes that are completely stuck and do not exit. Suitable for long-running PyTorch training. | | Full mode | **Automatic Fault Tolerance** + **Fault Tolerance Policy** + **Task Hang Detection** | Proactively detects tasks that have made no progress for an extended period. Suitable for PyTorch GPU tasks with a high risk of communication hangs, where reduced manual monitoring costs are needed and the risk of detection false positives is acceptable. | RayJob uses the basic mode and does not apply PyTorch-specific configurations. ### Understanding Recovery Mechanisms and Capability Boundaries After a qualifying fault occurs, the platform typically processes it through the following workflow: 1. **Detect faults**: Monitor exceptions such as process exits, GPU or hardware errors, and communication hangs, and use logs and error messages to help determine the fault type. After relevant capabilities are enabled, the platform may also perform health checks before training officially begins. 2. **Select a recovery action**: Based on the fault type and current configuration, restart the process in place, resubmit the job, avoid suspected faulty nodes, or trigger diagnostics. 3. **Continue training**: If the training program has saved a checkpoint, the new process or job can continue training from the checkpoint. Automatic fault tolerance may produce false positives or false negatives, and it cannot resolve deterministic issues such as code errors, incorrect data paths, or insufficient GPU memory. When a task is stopped manually, the retry budget is exhausted, or a fast-fail policy is triggered, the task may no longer be automatically resubmitted. When configuring Hang detection, account for normally long-running stages such as model loading, data preparation, evaluation, and communication initialization to avoid setting the detection window too short. ## Preparing a checkpoint Whether training progress can be resumed depends on whether the training program correctly saves and loads checkpoints. Before configuring automatic fault tolerance, complete the following preparations: | Preparation Item | Description | | - | - | | Fix experiment inputs | Fix the training code, image, and data versions to prevent retried tasks from using different experiment configurations. | | Use a persistent directory | Write checkpoints to an authorized persistent Volume to ensure they remain accessible after rescheduling. | | Save the complete state | In addition to model weights, save the optimizer, learning rate scheduler, training steps, and other states required to resume training, as required by the training framework. | | Configure the recovery entry point | Load an existing checkpoint in the startup command or training program, and continue from the correct training step. | | Avoid partial writes | Use secure saving methods supported by the training framework, and use separate output directories for different experiments to avoid loading unfinished files. | | Verify saving and recovery | First use a small-scale task to verify the checkpoint saving, loading, and training resumption logic before running formal long-running training. | The checkpoint save frequency should balance storage overhead against rerun costs. Saving only Model weights is generally suitable only for initialization or fine-tuning and may not support fully resuming training. ## Configure Automatic Fault Tolerance When creating a Training job, configure automatic fault tolerance in **Advanced Options**. ### 1. Configure the task retry budget Enable **Auto Failover** and set **Maximum Task Restart Attempts**. This parameter controls the number of times a task is automatically resubmitted after failure; it is counted separately from in-place process restarts. | Parameter | Description | | - | - | | **Maximum Task Restart Attempts** | For example, if set to `2`, the task can be rerun at the task level up to two additional times beyond the initial run. | When configured through the API, a task retry budget of `0` means that the task-level rerun process is not triggered; a negative value does not indicate unlimited retries. ### 2. Configure Advanced PyTorch Capabilities as Needed For PyTorchJob, you can continue to configure **Fault Tolerance Policy** and **Job Hang Detection**. | Parameter | Description | | - | - | | **Policy Type** | Select a recovery action based on clues such as exit codes and error messages. Choose from the policies supported on the current page. | | **Hang Detection Interval** | The sampling interval between consecutive detections. The page is currently prepopulated with `240` seconds. | | **Hang Determination Duration** | The time window during which the stack is observed to remain unchanged. The page is currently prepopulated with `600` seconds. | | **Maximum Restart Count for Communication Hang** | The maximum number of restarts for the training process on the original node. The page is currently prepopulated with `1`; this is counted separately from the task-level retry budget. | The page currently indicates that hang detection starts 30 minutes after the task is launched. The above value describes the current configuration and does not mean that different clusters and future versions will always use the same default value. ## Integrate checkpoint exit notifications Exit checkpoint notifications and automatic fault tolerance are independent. To notify the training program to save its state before the job exits, you must integrate the program interface and enable the corresponding configuration when creating a PyTorchJob. 1. Implement `/v1/preempt/notify` in the training program to receive checkpoint save notifications. 2. Implement `/v1/preempt/status` to return the save result. 3. On the PyTorchJob creation page, enable **Exit ckpt Notification** and configure the program listening port. The page currently defaults to `9000`, with a valid range of `9000` to `9999`. 4. Use a test job to verify notification receipt, save result reporting, and recovery logic. After integration, the corresponding exit process waits for the save operation to complete, fail, or time out. The maximum wait time currently displayed on the page is `600` seconds. Preemption, manual stopping, or deletion may therefore be delayed; you can check the training program's save logs to determine the progress. > [!WARNING] > > Exit checkpoint notifications cannot replace periodic checkpoints. In scenarios such as node power loss or unexpected process termination, pre-exit notifications may not be completed. For long-running production training, the state required to resume training should still be written to persistent Volume storage at regular intervals. ## View and Verify Recovery Results ### 1. View Fault Tolerance Records If automatic fault tolerance is enabled for a task, or the task status shows a fault-tolerance-related substatus, you can open the **Fault Tolerance Detail** tab to view the recovery process. 1. Click the task name to open the training task details page. 2. Open the **Fault Tolerance Detail** tab. 3. View the retry chain, fault description, and retry task status. **Fault Tolerance Detail** helps determine whether the platform has triggered automatic recovery, whether the retry limit has been reached, and whether manual resubmission, Debug resubmission, or task configuration adjustments are required. If the platform has resubmitted an associated task, use the associated task ID to open the latest attempt and verify whether it has obtained resources, loaded the checkpoint, and continued writing new output or checkpoints. On the original training task details page, view the status, fault description, start and end times, and processing result of the original task and each retry task in **Fault Tolerance Detail**. The summary status of the original task may differ from the latest attempt. Use the associated task ID to locate the attempt that is currently running or being recovered. ### 2. Determine the Recovery Action First confirm which type of recovery action the platform performed, then determine the next step. | Recovery Action | Where to Check | Key Points to Verify | | - | - | - | | Restart in place | Logs and events for the current task. | Whether the process has restarted and whether the expected checkpoint has been loaded. | | Resubmit | The associated retry task in **Fault Tolerance Detail**. | Whether the new task has obtained resources and continued from the checkpoint. | | Avoid suspected faulty nodes or perform diagnostics | **Fault Tolerance Detail**, events, and Bisect Diagnose records. | Whether the suspected nodes have been excluded and whether new queuing occurs after the node scope is narrowed. | | Fast failure or budget exhaustion | **Fault Tolerance Detail** and the failure description. | Whether the code, image, data path, or resource configuration must be fixed before manually submitting the task again. | If a fault type is displayed in **Fault Tolerance Detail**, use the following table to understand the classification result. The fault type is derived by matching logs and error messages to help explain recovery records; it does not represent a confirmed root cause. | Failure Type | Common Meaning | | - | - | | `HW_NODE` | Node-related exceptions involving the GPU, hardware, drivers, ECC, Xid, or segmentation faults. | | `NET_FABRIC` | NCCL, InfiniBand, RDMA, RoCE, or network communication exceptions. | | `OOM_HOST` | Insufficient host memory or container termination by the system. | | `OOM_GPU` | Insufficient GPU memory. | | `COMM_HANG` | Communication hangs or no progress in multi-node training for an extended period. | | `APP_ERROR` | Training code exception. | | `ENV_ERROR` | Image, dependency, environment, or external access exception. | | `UNKNOWN` | No known type matched. | For further troubleshooting, see [Troubleshoot Training Issues](/siverse/training/troubleshoot-training-tasks.md). To identify potentially faulty nodes in multi-node training, use [Bisect Diagnose](/siverse/training/diagnose-training-failures-with-bisect.md). ### 3. Verify Training Progress The platform restarting a process or resubmitting a task does not mean that training progress has been restored. Open the latest attempt and continue to verify: * The task has acquired resources, and the associated instances are ready. * The logs show that the training program loaded the expected checkpoint. * Training steps, loss, and business metrics can continue progressing seamlessly before and after recovery. * New checkpoints and output files are written successfully, with no abnormal duplicate output or data processing. Retain the original task ID, latest attempt ID, recovery actions, checkpoint path, and key Logs to facilitate subsequent troubleshooting. ## FAQ > **Why was no new task created after auto fault tolerance was enabled?** > > First, check **Fault Tolerance Detail** to confirm whether the platform performed an in-place restart, whether the **Maximum Task Restart Count** was reached, and whether recovery was stopped due to manual termination or the fast-failure policy. An in-place restart does not create a new task record. > **The task is running but training has stalled. Will auto fault tolerance always handle it?** > > Not necessarily. Hang detection applies to PyTorch GPU tasks and relies on the detection and recovery components in the current cluster. Low GPU utilization alone cannot be used to determine a hang. Please assess it together with training steps, logs, stack traces, monitoring data, and normally time-consuming phases. > **Why is a resubmitted task still queued?** > > The new attempt still needs to acquire resources again. Check the associated task's **Event** and verify the resource quota, resource pool availability, node avoidance policy, and scheduling constraints. After a fault-tolerance action avoids suspected faulty nodes, the range of available nodes may be reduced. > **Why does every recovery start from step 0?** > > Check the recovery command, checkpoint mount path, file integrity, and read/write permissions. The platform can relaunch the task, but the training program must actively load the checkpoint and resume from the correct training step. > **Should automatic retries continue when the same error occurs repeatedly?** > > If the task encounters an OOM error, path error, or dependency failure at the same step every time, first correct the batch size, resource configuration, data path, image, or program issue. Increasing the number of retries usually cannot resolve deterministic errors. > **Why does the task restart too frequently?** > > Compare the fault categories and training phases in **Fault Tolerance Detail**, and check whether the Hang detection window covers normally long-running phases such as model loading, data preparation, and evaluation. A window that is too short may increase the risk of false positives. > **Why does the number of fault-tolerance actions differ from the number of tasks?** > > Task-level resubmissions and in-place process restarts are counted separately, and diagnostic processes may also generate associated records. In **Fault Tolerance Detail**, distinguish between task reruns, in-place restarts, and diagnostic actions. Do not estimate the number of fault-tolerance actions solely based on the number of tasks in the task list. ## Related Documentation * [Training Advanced Config](/siverse/training/advanced-training-task-configuration.md) * [Troubleshoot Training Issues](/siverse/training/troubleshoot-training-tasks.md) * [Use Bisect Diagnose to Diagnose Training Failure Nodes](/siverse/training/diagnose-training-failures-with-bisect.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.