> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshoot Training Issues

> Troubleshoot Training issues in Siverse, including queuing, startup failures, runtime failures, hangs, preemption, and abnormal resource utilization.

Training jobs may fail because of insufficient resources, image issues, incorrect startup commands, invalid data paths, node exceptions, communication hangs, or resource preemption. When troubleshooting, first check the status, events, logs, and monitoring information on the training task details page to determine whether the issue occurred during scheduling, environment preparation, program execution, or recovery. Then choose Debug resubmission, automatic fault tolerance, or Bisect Diagnose as appropriate.

## Troubleshooting Task Issues

1. Go to **Training > Workloads** and locate the target task.
2. Click the task name to open the training task details page.
3. Review the lifecycle timeline to determine whether the task is stuck in the submission, queuing, environment preparation, running, or completion stage.
4. Review **Status** to verify the overall task status, Pod status, node, exit code, and status description.
5. Review **Event** to check for insufficient resources, image pull failures, Volume mount failures, scheduling blocks, or fault-tolerance-related events.
6. Review **Log** to identify training script errors, missing dependencies, incorrect data paths, or runtime exceptions.
7. Review **Monitoring** to determine whether there are bottlenecks or abnormal fluctuations in GPU, GPU memory, CPU, memory, or network.

## Handle Based on Symptoms

| Issue | Possible Cause | Recommended Action |
| - | - | - |
| Queued for a long time | Insufficient resource pool quota, low priority, unmatched resource guarantee requirements, node scheduling policies restricting available nodes, or inability to place a distributed task as a whole. | Check events, resource pool quota, single-replica specifications, and resource guarantees. If events indicate a node policy, check [Node Scheduling Policies](/siverse/configuration/manage-node-scheduling-policies.md). If necessary, adjust the number of replicas, instance specification, candidate resources, or resource guarantees. |
| The message `quota check failed due to node blacklist: XXX` is displayed | The task matches a node-blocking policy, resulting in insufficient schedulable nodes. | Check whether any [Node Scheduling Policies](/siverse/configuration/manage-node-scheduling-policies.md) apply to the current task, then edit or delete the policies as needed. |
| Startup failure | The image is unavailable, the command is incorrect, the mount path overwrites a system directory, dependencies are missing, or Volume/dataset mounting fails. | Check events and logs. Verify the image address, version, repository authorization, startup command, mount path, environment variables, and input data path. |
| Runtime failure | Training code errors, an incorrect data path, or insufficient memory or GPU memory. | Check user logs and monitoring, then adjust the code, data path, batch size, or instance specification. |
| Task preempted | Spot or Idle Resource is used, or resource guarantees are not enabled. | Resume from a checkpoint, or switch to a more stable resource type and resource guarantee configuration. If the status information contains `was preempted by another task`, the task has typically been preempted by another task. |
| Multi-node training hangs | Communication is stalled, a node is faulty, the network is abnormal, or a node is slow. | Check monitoring and logs. To enable automatic platform recovery, use [Auto Fault Tolerance](./recover-training-tasks-with-auto-fault-tolerance). To identify the faulty node, use [Bisect Diagnose](/siverse/training/diagnose-training-failures-with-bisect.md). |
| The task is running but training makes no progress | Data loading, distributed initialization, communication waits, a stalled training program, or the task has entered an abnormal stage but has not yet exited. | Compare the status and logs of each Pod, check GPU, CPU, and network metrics, and confirm whether training steps are progressing. |
| Training needs to continue after task interruption | The task was preempted, a node failed, the task was stopped manually, or the fault-tolerance limit was reached. | Check the actual resource pool, preemption or fault events, fault-tolerance records, and related tasks. After locating the latest attempt, confirm that the checkpoint was loaded successfully. |

## Key Troubleshooting Dir

### Long Queue Times

Even when total resources appear sufficient, the placement requirements for individual replicas and the overall scheduling requirements for distributed jobs must still be met. We recommend troubleshooting in the following order:

1. Open the task details and check **Status** and **Event** to determine whether the issue is related to quota, the resource pool, node policies, or the instance specification. If an event points to a node policy, check the Schedule Type, Scope, Effect Mode, Workload Type, Node Name, and expiration time in [Node Scheduling Policy](/siverse/configuration/manage-node-scheduling-policies.md).
2. Verify your personal quota, resource pool availability, and resource pool status.
3. Check whether a single replica can fit on an available node. For distributed tasks, all replicas must simultaneously meet the startup requirements.
4. If the task has network group requirements, ensure that the nodes are in compatible groups. RoCE POD on the page or in events refers to a network group, not a Kubernetes Pod.

For more troubleshooting methods for resource queuing, see [Troubleshoot insufficient task resources or long queue times](/siverse/resources/troubleshoot-insufficient-resources-or-queued-tasks.md).

### Running but Training Makes No Progress

Resource allocation does not mean that training computation has started. First, check whether each Pod is ready in **Status**, then use **Log** and **Monitoring** to identify the stage at which the issue occurs:

| Stage | Key Checks |
| - | - |
| Image pull or container startup | Check **Event** and **System Log** to determine whether there are issues with the image, dependencies, or startup process. |
| Storage or data preparation | Check mount events, user logs, and data read paths to verify whether the Volume, Dataset, or Model is accessible. |
| Distributed initialization | Compare logs across Pods to verify that environment variables such as `MASTER_ADDR`, `MASTER_PORT`, `WORLD_SIZE`, and `RANK`, as well as communication initialization, are configured correctly. |
| Training program stall | Check training steps, loss, GPU utilization, CPU, memory, and network metrics to determine whether the program has entered a prolonged state without progress. |

### Failed to Pull Image or Access File

| Issue | Items to Check |
| - | - |
| Image pull failed | Image address, image version, repository authorization, network connectivity, and pull error messages. |
| File not found | The selected source directory, container mount path, paths in the startup command, and the current working directory. |
| Unable to write | `r` / `rw` permissions for the Volume or directory, directory-level permissions, volume capacity, and directory quotas. |

Resubmit the task after fixing the deterministic configuration errors. Simply increasing the number of retries usually cannot resolve an incorrect image address, an invalid path, insufficient permissions, or insufficient capacity.

### Continue Training After an Interruption

After a task is preempted, a node fails, it is manually stopped, or it reaches the fault-tolerance limit, first check whether an automatic recovery action is already in progress, then decide whether to resubmit it manually.

1. View the actual resource pool and preemption, failure, or stop events to identify the cause of the interruption.
2. View **Fault Tolerance Detail** and associated tasks to check whether an in-place restart or automatic resubmission has already occurred.
3. Track the latest attempt and check the checkpoint loading logs, training steps after recovery, and new output.
4. Manual termination, budget exhaustion, or a fail-fast policy may halt automatic recovery. Verify the current attempt before manually resubmitting the task to avoid running two copies simultaneously.

## Resubmit Using Debug

Debug resubmission retains the original task’s resource requirements but does not immediately execute the original training command. Instead, it provides a live environment for troubleshooting.

Locate the target task in the task list, click **Resubmit**, and select Debug mode in the resubmission dialog. Before resubmitting, ensure that any checkpoints, logs, or model outputs to be retained have been written to a persistent Volume.

| Applicable Scenario | Usage Recommendation |
| - | - |
| The reason for task failure is unclear | Use Debug resubmission to enter the container and check the environment, paths, dependencies, and startup command. |
| You want to reuse the original task scheduling location as much as possible | Debug resubmission usually retains the original resource requirements, but starts a new task with a troubleshooting command rather than directly executing the original training command. |
| A RayJob requires investigation of the cluster state | After Debug is enabled for a RayJob, the Ray cluster can be retained for a period of time—typically 600 seconds by default—allowing you to enter the Head Pod and view the cluster status. |

For a RayJob, if the Submitter Pod has completed but the Head/Worker Pods are still retained, the job may be in the `DebugHolding` state. You can access the Head Pod to view the Ray cluster status.

## Related Documentation

* [Find, View, and Manage Training Tasks](/siverse/training/view-and-manage-training-tasks.md)
* [View Training Task Logs](/siverse/training/view-training-task-logs.md)
* [View Training Task Monitoring](/siverse/training/view-training-task-monitoring.md)
* [Use Automatic Fault Tolerance to Recover Training Tasks](./recover-training-tasks-with-auto-fault-tolerance)
* [Use Bisect Diagnose to Diagnose Training Failure Nodes](/siverse/training/diagnose-training-failures-with-bisect.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.