Troubleshoot Training Issues

Training jobs may fail because of insufficient resources, image issues, incorrect startup commands, invalid data paths, node exceptions, communication hangs, or resource preemption. When troubleshooting, first check the status, events, logs, and monitoring information on the training task details page to determine whether the issue occurred during scheduling, environment preparation, program execution, or recovery. Then choose Debug resubmission, automatic fault tolerance, or Bisect Diagnose as appropriate.

Troubleshooting Task Issues

Go to Training > Workloads and locate the target task.
Click the task name to open the training task details page.
Review the lifecycle timeline to determine whether the task is stuck in the submission, queuing, environment preparation, running, or completion stage.
Review Status to verify the overall task status, Pod status, node, exit code, and status description.
Review Event to check for insufficient resources, image pull failures, Volume mount failures, scheduling blocks, or fault-tolerance-related events.
Review Log to identify training script errors, missing dependencies, incorrect data paths, or runtime exceptions.
Review Monitoring to determine whether there are bottlenecks or abnormal fluctuations in GPU, GPU memory, CPU, memory, or network.

Handle Based on Symptoms

Issue	Possible Cause	Recommended Action
Queued for a long time	Insufficient resource pool quota, low priority, unmatched resource guarantee requirements, node scheduling policies restricting available nodes, or inability to place a distributed task as a whole.	Check events, resource pool quota, single-replica specifications, and resource guarantees. If events indicate a node policy, check Node Scheduling Policies. If necessary, adjust the number of replicas, instance specification, candidate resources, or resource guarantees.
The message `quota check failed due to node blacklist: XXX` is displayed	The task matches a node-blocking policy, resulting in insufficient schedulable nodes.	Check whether any Node Scheduling Policies apply to the current task, then edit or delete the policies as needed.
Startup failure	The image is unavailable, the command is incorrect, the mount path overwrites a system directory, dependencies are missing, or Volume/dataset mounting fails.	Check events and logs. Verify the image address, version, repository authorization, startup command, mount path, environment variables, and input data path.
Runtime failure	Training code errors, an incorrect data path, or insufficient memory or GPU memory.	Check user logs and monitoring, then adjust the code, data path, batch size, or instance specification.
Task preempted	Spot or Idle Resource is used, or resource guarantees are not enabled.	Resume from a checkpoint, or switch to a more stable resource type and resource guarantee configuration. If the status information contains `was preempted by another task`, the task has typically been preempted by another task.
Multi-node training hangs	Communication is stalled, a node is faulty, the network is abnormal, or a node is slow.	Check monitoring and logs. To enable automatic platform recovery, use Auto Fault Tolerance. To identify the faulty node, use Bisect Diagnose.
The task is running but training makes no progress	Data loading, distributed initialization, communication waits, a stalled training program, or the task has entered an abnormal stage but has not yet exited.	Compare the status and logs of each Pod, check GPU, CPU, and network metrics, and confirm whether training steps are progressing.
Training needs to continue after task interruption	The task was preempted, a node failed, the task was stopped manually, or the fault-tolerance limit was reached.	Check the actual resource pool, preemption or fault events, fault-tolerance records, and related tasks. After locating the latest attempt, confirm that the checkpoint was loaded successfully.

Key Troubleshooting Dir

Long Queue Times

Even when total resources appear sufficient, the placement requirements for individual replicas and the overall scheduling requirements for distributed jobs must still be met. We recommend troubleshooting in the following order:

Open the task details and check Status and Event to determine whether the issue is related to quota, the resource pool, node policies, or the instance specification. If an event points to a node policy, check the Schedule Type, Scope, Effect Mode, Workload Type, Node Name, and expiration time in Node Scheduling Policy.
Verify your personal quota, resource pool availability, and resource pool status.
Check whether a single replica can fit on an available node. For distributed tasks, all replicas must simultaneously meet the startup requirements.
If the task has network group requirements, ensure that the nodes are in compatible groups. RoCE POD on the page or in events refers to a network group, not a Kubernetes Pod.

For more troubleshooting methods for resource queuing, see Troubleshoot insufficient task resources or long queue times.

Running but Training Makes No Progress

Resource allocation does not mean that training computation has started. First, check whether each Pod is ready in Status, then use Log and Monitoring to identify the stage at which the issue occurs:

Stage	Key Checks
Image pull or container startup	Check Event and System Log to determine whether there are issues with the image, dependencies, or startup process.
Storage or data preparation	Check mount events, user logs, and data read paths to verify whether the Volume, Dataset, or Model is accessible.
Distributed initialization	Compare logs across Pods to verify that environment variables such as `MASTER_ADDR`, `MASTER_PORT`, `WORLD_SIZE`, and `RANK`, as well as communication initialization, are configured correctly.
Training program stall	Check training steps, loss, GPU utilization, CPU, memory, and network metrics to determine whether the program has entered a prolonged state without progress.

Failed to Pull Image or Access File

Issue	Items to Check
Image pull failed	Image address, image version, repository authorization, network connectivity, and pull error messages.
File not found	The selected source directory, container mount path, paths in the startup command, and the current working directory.
Unable to write	`r` / `rw` permissions for the Volume or directory, directory-level permissions, volume capacity, and directory quotas.

Resubmit the task after fixing the deterministic configuration errors. Simply increasing the number of retries usually cannot resolve an incorrect image address, an invalid path, insufficient permissions, or insufficient capacity.

Continue Training After an Interruption

After a task is preempted, a node fails, it is manually stopped, or it reaches the fault-tolerance limit, first check whether an automatic recovery action is already in progress, then decide whether to resubmit it manually.

View the actual resource pool and preemption, failure, or stop events to identify the cause of the interruption.
View Fault Tolerance Detail and associated tasks to check whether an in-place restart or automatic resubmission has already occurred.
Track the latest attempt and check the checkpoint loading logs, training steps after recovery, and new output.
Manual termination, budget exhaustion, or a fail-fast policy may halt automatic recovery. Verify the current attempt before manually resubmitting the task to avoid running two copies simultaneously.

Resubmit Using Debug

Debug resubmission retains the original task’s resource requirements but does not immediately execute the original training command. Instead, it provides a live environment for troubleshooting. Locate the target task in the task list, click Resubmit, and select Debug mode in the resubmission dialog. Before resubmitting, ensure that any checkpoints, logs, or model outputs to be retained have been written to a persistent Volume.

Applicable Scenario	Usage Recommendation
The reason for task failure is unclear	Use Debug resubmission to enter the container and check the environment, paths, dependencies, and startup command.
You want to reuse the original task scheduling location as much as possible	Debug resubmission usually retains the original resource requirements, but starts a new task with a troubleshooting command rather than directly executing the original training command.
A RayJob requires investigation of the cluster state	After Debug is enabled for a RayJob, the Ray cluster can be retained for a period of time—typically 600 seconds by default—allowing you to enter the Head Pod and view the cluster status.

For a RayJob, if the Submitter Pod has completed but the Head/Worker Pods are still retained, the job may be in the DebugHolding state. You can access the Head Pod to view the Ray cluster status.

Get Started

Resources

Development & Training

Models & Inference

Operations & Governance

Best Practices

Developer Access

FAQ

Troubleshoot Training Issues

Troubleshooting Task Issues

Handle Based on Symptoms

Key Troubleshooting Dir

Long Queue Times

Running but Training Makes No Progress

Failed to Pull Image or Access File

Continue Training After an Interruption

Resubmit Using Debug

​Troubleshooting Task Issues

​Handle Based on Symptoms

​Key Troubleshooting Dir

​Long Queue Times

​Running but Training Makes No Progress

​Failed to Pull Image or Access File

​Continue Training After an Interruption

​Resubmit Using Debug

​Related Documentation

Troubleshooting Task Issues

Handle Based on Symptoms

Key Troubleshooting Dir

Long Queue Times

Running but Training Makes No Progress

Failed to Pull Image or Access File

Continue Training After an Interruption

Resubmit Using Debug

Related Documentation