This page helps distinguish issues occurring during resource scheduling, environment preparation, and program execution. First, use status details and events to identify the stage at which the issue occurred, then review the relevant configuration or logs.
Why Is the Task Still Queued When There Is Remaining Quota?
Quota only specifies how many resources may be used. Actual scheduling also requires that the resource pool be ready, the workload type be compatible, the capacity of a single node meet the requirements of a single replica, and that both node and network topology conditions be satisfied. Therefore, having remaining quota does not mean that schedulable resources matching the task configuration are currently available.
- Open the task details and check Status and Event to determine whether there are any messages indicating quota, resource pool, instance specification, or node scheduling issues.
- Verify your personal quota, the total quota of the resource pool, and the resource pool status.
- Confirm that the resource pool supports the current workload type and that the instance specification and number of replicas selected for the task comply with the resource pool configuration.
- If the task is configured with candidate specifications, resource guarantees, or node scheduling policies, confirm that these constraints have not further reduced the available resource scope.
- Check the current resource utilization in the Resource Dashboard and Task Dashboard to determine whether you need to wait for resources to be released, adjust the task configuration, or scale out the resource pool.
When viewing schedulable capacity, do not confuse 0 with a query failure: 0 means that no replicas can be scheduled under the current criteria; if “—” is displayed, loading is still in progress, or the request returns an error, the result is unreliable. Refresh the page before making a judgment.
For the complete troubleshooting procedure, see Troubleshoot insufficient task resources or long queue times.
Submission Succeeded, but the Instance Remains Not Ready
A successful submission only indicates that the platform has received the creation request. The instance must still complete resource scheduling, image pulling, storage mount, and application startup.
- View Event to determine whether the instance is stuck in scheduling, image pulling, Volume mounting, or startup.
- If the events indicate insufficient resources or queuing, follow Troubleshoot Insufficient Task Resources or Long Queue Times to check the resource pool and quota.
- If the events indicate an image or mounting issue, verify the image address, version, repository access, Volume status, source directory, and container mount path.
- If the instance has started but is still not ready, review the system logs and application logs, and check the startup command, environment variables, ports, or health checks.
If the list does not refresh immediately, first search by name or refresh the page. Confirm that the object was not created successfully before resubmitting to avoid creating duplicate objects.
Training Is Running, but Training Steps Are Not Progressing
A task entering the running state only indicates that resources have been allocated and the training process has started; it does not mean that training computation is progressing normally.
- In Status, verify whether each Pod is ready and whether any replica has an abnormal status.
- Check the logs of each replica to determine whether the task is stuck during distributed initialization, data reading, communication waiting, or execution of the training program itself.
- Combine Monitoring with user logs to confirm whether training steps, loss, or business metrics continue to change, and whether checkpoints and output files are still being written to the persistent Volume.
For detailed symptoms and handling methods, see Troubleshoot Training Task Issues.