Skip to main content
Bisect Diagnose is suitable for identifying problematic nodes in multi-node training when hangs, communication errors, or suspected slow nodes occur. It splits the nodes participating in training into two groups, runs diagnostic tasks for each group, and progressively narrows the scope based on which group reproduces the issue, until the suspected faulty node or abnormal communication link is identified.

​
Confirmation Before Diagnosis

  • The Task Type is PyTorchJob.
  • The task runs on two or more nodes.
  • The task status is Running, Failed, Succeeded, or Abnormal.
  • The impact of diagnostics on the current task has been assessed. When diagnostics are initiated for a Running task, the platform may first stop the current task and then start a new diagnostic task based on the current machine list. Therefore, ensure that training outputs, checkpoints, or important logs have been written to a persistent Volume.
  • After diagnostics are complete, clear the diagnostic session to prevent node locks from affecting subsequent scheduling.

​
How It Works

A diagnostic cycle typically includes:
  1. Pre-check and lock nodes: Check node health, GPU resources, and whether the nodes are occupied by other diagnostic sessions. After the pre-check passes, the platform locks the relevant nodes to prevent other tasks from interfering with the diagnostic process.
  2. Perform bisection: Split the nodes into Group A and Group B, and create independent subtasks for each group to run the original training command or diagnostic command.
  3. Identify the faulty side: Determine which group is more likely to contain the fault based on whether the tasks in the two groups reproduce the issue.
  4. Clean up the diagnostic session: After diagnosis is complete, stop unfinished subtasks and release the node locks.
If an anomaly recurs in a group, the platform continues bisecting the affected group, repeating prechecks, grouped runs, and result evaluations until the scope converges on suspected faulty nodes or can no longer be bisected. Diagnostic details are typically displayed in a tree structure for the subtasks in each round. Each subtask card displays the task name, associated group, status, and a link to details. The following figure uses four nodes as an example to illustrate how Bisect Diagnose continues bisecting along the side where the anomaly is reproduced. Bisect Diagnose binary diagnostic tree diagram After diagnostics are complete, clear the diagnostic session; otherwise, node locking may affect subsequent task scheduling.

​
Identify the Faulty Side

When only one node remains in a group and the issue can still be reproduced, that node can usually be treated as a suspected faulty node for further handling.

​
Diagnostic Mode

​
View Diagnostic Conclusions

Diagnostic results typically display the grouping result for each round, the basis for determination, and handling recommendations. The conclusions distinguish between faulty nodes and slow nodes: faulty nodes usually indicate reproduction failures or exceptions and may be represented as faultyNode in the results; slow nodes usually indicate nodes with significantly lagging performance but not necessarily direct failures and may be represented as slowNode in the results. For perf mode, the results may also include performance metrics such as the average bandwidth, average compute power, or average Model MFU for each round group, which can be used to determine whether there are any significantly lagging nodes. Bisect Diagnose can be initiated manually or triggered as a remediation action for automatic fault tolerance. When automatic fault tolerance needs to further identify the faulty node, the platform may initiate bisect diagnosis through remediation actions such as NODE_BISECTION. The specific trigger conditions depend on the current fault-tolerance configuration and platform capabilities.