Skip to main content
While a task is running or after it is completed, you can view resource metrics, anomaly metrics, and custom metrics on the training task details page to assess resource usage and identify performance bottlenecks.

​
Select a Monitoring Perspective

  1. Click the task name to go to the training task details page.
  2. Open the Monitoring tab.
  3. Select a monitoring perspective on the left.
  4. If you select Task Level, Pod Level, or Node Level, select a metric category, such as GPU & RDMA, CPU System, or Anomalies.
  5. If you select Pod Level or Node Level, select the Pod or node to view.
  6. Adjust the time range as needed, and click Refresh to get the latest data.

​
View Platform Preset Metrics

Resource, system, and anomaly metrics for task-level, Pod-level, and node-level monitoring are collected by default by the platform and do not require additional configuration when creating a task. To view them, first select a monitoring perspective, then choose a metric category based on your troubleshooting objective. Monitoring can be used to determine whether tasks are making full use of GPUs, whether CPU, memory, or network resources are becoming bottlenecks, and to inform subsequent adjustments to instance specifications, resource pools, concurrency, or startup parameters. After a task is completed, monitoring data can still be used for post-analysis.

​
View Custom Metrics

Custom metrics are used to view business metrics exposed by training code, such as request volume, queue length, training loss, and other business-side statuses. This topic explains how to view integrated custom metrics. To add new metric endpoints, configure multi-port collection, or configure custom metric charts, see Integrate Custom Metrics into Training.
  1. Go to the training task details page and open the Monitoring tab.
  2. Select Custom on the left.
  3. Select the metrics or charts you want to view.
  4. Adjust the time range as needed, and click Refresh.
If custom metrics include business labels, the platform retains these dimensions when displaying them, making it easier to distinguish curves by business status. The platform also adds dimensions such as tasks, Pods, and collection endpoints to associate metrics with specific task instances. Available labels are subject to the current metrics and console display.

​
Troubleshoot No Data in Monitor

If the chart shows no data, first ensure that the selected time range covers the period when the task was running, and refresh the page as needed. For different types of monitoring data, continue troubleshooting as follows.

​
No Data for Platform Preset Metrics

  1. If viewing Pod Level or Node Level monitoring, ensure that the correct Pod or node is selected.
  2. Ensure that the current metric category applies to the selected monitoring perspective and task resource type.
  3. Metrics collection and display may be subject to some delay. You can refresh later or expand the time range before viewing.

​
No Data for Custom Metrics

  1. Ensure that Metrics Monitor is enabled when creating the task and that the port and path are configured correctly.
  2. Ensure that the training code exposes a metrics endpoint and that metrics are generated while the task is running. For end-to-end integration and endpoint exposure examples, see Integrate Custom Metrics into Training.
  3. For further verification, access the corresponding metrics endpoint from within the training task container.
  4. If multiple metric endpoints are configured, check each port and path separately. If an endpoint can return metric samples, the training code has exposed the metrics. If access fails or the expected metrics are not returned, first check the training code and the port and path configuration.
  5. Metric collection and display may be subject to some delay. You can refresh later or expand the time range before viewing.