> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# View Training Task Monitoring

> View resource metrics, anomaly metrics, and custom metrics for training tasks in Siverse.

While a task is running or after it is completed, you can view resource metrics, anomaly metrics, and custom metrics on the training task details page to assess resource usage and identify performance bottlenecks.

## Select a Monitoring Perspective

1. Click the task name to go to the training task details page.

2. Open the **Monitoring** tab.

3. Select a monitoring perspective on the left.

   | Monitoring Perspective | Description |
   | - | - |
   | **Task Level** | View overall resource trends for the task. |
   | **Pod Level** | Select a specific Pod to view resource usage and anomaly metrics for an individual replica. |
   | **Node Level** | Select a specific node to view resource usage and anomaly metrics for the node where the task is running. |
   | **Custom** | View custom metrics exposed by the training code. |

4. If you select **Task Level**, **Pod Level**, or **Node Level**, select a metric category, such as **GPU & RDMA**, **CPU System**, or **Anomalies**.

5. If you select **Pod Level** or **Node Level**, select the Pod or node to view.

6. Adjust the time range as needed, and click **Refresh** to get the latest data.

## View Platform Preset Metrics

Resource, system, and anomaly metrics for task-level, Pod-level, and node-level monitoring are collected by default by the platform and do not require additional configuration when creating a task. To view them, first select a monitoring perspective, then choose a metric category based on your troubleshooting objective.

| Diagnostic Objective | How to View |
| - | - |
| Assess the overall task load | Use **Task Level** to view trends related to GPU, GPU memory, CPU, memory, network, disk, and storage. |
| Determine whether a specific replica is abnormal | Use **Pod Level** to select the target Pod and view its GPU, CPU, memory, network, GPFS, or anomaly metrics. Use logs together with these metrics to locate the issue. |
| Determine whether an issue is node-related | Use **Node Level** to select the target node and view node-side GPU, RDMA, CPU, memory, network, disk, GPFS, and anomaly metrics. |
| Analyze GPU and communication links | In **GPU & RDMA**, view metrics such as GPU, GPU memory, SM, NVLink, and RDMA network metrics. |
| Troubleshoot system resource bottlenecks | In **CPU System**, view metrics such as CPU, memory, network, disk, and GPFS. |
| Troubleshoot hardware, network, or congestion anomalies | In **Anomalies**, view metrics such as GPU hardware errors, RDMA or network anomalies, straggler alerts, and ECN or PFC congestion control. |

Monitoring can be used to determine whether tasks are making full use of GPUs, whether CPU, memory, or network resources are becoming bottlenecks, and to inform subsequent adjustments to instance specifications, resource pools, concurrency, or startup parameters. After a task is completed, monitoring data can still be used for post-analysis.

## View Custom Metrics

Custom metrics are used to view business metrics exposed by training code, such as request volume, queue length, training loss, and other business-side statuses. This topic explains how to view integrated custom metrics. To add new metric endpoints, configure multi-port collection, or configure custom metric charts, see [Integrate Custom Metrics into Training](/siverse/training/collect-training-task-custom-metrics.md).

1. Go to the training task details page and open the **Monitoring** tab.
2. Select **Custom** on the left.
3. Select the metrics or charts you want to view.
4. Adjust the time range as needed, and click **Refresh**.

If custom metrics include business labels, the platform retains these dimensions when displaying them, making it easier to distinguish curves by business status. The platform also adds dimensions such as tasks, Pods, and collection endpoints to associate metrics with specific task instances. Available labels are subject to the current metrics and console display.

## Troubleshoot No Data in Monitor

If the chart shows no data, first ensure that the selected time range covers the period when the task was running, and refresh the page as needed. For different types of monitoring data, continue troubleshooting as follows.

### No Data for Platform Preset Metrics

1. If viewing **Pod Level** or **Node Level** monitoring, ensure that the correct Pod or node is selected.
2. Ensure that the current metric category applies to the selected monitoring perspective and task resource type.
3. Metrics collection and display may be subject to some delay. You can refresh later or expand the time range before viewing.

### No Data for Custom Metrics

1. Ensure that **Metrics Monitor** is enabled when creating the task and that the port and path are configured correctly.

2. Ensure that the training code exposes a metrics endpoint and that metrics are generated while the task is running. For end-to-end integration and endpoint exposure examples, see [Integrate Custom Metrics into Training](/siverse/training/collect-training-task-custom-metrics.md).

3. For further verification, access the corresponding metrics endpoint from within the training task container.

   ```bash theme={null}
   curl localhost:8000/metrics
   curl localhost:8001/metrics
   ```

4. If multiple metric endpoints are configured, check each port and path separately. If an endpoint can return metric samples, the training code has exposed the metrics. If access fails or the expected metrics are not returned, first check the training code and the port and path configuration.

5. Metric collection and display may be subject to some delay. You can refresh later or expand the time range before viewing.

## Related Documentation

* [Find, View, and Manage Training Tasks](/siverse/training/view-and-manage-training-tasks.md)
* [Create Training](/siverse/training/create-training-tasks.md)
* [Integrate Custom Metrics into Training](/siverse/training/collect-training-task-custom-metrics.md)
* [Training Advanced Config](/siverse/training/advanced-training-task-configuration.md)
* [View Training Task Logs](/siverse/training/view-training-task-logs.md)
* [Troubleshoot Training Issues](/siverse/training/troubleshoot-training-tasks.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.