> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Integrate Custom Metrics into Training

> Expose custom metrics in a training task, configure multi-port collection, and view custom metrics on the training task details page.

If a training task needs to report request counts, queue length, training loss, or other business metrics, you can expose a metrics endpoint in the training code and configure the metric collection endpoint when creating the training task. After the task runs, the platform collects these metrics and displays them under **Monitoring > Custom** on the training task details page.

## 1. Expose a Metrics Endpoint in the Training Code

The training code must expose ports and paths, ensuring that they match the metrics endpoints configured when the task is created. The following example exposes `/metrics` through two ports in the same training task.

| Metrics Port | Metrics Path | Example Metrics |
| - | - | - |
| `8000` | `/metrics` | Application metrics, such as request count, connection count, and queue length. |
| `8001` | `/metrics` | System metrics, such as CPU utilization, memory usage, and uptime. |

**Code example:**

The metric update logic in the example is for demonstration purposes only. In actual integration, replace it with real business data from the training code, and ensure that the port and path exposed by the code are consistent with the metric endpoint specified when creating the task.

> **View Python Sample Code**
>
> ```python theme={null}
> #!/usr/bin/env python3
> # -*- coding: utf-8 -*-
> """
> Prometheus-style metrics exporter with two ports.
>
> - Port 8000: application metrics (app_registry)
> - Port 8001: system metrics (sys_registry)
> """
>
> import time
> import random
> import threading
> import logging
> from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
> from urllib.parse import urlparse
>
> logging.basicConfig(
>     level=logging.INFO,
>     format='%(asctime)s %(message)s',
> )
> L = logging.getLogger(__name__)
>
>
> class Metric:
>     """Base metric class."""
>
>     def __init__(self, name, help_text, mtype, labelnames=None):
>         self.name = name
>         self.help = help_text
>         self.type = mtype
>         self.labelnames = labelnames or []
>         self._lock = threading.Lock()
>
>     def _fmt(self, labels):
>         if not labels:
>             return ''
>         parts = [f'{k}="{str(labels.get(k, ""))}"' for k in self.labelnames]
>         return '{' + ','.join(parts) + '}'
>
>     def hdr(self):
>         return [
>             f'# HELP {self.name} {self.help}',
>             f'# TYPE {self.name} {self.type}',
>         ]
>
>
> class Counter(Metric):
>     """Monotonically increasing counter."""
>
>     def __init__(self, name, help_text, labelnames=None):
>         super().__init__(name, help_text, 'counter', labelnames)
>         self._values = {}
>
>     def _key(self, labels):
>         return tuple(labels.get(k, '') for k in self.labelnames)
>
>     def inc(self, amount=1, **labels):
>         with self._lock:
>             k = self._key(labels)
>             self._values[k] = self._values.get(k, 0) + amount
>
>     def expose(self):
>         lines = self.hdr()
>         with self._lock:
>             if not self._values and not self.labelnames:
>                 lines.append(f'{self.name} 0')
>             for k, v in self._values.items():
>                 lbl = dict(zip(self.labelnames, k))
>                 lines.append(f'{self.name}{self._fmt(lbl)} {v}')
>         return lines
>
>
> class Gauge(Metric):
>     """Instantaneous value that can increase or decrease."""
>
>     def __init__(self, name, help_text, labelnames=None):
>         super().__init__(name, help_text, 'gauge', labelnames)
>         self._values = {}
>
>     def _key(self, labels):
>         return tuple(labels.get(k, '') for k in self.labelnames)
>
>     def set(self, value, **labels):
>         with self._lock:
>             self._values[self._key(labels)] = value
>
>     def expose(self):
>         lines = self.hdr()
>         with self._lock:
>             if not self._values and not self.labelnames:
>                 lines.append(f'{self.name} 0')
>             for k, v in self._values.items():
>                 lbl = dict(zip(self.labelnames, k))
>                 lines.append(f'{self.name}{self._fmt(lbl)} {v}')
>         return lines
>
>
> class Registry:
>     """Metric registry that collects and exports all metrics in text format."""
>
>     def __init__(self):
>         self._metrics = []
>         self._lock = threading.Lock()
>
>     def register(self, metric):
>         with self._lock:
>             self._metrics.append(metric)
>         return metric
>
>     def expose(self):
>         lines = []
>         with self._lock:
>             for m in self._metrics:
>                 lines.extend(m.expose())
>         return '\n'.join(lines) + '\n'
>
>
> # ---------- Application metrics ----------
> app_registry = Registry()
> http_requests = app_registry.register(
>     Counter('myapp_http_requests_total', 'Total requests', ['method', 'status'])
> )
> active_conns = app_registry.register(
>     Gauge('myapp_active_connections', 'Active connections')
> )
> queue = app_registry.register(
>     Gauge('myapp_queue_size', 'Queue size')
> )
>
> # ---------- System metrics ----------
> sys_registry = Registry()
> cpu = sys_registry.register(
>     Gauge('system_cpu_usage_percent', 'CPU usage', ['core'])
> )
> mem = sys_registry.register(
>     Gauge('system_memory_usage_bytes', 'Memory usage', ['type'])
> )
> uptime_metric = sys_registry.register(
>     Gauge('system_uptime_seconds', 'Uptime')
> )
>
>
> class MetricsHandler(BaseHTTPRequestHandler):
>     registry = None
>
>     def do_GET(self):
>         path = urlparse(self.path).path
>         if path == '/metrics':
>             body = self.registry.expose().encode('utf-8')
>             self.send_response(200)
>             self.send_header('Content-Type', 'text/plain')
>             self.send_header('Content-Length', str(len(body)))
>             self.end_headers()
>             self.wfile.write(body)
>         else:
>             self.send_response(404)
>             self.end_headers()
>
>     def log_message(self, fmt, *args):
>         # Disable the default access log
>         pass
>
>
> def make_handler(registry):
>     """Generate a handler class for the specified registry."""
>
>     class _Handler(MetricsHandler):
>         pass
>
>     _Handler.registry = registry
>     return _Handler
>
>
> def update_app():
>     """Simulate application metric updates."""
>     while True:
>         try:
>             http_requests.inc(
>                 method=random.choice(['GET', 'POST']),
>                 status=random.choice(['200', '500']),
>             )
>             active_conns.set(random.randint(10, 200))
>             queue.set(random.randint(0, 50))
>             time.sleep(1)
>         except Exception as e:
>             L.error(f'app:{e}')
>             time.sleep(1)
>
>
> def update_sys():
>     """Simulate system metric updates."""
>     start = time.time()
>     while True:
>         try:
>             for i in range(4):
>                 cpu.set(random.uniform(0, 100), core=f'cpu{i}')
>             total = 16 * 1024 ** 3
>             mem.set(total, type='total')
>             mem.set(total * random.uniform(0.3, 0.8), type='used')
>             uptime_metric.set(time.time() - start)
>             time.sleep(2)
>         except Exception as e:
>             L.error(f'sys:{e}')
>             time.sleep(1)
>
>
> def start_server(port, registry):
>     """Start an HTTP service that exposes /metrics for the specified registry."""
>     httpd = ThreadingHTTPServer(('0.0.0.0', port), make_handler(registry))
>     threading.Thread(target=httpd.serve_forever, daemon=True).start()
>     L.info(f'Server started: port {port}')
>
>
> def main():
>     L.info('Starting dual-port exporter')
>     start_server(8000, app_registry)
>     start_server(8001, sys_registry)
>     threading.Thread(target=update_app, daemon=True).start()
>     threading.Thread(target=update_sys, daemon=True).start()
>     while True:
>         time.sleep(1)
>
>
> if __name__ == '__main__':
>     main()
> ```

## 2. Create a Training Task and Configure the Metrics Endpoint

When creating a training task, enable **Metrics Monitor** under **Advanced Options** and configure metrics endpoints for the ports exposed by the training code.

| Configuration Item | Description |
| - | - |
| Metrics Monitor | When enabled, the platform collects custom metrics exposed by the training code. Preset resource metrics are collected by default and do not need to be configured here. |
| Metrics Port | The metrics port exposed by the training container, for example, `8000` or `8001`. |
| Metrics Path | The metrics collection path, for example, `/metrics`. |

If you need to collect data from multiple endpoints, click **Add Metrics Endpoint** to add them one by one. Within the same training task, each combination of **Metrics Port** and **Metrics Path** must be unique. Based on the example above, add the following two endpoints.

| Metrics Port | Metrics Path |
| - | - |
| `8000` | `/metrics` |
| `8001` | `/metrics` |

For other configuration options for creating a training task, see [Create Training](/siverse/training/create-training-tasks.md).

## 3. Verify the Metrics Endpoint

After the task starts running, access the corresponding metrics endpoint within the training task container and verify that the endpoint can return metric samples.

```bash theme={null}
curl localhost:8000/metrics
curl localhost:8001/metrics
```

> **View Response Example**
>
> ```bash theme={null}
> root@e2e-multiport-v3-7e9039ba-master-0:/workspace# curl localhost:8000/metrics
> # HELP myapp_http_requests_total Total requests
> # TYPE myapp_http_requests_total counter
> myapp_http_requests_total{method="GET",status="500"} 461
> myapp_http_requests_total{method="POST",status="500"} 431
> myapp_http_requests_total{method="GET",status="200"} 423
> myapp_http_requests_total{method="POST",status="200"} 413
> # HELP myapp_active_connections Active connections
> # TYPE myapp_active_connections gauge
> myapp_active_connections 178
> # HELP myapp_queue_size Queue size
> # TYPE myapp_queue_size gauge
> myapp_queue_size 42
>
> root@e2e-multiport-v3-7e9039ba-master-0:/workspace# curl localhost:8001/metrics
> # HELP system_cpu_usage_percent CPU usage
> # TYPE system_cpu_usage_percent gauge
> system_cpu_usage_percent{core="cpu0"} 60.21940599671388
> system_cpu_usage_percent{core="cpu1"} 21.640438681845218
> system_cpu_usage_percent{core="cpu2"} 9.09251066506329
> system_cpu_usage_percent{core="cpu3"} 55.24733876700996
> # HELP system_memory_usage_bytes Memory usage
> # TYPE system_memory_usage_bytes gauge
> system_memory_usage_bytes{type="total"} 17179869184
> system_memory_usage_bytes{type="used"} 6094192600.644531
> # HELP system_uptime_seconds Uptime
> # TYPE system_uptime_seconds gauge
> system_uptime_seconds 1730.1035702228546
> ```

If multiple metrics endpoints are configured, check each port and path separately. If an endpoint returns metric samples, the metrics have been exposed by the training code. If access fails or the expected metrics are not returned, first check the training code, port, and path configuration.

## 4. Configure and View Custom Metrics Charts

After the metrics endpoint collects data, go to the training task details page and open the **Monitoring** tab. Select **Custom** on the left, then configure charts based on the metric name and label to view the corresponding curves. The available modules, submodules, and classes may vary by environment; refer to the console for the actual options.

When configuring, pay particular attention to the following fields.

| Configuration Item | Description |
| - | - |
| Module | Select a module related to the training task. |
| SubModule | Select a submodule for carrying custom metrics. |
| Class | Charts in the same class are grouped and displayed on the business page. We recommend naming them by business scenario or metric type. |
| Legend | The display name of the chart curve. You can use label variables, such as `{{pod}} {{endpoint}}`, to distinguish different Pods, collection endpoints, or business dimensions. |
| PromQL | Enter a query expression that matches the custom metrics. Metric names, labels, and available scopes are based on the data actually exposed by the current task. |

When configuring the Legend and PromQL, you can use the metric’s own business labels or labels added by the platform, such as Pod, scrape endpoint, and container labels. Common labels are as follows.

| Label | Description |
| - | - |
| `pod` | The Pod to which the metric belongs. |
| `endpoint` | The metric collection endpoint. |
| `container` | The container to which the metric belongs. |
| Service label | A label exposed by the training code itself in metrics, such as `method` and `status`. |

Available labels are subject to the metrics actually exposed by the current task and those displayed in the console.

The following example shows how to configure a chart for `myapp_http_requests_total`:

```json theme={null}
[
  {
    "module": "training",
    "subModule": "Custom",
    "class": "myapp_http_requests_total",
    "format": "TimeSeries",
    "title": "HTTP Requests",
    "description": "View request metrics exposed by the training task.",
    "metrics": [
      {
        "legend": "{{pod}} {{endpoint}}",
        "query": "myapp_http_requests_total{}"
      }
    ]
  }
]
```

After configuration is complete, select the metrics or charts you want to view, set the time range as needed, and click **Refresh**.

For more information about creating and updating dashboards, see [Manage monitoring dashboards](/siverse/configuration/manage-monitoring-dashboards.md).

## Troubleshoot No Data for Custom Metrics

If there is no data in **Monitoring > Custom**, troubleshoot in the following order:

1. Ensure that **Metrics Monitor** was enabled when the training task was created, and that the port and path match the training code.
2. Ensure that the training code exposes a metrics endpoint and that metrics are generated while the task is running.
3. Access the corresponding metrics endpoint from within the training task container and verify that the endpoint can return metric samples.
4. If multiple metrics endpoints are configured, check each port and path separately.
5. Refresh later or expand the time range and check again.

## Related Documentation

* [Create Training](/siverse/training/create-training-tasks.md)
* [Training Advanced Config](/siverse/training/advanced-training-task-configuration.md)
* [View Training Task Monitoring](/siverse/training/view-training-task-monitoring.md)
* [Manage monitoring dashboards](/siverse/configuration/manage-monitoring-dashboards.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.