Integrate Custom Metrics into Training

If a training task needs to report request counts, queue length, training loss, or other business metrics, you can expose a metrics endpoint in the training code and configure the metric collection endpoint when creating the training task. After the task runs, the platform collects these metrics and displays them under Monitoring > Custom on the training task details page.

1. Expose a Metrics Endpoint in the Training Code

The training code must expose ports and paths, ensuring that they match the metrics endpoints configured when the task is created. The following example exposes /metrics through two ports in the same training task.

Metrics Port	Metrics Path	Example Metrics
`8000`	`/metrics`	Application metrics, such as request count, connection count, and queue length.
`8001`	`/metrics`	System metrics, such as CPU utilization, memory usage, and uptime.

Code example: The metric update logic in the example is for demonstration purposes only. In actual integration, replace it with real business data from the training code, and ensure that the port and path exposed by the code are consistent with the metric endpoint specified when creating the task.

View Python Sample Code

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Prometheus-style metrics exporter with two ports.

- Port 8000: application metrics (app_registry)
- Port 8001: system metrics (sys_registry)
"""

import time
import random
import threading
import logging
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlparse

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s %(message)s',
)
L = logging.getLogger(__name__)


class Metric:
    """Base metric class."""

    def __init__(self, name, help_text, mtype, labelnames=None):
        self.name = name
        self.help = help_text
        self.type = mtype
        self.labelnames = labelnames or []
        self._lock = threading.Lock()

    def _fmt(self, labels):
        if not labels:
            return ''
        parts = [f'{k}="{str(labels.get(k, ""))}"' for k in self.labelnames]
        return '{' + ','.join(parts) + '}'

    def hdr(self):
        return [
            f'# HELP {self.name} {self.help}',
            f'# TYPE {self.name} {self.type}',
        ]


class Counter(Metric):
    """Monotonically increasing counter."""

    def __init__(self, name, help_text, labelnames=None):
        super().__init__(name, help_text, 'counter', labelnames)
        self._values = {}

    def _key(self, labels):
        return tuple(labels.get(k, '') for k in self.labelnames)

    def inc(self, amount=1, **labels):
        with self._lock:
            k = self._key(labels)
            self._values[k] = self._values.get(k, 0) + amount

    def expose(self):
        lines = self.hdr()
        with self._lock:
            if not self._values and not self.labelnames:
                lines.append(f'{self.name} 0')
            for k, v in self._values.items():
                lbl = dict(zip(self.labelnames, k))
                lines.append(f'{self.name}{self._fmt(lbl)} {v}')
        return lines


class Gauge(Metric):
    """Instantaneous value that can increase or decrease."""

    def __init__(self, name, help_text, labelnames=None):
        super().__init__(name, help_text, 'gauge', labelnames)
        self._values = {}

    def _key(self, labels):
        return tuple(labels.get(k, '') for k in self.labelnames)

    def set(self, value, **labels):
        with self._lock:
            self._values[self._key(labels)] = value

    def expose(self):
        lines = self.hdr()
        with self._lock:
            if not self._values and not self.labelnames:
                lines.append(f'{self.name} 0')
            for k, v in self._values.items():
                lbl = dict(zip(self.labelnames, k))
                lines.append(f'{self.name}{self._fmt(lbl)} {v}')
        return lines


class Registry:
    """Metric registry that collects and exports all metrics in text format."""

    def __init__(self):
        self._metrics = []
        self._lock = threading.Lock()

    def register(self, metric):
        with self._lock:
            self._metrics.append(metric)
        return metric

    def expose(self):
        lines = []
        with self._lock:
            for m in self._metrics:
                lines.extend(m.expose())
        return '\n'.join(lines) + '\n'


# ---------- Application metrics ----------
app_registry = Registry()
http_requests = app_registry.register(
    Counter('myapp_http_requests_total', 'Total requests', ['method', 'status'])
)
active_conns = app_registry.register(
    Gauge('myapp_active_connections', 'Active connections')
)
queue = app_registry.register(
    Gauge('myapp_queue_size', 'Queue size')
)

# ---------- System metrics ----------
sys_registry = Registry()
cpu = sys_registry.register(
    Gauge('system_cpu_usage_percent', 'CPU usage', ['core'])
)
mem = sys_registry.register(
    Gauge('system_memory_usage_bytes', 'Memory usage', ['type'])
)
uptime_metric = sys_registry.register(
    Gauge('system_uptime_seconds', 'Uptime')
)


class MetricsHandler(BaseHTTPRequestHandler):
    registry = None

    def do_GET(self):
        path = urlparse(self.path).path
        if path == '/metrics':
            body = self.registry.expose().encode('utf-8')
            self.send_response(200)
            self.send_header('Content-Type', 'text/plain')
            self.send_header('Content-Length', str(len(body)))
            self.end_headers()
            self.wfile.write(body)
        else:
            self.send_response(404)
            self.end_headers()

    def log_message(self, fmt, *args):
        # Disable the default access log
        pass


def make_handler(registry):
    """Generate a handler class for the specified registry."""

    class _Handler(MetricsHandler):
        pass

    _Handler.registry = registry
    return _Handler


def update_app():
    """Simulate application metric updates."""
    while True:
        try:
            http_requests.inc(
                method=random.choice(['GET', 'POST']),
                status=random.choice(['200', '500']),
            )
            active_conns.set(random.randint(10, 200))
            queue.set(random.randint(0, 50))
            time.sleep(1)
        except Exception as e:
            L.error(f'app:{e}')
            time.sleep(1)


def update_sys():
    """Simulate system metric updates."""
    start = time.time()
    while True:
        try:
            for i in range(4):
                cpu.set(random.uniform(0, 100), core=f'cpu{i}')
            total = 16 * 1024 ** 3
            mem.set(total, type='total')
            mem.set(total * random.uniform(0.3, 0.8), type='used')
            uptime_metric.set(time.time() - start)
            time.sleep(2)
        except Exception as e:
            L.error(f'sys:{e}')
            time.sleep(1)


def start_server(port, registry):
    """Start an HTTP service that exposes /metrics for the specified registry."""
    httpd = ThreadingHTTPServer(('0.0.0.0', port), make_handler(registry))
    threading.Thread(target=httpd.serve_forever, daemon=True).start()
    L.info(f'Server started: port {port}')


def main():
    L.info('Starting dual-port exporter')
    start_server(8000, app_registry)
    start_server(8001, sys_registry)
    threading.Thread(target=update_app, daemon=True).start()
    threading.Thread(target=update_sys, daemon=True).start()
    while True:
        time.sleep(1)


if __name__ == '__main__':
    main()

2. Create a Training Task and Configure the Metrics Endpoint

When creating a training task, enable Metrics Monitor under Advanced Options and configure metrics endpoints for the ports exposed by the training code.

Configuration Item	Description
Metrics Monitor	When enabled, the platform collects custom metrics exposed by the training code. Preset resource metrics are collected by default and do not need to be configured here.
Metrics Port	The metrics port exposed by the training container, for example, `8000` or `8001`.
Metrics Path	The metrics collection path, for example, `/metrics`.

If you need to collect data from multiple endpoints, click Add Metrics Endpoint to add them one by one. Within the same training task, each combination of Metrics Port and Metrics Path must be unique. Based on the example above, add the following two endpoints.

Metrics Port	Metrics Path
`8000`	`/metrics`
`8001`	`/metrics`

For other configuration options for creating a training task, see Create Training.

3. Verify the Metrics Endpoint

After the task starts running, access the corresponding metrics endpoint within the training task container and verify that the endpoint can return metric samples.

curl localhost:8000/metrics
curl localhost:8001/metrics

View Response Example

root@e2e-multiport-v3-7e9039ba-master-0:/workspace# curl localhost:8000/metrics
# HELP myapp_http_requests_total Total requests
# TYPE myapp_http_requests_total counter
myapp_http_requests_total{method="GET",status="500"} 461
myapp_http_requests_total{method="POST",status="500"} 431
myapp_http_requests_total{method="GET",status="200"} 423
myapp_http_requests_total{method="POST",status="200"} 413
# HELP myapp_active_connections Active connections
# TYPE myapp_active_connections gauge
myapp_active_connections 178
# HELP myapp_queue_size Queue size
# TYPE myapp_queue_size gauge
myapp_queue_size 42

root@e2e-multiport-v3-7e9039ba-master-0:/workspace# curl localhost:8001/metrics
# HELP system_cpu_usage_percent CPU usage
# TYPE system_cpu_usage_percent gauge
system_cpu_usage_percent{core="cpu0"} 60.21940599671388
system_cpu_usage_percent{core="cpu1"} 21.640438681845218
system_cpu_usage_percent{core="cpu2"} 9.09251066506329
system_cpu_usage_percent{core="cpu3"} 55.24733876700996
# HELP system_memory_usage_bytes Memory usage
# TYPE system_memory_usage_bytes gauge
system_memory_usage_bytes{type="total"} 17179869184
system_memory_usage_bytes{type="used"} 6094192600.644531
# HELP system_uptime_seconds Uptime
# TYPE system_uptime_seconds gauge
system_uptime_seconds 1730.1035702228546

If multiple metrics endpoints are configured, check each port and path separately. If an endpoint returns metric samples, the metrics have been exposed by the training code. If access fails or the expected metrics are not returned, first check the training code, port, and path configuration.

4. Configure and View Custom Metrics Charts

After the metrics endpoint collects data, go to the training task details page and open the Monitoring tab. Select Custom on the left, then configure charts based on the metric name and label to view the corresponding curves. The available modules, submodules, and classes may vary by environment; refer to the console for the actual options. When configuring, pay particular attention to the following fields.

Configuration Item	Description
Module	Select a module related to the training task.
SubModule	Select a submodule for carrying custom metrics.
Class	Charts in the same class are grouped and displayed on the business page. We recommend naming them by business scenario or metric type.
Legend	The display name of the chart curve. You can use label variables, such as `{{pod}} {{endpoint}}`, to distinguish different Pods, collection endpoints, or business dimensions.
PromQL	Enter a query expression that matches the custom metrics. Metric names, labels, and available scopes are based on the data actually exposed by the current task.

When configuring the Legend and PromQL, you can use the metric’s own business labels or labels added by the platform, such as Pod, scrape endpoint, and container labels. Common labels are as follows.

Label	Description
`pod`	The Pod to which the metric belongs.
`endpoint`	The metric collection endpoint.
`container`	The container to which the metric belongs.
Service label	A label exposed by the training code itself in metrics, such as `method` and `status`.

Available labels are subject to the metrics actually exposed by the current task and those displayed in the console. The following example shows how to configure a chart for myapp_http_requests_total:

[
  {
    "module": "training",
    "subModule": "Custom",
    "class": "myapp_http_requests_total",
    "format": "TimeSeries",
    "title": "HTTP Requests",
    "description": "View request metrics exposed by the training task.",
    "metrics": [
      {
        "legend": "{{pod}} {{endpoint}}",
        "query": "myapp_http_requests_total{}"
      }
    ]
  }
]

After configuration is complete, select the metrics or charts you want to view, set the time range as needed, and click Refresh. For more information about creating and updating dashboards, see Manage monitoring dashboards.

Troubleshoot No Data for Custom Metrics

If there is no data in Monitoring > Custom, troubleshoot in the following order:

Ensure that Metrics Monitor was enabled when the training task was created, and that the port and path match the training code.
Ensure that the training code exposes a metrics endpoint and that metrics are generated while the task is running.
Access the corresponding metrics endpoint from within the training task container and verify that the endpoint can return metric samples.
If multiple metrics endpoints are configured, check each port and path separately.
Refresh later or expand the time range and check again.

Get Started

Resources

Development & Training

Models & Inference

Operations & Governance

Best Practices

Developer Access

FAQ

Integrate Custom Metrics into Training

1. Expose a Metrics Endpoint in the Training Code

2. Create a Training Task and Configure the Metrics Endpoint

3. Verify the Metrics Endpoint

4. Configure and View Custom Metrics Charts

Troubleshoot No Data for Custom Metrics

​1. Expose a Metrics Endpoint in the Training Code

​2. Create a Training Task and Configure the Metrics Endpoint

​3. Verify the Metrics Endpoint

​4. Configure and View Custom Metrics Charts

​Troubleshoot No Data for Custom Metrics

​Related Documentation

1. Expose a Metrics Endpoint in the Training Code

2. Create a Training Task and Configure the Metrics Endpoint

3. Verify the Metrics Endpoint

4. Configure and View Custom Metrics Charts

Troubleshoot No Data for Custom Metrics

Related Documentation