Manage monitoring dashboards

Monitoring dashboards let you configure PromQL queries as reusable charts and attach them to business modules that support dashboards, such as training and inference. After creation, you can view custom metrics on the details page of the corresponding business module.

​

Usage recommendations

• First, determine which business module, submodule, and category the chart will be associated with, then configure the metric query.
• PromQL should match the monitoring metrics and variables of the target module. Refer to the console prompts for variable names, label definitions, and availability scope.
• For the legend, use stable variables or metric dimensions to facilitate comparison across multiple curves or instances.

​

Create a dashboard

A dashboard chart requires configuration of its placement, display mode, and metric query. The placement is determined jointly by the module, submodule, and category; the metric query consists of the legend and PromQL.

1. Go to Config Management > Dashboard Management.
2. Select a region.
3. Click Create.
4. Select a creation method and complete the dashboard configuration.
   • Form
   • JSON

   Use the Form tab to enter each setting in the console.

   Configuration Item	Description
   Module	Select the business module where the dashboard is mounted.
   SubModule	Select the specific location under the module.
   Class	Select the category to which the chart belongs.
   Format	Select how metrics are displayed, such as Timeseries or HeatMap.
   Title	The chart title.
   Description	The chart description.
   Legend	The display name of a metric curve or series. Variable combinations can be used.
   PromQL	The metric query expression. The expression must match the monitoring metrics and variables of the target module to avoid no data being displayed or query failures.

   To display multiple metrics, click Add Metric and continue configuring.

   Use the JSON tab to reuse existing configurations, manage dashboards in bulk, or edit their definitions precisely. For example:

   {
     "module": "llm-inference",
     "subModule": "Vllm",
     "class": "By Instance",
     "format": "TimeSeries",
     "title": "Number of successful vLLM requests",
     "description": "Number of successful requests in the last minute, grouped by service.",
     "metrics": [
       {
         "legend": "{{service}}",
         "query": "sum by (service) (increase(vllm:request_success_total{service=\"$service_name\",cluster=\"$cluster\"}[1m]))"
       }
     ]
   }
   
5. Click Create.

After creation is complete, go to the details page of the relevant business module to verify that the chart display and returned data meet expectations.

​

Update and delete dashboards

In Config Management > Dashboard Management, select a region and locate the target dashboard. You can then update or delete the dashboard in the Actions column.

• Update: Modify the chart mounting location, display configuration, or PromQL. After updating, verify on the relevant business module details page that the chart display and returned data meet expectations.
• Delete: Delete dashboards that are no longer needed. Before deleting, confirm that the team no longer relies on the chart for monitoring or troubleshooting.

​

Related documentation

• View Training Task Monitoring
• Integrate Custom Metrics into Training
• Advanced training task configuration
• Advanced inference service operations