Manage LLM Inference Templates

LLM inference templates are used to create reusable service configurations. After creating a template, you can enable Use Recommended Template when creating an LLM inference service to quickly reuse the Model, Engine, Execute Type, probe, Monitor, and some Advanced Config settings.

Create Template

1. Go to the creation page

Go to Inference > LLM Inference Template.
Select a Region and Cluster.
Click Create.

2. Enter configuration information

Basic Setting

Configuration	Description
Template Name	Used to identify and select the template when creating a service. It is recommended to include the model, engine, or deployment type.
Is Public	Controls whether the template can be viewed or reused by other users.

Model and Engine

Configuration	Description
Model	Select the model source, model version, and access path.
LoRA Model	To load a LoRA Adapter, enable it and configure the corresponding model.
Engine	Select a configured inference engine and version. To add or maintain engine versions, see Manage LLM Engine Config.
Engine Params	Add startup parameters for the inference engine. System parameters required for distributed deployment and PD disaggregation are typically generated by the platform and do not need to be entered again.
Served Model Name	Set the model name exposed by the service.

Probes, Monitor, and Advanced Config

Configuration	Description
Liveness Probe	Determines whether the service process is alive. If it fails, the platform can restart the instance according to the policy.
Readiness Probe	Determines whether the service can receive traffic. If it fails, the platform no longer assigns new requests to the instance.
Metrics	Configures the metrics collection path and port for monitoring, scaling, or alerts.
Environment Variables	Injects runtime configuration for the service.
Storage	Mounts additional volumes.
Service Port	Configures the port on which the service listens or is exposed.
Auto Scaling Configuration	Configures service replica scaling policies based on time or metrics.
Gateway Configuration	Configures the service access entry point, authentication, and routing.
Whether to Send Alerts	Controls whether service exceptions enter the alerting workflow.

Probes support check methods such as HTTP, TCP, and Exec. If the service provides a health check endpoint, configure a Readiness Probe first to prevent traffic from being routed to the service before the Model has finished loading. If automatic recovery is also required when the process hangs or encounters an error, configure a Liveness Probe.

3. Submit Creation

Click Submit. After it is created, you can select this template and reuse its configuration when creating an LLM inference service.

Create from a Service Template

If an existing service configuration is suitable for reuse, you can save it as a template from the Create as Template entry of the target LLM inference service. Before saving, verify that the model path, engine, probes, monitoring, gateway, and public access scope are suitable for reuse by subsequent services.

Manage Templates

After locating the target template under Inference > LLM Inference Templates, you can edit, clone, or delete it as needed.

Actions	Description
Edit	Modify the template configuration. After making changes, recheck the Model, Engine, and Advanced Config before next using the template to create a service.
Clone	Copy an existing template to create a new one. This is suitable when you want to retain the original template while adjusting the Model, Engine, or Advanced Config.
Delete	Delete templates that are no longer in use. Before deletion, confirm that your team no longer relies on the template to create services.

Next Steps

Create an LLM inference service

Get Started

Resources

Development & Training

Models & Inference

Operations & Governance

Best Practices

Developer Access

FAQ

Manage LLM Inference Templates

Create Template

1. Go to the creation page

2. Enter configuration information

Basic Setting

Model and Engine

Probes, Monitor, and Advanced Config

3. Submit Creation

Create from a Service Template

Manage Templates

Next Steps

​Create Template

​1. Go to the creation page

​2. Enter configuration information

​Basic Setting

​Model and Engine

​Probes, Monitor, and Advanced Config

​3. Submit Creation

​Create from a Service Template

​Manage Templates

​Next Steps

Create Template

1. Go to the creation page

2. Enter configuration information

Basic Setting

Model and Engine

Probes, Monitor, and Advanced Config

3. Submit Creation

Create from a Service Template

Manage Templates

Next Steps