Inference Overview

Inference services enable you to publish LLMs or containerized applications as online services and expose them through endpoints. Before proceeding, determine whether you want to publish an LLM inference workload or a standard HTTP, gRPC, or other containerized service, and then follow the corresponding creation path.

Service Type

Service Type	Suitable Scenarios	Key Considerations Before Creation
LLM inference service	Deploy large language models and provide online inference using vLLM, SGLang, Dynamo, or a custom engine.	Confirm the model source, inference engine, execute type, role resources, and whether an inference template is required.
General Service	Deploy containerized online applications such as OCR, embedding, retrieval, business APIs, post-processing services, or self-developed inference APIs.	Confirm the fixed-version image, startup command, service port, resource specifications, access method, and health check.

To deploy a large language model, we recommend using LLM Inference Service. If the service is essentially a custom HTTP, gRPC, or other containerized application, General Service is more suitable.

Basic Objects

Concept	Description
LLM Engine Config	Defines the engine name, version, runtime image, and role rules for different execute types used for LLM inference. If an existing configuration meets deployment requirements, it can be reused directly; you do not need to create one separately for each service.
LLM Inference Template	Stores reusable models, engines, execute types, probes, monitoring settings, and some advanced configurations. Use it when you need to standardize team configurations or repeatedly create similar services.
Service	An online service that is available for external invocation and is the managed object of the inference module.
Runtime Resources	The resource pool, resource type, and instance specification consumed while the service is running. Resource selection affects whether the service can be scheduled or scaled out, as well as its operating cost.
Instance / Replicas	The runtime units that actually handle requests. Increasing replicas is typically used to improve concurrent request capacity; for multi-role services, pay attention to the replicas of each role separately.
Endpoint	The address clients use to invoke the service. Depending on the service type and configuration, it can be used for access from outside the cluster, within the cluster, or through an independent gateway.
Gateway	Manages service access entries, paths, routing, and authentication. After authentication is enabled, callers must include an access token.

Dir

LLM Inference Service

Prepare the model, resource pool, and volume. The model can come from the model repository, Volume, Object Storage, or other available sources.
To add or maintain an engine version, first manage LLM engine configurations. To reuse team service configurations, create a template in Manage LLM Inference Templates.
When creating a service, select the model source, inference engine, and execute type, then configure resources by role. Different role configurations are displayed for single-node, multi-node, and PD-disaggregated deployments.
After submission, wait for the service to enter Running, then view the endpoint provided by the service and send a minimal request.
After the service goes live, adjust the size as needed, perform functional testing, conduct performance stress testing, or configure alerts. When you need to change the model or service configuration, choose direct release or canary release, and roll back the service if the target version encounters an exception.

General Service

Prepare a fixed-version image, startup command, application port, resource pool, minimum validation request, and required storage resources.
When creating a service, configure Basic information, Resource Information, Environment Information, enhanced features, gateway, and monitoring.
If authentication or Base URL is enabled, ensure that callers can include a Token and that the service can handle the corresponding access path.
After the service enters the Running state, view the endpoint and send a minimum business request. Also verify that the Volume, external dependencies, and authentication method are functioning properly.
After the service passes initial validation, choose direct release or canary release based on change risk. Perform deployment, takedown, restart, scheduled tasks, or rollback as needed during routine operations.

Get Started

Resources

Development & Training

Models & Inference

Operations & Governance

Best Practices

Developer Access

FAQ

Inference Overview

Service Type

Basic Objects

Dir

LLM Inference Service

General Service

​Service Type

​Basic Objects

​Dir

​LLM Inference Service

​General Service

​Related Documentation

Service Type

Basic Objects

Dir

LLM Inference Service

General Service

Related Documentation