Create an LLM inference service

After you create an LLM inference service, the platform starts service instances based on the model, inference engine, resource pool, and service configuration, and provides an endpoint for online invocation. You can create the service using a form or reuse an existing configuration through JSON.

Create a Service Using a Form

1. Go to the creation page

Go to Inference > LLM Inference.
Click Create and select Create with Form.

2. Enter configuration information

Basic Setting

Basic settings determine the service scope, service name, and Project Group visibility.

Configuration	Description
Region, Cluster	Determine the resource scope in which the service is created. The available resource pools, models, storage, and images are affected by the selected region and cluster.
Service Name	The unique name of the service. We recommend including the model name, purpose, or environment to facilitate subsequent search and O&M.
Service Description	Records the service purpose, model version, callers, or deployment notes.
Project Group	Determines the service ownership and collaboration scope.
Public within Project Group	When enabled, other users in the project group can view or reuse the service. Specific permissions are subject to the project group configuration.

Recommendation Templates and Resource Pools

Configuration	Description
Use Recommended Template	To reuse an existing template, enable this option and select the target template. The template automatically populates the Model, Engine, probe, monitoring, and some Advanced Config settings. Before submitting, verify that they meet the requirements of the current service.
Resource Pool	Select the resource source for the service. The page may display a combination of Resource Type and resource pool type, for example, Reserved / Shared for a reserved instance in a shared resource pool. The resource pool affects available quota, scheduling location, and the resource guarantee method. After creation, you can confirm the actual resources in use under Resource Pool and Occupied Resources in the service list.

To create or maintain templates in advance, see Manage LLM Inference Templates.

Inference Engine

Configure	Description
Engine	Select the inference engine and version, for example, SGLang / v0.5.1.post1-cu126. Available options are based on the current Cluster. To add an engine version, use another runtime image, or adjust the role rules for an Execute Type, see Manage LLM Engine Config.
Engine Params	Click Add more fields to add engine parameters. The system-distributed parameters required by vLLM and SGLang in distributed and PD-disaggregated scenarios are generated by the platform and do not need to be specified again.
Execute Type	Select `distributed`, `pd-disaggregated`, or `single-node`. The Execute Type affects the roles and resources that must be configured later.
Served Model Name	Set the model identifier used in client requests. When using the OpenAI-compatible API, the model name in the request must match this configuration.

After you select Engine and Execute Type, the page displays the corresponding role resource configuration. The actual roles depend on the engine, engine version, and execute type, and should not be inferred solely from the execute type name. The Execute Type is used as follows:

Execute Type	Applicable Scenario	Configuration Focus
`single-node`	The Model and runtime state can fit on a single node. A single node can also use multiple GPUs.	Configure the single-node specifications, number of GPUs, and parallelism parameters.
`distributed`	The Model or execution plan needs to run across multiple nodes.	Configure the group size, node resources, communication network, and parallelism parameters.
`pd-disaggregated`	The capacity for the input processing and generation stages needs to be planned separately.	Configure Prefill, Decode, routing, and other roles displayed on the page separately. For Prefill, focus on input length and request concurrency; for Decode, focus on output length and request concurrency.

When selecting an execute type, evaluate the model size, status, representative requests, and target concurrency. The execute types and role combinations actually supported are subject to the current engine version and cluster display. After changing the Engine or Execute Type, the existing role configuration may no longer apply. Before submission, recheck the roles displayed on the page, the number of replicas for each role, group size, instance specification, and total GPU count. For multi-node or multi-role services, calculate the resource requirements for each role separately. The number of GPUs for a role can be estimated as number of role replicas × number of nodes per group × number of GPUs per node; the number of GPUs per node must also account for the Count Per Node of the instance specification. For example, if a role is configured with 2 replicas, 2 nodes per group, and 4 GPUs per node, the role uses a total of 16 GPUs. If both the old and new capacity are retained during an update, the additional resources must be calculated separately. When vLLM and SGLang use the platform’s preset startup method, Engine Params are appended to the platform-generated startup command, for example, vllm serve --port <PORT> --host <HOST> {{Engine Params}} or python3 -m sglang.launch_server --port <PORT> --host <HOST> {{Engine Params}}. In Custom mode, you must enter the complete startup command and ensure that the command, port, and service protocol are consistent with the subsequent port, probe, and gateway configurations. To read the model path or Engine Params passed by the platform, you can reference environment variables such as MODEL_PATH and ENGINE_PARAMS in the command.

Model Configuration

Configure	Description
Model	Select the model source. Hugging Face: Enter the model name and Hugging Face Token. Volume: Select an authorized Volume, then enter the container mount path and the model path under the mount directory. Model Repository: Select a model and version from the model repository. OSS: Enter information such as the Object Storage bucket, path, and Access Keys. Do not include keys in screenshots, logs, or shared materials.
Lora Model	To load a LoRA Adapter, enable this option and select the corresponding model.

When Volume is selected as the model source, distinguish between the in-container mount path and the model path under the mount directory. The model path ultimately passed to the inference engine is MountPath/ModelPath. If SubPath is specified, it is used only to restrict the PVC to a specific subdirectory and perform directory-level authorization. Do not use the model path as the Kubernetes SubPath. The same rules apply when Lora Model uses Volume as its source.

Cache Configuration

Configuration	Description
Enable Distributed Cache	Enables distributed caching capability for supported engines. Availability and any required roles, ports, or parameters depend on the current engine and the options displayed on the page.

Service Configuration

Service configuration is used to control Health Checks for service instances.

Configure	Description
Liveness Probe Check	Determines whether the service process is alive. If checks continue to fail, the platform may restart the service instance. If the model takes a long time to load, do not configure the checks too aggressively to avoid incorrectly identifying the startup phase as abnormal.
Readiness Probe Check	Determines whether the service is ready to receive traffic. If the check fails, the platform will not route new requests to the instance. We recommend configuring a readiness probe first and routing traffic only after the model has finished loading.

Probes support HTTP, TCP, and Exec. HTTP is suitable for services that already provide health check endpoints such as /health or /ready; TCP is suitable for services that only need to verify port connectivity; Exec is suitable for running custom commands within a container to check files, processes, or the status of multiple ports. A common Health Check path for vLLM and SGLang is /health. The actual path and port should be based on the selected Engine configuration and the capabilities actually exposed by the service.

Monitor Config

Configuration	Description
Metrics	Enter the metrics path and metrics port for service monitoring, elastic scaling, or alarms. Common default paths for SGLang, vLLM, and Custom are `/metrics`; common default ports are `8000` for vLLM, `30000` for SGLang, and `8080` for Custom. For deployment methods such as PD disaggregation, the monitoring port may differ from the service port. Refer to the configuration generated or displayed on the current page.

Advanced Config

Configuration	Description
Environment Variables	Click Add more fields to inject runtime configuration into the service.
Volume, Mount Dir	Mount additional Volumes to read models, configuration files, or dependency data. Directory-mode volumes support mounting a specified subdirectory through `SubPath`; mount paths within the same container must be unique.
Service Port	Configure the port on which the service listens or that it exposes. This port must match the port actually listened on by the engine or custom command.
Auto Scaling Configuration	Adjust the number of replicas based on time or metrics. Before the initial release, it is recommended to manually verify service capacity before enabling automatic policies. For more complete scaling policies, see Advanced Inference Service Operations.
Gateway Configuration	Click Add Gateway to configure the service access endpoint, routing, and authentication. Select Gateway Type, Enable Route Hash, and Auth Token as displayed on the page. For Gateway Type, you can select supported types such as Shared Gateway. After Auth Token is enabled, authentication information must be included when calling the service. Use the corresponding OpenAI-compatible path only when the engine or route explicitly provides a compatible interface.
Send Alarms	When enabled, service exceptions can enter the alarm workflow. Configure the notification method in the alarm center.
Priority	Set the scheduling priority of the service.
Preemptible	Specify whether the service can be preempted.
Enable Gang Scheduling	When enabled, multiple roles within the same replica are scheduled only after their resource requirements can be met as a whole. This is suitable for services with strong dependencies among multiple roles.
Fast Fail	Enables faster exposure of failure causes for creation-stage exceptions, such as image pull failures, facilitating troubleshooting.

Container local disks (such as local directories /root and /tmp) have limited capacity. Writing large amounts of data may cause the workload to be evicted, and local data will be cleared after a restart or migration. Store important files, such as datasets, models, and output results, in mounted Volume shared storage. We recommend using mount paths that begin with /volume or /mnt to avoid overwriting default directories in the image, such as the root directory, /opt, /workspace, and /usr.

3. Submit and Verify

Before submission, verify the Model source, Engine Version, Execute Type, total resources for each role, ports, gateway, and public scope. If necessary, record the Model Version and key configurations.
Click Submit.
After submission, check the target instances and ready instances for each role, and use events and logs to confirm the Model loading progress.
After the service enters the Running state, go to the Service Gateway tab on the details page to view the actual access address and authentication information.
Send a minimal request according to the protocol, path, port, and authentication method, and verify that the Model identifier, response content, and streaming or non-streaming invocation method meet expectations. Use the corresponding OpenAI-compatible path only when the Engine or route explicitly provides a compatible API. In Custom mode, invoke the service according to its actual protocol.

Access Tokens must not appear in screenshots, Logs, or shared materials.

Create a Service Using JSON

If you need to reuse existing configurations, migrate services in batches, or generate service parameters through automated workflows, you can use JSON to create them.

Go to Inference > LLM Inference.
Click Create and select Create with JSON.
Select a Region and Cluster.
Enter the service configuration JSON in the editor.
Click Create.

Configuration Example

The following example uses a model from the model repository to create an LLM inference service with vLLM in single-node mode. Before submitting, replace the placeholder values based on the actual Region, Cluster, ResourcePool, Instance Spec, model path, and access policy.

{
  "name": "qwen-vllm-service",
  "description": "Qwen vLLM inference service",
  "modelConfig": {
    "modelSource": {
      "storageType": "model-repo",
      "storage": {
        "modelRepo": {
          "name": "Qwen/Qwen2.5-0.5B-Instruct",
          "version": "v1.0"
        }
      }
    }
  },
  "servingEngineConfig": {
    "engineType": "vllm",
    "engineVersion": "v0.8.5",
    "executeType": "single-node",
    "engineParams": {}
  },
  "resourcePool": "<RESOURCE_POOL_NAME>",
  "resourcePoolType": "dedicated",
  "resourcePoolQosType": "reserved",
  "roleConfig": {
    "server": {
      "replicas": 1,
      "ports": [
        {
          "name": "router",
          "containerPort": 8000
        }
      ],
      "resourceConfig": {
        "instanceName": "sci.g20-3",
        "instanceQuantity": 1
      },
      "routerConfig": {
        "routerPolicy": "round_robin",
        "resourcePool": "<ROUTER_RESOURCE_POOL_NAME>",
        "resourcePoolType": "shared",
        "resourcePoolQosType": "ondemand"
      }
    },
    "worker": {
      "replicas": 1,
      "resourceConfig": {
        "instanceName": "sci.g20-3",
        "instanceQuantity": 1
      }
    }
  },
  "serviceConfig": {
    "replicas": 0,
    "servicePort": {
      "name": "server",
      "port": 8000
    },
    "registerConfig": {
      "servedModelName": "qwen2.5"
    },
    "readinessProbe": {
      "probeType": "",
      "initialDelaySeconds": 0,
      "periodSeconds": 0,
      "timeoutSeconds": 0,
      "probeConfig": {}
    },
    "livenessProbe": {
      "probeType": "",
      "initialDelaySeconds": 0,
      "periodSeconds": 0,
      "timeoutSeconds": 0,
      "probeConfig": {}
    },
    "autoScaleConfig": {
      "autoScaleType": "timeseries",
      "timeseriesConfig": {
        "timePoints": []
      }
    }
  },
  "gatewayConfigs": [
    {
      "type": "Shared",
      "authToken": "",
      "enableHashRoute": false
    }
  ],
  "metricsConfig": {
    "metricsType": "prometheus",
    "metricsPath": "/metrics",
    "metricsPort": 8000
  },
  "monitoringConfig": {
    "enableAlarmPush": true
  },
  "env": {
    "MODEL_PATH": "/mnt/model/Qwen/Qwen2.5-0.5B-Instruct",
    "NCCL_IB_ADDR_FAMILY": "AF_INET6",
    "NCCL_IB_ADDR_RANGE": "<NCCL_IB_ADDR_RANGE>",
    "TZ": "Asia/Shanghai"
  },
  "storageConfig": {
    "fileSystemVolumes": []
  }
}

Configuration Description

Configuration Block	Description
`name`	Service name.
`modelConfig`	Model source, model name, version, and storage configuration.
`servingEngineConfig`	Inference engine type, version, execute type, and engine parameters. `executeType` affects the roles that need to be configured in `roleConfig`.
`resourcePool` / `resourcePoolType` / `resourcePoolQosType`	The resource pool, resource pool type, and resource type used by default for the service.
`roleConfig`	The number of replicas, ports, and resource configuration for the roles used in the current example. The `server` and `worker` roles in the example apply only to this configuration and do not imply that different engines, versions, and execute types use the same roles. Refer to the creation page for the roles that actually need to be configured.
`serviceConfig`	Configuration for auto scaling, probes, registration information, service ports, and more. `registerConfig.servedModelName` is the model identifier used in client requests and must match the model name in the request.
`gatewayConfigs`	Gateway, routing, authentication, and access endpoint configuration. If `authToken` is empty, verify whether it is generated by the platform or needs to be configured separately based on the actual authentication policy.
`metricsConfig`	Metrics collection path and port configuration.
`monitoringConfig`	Monitoring-related configuration, such as alarm notifications.
`env`	Environment variables for the service runtime.
`storageConfig`	Additional storage configuration, such as Volume. If no additional storage is mounted, `fileSystemVolumes` can be an empty array.

When creating with JSON, ensure that the resource pool, model, image, ports, environment variables, and gateway information are consistent with the target region and cluster. Before submitting, you can copy the configuration from an existing service or template, then modify the required fields for the current service.

Get Started

Resources

Development & Training

Models & Inference

Operations & Governance

Best Practices

Developer Access

FAQ

Create an LLM inference service

Create a Service Using a Form

1. Go to the creation page

2. Enter configuration information

Basic Setting

Recommendation Templates and Resource Pools

Inference Engine

Model Configuration

Cache Configuration

Service Configuration

Monitor Config

Advanced Config

3. Submit and Verify

Create a Service Using JSON

Configuration Example

Configuration Description

Next Steps

​Create a Service Using a Form

​1. Go to the creation page

​2. Enter configuration information

​Basic Setting

​Recommendation Templates and Resource Pools

​Inference Engine

​Model Configuration

​Cache Configuration

​Service Configuration

​Monitor Config

​Advanced Config

​3. Submit and Verify

​Create a Service Using JSON

​Configuration Example

​Configuration Description

​Next Steps

Create a Service Using a Form

1. Go to the creation page

2. Enter configuration information

Basic Setting

Recommendation Templates and Resource Pools

Inference Engine

Model Configuration

Cache Configuration

Service Configuration

Monitor Config

Advanced Config

3. Submit and Verify

Create a Service Using JSON

Configuration Example

Configuration Description

Next Steps