> ## Documentation Index > Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt > Use this file to discover all available pages before exploring further. # Create an LLM inference service > Create an LLM inference service on Siverse and configure the model source, inference engine, resource pool, service port, service gateway, and advanced parameters. After you create an LLM inference service, the platform starts service instances based on the model, inference engine, resource pool, and service configuration, and provides an endpoint for online invocation. You can create the service using a form or reuse an existing configuration through JSON. ## Create a Service Using a Form ### 1. Go to the creation page 1. Go to **Inference > LLM Inference**. 2. Click **Create** and select **Create with Form**. ### 2. Enter configuration information #### Basic Setting Basic settings determine the service scope, service name, and Project Group visibility. | Configuration | Description | | - | - | | **Region**, **Cluster** | Determine the resource scope in which the service is created. The available resource pools, models, storage, and images are affected by the selected region and cluster. | | **Service Name** | The unique name of the service. We recommend including the model name, purpose, or environment to facilitate subsequent search and O\&M. | | **Service Description** | Records the service purpose, model version, callers, or deployment notes. | | **Project Group** | Determines the service ownership and collaboration scope. | | **Public within Project Group** | When enabled, other users in the project group can view or reuse the service. Specific permissions are subject to the project group configuration. | #### Recommendation Templates and Resource Pools | Configuration | Description | | - | - | | **Use Recommended Template** | To reuse an existing template, enable this option and select the target template. The template automatically populates the Model, Engine, probe, monitoring, and some Advanced Config settings. Before submitting, verify that they meet the requirements of the current service. | | **Resource Pool** | Select the resource source for the service. The page may display a combination of Resource Type and resource pool type, for example, **Reserved / Shared** for a reserved instance in a shared resource pool. The resource pool affects available quota, scheduling location, and the resource guarantee method. After creation, you can confirm the actual resources in use under **Resource Pool** and **Occupied Resources** in the service list. | To create or maintain templates in advance, see [Manage LLM Inference Templates](/siverse/inference/manage-llm-inference-templates.md). #### Inference Engine | Configure | Description | | - | - | | **Engine** | Select the inference engine and version, for example, **SGLang / v0.5.1.post1-cu126**. Available options are based on the current Cluster. To add an engine version, use another runtime image, or adjust the role rules for an Execute Type, see [Manage LLM Engine Config](/siverse/inference/manage-llm-inference-engine-configurations.md). | | **Engine Params** | Click **Add more fields** to add engine parameters. The system-distributed parameters required by **vLLM** and **SGLang** in distributed and PD-disaggregated scenarios are generated by the platform and do not need to be specified again. | | **Execute Type** | Select `distributed`, `pd-disaggregated`, or `single-node`. The Execute Type affects the roles and resources that must be configured later. | | **Served Model Name** | Set the model identifier used in client requests. When using the OpenAI-compatible API, the model name in the request must match this configuration. | After you select **Engine** and **Execute Type**, the page displays the corresponding role resource configuration. The actual roles depend on the engine, engine version, and execute type, and should not be inferred solely from the execute type name. The Execute Type is used as follows: | Execute Type | Applicable Scenario | Configuration Focus | | - | - | - | | `single-node` | The Model and runtime state can fit on a single node. A single node can also use multiple GPUs. | Configure the single-node specifications, number of GPUs, and parallelism parameters. | | `distributed` | The Model or execution plan needs to run across multiple nodes. | Configure the group size, node resources, communication network, and parallelism parameters. | | `pd-disaggregated` | The capacity for the input processing and generation stages needs to be planned separately. | Configure Prefill, Decode, routing, and other roles displayed on the page separately. For Prefill, focus on input length and request concurrency; for Decode, focus on output length and request concurrency. | When selecting an execute type, evaluate the model size, status, representative requests, and target concurrency. The execute types and role combinations actually supported are subject to the current engine version and cluster display. After changing the **Engine** or **Execute Type**, the existing role configuration may no longer apply. Before submission, recheck the roles displayed on the page, the number of replicas for each role, group size, instance specification, and total GPU count. For multi-node or multi-role services, calculate the resource requirements for each role separately. The number of GPUs for a role can be estimated as `number of role replicas × number of nodes per group × number of GPUs per node`; the number of GPUs per node must also account for the Count Per Node of the instance specification. For example, if a role is configured with 2 replicas, 2 nodes per group, and 4 GPUs per node, the role uses a total of 16 GPUs. If both the old and new capacity are retained during an update, the additional resources must be calculated separately. When vLLM and SGLang use the platform's preset startup method, Engine Params are appended to the platform-generated startup command, for example, `vllm serve --port --host {{Engine Params}}` or `python3 -m sglang.launch_server --port --host {{Engine Params}}`. In Custom mode, you must enter the complete startup command and ensure that the command, port, and service protocol are consistent with the subsequent port, probe, and gateway configurations. To read the model path or Engine Params passed by the platform, you can reference environment variables such as `MODEL_PATH` and `ENGINE_PARAMS` in the command. #### Model Configuration | Configure | Description | | - | - | | **Model** | Select the model source.
  • **Hugging Face**: Enter the model name and Hugging Face Token.
  • **Volume**: Select an authorized Volume, then enter the container mount path and the model path under the mount directory.
  • **Model Repository**: Select a model and version from the model repository.
  • **OSS**: Enter information such as the Object Storage bucket, path, and Access Keys. Do not include keys in screenshots, logs, or shared materials.
| | **Lora Model** | To load a LoRA Adapter, enable this option and select the corresponding model. | When Volume is selected as the model source, distinguish between the in-container mount path and the model path under the mount directory. The model path ultimately passed to the inference engine is `MountPath/ModelPath`. If `SubPath` is specified, it is used only to restrict the PVC to a specific subdirectory and perform directory-level authorization. Do not use the model path as the Kubernetes `SubPath`. The same rules apply when Lora Model uses Volume as its source. #### Cache Configuration | Configuration | Description | | - | - | | **Enable Distributed Cache** | Enables distributed caching capability for supported engines. Availability and any required roles, ports, or parameters depend on the current engine and the options displayed on the page. | #### Service Configuration Service configuration is used to control Health Checks for service instances. | Configure | Description | | - | - | | **Liveness Probe Check** | Determines whether the service process is alive. If checks continue to fail, the platform may restart the service instance. If the model takes a long time to load, do not configure the checks too aggressively to avoid incorrectly identifying the startup phase as abnormal. | | **Readiness Probe Check** | Determines whether the service is ready to receive traffic. If the check fails, the platform will not route new requests to the instance. We recommend configuring a readiness probe first and routing traffic only after the model has finished loading. | Probes support HTTP, TCP, and Exec. HTTP is suitable for services that already provide health check endpoints such as `/health` or `/ready`; TCP is suitable for services that only need to verify port connectivity; Exec is suitable for running custom commands within a container to check files, processes, or the status of multiple ports. A common Health Check path for vLLM and SGLang is `/health`. The actual path and port should be based on the selected Engine configuration and the capabilities actually exposed by the service. #### Monitor Config | Configuration | Description | | - | - | | **Metrics** | Enter the metrics path and metrics port for service monitoring, elastic scaling, or alarms. Common default paths for SGLang, vLLM, and Custom are `/metrics`; common default ports are `8000` for vLLM, `30000` for SGLang, and `8080` for Custom. For deployment methods such as PD disaggregation, the monitoring port may differ from the service port. Refer to the configuration generated or displayed on the current page. | #### Advanced Config | Configuration | Description | | - | - | | **Environment Variables** | Click **Add more fields** to inject runtime configuration into the service. | | **Volume**, **Mount Dir** | Mount additional Volumes to read models, configuration files, or dependency data. Directory-mode volumes support mounting a specified subdirectory through `SubPath`; mount paths within the same container must be unique. | | **Service Port** | Configure the port on which the service listens or that it exposes. This port must match the port actually listened on by the engine or custom command. | | **Auto Scaling Configuration** | Adjust the number of replicas based on time or metrics. Before the initial release, it is recommended to manually verify service capacity before enabling automatic policies. For more complete scaling policies, see [Advanced Inference Service Operations](/siverse/inference/advanced-inference-service-operations.md). | | **Gateway Configuration** | Click **Add Gateway** to configure the service access endpoint, routing, and authentication. Select **Gateway Type**, **Enable Route Hash**, and **Auth Token** as displayed on the page. For **Gateway Type**, you can select supported types such as **Shared Gateway**. After **Auth Token** is enabled, authentication information must be included when calling the service. Use the corresponding OpenAI-compatible path only when the engine or route explicitly provides a compatible interface. | | **Send Alarms** | When enabled, service exceptions can enter the alarm workflow. Configure the notification method in the alarm center. | | **Priority** | Set the scheduling priority of the service. | | **Preemptible** | Specify whether the service can be preempted. | | **Enable Gang Scheduling** | When enabled, multiple roles within the same replica are scheduled only after their resource requirements can be met as a whole. This is suitable for services with strong dependencies among multiple roles. | | **Fast Fail** | Enables faster exposure of failure causes for creation-stage exceptions, such as image pull failures, facilitating troubleshooting. | Container local disks (such as local directories `/root` and `/tmp`) have limited capacity. Writing large amounts of data may cause the workload to be evicted, and local data will be cleared after a restart or migration. Store important files, such as datasets, models, and output results, in mounted Volume shared storage. We recommend using mount paths that begin with `/volume` or `/mnt` to avoid overwriting default directories in the image, such as the root directory, `/opt`, `/workspace`, and `/usr`. ### 3. Submit and Verify 1. Before submission, verify the Model source, Engine Version, Execute Type, total resources for each role, ports, gateway, and public scope. If necessary, record the Model Version and key configurations. 2. Click **Submit**. 3. After submission, check the target instances and ready instances for each role, and use events and logs to confirm the Model loading progress. 4. After the service enters the **Running** state, go to the **Service Gateway** tab on the details page to view the actual access address and authentication information. 5. Send a minimal request according to the protocol, path, port, and authentication method, and verify that the Model identifier, response content, and streaming or non-streaming invocation method meet expectations. Use the corresponding OpenAI-compatible path only when the Engine or route explicitly provides a compatible API. In **Custom** mode, invoke the service according to its actual protocol. Access Tokens must not appear in screenshots, Logs, or shared materials. ## Create a Service Using JSON If you need to reuse existing configurations, migrate services in batches, or generate service parameters through automated workflows, you can use JSON to create them. 1. Go to **Inference > LLM Inference**. 2. Click **Create** and select **Create with JSON**. 3. Select a **Region** and **Cluster**. 4. Enter the service configuration JSON in the editor. 5. Click **Create**. ### Configuration Example The following example uses a model from the model repository to create an LLM inference service with **vLLM** in **single-node** mode. Before submitting, replace the placeholder values based on the actual Region, Cluster, ResourcePool, Instance Spec, model path, and access policy. ```json theme={null} { "name": "qwen-vllm-service", "description": "Qwen vLLM inference service", "modelConfig": { "modelSource": { "storageType": "model-repo", "storage": { "modelRepo": { "name": "Qwen/Qwen2.5-0.5B-Instruct", "version": "v1.0" } } } }, "servingEngineConfig": { "engineType": "vllm", "engineVersion": "v0.8.5", "executeType": "single-node", "engineParams": {} }, "resourcePool": "", "resourcePoolType": "dedicated", "resourcePoolQosType": "reserved", "roleConfig": { "server": { "replicas": 1, "ports": [ { "name": "router", "containerPort": 8000 } ], "resourceConfig": { "instanceName": "sci.g20-3", "instanceQuantity": 1 }, "routerConfig": { "routerPolicy": "round_robin", "resourcePool": "", "resourcePoolType": "shared", "resourcePoolQosType": "ondemand" } }, "worker": { "replicas": 1, "resourceConfig": { "instanceName": "sci.g20-3", "instanceQuantity": 1 } } }, "serviceConfig": { "replicas": 0, "servicePort": { "name": "server", "port": 8000 }, "registerConfig": { "servedModelName": "qwen2.5" }, "readinessProbe": { "probeType": "", "initialDelaySeconds": 0, "periodSeconds": 0, "timeoutSeconds": 0, "probeConfig": {} }, "livenessProbe": { "probeType": "", "initialDelaySeconds": 0, "periodSeconds": 0, "timeoutSeconds": 0, "probeConfig": {} }, "autoScaleConfig": { "autoScaleType": "timeseries", "timeseriesConfig": { "timePoints": [] } } }, "gatewayConfigs": [ { "type": "Shared", "authToken": "", "enableHashRoute": false } ], "metricsConfig": { "metricsType": "prometheus", "metricsPath": "/metrics", "metricsPort": 8000 }, "monitoringConfig": { "enableAlarmPush": true }, "env": { "MODEL_PATH": "/mnt/model/Qwen/Qwen2.5-0.5B-Instruct", "NCCL_IB_ADDR_FAMILY": "AF_INET6", "NCCL_IB_ADDR_RANGE": "", "TZ": "Asia/Shanghai" }, "storageConfig": { "fileSystemVolumes": [] } } ``` ### Configuration Description | Configuration Block | Description | | - | - | | `name` | Service name. | | `modelConfig` | Model source, model name, version, and storage configuration. | | `servingEngineConfig` | Inference engine type, version, execute type, and engine parameters. `executeType` affects the roles that need to be configured in `roleConfig`. | | `resourcePool` / `resourcePoolType` / `resourcePoolQosType` | The resource pool, resource pool type, and resource type used by default for the service. | | `roleConfig` | The number of replicas, ports, and resource configuration for the roles used in the current example. The `server` and `worker` roles in the example apply only to this configuration and do not imply that different engines, versions, and execute types use the same roles. Refer to the creation page for the roles that actually need to be configured. | | `serviceConfig` | Configuration for auto scaling, probes, registration information, service ports, and more. `registerConfig.servedModelName` is the model identifier used in client requests and must match the model name in the request. | | `gatewayConfigs` | Gateway, routing, authentication, and access endpoint configuration. If `authToken` is empty, verify whether it is generated by the platform or needs to be configured separately based on the actual authentication policy. | | `metricsConfig` | Metrics collection path and port configuration. | | `monitoringConfig` | Monitoring-related configuration, such as alarm notifications. | | `env` | Environment variables for the service runtime. | | `storageConfig` | Additional storage configuration, such as Volume. If no additional storage is mounted, `fileSystemVolumes` can be an empty array. | When creating with JSON, ensure that the resource pool, model, image, ports, environment variables, and gateway information are consistent with the target region and cluster. Before submitting, you can copy the configuration from an existing service or template, then modify the required fields for the current service. ## Next Steps * [Release and Roll Back LLM Inference Services](/siverse/inference/release-and-rollback-llm-inference-services.md) * [Manage LLM Inference Services](/siverse/inference/manage-llm-inference-services.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.