> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Inference Overview

> Learn about inference service types in Siverse, key concepts before creation, and how to use LLM inference services and general services.

Inference services enable you to publish LLMs or containerized applications as online services and expose them through endpoints. Before proceeding, determine whether you want to publish an LLM inference workload or a standard HTTP, gRPC, or other containerized service, and then follow the corresponding creation path.

## Service Type

| Service Type | Suitable Scenarios | Key Considerations Before Creation |
| - | - | - |
| **LLM inference service** | Deploy large language models and provide online inference using **vLLM**, **SGLang**, **Dynamo**, or a custom engine. | Confirm the model source, inference engine, execute type, role resources, and whether an inference template is required. |
| **General Service** | Deploy containerized online applications such as OCR, embedding, retrieval, business APIs, post-processing services, or self-developed inference APIs. | Confirm the fixed-version image, startup command, service port, resource specifications, access method, and health check. |

To deploy a large language model, we recommend using **LLM Inference Service**. If the service is essentially a custom HTTP, gRPC, or other containerized application, **General Service** is more suitable.

## Basic Objects

| Concept | Description |
| - | - |
| **LLM Engine Config** | Defines the engine name, version, runtime image, and role rules for different execute types used for LLM inference. If an existing configuration meets deployment requirements, it can be reused directly; you do not need to create one separately for each service. |
| **LLM Inference Template** | Stores reusable models, engines, execute types, probes, monitoring settings, and some advanced configurations. Use it when you need to standardize team configurations or repeatedly create similar services. |
| **Service** | An online service that is available for external invocation and is the managed object of the inference module. |
| **Runtime Resources** | The resource pool, resource type, and instance specification consumed while the service is running. Resource selection affects whether the service can be scheduled or scaled out, as well as its operating cost. |
| **Instance / Replicas** | The runtime units that actually handle requests. Increasing replicas is typically used to improve concurrent request capacity; for multi-role services, pay attention to the replicas of each role separately. |
| **Endpoint** | The address clients use to invoke the service. Depending on the service type and configuration, it can be used for access from outside the cluster, within the cluster, or through an independent gateway. |
| **Gateway** | Manages service access entries, paths, routing, and authentication. After authentication is enabled, callers must include an access token. |

## Dir

### LLM Inference Service

1. Prepare the model, resource pool, and volume. The model can come from the model repository, Volume, Object Storage, or other available sources.
2. To add or maintain an engine version, first [manage LLM engine configurations](/siverse/inference/manage-llm-inference-engine-configurations.md). To reuse team service configurations, create a template in [Manage LLM Inference Templates](/siverse/inference/manage-llm-inference-templates.md).
3. When creating a service, select the model source, inference engine, and execute type, then configure resources by role. Different role configurations are displayed for single-node, multi-node, and PD-disaggregated deployments.
4. After submission, wait for the service to enter **Running**, then view the endpoint provided by the service and send a minimal request.
5. After the service goes live, adjust the size as needed, perform functional testing, conduct performance stress testing, or configure alerts. When you need to change the model or service configuration, choose direct release or canary release, and roll back the service if the target version encounters an exception.

### General Service

1. Prepare a fixed-version image, startup command, application port, resource pool, minimum validation request, and required storage resources.
2. When creating a service, configure Basic information, Resource Information, Environment Information, enhanced features, gateway, and monitoring.
3. If authentication or **Base URL** is enabled, ensure that callers can include a Token and that the service can handle the corresponding access path.
4. After the service enters the **Running** state, view the endpoint and send a minimum business request. Also verify that the Volume, external dependencies, and authentication method are functioning properly.
5. After the service passes initial validation, choose direct release or canary release based on change risk. Perform deployment, takedown, restart, scheduled tasks, or rollback as needed during routine operations.

## Related Documentation

* [Create an LLM inference service](/siverse/inference/create-llm-inference-services.md)
* [Create a general service](/siverse/inference/create-general-inference-services.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.