> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Manage LLM Inference Services

> Find and manage LLM inference services, view access endpoints, instance status, events, logs, monitoring, dry-run tasks, and version information, and adjust service capacity and visibility.

After an LLM inference service is created, you can find it in the service list. On the details page, you can obtain the access endpoint, troubleshoot instance status, view events, logs, and monitoring data, and perform scale-out/scale-in, publish updates, take the service offline, copy, share, make it available to a Project Group, or delete it based on business requirements.

## Find Services

1. Go to **Inference > LLM Inference**.
2. Select **Region**, **Cluster**, and **Project Group**.
3. Enter a service name in the search box, or use filters such as **Engine Type**, **Service Status**, and **ResourcePool**.
4. View services on the **Personal Services** or **Shared Services** tab.

Focus on the following information in the list:

| Objective | Information to Monitor | How to Determine |
| - | - | - |
| Determine whether the service can be invoked | **Service Status**, endpoint. | After the service enters the **Running** state, view the access address and send a minimal request. If the service is in a state such as **Queued** or **Scaling**, first go to the details page to view events and instances. |
| Determine whether resources are properly allocated | **Resource Pool**, **Allocated Resources**. | Monitor the instance specifications and quantities used by each role. If the desired replicas and ready replicas remain inconsistent for an extended period, check events and instance logs. |
| Perform routine O\&M | **Details**, **Scale**, **Update**, **Take Offline**, **More**. | First use **Details** to confirm the current status. To adjust capacity, click **Scale**; to change the configuration, select **Update > Direct Release** or **Update > Canary Release**. To obtain an access entry, go to **Service Gateway** on the details page. In the **More** menu, select **Copy**, **Share**, **Create as Template**, **Delete**, or **Make Public to Project Group** as needed. |
| Confirm the scope of collaboration | **Personal Services**, **Shared Services**, creator, project group, and visibility. | Services that you created or have permission to manage are available under **Personal Services**; services shared by others are available under **Shared Services**. |

## View Service Details

In the service list, click **Details** for the target service. The top of the Details page displays the service status, engine, engine version, creator, region, cluster, and visibility. The tabs below provide access, troubleshooting, monitoring, and testing.

| Objective | Tab to Open | Key Information and Actions |
| - | - | - |
| Verify the current configuration | **Basic Info** | Check whether the model, engine, resources, ports, gateway, and advanced configuration meet expectations, then decide whether to update, scale, or recreate the service. |
| Troubleshoot instances that are not ready or have restarted | **Instance List** | Search by Pod name and check the role, container status, host IP, container IP, restart count, and status. For an individual Pod, click **Restart**, **Live Logs**, **View Monitor**, or **Terminal**. |
| View storage information | **Storage** | View the usage of the service instance's **Pod Ephemeral Storage** and **Node Disk**. Pod ephemeral storage helps determine whether container-local logs, cache, or temporary files continue to grow; node disk usage helps determine whether disk utilization on the node hosting the instance may affect service stability. Data that needs to be retained or continues to grow should be written to a mounted Volume. For details, see [Best practices for ephemeral storage](/siverse/best-practices/use-ephemeral-storage.md). |
| Identify bottlenecks in creation, scheduling, or scaling | **Event** | Use the event timeline to view changes as the service progresses from creation and initialization to queuing and running. Use the event type, subtype, occurrence time, and event details to identify bottlenecks. For ongoing monitoring, use the alarm subscription action in the event row. |
| View resource and engine metrics | **Monitoring** | View data in **Config**, **Resource**, the monitoring tab for the selected engine (such as **Sglang Monitor**), or **Custom**. You can filter by Pod, monitoring data, time range, and sampling interval. |
| Obtain the endpoint address | **Service Gateway** | View the gateway domain name, path, **Auth Token**, and **Enable Route Hash** status for **Shared Gateway** or **In-Cluster Service**. To verify an endpoint, click **Dry-run** for the corresponding gateway. |
| View version information | **Publish Versions** | View the version number, version status, creation time, and update time. For full rollout, rollback, and old-version cleanup during a canary release, follow the current canary release process. |
| Troubleshoot startup or request exceptions | **Log** | View user logs or system logs, select the time range, Pod, and number of lines, then use filters or keyword search to locate issues. If necessary, click **Download** to save the logs. |
| Verify APIs or evaluate performance | **Test** | Click **Start Functional Test** or **Add Load Test Task**. After the task is complete, you can **View Report**, **Copy**, or **Delete** historical test tasks. |

![LLM Inference Service Instance List](https://mintcdn.com/scitix-adfc65c2/ZqwdmfntBUDyGPkJ/siverse/inference/media/llm-inference-service-instance-list.png?fit=max\&auto=format\&n=ZqwdmfntBUDyGPkJ\&q=85\&s=ed903f8e4669c2212df2b60d35a85c0e)

The default time range for Logs is 1 hour. Platform logs are currently retained for only the most recent 30 days. If the selected start time is outside the log retention period, no logs may be displayed.

## Obtain and Verify Access Entry Points

After the service enters the **Running** state, click **Details** in the service list and go to the **Service Gateway** tab to view the gateway configuration, access address, and authentication information, and test the access endpoint as needed.

| Access Method | Suitable Scenarios | Usage |
| - | - | - |
| **Shared Gateway** | Invoke services from outside the cluster or through a unified gateway entry point. | Obtain the gateway domain name and path. If **Auth Token** is enabled, the caller must provide authentication information. If **Routing Hash** is enabled, confirm the routing behavior as required by your business. |
| **In-Cluster Service** | Invoke services from other workloads within the same cluster. | Use the in-cluster domain name and path displayed on the page. Whether it is suitable as a production entry point depends on the network environment and the caller's deployment location. |

If the page provides both a dry-run entry point and a production entry point, the dry-run entry point is for validation only and is not recommended as the formal endpoint. Auth Token must not appear in screenshots, Logs, or shared materials.

## Adjust Service Capacity

When service capacity is insufficient, business traffic changes, or role resources need to be adjusted, click **Scale Out/In** in the service list. Before scaling out, confirm the resource pool quota and available nodes. Before scaling in, ensure that the remaining capacity can handle the current traffic, and pay attention to long-lived connections, streaming requests, and in-flight requests.

| Configuration | Description |
| - | - |
| **Engine Params** | To adjust engine runtime parameters, click **Add more fields**. Changes to engine parameters may affect the startup command, model loading, and API behavior. Before submitting, ensure that the changes are compatible with the current engine, model, and port configuration. |
| **ResourcePool** | Select the resource pool to use for this scale-out or scale-in operation. The resource pool affects available quota, scheduling location, and resource guarantee methods. |
| Role tab | Configure each role displayed on the page separately, such as **worker**. For multi-role services, confirm the number of replicas and resource configuration for each role. |
| **Number of Replicas** | Set the target number of replicas. Changing only the number of replicas typically does not trigger a rolling update. After scaling out, wait until the target and ready replica counts reach the expected values. Before scaling in, confirm that the remaining replicas can handle the current traffic. |
| **Instance Configuration** | Select an instance specification. The page displays information such as CPU, memory, and the number of available nodes for the selected specification. If there are insufficient available nodes, scaling out may be queued or fail. |
| **Image**, **Command** | If these two configurations are displayed on the page, the change may involve the service runtime image or startup command. Before making changes, record the current working configuration and ensure that the new image, command, ports, and model path can still start properly. |
| **Intelligent Routing** | When enabled, configure routing-side resources as required by the page. You can choose whether to use another resource pool and specify the number of routing replicas, instance configuration, and **Request Distribution Policy**, such as `cache_aware_no_queue`. If you are unsure about the meaning of a policy, retain the existing configuration whenever possible. |

After clicking **OK**, go to the details page to view **Event** and **Instance List**, and verify that the target replicas and ready replicas gradually reach the expected numbers. If instances remain unready for an extended period, check **Log** and **Monitoring**. After scaling out is complete, send minimal requests and observe the error rate, latency, and resource usage. After scaling in is complete, verify that the remaining instances can still process requests reliably.

## Manage Services

| Actions | Entry Point and Purpose | Considerations Before Action |
| - | - | - |
| **Update** | In the service list, select **Update > Direct Release** or **Update > Canary Release**. Direct release updates the running version in a single operation. Canary release is used to validate the target version in parallel and to complete full rollout, rollback, or cleanup of the old version within the current canary process. For the complete procedure, see [Release and Roll Back LLM Inference Services](/siverse/inference/release-and-rollback-llm-inference-services.md). | Before modifying the model, engine, image, startup parameters, role structure, or probe, it is recommended that you record the current working configuration, model version, and business baseline. Use canary release for high-risk changes whenever possible. |
| **Take Offline / Bring Online** | For a running service, click **Take Offline** in the list to stop the service. For an offline service, if the **Bring Online** option is available on the page, you can restart the service. | Migrate or stop business traffic before taking the service offline. After bringing it online, wait until the service enters the **Running** state, then verify the endpoint. |
| **Copy** | Select **Copy** from the **More** menu to create a similar service by reusing the current service configuration. | Before submitting, verify the service name, project group, resource pool, access entry, and sensitive settings. Do not directly reuse tokens, paths, or model versions that are unsuitable for the new scenario. |
| **Share** | Select **Share** from the **More** menu to share the service with other users for viewing or use. | Confirm whether the service contains sensitive models, endpoints, or runtime configurations. After sharing, shared users can view it on the **Shared Services** tab. |
| **Make Public to Project Group** | Select **Make Public to Project Group** from the **More** menu to make the service visible within the project group. | Before making the service public, confirm that all project group members are authorized to view or use it. |
| **Create as Template** | Select **Create as Template** from the **More** menu to save the current service configuration as an [LLM Inference Template](/siverse/inference/manage-llm-inference-templates.md). | Before saving, verify that the model path, engine, probe, and gateway configurations are suitable for reuse. |
| **Delete** | Select **Delete** from the **More** menu to delete the service record and related runtime resources. | Before deletion, confirm that no callers depend on the service, and save any required configurations, logs, or troubleshooting information. Deleting services, models, images, and Volumes are separate operations; confirm their dependencies individually. |

## Troubleshoot Service Exceptions

When troubleshooting service unavailability or performance issues, we recommend checking the following in order:

1. Check the service status and events to determine whether there are insufficient resources, image pull failures, probe failures, or gateway configuration issues.
2. Check the instance status to identify affected roles and instances.
3. Check the logs to determine whether there are issues with Model loading, Engine startup, or request processing.
4. Check resource monitoring and Engine monitoring to determine whether GPU, GPU memory, CPU, memory, throughput, or latency is a bottleneck.
5. If you need to verify API correctness, initiate functional testing; if you need to assess capacity, initiate performance stress testing.

## Related Documentation

* [Release and roll back LLM inference services](/siverse/inference/release-and-rollback-llm-inference-services.md)
* [Test LLM inference service functionality](/siverse/inference/test-llm-inference-service-functionality.md)
* [Load test LLM inference services](/siverse/inference/load-test-llm-inference-services.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.