Manage LLM Inference Services

After an LLM inference service is created, you can find it in the service list. On the details page, you can obtain the access endpoint, troubleshoot instance status, view events, logs, and monitoring data, and perform scale-out/scale-in, publish updates, take the service offline, copy, share, make it available to a Project Group, or delete it based on business requirements.

Find Services

Go to Inference > LLM Inference.
Select Region, Cluster, and Project Group.
Enter a service name in the search box, or use filters such as Engine Type, Service Status, and ResourcePool.
View services on the Personal Services or Shared Services tab.

Focus on the following information in the list:

Objective	Information to Monitor	How to Determine
Determine whether the service can be invoked	Service Status, endpoint.	After the service enters the Running state, view the access address and send a minimal request. If the service is in a state such as Queued or Scaling, first go to the details page to view events and instances.
Determine whether resources are properly allocated	Resource Pool, Allocated Resources.	Monitor the instance specifications and quantities used by each role. If the desired replicas and ready replicas remain inconsistent for an extended period, check events and instance logs.
Perform routine O&M	Details, Scale, Update, Take Offline, More.	First use Details to confirm the current status. To adjust capacity, click Scale; to change the configuration, select Update > Direct Release or Update > Canary Release. To obtain an access entry, go to Service Gateway on the details page. In the More menu, select Copy, Share, Create as Template, Delete, or Make Public to Project Group as needed.
Confirm the scope of collaboration	Personal Services, Shared Services, creator, project group, and visibility.	Services that you created or have permission to manage are available under Personal Services; services shared by others are available under Shared Services.

View Service Details

In the service list, click Details for the target service. The top of the Details page displays the service status, engine, engine version, creator, region, cluster, and visibility. The tabs below provide access, troubleshooting, monitoring, and testing.

Objective	Tab to Open	Key Information and Actions
Verify the current configuration	Basic Info	Check whether the model, engine, resources, ports, gateway, and advanced configuration meet expectations, then decide whether to update, scale, or recreate the service.
Troubleshoot instances that are not ready or have restarted	Instance List	Search by Pod name and check the role, container status, host IP, container IP, restart count, and status. For an individual Pod, click Restart, Live Logs, View Monitor, or Terminal.
View storage information	Storage	View the usage of the service instance’s Pod Ephemeral Storage and Node Disk. Pod ephemeral storage helps determine whether container-local logs, cache, or temporary files continue to grow; node disk usage helps determine whether disk utilization on the node hosting the instance may affect service stability. Data that needs to be retained or continues to grow should be written to a mounted Volume. For details, see Best practices for ephemeral storage.
Identify bottlenecks in creation, scheduling, or scaling	Event	Use the event timeline to view changes as the service progresses from creation and initialization to queuing and running. Use the event type, subtype, occurrence time, and event details to identify bottlenecks. For ongoing monitoring, use the alarm subscription action in the event row.
View resource and engine metrics	Monitoring	View data in Config, Resource, the monitoring tab for the selected engine (such as Sglang Monitor), or Custom. You can filter by Pod, monitoring data, time range, and sampling interval.
Obtain the endpoint address	Service Gateway	View the gateway domain name, path, Auth Token, and Enable Route Hash status for Shared Gateway or In-Cluster Service. To verify an endpoint, click Dry-run for the corresponding gateway.
View version information	Publish Versions	View the version number, version status, creation time, and update time. For full rollout, rollback, and old-version cleanup during a canary release, follow the current canary release process.
Troubleshoot startup or request exceptions	Log	View user logs or system logs, select the time range, Pod, and number of lines, then use filters or keyword search to locate issues. If necessary, click Download to save the logs.
Verify APIs or evaluate performance	Test	Click Start Functional Test or Add Load Test Task. After the task is complete, you can View Report, Copy, or Delete historical test tasks.

The default time range for Logs is 1 hour. Platform logs are currently retained for only the most recent 30 days. If the selected start time is outside the log retention period, no logs may be displayed.

Obtain and Verify Access Entry Points

After the service enters the Running state, click Details in the service list and go to the Service Gateway tab to view the gateway configuration, access address, and authentication information, and test the access endpoint as needed.

Access Method	Suitable Scenarios	Usage
Shared Gateway	Invoke services from outside the cluster or through a unified gateway entry point.	Obtain the gateway domain name and path. If Auth Token is enabled, the caller must provide authentication information. If Routing Hash is enabled, confirm the routing behavior as required by your business.
In-Cluster Service	Invoke services from other workloads within the same cluster.	Use the in-cluster domain name and path displayed on the page. Whether it is suitable as a production entry point depends on the network environment and the caller’s deployment location.

If the page provides both a dry-run entry point and a production entry point, the dry-run entry point is for validation only and is not recommended as the formal endpoint. Auth Token must not appear in screenshots, Logs, or shared materials.

Adjust Service Capacity

When service capacity is insufficient, business traffic changes, or role resources need to be adjusted, click Scale Out/In in the service list. Before scaling out, confirm the resource pool quota and available nodes. Before scaling in, ensure that the remaining capacity can handle the current traffic, and pay attention to long-lived connections, streaming requests, and in-flight requests.

Configuration	Description
Engine Params	To adjust engine runtime parameters, click Add more fields. Changes to engine parameters may affect the startup command, model loading, and API behavior. Before submitting, ensure that the changes are compatible with the current engine, model, and port configuration.
ResourcePool	Select the resource pool to use for this scale-out or scale-in operation. The resource pool affects available quota, scheduling location, and resource guarantee methods.
Role tab	Configure each role displayed on the page separately, such as worker. For multi-role services, confirm the number of replicas and resource configuration for each role.
Number of Replicas	Set the target number of replicas. Changing only the number of replicas typically does not trigger a rolling update. After scaling out, wait until the target and ready replica counts reach the expected values. Before scaling in, confirm that the remaining replicas can handle the current traffic.
Instance Configuration	Select an instance specification. The page displays information such as CPU, memory, and the number of available nodes for the selected specification. If there are insufficient available nodes, scaling out may be queued or fail.
Image, Command	If these two configurations are displayed on the page, the change may involve the service runtime image or startup command. Before making changes, record the current working configuration and ensure that the new image, command, ports, and model path can still start properly.
Intelligent Routing	When enabled, configure routing-side resources as required by the page. You can choose whether to use another resource pool and specify the number of routing replicas, instance configuration, and Request Distribution Policy, such as `cache_aware_no_queue`. If you are unsure about the meaning of a policy, retain the existing configuration whenever possible.

After clicking OK, go to the details page to view Event and Instance List, and verify that the target replicas and ready replicas gradually reach the expected numbers. If instances remain unready for an extended period, check Log and Monitoring. After scaling out is complete, send minimal requests and observe the error rate, latency, and resource usage. After scaling in is complete, verify that the remaining instances can still process requests reliably.

Manage Services

Actions	Entry Point and Purpose	Considerations Before Action
Update	In the service list, select Update > Direct Release or Update > Canary Release. Direct release updates the running version in a single operation. Canary release is used to validate the target version in parallel and to complete full rollout, rollback, or cleanup of the old version within the current canary process. For the complete procedure, see Release and Roll Back LLM Inference Services.	Before modifying the model, engine, image, startup parameters, role structure, or probe, it is recommended that you record the current working configuration, model version, and business baseline. Use canary release for high-risk changes whenever possible.
Take Offline / Bring Online	For a running service, click Take Offline in the list to stop the service. For an offline service, if the Bring Online option is available on the page, you can restart the service.	Migrate or stop business traffic before taking the service offline. After bringing it online, wait until the service enters the Running state, then verify the endpoint.
Copy	Select Copy from the More menu to create a similar service by reusing the current service configuration.	Before submitting, verify the service name, project group, resource pool, access entry, and sensitive settings. Do not directly reuse tokens, paths, or model versions that are unsuitable for the new scenario.
Share	Select Share from the More menu to share the service with other users for viewing or use.	Confirm whether the service contains sensitive models, endpoints, or runtime configurations. After sharing, shared users can view it on the Shared Services tab.
Make Public to Project Group	Select Make Public to Project Group from the More menu to make the service visible within the project group.	Before making the service public, confirm that all project group members are authorized to view or use it.
Create as Template	Select Create as Template from the More menu to save the current service configuration as an LLM Inference Template.	Before saving, verify that the model path, engine, probe, and gateway configurations are suitable for reuse.
Delete	Select Delete from the More menu to delete the service record and related runtime resources.	Before deletion, confirm that no callers depend on the service, and save any required configurations, logs, or troubleshooting information. Deleting services, models, images, and Volumes are separate operations; confirm their dependencies individually.

Troubleshoot Service Exceptions

When troubleshooting service unavailability or performance issues, we recommend checking the following in order:

Check the service status and events to determine whether there are insufficient resources, image pull failures, probe failures, or gateway configuration issues.
Check the instance status to identify affected roles and instances.
Check the logs to determine whether there are issues with Model loading, Engine startup, or request processing.
Check resource monitoring and Engine monitoring to determine whether GPU, GPU memory, CPU, memory, throughput, or latency is a bottleneck.
If you need to verify API correctness, initiate functional testing; if you need to assess capacity, initiate performance stress testing.

Get Started

Resources

Development & Training

Models & Inference

Operations & Governance

Best Practices

Developer Access

FAQ

Manage LLM Inference Services

Find Services

View Service Details

Obtain and Verify Access Entry Points

Adjust Service Capacity

Manage Services

Troubleshoot Service Exceptions

​Find Services

​View Service Details

​Obtain and Verify Access Entry Points

​Adjust Service Capacity

​Manage Services

​Troubleshoot Service Exceptions

​Related Documentation

Find Services

View Service Details

Obtain and Verify Access Entry Points

Adjust Service Capacity

Manage Services

Troubleshoot Service Exceptions

Related Documentation