Publish and Roll Back LLM Inference Services

LLM inference services support direct release and canary release. Direct release is suitable for changes with controllable impact. Canary release retains both the old version and the target version, using a portion of traffic to validate the target version before deciding whether to continue the release or roll back.

​

Select a Release Method

Only one canary rollout can be in progress for the same service at a time. After the canary rollout is fully switched to the target version, the old version must still be cleaned up before the process is considered complete. You can roll back before cleanup; after cleanup, fast rollback for the current canary rollout is no longer supported.

​

Direct Publishing

In the service list, click Update > Direct Release, configure and submit the target version. After release, go to the service details to view Event, Instance List, and Log, and wait for the instances to become ready. Then, on the Service Gateway tab, confirm the access endpoint and send a minimal request. When changes involve models, inference engines, images, startup commands, or role configurations, we recommend using a canary release first to validate the target version while the previous version remains available.

​

Canary Release

​

1. Configure the Target Version

In the service list, click Update > Canary Release. On the canary release page, the Version Change section allows you to modify only fields related to the target version. The service must be Running and have no ongoing canary rollout. Before initiating a canary rollout, we recommend recording the current model, inference engine, image, access endpoint, target replicas, and business baseline. The page also includes a Fields Not Modifiable in Canary Release section. Expand it to view the fields that cannot be changed in this canary release. To modify these fields, follow the on-screen instructions to use direct release or the corresponding standalone O&M entry point.

​

2. Select a Canary Deployment Strategy

Select a capacity policy:

• Start Before Stop: Start the target version instances before stopping the old version instances. This reduces the risk of capacity reduction during the transition period but requires additional resources.
• Stop Before Start: Stop the old version instances before starting the target version instances. This is suitable for scenarios with limited resources where capacity changes during the transition period are acceptable.

​

3. Preview and Start the Canary Rollout

In Canary Preview, verify the following information:

• Traffic proportions for the old and target versions.
• The number of target instances on both sides and the total number of instances.
• Current capacity policy.

After confirming that the resource and traffic allocation meets expectations, click Create and Start Canary.

​

Verify the Target Version

After the canary rollout is started, first confirm that the target version can run stably, and then continue the release:

1. View Event and Instance List to confirm that the target instance is ready and is not restarting continuously.
2. View Log to confirm that model loading, inference engine startup, and request processing are functioning properly.
3. On the Service Gateway tab, confirm the access endpoint, send a minimal request, and verify the response.
4. Review resource and engine monitoring to confirm that errors, latency, throughput, and resource utilization, including GPU and GPU memory, meet expectations.

If the target version is not yet stable, do not continue expanding its scope of use.

​

Gradual Rollout

After confirming that the target version meets expectations, gradually roll out the canary release based on the validation results. After each target adjustment, wait for the instance to become ready and monitor errors, latency, throughput, business results, and alarms before proceeding. The actual traffic proportion of small-sample requests may differ from the configured proportion; make judgments based on overall metrics.

​

Rollback and Clean Up Old Versions

If the target version encounters an issue, use the rollback action in the current canary rollout process to restore traffic to the earlier version. Before rolling back, ensure that the earlier version’s instances and capacity can continue handling requests. After the rollback, wait for the service to return to Running, then recheck Event, Instance List, and Log. Finally, confirm the access endpoint on the Service Gateway tab, send a minimal request, and verify that the service has recovered. After traffic is fully switched to the target version, the old version is retained for rollback during the observation period and continues to consume the corresponding resources. After confirming that the target version is stable, click Clean Up Old Version to release resources and complete this canary rollout. After cleanup, you can no longer quickly roll back through the current canary rollout process.

​

Roll Back to a Historical Version

Rolling back to a historical version restores a service configuration that was previously in effect. It is not the same as withdrawing the current canary release.

1. In the service version history, select a version that was previously active in production, and verify the model, inference engine, image, startup command, and related dependencies.
2. Ensure that no canary rollout is currently in progress, and confirm that resource quotas, model access, and storage permissions still meet the requirements of the earlier version’s configuration.
3. Submit the rollback as prompted on the page, then wait for the instance to be updated and ready.
4. Revalidate the access endpoint, minimum request, and required dependencies.

​

Frequently Asked Questions

​

Related Documentation

• Manage LLM Inference Services
• Test LLM Inference Service Functionality
• Load Test LLM Inference Services