> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Publish and Roll Back LLM Inference Services

> Directly release or canary release configuration changes to an LLM inference service, and roll back the service if the target version is abnormal.

LLM inference services support direct release and canary release. Direct release is suitable for changes with controllable impact. Canary release retains both the old version and the target version, using a portion of traffic to validate the target version before deciding whether to continue the release or roll back.

## Select a Release Method

| Method | Suitable Scenarios | Key Considerations |
| - | - | - |
| Direct release | Validated in an independent service, or configuration changes with limited impact. | The running version is updated directly after submission. Before release, record the currently available configuration and confirm that resources can support the update process. |
| Canary release | Changes to the inference engine, model, image, startup command, or other configurations that may affect production requests. | The current version and target version run in parallel. Monitor resource usage, instance readiness, and actual request results. |
| Roll back current canary | The canary target version fails to start, request results are abnormal, or performance and resource usage do not meet expectations. | Restore traffic to the previous version during the current canary process, and revalidate the access endpoint and request results after the rollback. |

Only one canary rollout can be in progress for the same service at a time. After the canary rollout is fully switched to the target version, the old version must still be cleaned up before the process is considered complete. You can roll back before cleanup; after cleanup, fast rollback for the current canary rollout is no longer supported.

## Direct Publishing

In the service list, click **Update > Direct Release**, configure and submit the target version. After release, go to the service details to view **Event**, **Instance List**, and **Log**, and wait for the instances to become ready. Then, on the **Service Gateway** tab, confirm the access endpoint and send a minimal request.

When changes involve models, inference engines, images, startup commands, or role configurations, we recommend using a canary release first to validate the target version while the previous version remains available.

## Canary Release

### 1. Configure the Target Version

In the service list, click **Update > Canary Release**. On the canary release page, the **Version Change** section allows you to modify only fields related to the target version.

The service must be **Running** and have no ongoing canary rollout. Before initiating a canary rollout, we recommend recording the current model, inference engine, image, access endpoint, target replicas, and business baseline.

| Configuration Area | Description |
| - | - |
| **Inference Engine** | Select the engine for the target version, and adjust **Engine Params** and **Execute Type** as needed. After making changes, confirm compatibility with the Model, role configuration, and service ports. |
| **Role Configuration** | Configure **Instance Configuration**, image, and **Command** for each role. For multi-role services, verify the runtime configuration of each role individually. |
| **Model Configuration** | Select the model and version for the target version. To use a LoRA model, enable it according to the configuration options provided on the page. |
| **Service Configuration** | Configure the service parameters for the target version within the scope supported by the page. |
| **Advanced Config** | Configure environment variables and storage mounts. Container local disk capacity is limited; data that needs to be persistently stored or shared should be mounted to a Volume. |

The page also includes a **Fields Not Modifiable in Canary Release** section. Expand it to view the fields that cannot be changed in this canary release. To modify these fields, follow the on-screen instructions to use direct release or the corresponding standalone O\&M entry point.

### 2. Select a Canary Deployment Strategy

| Mode | Configuration Method | Suitable Scenario |
| - | - | - |
| **Simple Mode** | Set the **initial canary percentage**. After submission, the platform allocates initial traffic and instances to the target version according to this percentage. | Use a small amount of traffic first to validate the target version. |
| **Custom Mode** | Set the target number of instances for the old version and the target version separately. | Independent control of capacity on both sides is required. |

Select a capacity policy:

* **Start Before Stop**: Start the target version instances before stopping the old version instances. This reduces the risk of capacity reduction during the transition period but requires additional resources.
* **Stop Before Start**: Stop the old version instances before starting the target version instances. This is suitable for scenarios with limited resources where capacity changes during the transition period are acceptable.

### 3. Preview and Start the Canary Rollout

In **Canary Preview**, verify the following information:

* Traffic proportions for the old and target versions.
* The number of target instances on both sides and the total number of instances.
* Current capacity policy.

After confirming that the resource and traffic allocation meets expectations, click **Create and Start Canary**.

## Verify the Target Version

After the canary rollout is started, first confirm that the target version can run stably, and then continue the release:

1. View **Event** and **Instance List** to confirm that the target instance is ready and is not restarting continuously.
2. View **Log** to confirm that model loading, inference engine startup, and request processing are functioning properly.
3. On the **Service Gateway** tab, confirm the access endpoint, send a minimal request, and verify the response.
4. Review resource and engine monitoring to confirm that errors, latency, throughput, and resource utilization, including GPU and GPU memory, meet expectations.

If the target version is not yet stable, do not continue expanding its scope of use.

## Gradual Rollout

After confirming that the target version meets expectations, gradually roll out the canary release based on the validation results.

| Action | Applicable Stage | Result |
| - | - | - |
| **Adjust Target** | Further scale-out, scale-in, or target version capacity adjustments are required. | The platform gradually adjusts traffic and instances for the old version and target version according to the new target. |
| **Switch All Traffic to Target Version** | The target version has been verified as stable and is ready to handle all traffic. | Traffic is switched to the target version, while the old version is retained for rollback during the observation period. |
| **Rollback** | The target version is faulty, has insufficient capacity, or business metrics do not meet expectations. | Traffic is restored to the old version, and target version resources are removed according to the canary release process. |
| **Clean Up Old Version** | After the full traffic switch, the target version is stable and rapid rollback is no longer required. | Delete the old version resources and complete this canary release. |

After each target adjustment, wait for the instance to become ready and monitor errors, latency, throughput, business results, and alarms before proceeding. The actual traffic proportion of small-sample requests may differ from the configured proportion; make judgments based on overall metrics.

## Rollback and Clean Up Old Versions

If the target version encounters an issue, use the rollback action in the current canary rollout process to restore traffic to the earlier version. Before rolling back, ensure that the earlier version’s instances and capacity can continue handling requests.

After the rollback, wait for the service to return to **Running**, then recheck **Event**, **Instance List**, and **Log**. Finally, confirm the access endpoint on the **Service Gateway** tab, send a minimal request, and verify that the service has recovered.

After traffic is fully switched to the target version, the old version is retained for rollback during the observation period and continues to consume the corresponding resources. After confirming that the target version is stable, click **Clean Up Old Version** to release resources and complete this canary rollout. After cleanup, you can no longer quickly roll back through the current canary rollout process.

## Roll Back to a Historical Version

Rolling back to a historical version restores a service configuration that was previously in effect. It is not the same as withdrawing the current canary release.

1. In the service version history, select a version that was previously active in production, and verify the model, inference engine, image, startup command, and related dependencies.
2. Ensure that no canary rollout is currently in progress, and confirm that resource quotas, model access, and storage permissions still meet the requirements of the earlier version’s configuration.
3. Submit the rollback as prompted on the page, then wait for the instance to be updated and ready.
4. Revalidate the access endpoint, minimum request, and required dependencies.

> [!WARNING]
>
> Rolling back to a historical version restores only the service configuration of the corresponding version; it does not restore data in the database or Volume. A canary target version that has not officially taken effect cannot be used as a historical version rollback point.

## Frequently Asked Questions

> **Why are other operations disabled during the canary release?**
>
> The canary release process is managing traffic and instances for the old version and the target version. To prevent direct updates, scaling, bringing services online or offline, or ingress changes from overriding the canary release target, some operations are disabled. To resume normal operations, first clean up the old version or end the canary release by rolling back.

> **Why are resources of the old version still occupied after a full rollout?**
>
> A full rollout only switches traffic to the target version. The old version is retained for rollback during the observation period. After confirming that the target version is stable, perform **Clean Up Old Versions**.

> **What should I check first if canary rollout progression fails?**
>
> First, check the alarms, target instance count, and ready instance count in the canary rollout details, then check the events and instance logs for the target version. If resources are insufficient, reduce the capacity of the target version, adjust the capacity policy, or roll back.

## Related Documentation

* [Manage LLM Inference Services](/siverse/inference/manage-llm-inference-services.md)
* [Test LLM Inference Service Functionality](/siverse/inference/test-llm-inference-service-functionality.md)
* [Load Test LLM Inference Services](/siverse/inference/load-test-llm-inference-services.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.