Functional Testing of LLM Inference Services

Functional testing is used to verify whether the APIs and capabilities of the LLM inference service meet expectations. The platform sends actual requests to the running service, checks the selected test suite, and outputs results such as passed, failed, warning, and skipped. Testing generates actual requests; please choose an appropriate time to run it.

Initiate a Dry-run

1. Go to the Dry-run page

Go to the details page of the target LLM inference service, open the Test tab, and click Start Feature Test. Functional testing requires the service to be Running. A service can have only one active test task at a time, and this limit is shared by functional testing and performance stress testing. If functional testing is not enabled for the current cluster, or the test image is not configured, the corresponding entry point may not be displayed on the page.

2. Enter the test configuration

Configuration	Description
Test Suite	You can select openai, messages, gemini, embedding, audio, and image. By default, only openai is run. Select based on the APIs actually supported by the service; test conclusions apply only to the suites selected for this run. For services that provide only OpenAI APIs, do not select gemini, as this will produce meaningless failures.
Per-Case Timeout (Seconds)	The time a single test case waits for a service response. The value ranges from `10` to `600` seconds, with a default of `120` seconds. Increase it as appropriate for larger models or slower initial loading.
Number of function calling Samples	Controls the number of samples for function calling test cases. If left blank, `10` samples are used; if set to `0`, the full dataset is used.
Token Count for Overlong Text Attacks	Set to `0` or leave blank to skip this test case. To trigger the test, the value must be greater than the model’s context window. For example, when the context window is `32k`, you can enter `131072`. This case belongs to the attack test case group; selecting `skip_long_context` does not skip it.
Skip Test Case Groups	You can select `skip_long_context`, `skip_attack`, or `skip_sampling_params` to skip test cases related to long context, attacks, or sampling parameters, respectively. Long-context test cases take longer to run and can be skipped when verification is not required to reduce task duration.
Resource Pool	Select the resource pool that runs the test client. Insufficient resources may cause test tasks to queue or fail.
Instance Spec	Select the instance specification used by the test client.
Target Endpoint	By default, requests go through the service’s actual gateway endpoint. If the service has multiple gateway endpoints, select the endpoint to be validated in this run.

3. Submit Dry-run

Click Submit. The task runs asynchronously. You can view its status in the test task list on the Test tab.

View Dry-run Results

After the Dry-run task reaches a terminal state, click View Report in the task list on the Test tab to view the overall conclusion and per-case details in the report.

Item to Review	Interpretation
Task status	Completed only indicates that the test run has finished; it does not mean the test passed. Failed indicates that the test task itself failed to execute, for example, due to a timeout, insufficient resources, or a client exception.
Overall conclusion	Refer to the pass or fail result in the report. If any test case fails, the functional test should generally be considered unsuccessful.
Four-state counts	`PASS` indicates success, `FAIL` indicates that the result does not meet expectations, `WARN` indicates risks in optional capabilities or fields, and `SKIP` indicates that prerequisites were not met or the test was not executed.
Pass rate	Calculated as `PASS / (PASS + FAIL)`. `WARN` and `SKIP` are not included in the denominator.
Per-test-case details	Focus on the target endpoint, HTTP status code, latency, failure reason, and supplementary information for failed test cases.

If you initiate tests as soon as the service enters Running, the model may still be loading, which can easily result in numerous gateway errors or failed test cases. It is recommended to first send a minimal request to confirm that the service is truly available before performing functional testing.

Stop or Delete Dry-run Tasks

Action	Impact
Stop	Terminates test clients that are still running. Test results that have already been returned are retained. After stopping, you can troubleshoot issues based on the existing results or initiate the test again.
Delete	Deletes the test task and its report, and releases the active test task quota for the service. The report cannot be viewed after deletion.

Frequently Asked Questions

What should I do if the feature testing entry is unavailable?

The feature testing entry depends on the capabilities of the current Cluster and the test Image configuration. Ensure that the service is in a Cluster that supports this capability and that feature testing is enabled.

What should I do if there are already active tasks?

This service has unfinished functional testing or performance stress testing tasks. Wait for the tasks to finish, or stop the existing tasks before starting a new test.

Does task completion with an overall conclusion of failure indicate that the platform execution failed?

No. Task completion indicates that the Dry-run process ended normally. An overall conclusion of failure indicates that the APIs or capabilities of the service under test did not meet the expectations of the current test case. Check the reasons for the failed test cases.

Why are there many 502 errors or failed test cases?

When the service has just entered the Running state, the model may still be loading. First, send a minimal request to confirm that the service can respond normally, then run the functional test again.

Load Testing LLM Inference Services

Get Started

Resources

Development & Training

Models & Inference

Operations & Governance

Best Practices

Developer Access

FAQ

Functional Testing of LLM Inference Services

Initiate a Dry-run

1. Go to the Dry-run page

2. Enter the test configuration

3. Submit Dry-run

View Dry-run Results

Stop or Delete Dry-run Tasks

Frequently Asked Questions

​Initiate a Dry-run

​1. Go to the Dry-run page

​2. Enter the test configuration

​3. Submit Dry-run

​View Dry-run Results

​Stop or Delete Dry-run Tasks

​Frequently Asked Questions

​Related Documentation

Initiate a Dry-run

1. Go to the Dry-run page

2. Enter the test configuration

3. Submit Dry-run

View Dry-run Results

Stop or Delete Dry-run Tasks

Frequently Asked Questions

Related Documentation