Initiate Performance Stress Testing
1. Go to the load testing page
Go to the details page of the target LLM inference service, open the Test tab, and click Add Stress Test Task. Load testing requires the service to be Running. A service can have only one active test task at a time; this limit is shared by performance load testing and functional testing. Load testing adds actual load to the target service. We recommend performing it during a time window when the service can handle additional traffic, or running load tests on canary instances or dedicated instances.2. Configure Stress Testing
Select the Load Testing Configuration
The stress test type determines the performance scope to be monitored for this task.
Matrix jobs run multiple test configurations and typically take longer than single-point tests. Refer to the page for the available load-testing configurations.
Select a Load Generation Method
The load generation method determines how the stress testing client sends requests to the service.
After selecting a method, enter the corresponding parameters on the page, such as concurrency or request rate, runtime duration, warm-up duration, and maximum number of requests. Start with a workload close to real-world business traffic and gradually increase it to avoid placing excessive pressure on the service all at once.
Select API Type
The API type must match the interface actually provided by the service.
Select Data Source
Configure Load Testing Clients and Request Options
3. Submit the stress test
Click Submit. The load testing task runs asynchronously. You can view its status in the task list on the Test tab. The total duration displayed for a task includes queuing, resource creation, Pod startup, and actual execution time. It is not equivalent to the actual runtime of the load testing request.View Stress Test Results
After the stress test task reaches a terminal state, click View Report in the task list on the Dry-run tab. Performance testing outputs metrics only. Whether the requirements are met must be determined in conjunction with business SLAs, cost, and capacity targets.Interpreting Matrix Results
- Cache Hit Rate Matrix: Compare different combinations of input, output, and cache hit rates using a heatmap. Select a specific cell to view throughput and latency details for that combination.
- Rate Sweep Matrix: Use rate curves to observe throughput and latency changes as the request rate increases, helping identify capacity inflection points. Maximum throughput represents the upper throughput limit the service can achieve and does not reflect typical latency performance at that rate.
Stop or Delete Stress Test Tasks
Identifying Bottlenecks
Frequently Asked Questions
Will load testing affect real production requests?
Will load testing affect real production requests?
Yes. Load testing sends real requests to the target service and increases the actual load. Before load testing a production service, select an off-peak business window, canary instances, or dedicated instances.
Why does the report not provide a pass or fail conclusion?
Why does the report not provide a pass or fail conclusion?
Performance stress testing outputs only metrics. Whether the requirements are met must be determined based on business SLAs, cost, and capacity targets. If you need an API correctness conclusion, use functional testing.
What should I do if there is already an active task?
What should I do if there is already an active task?
This service already has an unfinished performance stress test or functional test task. Wait for the task to finish, or stop the existing task before starting a new test.