> ## Documentation Index > Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt > Use this file to discover all available pages before exploring further. # Performance Testing for LLM Inference Services > Run performance stress tests on active LLM inference services to evaluate throughput, latency, capacity limits, and stress test report metrics. Performance testing is used to evaluate the throughput, latency, and capacity limits of an LLM inference service. The platform launches a load-testing client, sends real requests to the service based on the configuration, and reports metrics such as QPS, time to first token, token generation speed, end-to-end latency, token throughput, and HTTP status distribution. The performance test report provides metrics only and does not determine pass or fail. ## Initiate Performance Stress Testing ### 1. Go to the load testing page Go to the details page of the target LLM inference service, open the **Test** tab, and click **Add Stress Test Task**. Load testing requires the service to be **Running**. A service can have only one active test task at a time; this limit is shared by performance load testing and functional testing. Load testing adds actual load to the target service. We recommend performing it during a time window when the service can handle additional traffic, or running load tests on canary instances or dedicated instances. ### 2. Configure Stress Testing #### Select the Load Testing Configuration The stress test type determines the performance scope to be monitored for this task. | Benchmark Type | Suitable Scenarios | Result Interpretation | | - | - | - | | **Single-Point Test** | Validate throughput and latency under a single load configuration. | The report shows the performance metrics for the current configuration. | | **Cache Hit Rate Matrix** | Compare performance variations across different input, output, and cache hit conditions. | Compare combinations using a heatmap to identify the impact of cache hit rate changes on throughput and latency. | | **Rate Scan Matrix** | Scan different request rates to identify the service capacity boundary. | Observe throughput and latency changes through rate curves to identify capacity inflection points. | Matrix jobs run multiple test configurations and typically take longer than single-point tests. Refer to the page for the available load-testing configurations. #### Select a Load Generation Method The load generation method determines how the stress testing client sends requests to the service. | Load Generation Method | Description | | - | - | | **Fixed Concurrency** | Maintains a specified number of in-flight requests at all times. Suitable for validating stable performance at a given concurrency level. | | **Fixed Request Rate** | Continuously sends requests at a specified arrival rate. Suitable for simulating relatively stable business traffic. | | **Maximum Throughput** | Increases request pressure as much as possible within the configured range. Suitable for exploring the maximum throughput the service can achieve. | After selecting a method, enter the corresponding parameters on the page, such as concurrency or request rate, runtime duration, warm-up duration, and maximum number of requests. Start with a workload close to real-world business traffic and gradually increase it to avoid placing excessive pressure on the service all at once. #### Select API Type | API Type | Corresponding Endpoint | | - | - | | `completions` | `/v1/completions` | | `chat_completions` | `/v1/chat/completions` | The API type must match the interface actually provided by the service. #### Select Data Source | Data Source | Description | | - | - | | **Synthetic Data** | Generates request data based on token specifications, suitable for standard load testing and matrix load testing. Synthetic data depends on an available tokenizer; this option may be unavailable if the platform cannot infer the tokenizer from the service. | | **Single Data Item** | Reuses a single request payload, suitable for quick testing with fixed input. | | **Remote URL** | Reads a dataset from a URL accessible to the load-testing client. | | **OSS** | Reads a dataset from object storage. Enter the storage address, dataset path, and access credentials as prompted on the page. | | **Volume** | Reads a dataset from a platform volume. Select a Volume and enter the dataset path as prompted on the page. | #### Configure Load Testing Clients and Request Options | Configuration | Description | | - | - | | **Resource Pool** | Select the resource pool to be used by the load-testing client. The load-testing client itself consumes resources; insufficient resources may cause tasks to queue or fail. | | **Instance Specification** | Select the instance specification for the load-testing client. Increasing client resources improves load generation capacity only when the load-testing client is the bottleneck. | | **Custom Header** | Add headers to load-testing requests. Custom headers cannot override the authentication headers used by the platform to access the service. | | **Skip SSL Verification** | Enable this option when the target endpoint uses a self-signed certificate, a test certificate, or an incomplete certificate chain. For production load testing, it is recommended to correct the certificate configuration first. | ### 3. Submit the stress test Click **Submit**. The load testing task runs asynchronously. You can view its status in the task list on the **Test** tab. The total duration displayed for a task includes queuing, resource creation, Pod startup, and actual execution time. It is not equivalent to the actual runtime of the load testing request. ## View Stress Test Results After the stress test task reaches a terminal state, click **View Report** in the task list on the **Dry-run** tab. Performance testing outputs metrics only. Whether the requirements are met must be determined in conjunction with business SLAs, cost, and capacity targets. | Metrics | Description | Recommendations | | - | - | - | | **QPS** | Measured request throughput. | Use this metric to assess the service's request-processing capacity under the current configuration. | | **Success Rate** | Percentage of successfully completed requests. | Evaluate whether requests are completed reliably together with the HTTP status distribution. | | **TTFT** | Time to first token. | Focus on this metric for streaming output; tail latency is usually more important than the average. | | **TPOT / ITL** | Token generation speed or the interval between adjacent tokens. | Use this metric to determine whether the generation stage has become a bottleneck. | | **E2E** | End-to-end latency. | Review P50, P90, P95, and P99 to assess the overall and tail-end experience. | | **Token Throughput** | Number of tokens processed per unit of time. | Use this metric to compare changes in models, engine parameters, and resource configurations. | | **HTTP Status Distribution** | Counts of various status codes and `incomplete` requests. | `incomplete` indicates in-flight requests that were still unfinished when the task ended. It is a backpressure signal and should not be treated directly as a failure. | ### Interpreting Matrix Results * **Cache Hit Rate Matrix**: Compare different combinations of input, output, and cache hit rates using a heatmap. Select a specific cell to view throughput and latency details for that combination. * **Rate Sweep Matrix**: Use rate curves to observe throughput and latency changes as the request rate increases, helping identify capacity inflection points. Maximum throughput represents the upper throughput limit the service can achieve and does not reflect typical latency performance at that rate. When comparing results from multiple load tests, ensure that the model, engine version, service specifications, request data, load test pattern, load generation method, and target endpoint are consistent. If any of these differ, throughput and latency differences may not be comparable. ## Stop or Delete Stress Test Tasks | Action | Impact | | - | - | | **Stop** | Terminates any load testing clients that are still running. Results that have already been returned are retained. | | **Delete** | Deletes the load testing task and its report, and releases the active test task quota for the service. The report cannot be viewed after deletion. | ## Identifying Bottlenecks | Symptom | Possible Cause | Recommended Action | | - | - | - | | Server GPU, GPU memory, or CPU utilization is near its limit, and latency rises rapidly with load. | Server capacity has reached a bottleneck. | Consider adding replicas, adjusting resources per replica, or optimizing engine parameters. | | The load-testing client CPU is near its limit, while server resources still have headroom. | The load-testing client has become the bottleneck. | Increase load-testing client resources or reduce the load per task. | | A large number of `5xx` or gateway errors occur. | The service is not fully ready, the ingress configuration is abnormal, or the server is overloaded. | First send a minimal request to verify the service, then check logs, events, and monitoring. | | **Synthetic Data** cannot be selected. | The platform cannot infer the tokenizer from the service model source. | Use **Single Data**, **Remote URL**, **OSS**, or **Volume** instead, or configure an available tokenizer as prompted on the page. | ## Frequently Asked Questions > **Will load testing affect real production requests?** > > Yes. Load testing sends real requests to the target service and increases the actual load. Before load testing a production service, select an off-peak business window, canary instances, or dedicated instances. > **Why does the report not provide a pass or fail conclusion?** > > Performance stress testing outputs only metrics. Whether the requirements are met must be determined based on business SLAs, cost, and capacity targets. If you need an API correctness conclusion, use functional testing. > **What should I do if there is already an active task?** > > This service already has an unfinished performance stress test or functional test task. Wait for the task to finish, or stop the existing task before starting a new test. > **Why is synthetic data unavailable?** > > Using synthetic data requires a tokenizer. If the platform cannot infer a tokenizer from the source of the current service model, you can instead use **Single Data**, **Remote URL**, **OSS**, or **Volume**, or configure an available tokenizer as prompted on the page. ## Related Documentation * [Test LLM Inference Service Functionality](/siverse/inference/test-llm-inference-service-functionality.md) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.