Skip to main content
This page helps troubleshoot common issues after general services and LLM inference services go live. First confirm the service type, then refer to the access, release, and testing documentation for the corresponding service.

​
The Service Instance Is Running, but Requests Fail

A service instance being in the running state only indicates that the workload has started; it does not mean that the gateway, authentication, and service APIs are all accessible.
  1. On the service details page, confirm the current access address and the corresponding protocol, port, and ingress path.
  2. Verify the authentication method, request headers, model identifier, and business API path.
  3. The general service should listen on an address accessible from outside the container; if it listens only on 127.0.0.1, it usually cannot be accessed through the gateway.
  4. Check whether the target instance is ready, and review events, health checks, and instance logs.
  5. Use a minimal request to validate the business response, then gradually add the complete request parameters and production traffic.
For general services, see Create a General Service; for LLM inference services, use Functionality Testing to validate large model inference requests.

​
Still Unable to Handle Traffic After Capacity Expansion

First determine whether a single replica can host the model or application, then assess whether the total capacity of the ready replicas is sufficient. A single replica cannot host the model or application
  1. Check whether the resource specifications of a single replica are sufficient to load the model or run the application.
  2. If the capacity of a single replica is insufficient, adjust the resource specifications, parallelization strategy, or model.
Replicas are ready, but total throughput is insufficient
  1. Before scaling out, allow time for model loading and warm-up.
  2. After increasing capacity, confirm that the new replicas are ready.
  3. Monitor latency, error rates, and request queueing to determine whether the total capacity after scaling out meets demand.
For the configuration limits of manual and automatic scaling, see Advanced Operations for Inference Services.

​
Concerned About Interrupting Requests During Deployment or Graceful Shutdown

  • Boundaries of graceful shutdown: The graceful shutdown delay only extends the Pod termination window. The application must still handle termination signals correctly: stop accepting new requests, process or hand off in-flight requests, and persist its state before exiting.
  • Post-deployment validation: After deployment, first validate under low traffic; do not use extended wait times as a substitute for probe, retries, and data consistency design.
For LLM inference service, see Release and Roll Back LLM Inference Services. For general service, see Release and Roll Back General Inference Services.

​
Does an inference service having no metric data indicate an anomaly?

Not necessarily. There may have been no requests within the query time range, monitoring may not have collected any samples, or the current engine may not support this metric.
  1. Confirm that the service is running, and adjust the query time range to when the actual request occurred.
  2. Check the metric collection status. If only some metrics are missing, confirm whether the current engine supports those metrics.
  3. Assess whether the service is functioning properly based on service details, events, and logs; do not draw conclusions solely from whether metrics display data.
For monitoring scope and usage, see Advanced Operations for Inference Services.