Model Inference Release Notes

MetricsUsageBillingTroubleshooting

Improved metrics, billing, and troubleshooting

Console
- Improved per-user usage queries on the Metrics page to improve responsiveness under high concurrency.
- Reduced data refresh delays on the Metrics page for more timely usage visibility.
- Updated Playground so that the default output limit follows the model configuration and the output display includes a detailed token breakdown.
Billing
- Adjusted the billing policy for client disconnections (499). For details, see Token Usage and Billing.
Request troubleshooting
- Error responses now include request_id to support troubleshooting. For details, see Error Handling.
Stability
- Improved service stability for high-concurrency workloads.

Fixed a data-alignment issue in Metrics page trend charts when aggregating data by time range.
Fixed an issue where Playground parameters set to 0 did not take effect.

ObservabilityUsageStability

Improved observability, usage and billing, call paths, and stability

Usage and cost analysis now supports Top Models, Top API Keys, and member usage, helping you quickly identify major traffic and cost sources. Member usage is visible only to administrators. For more information, see View Metrics.
The Metrics page now includes Time to First Token (TTFT), end-to-end latency (Latency), and prompt length distribution (prompt bucket) metrics to help analyze call performance and response latency. For more information, see View Metrics.
The Usage Detail page now provides more complete request details, including request time, API Key, token usage, cost, TTFT, and end-to-end latency.

Model information
- Improved context window and max output token display on model details pages, helping you evaluate context and output capabilities more accurately.
Call path
- Improved error code semantics: recoverable retryable errors now use 4xx responses, making it easier for callers to identify and configure handling logic.
- Improved platform request retry strategy to reduce unnecessary retries in some temporary error scenarios.
Stability and performance
- Improved caching and load balancing for high-concurrency scenarios, increasing service stability for long-context or high-throughput workloads.
- Improved inference service deployment and startup capabilities, reducing service preparation time for some models.

Fixed cache hit issues in some long-context call scenarios, improving stability for continuous requests.
Fixed request timeout issues in some long-context call scenarios.