New Features
- Usage and cost analysis now supports Top Models, Top API Keys, and member usage, helping you quickly identify major traffic and cost sources. Member usage is visible only to administrators. For more information, see View Metrics.
- The Metrics page now includes Time to First Token (TTFT), end-to-end latency (Latency), and prompt length distribution (prompt bucket) metrics to help analyze call performance and response latency. For more information, see View Metrics.
- The Usage Detail page now provides more complete request details, including request time, API Key, token usage, cost, TTFT, and end-to-end latency.
Improvements
-
Model information
- Improved
context windowandmax output tokendisplay on model details pages, helping you evaluate context and output capabilities more accurately.
- Improved
-
Call path
- Improved error code semantics: recoverable retryable errors now use 4xx responses, making it easier for callers to identify and configure handling logic.
- Improved platform request retry strategy to reduce unnecessary retries in some temporary error scenarios.
-
Stability and performance
- Improved caching and load balancing for high-concurrency scenarios, increasing service stability for long-context or high-throughput workloads.
- Improved inference service deployment and startup capabilities, reducing service preparation time for some models.
Bug Fixes
- Fixed cache hit issues in some long-context call scenarios, improving stability for continuous requests.
- Fixed request timeout issues in some long-context call scenarios.