Prompt Caching

​

Use Cases

• Long system prompts: Multiple requests reuse the same role definition, output rules, safety boundaries, or product instructions.
• Fixed tool definitions: In Function Calling scenarios, the tool list, parameter schemas, and tool descriptions remain stable across multiple turns.
• Long-document follow-up questions: Users ask multiple questions about the same document, contract, paper, or meeting notes.
• Codebase analysis: Multiple requests reuse the same code snippets, project structure, or constraints.
• RAG follow-up questions: The first request passes in long retrieval results, and later requests continue asking about the same material.

​

Supported Models

​

How It Works

• A stable system prompt.
• Fixed tool definitions, output formats, or constraint descriptions.
• Reused long documents, code snippets, knowledge base content, or retrieval results.
• A user question, small amount of new context, or conversation update that changes on each request.

1. Put long-lived reusable content in the first half of the request and keep its order stable.
2. Put the user question or temporary context that changes on each request after the stable content.
3. Reuse the same prefix and message order across requests, and avoid unnecessary rewriting, reordering, or insertion of dynamic metadata.
4. Use the response usage or usage page to check whether cache-related fields appear.
5. Based on latency, cost, and hit behavior, decide whether to continue using long context, switch to RAG, or adjust how context is organized.

​

Relationship with Long Context and RAG

​

Key Fields

• messages: Carries system prompts, history, and reusable context. To improve the chance of reuse, keep the reusable part stable in content, order, and structure.
• model: Selects the model. Prompt Caching support, hit rules, and usage fields may vary by model.
• usage.prompt_tokens: The input token count for the request. Long-context requests usually increase this value.
• usage.prompt_tokens_details.cached_tokens: If returned, this field can help you observe how many input tokens may have hit the cache. Some models may not return this field. Use the API guides as the source of truth for field meaning and billing.

​

Usage Recommendations

• Keep reusable content and message order stable: Put system prompts, tool definitions, long documents, or fixed constraints in the first half of the request, and keep their content, order, and structure consistent across requests.
• Put dynamic content later: Put the current user question, timestamps, temporary variables, retrieval additions, or new conversation content after the stable prefix.
• Avoid unnecessary rewrites: Frequent changes to whitespace, headings, field order, or tool descriptions may reduce the chance of cache hits.
• Control repeated context size: Caching can reduce repeated processing cost or latency, but it should not be used as a reason to put unlimited content into the context.
• Observe usage and latency together: If the response returns cached_tokens, or if the usage page shows cache-related metrics, evaluate effectiveness together with request latency and cost.
• Use with RAG when appropriate: For large knowledge bases, retrieve the most relevant snippets first, while keeping reusable system prompts, tool definitions, or fixed materials stable.
• Reserve output budget: Prompt Caching mainly applies to input context. Generated output still consumes the max_completion_tokens budget.

​

Limitations

• Prompt Caching does not guarantee a cache hit for every request. Hit behavior may be affected by the model, request content, message order, API entry point, and server-side scheduling.
• The first request, or the first few requests, may not return cached_tokens. Even when later requests hit the cache, hit timing and hit scope may vary by model.
• Prompt Caching does not make the model remember content outside the request. Materials the model needs must still be provided in the current request, or referenced through a confirmed caching mechanism.
• Prompt Caching does not change response quality and does not guarantee identical output for the same input.
• cached_tokens can be used as a signal for observing cache hits, but it is not a complete billing rule. Cache lifetime, whether explicit enablement is required, whether additional fees apply, and how cache hits are billed must be checked in the API guides and model details.
• If the input exceeds the model’s context limit, Prompt Caching does not replace the context window. You still need to reduce the input, split the task, or choose a model with a larger context window.
• Do not put sensitive information, one-time credentials, or private user data that should not be reused in reusable context.

​

Example

export API_KEY="YOUR_API_KEY"

1. Put fixed rules and long material first.
2. Put the question that changes on each request later.
3. Reuse the same stable_context across requests.
4. If the response returns cached_tokens, print it for observation.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["API_KEY"],
    base_url="https://api.scitix.ai/model-api",
)

stable_context = """
You are a precise document analysis assistant.
Answer questions only based on the material inside <document>.
If the answer is not in the material, say that it cannot be confirmed from the material.

<document>
Put the long document, code snippets, product description, or retrieval results
that will be reused across requests here.
Try to keep this part unchanged in later requests.
</document>
"""


def ask(question: str):
    response = client.chat.completions.create(
        model="glm-5.2",
        messages=[
            {"role": "system", "content": stable_context},
            {"role": "user", "content": question},
        ],
        max_completion_tokens=512,
    )

    print(response.choices[0].message.content)

    prompt_details = getattr(response.usage, "prompt_tokens_details", None)
    if prompt_details:
        cached_tokens = getattr(prompt_details, "cached_tokens", None)
        print("cached_tokens:", cached_tokens)


ask("Summarize the three key conclusions in this material.")
ask("List the questions that still need confirmation from this material.")