Cutting LLM spend ~95% with 24h prompt caching

The cost problem

Meraki generates AI coaching reports after every sales call. Each report requires a large system prompt: coaching rubric, product knowledge base, rep context, and call transcript. The system prompt alone was ~4,000 tokens. Multiplied across 50K+ calls, that adds up fast.

The transcript changes every call. The system prompt changes maybe once a week when we update the rubric. We were paying full price to re-send the same 4,000 tokens on every single request.

How prompt caching works

Anthropic's prompt caching feature (and similar offerings from other providers) lets you mark a prefix of your prompt as cacheable. On the first request, they compute and store the KV cache for that prefix. Subsequent requests within the TTL skip recomputation and pay a fraction of the input token price.

response = anthropic.messages.create(
    model="claude-opus-4-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": COACHING_RUBRIC + PRODUCT_KNOWLEDGE,
            "cache_control": {"type": "ephemeral"}  # 5 min TTL
        }
    ],
    messages=[
        {"role": "user", "content": call_transcript}
    ]
)

The 24h trick

The default TTL for ephemeral caching is 5 minutes. For our use case, the system prompt was stable for days. We structured the prompt so the stable parts came first (rubric, product knowledge) and the dynamic parts came last (rep context, transcript).

We then batched calls during off-peak hours and pre-warmed the cache at the start of each day. Combined with the natural clustering of calls during business hours, cache hit rate exceeded 95%.

The numbers

Before caching: ~4,200 input tokens per report at full price. After caching: ~4,200 tokens on cache miss (rare), ~200 tokens on cache hit (the transcript prefix check). At a cache hit rate of 95%, blended input cost dropped by ~95%.

The output tokens — the actual report — stayed the same. But input tokens dominated the bill for this workload.

Caveats

Caching only helps when your system prompt is large and stable. If your system prompt changes frequently or is short, the overhead of cache management exceeds the savings. Also, different providers implement caching differently — Gemini uses implicit caching via context caching, OpenAI has a similar mechanism. Test each one.