langchain-ai/deepagentsjs
1
0
Fork 0
mirror of https://github.com/langchain-ai/deepagentsjs.git synced 2026-07-18 16:04:44 -04:00
Code Issues 42 Packages Projects Releases 10 Wiki Activity

[PR #285] [MERGED] fix(deepagents): prompt caching with anthropic models results in higher than expected cache miss rates #326

New Issue
Closed
opened 2026-06-05 17:22:41 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 17:22:41 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/deepagentsjs/pull/285
Author: @colifran
Created: 3/7/2026
Status: ✅ Merged
Merged: 3/9/2026
Merged by: @colifran

Base: main ← Head: colifran/anthropic-cache-improvements

📝 Commits (8)

  • 0f8baa9 improve caching performance
  • 56cc246 add changeset
  • 4616c85 add check for anthropic models
  • 1812b42 use content blocks
  • bbdcd35 test isAnthropicModel
  • 76e56d5 update logic to ensure all stable content BEFORE the memory block is cached - not just the system prompt
  • 2837bc3 fix type check error
  • 9c8f6f7 add cache breakpoint after subagent middleware for default and general purpose

📊 Changes

7 files changed (+483 additions, -28 deletions)

View changed files

➕ .changeset/polite-lands-do.md (+5 -0)
➕ libs/deepagents/src/agent.test.ts (+120 -0)
📝 libs/deepagents/src/agent.ts (+43 -15)
➕ libs/deepagents/src/middleware/cache.test.ts (+73 -0)
➕ libs/deepagents/src/middleware/cache.ts (+49 -0)
📝 libs/deepagents/src/middleware/memory.test.ts (+164 -8)
📝 libs/deepagents/src/middleware/memory.ts (+29 -5)

📄 Description

Description

Prompt caching with Anthropic models was resulting in higher than expected cache miss rates due to two issues:

  1. The minimum messages threshold was too high (3)
  2. The system prompt and memory content shared a single cache breakpoint, meaning any memory update would invalidate the entire system prompt cache

This PR improves cache hit rates by caching earlier and introducing separate cache breakpoints for static content and memory, so a memory update only invalidates the memory breakpoint rather than the full system prompt.

Changes

  • Reduce minMessagesToCache from 3 to 1 so caching begins on the first turn
  • Add isAnthropicModel() helper to gate Anthropic-specific caching behavior
  • Add createCacheBreakpointMiddleware which places a cache_control breakpoint on the last block of the system message after all static middleware (todo, filesystem, subagent instructions, skills) have run
  • Add addCacheControl option to createMemoryMiddleware which tags the memory block with its own separate cache_control breakpoint
  • Wire both into createDeepAgent for Anthropic models, resulting in two breakpoints: one covering all stable static content, one covering memory

Tests

  • Unit tests for isAnthropicModel covering string detection (claude-, anthropic:, non-Anthropic) and model object detection
  • Unit tests for createCacheBreakpointMiddleware covering single-block, multi-block (last-only), immutability, and empty passthrough
  • Unit tests for createMemoryMiddleware cache control covering addCacheControl: true/false, multi-block system messages, and block stability when memory changes
  • Integration test verifying both breakpoints are present and correctly placed when using an Anthropic model with system prompt and memory
Cache verification — model: claude-haiku-4-5-20251001

Scenario 1: system prompt only (no memory)
Expected: cache write on first turn, cache read on second turn
────────────────────────────────────────────────────────────
Turn 1:
  input tokens:        7820
  cache read tokens:   0
  cache write tokens:  6622
  output tokens:       13

Turn 2:
  input tokens:        7845
  cache read tokens:   6622
  cache write tokens:  0
  output tokens:       13

Scenario 2: system prompt + memory
Expected: two cache breakpoints — static content and memory block
────────────────────────────────────────────────────────────
Turn 1:
  input tokens:        8897
  cache read tokens:   6622
  cache write tokens:  1950
  output tokens:       22

Turn 2 (same memory):
  input tokens:        8927
  cache read tokens:   8572
  cache write tokens:  0
  output tokens:       28

Scenario 3: multi-turn, cache reads should increase across turns
Expected: cache_read grows with each turn as history accumulates
────────────────────────────────────────────────────────────

Turn 1: "What is the capital of France?"
  input tokens:        7818
  cache read tokens:   6622
  cache write tokens:  0
  output tokens:       11

Turn 2: "What is the capital of Germany?"
  input tokens:        7839
  cache read tokens:   6622
  cache write tokens:  0
  output tokens:       11

Turn 3: "What is the capital of Japan?"
  input tokens:        7860
  cache read tokens:   6622
  cache write tokens:  0
  output tokens:       11

Scenario 4: main agent delegates to a subagent
Expected: subagent response has cache writes on its own static content
────────────────────────────────────────────────────────────

Main agent cache usage (last AI message):
  input tokens:        8011
  cache read tokens:   6635
  cache write tokens:  0
  output tokens:       53

Total AI messages in thread: 2
(subagent responses appear as tool result messages — its cache usage
 is on its own API call and not visible here, but no errors = it ran)

────────────────────────────────────────────────────────────
Done.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/deepagentsjs/pull/285 **Author:** [@colifran](https://github.com/colifran) **Created:** 3/7/2026 **Status:** ✅ Merged **Merged:** 3/9/2026 **Merged by:** [@colifran](https://github.com/colifran) **Base:** `main` ← **Head:** `colifran/anthropic-cache-improvements` --- ### 📝 Commits (8) - [`0f8baa9`](https://github.com/langchain-ai/deepagentsjs/commit/0f8baa90e88c0fc137e666fbbf0a37ce80826402) improve caching performance - [`56cc246`](https://github.com/langchain-ai/deepagentsjs/commit/56cc24620b4d22b410c2b754cfa74b0757984c1b) add changeset - [`4616c85`](https://github.com/langchain-ai/deepagentsjs/commit/4616c85c40e73e6d778edb4bde7a67cc1a015c16) add check for anthropic models - [`1812b42`](https://github.com/langchain-ai/deepagentsjs/commit/1812b424320721666d87208747642a3173863bba) use content blocks - [`bbdcd35`](https://github.com/langchain-ai/deepagentsjs/commit/bbdcd358964999d8db223d8f41048ad7f0023b54) test isAnthropicModel - [`76e56d5`](https://github.com/langchain-ai/deepagentsjs/commit/76e56d5eb0454093740c32d9bef46146bc521746) update logic to ensure all stable content BEFORE the memory block is cached - not just the system prompt - [`2837bc3`](https://github.com/langchain-ai/deepagentsjs/commit/2837bc3b698ba487beb915952643047b762ac74c) fix type check error - [`9c8f6f7`](https://github.com/langchain-ai/deepagentsjs/commit/9c8f6f7c3c4c4fab6e4b5480c8a95d9fddd46c05) add cache breakpoint after subagent middleware for default and general purpose ### 📊 Changes **7 files changed** (+483 additions, -28 deletions) <details> <summary>View changed files</summary> ➕ `.changeset/polite-lands-do.md` (+5 -0) ➕ `libs/deepagents/src/agent.test.ts` (+120 -0) 📝 `libs/deepagents/src/agent.ts` (+43 -15) ➕ `libs/deepagents/src/middleware/cache.test.ts` (+73 -0) ➕ `libs/deepagents/src/middleware/cache.ts` (+49 -0) 📝 `libs/deepagents/src/middleware/memory.test.ts` (+164 -8) 📝 `libs/deepagents/src/middleware/memory.ts` (+29 -5) </details> ### 📄 Description ### Description Prompt caching with Anthropic models was resulting in higher than expected cache miss rates due to two issues: 1. The minimum messages threshold was too high (3) 2. The system prompt and memory content shared a single cache breakpoint, meaning any memory update would invalidate the entire system prompt cache This PR improves cache hit rates by caching earlier and introducing separate cache breakpoints for static content and memory, so a memory update only invalidates the memory breakpoint rather than the full system prompt. ### Changes - Reduce `minMessagesToCache` from 3 to 1 so caching begins on the first turn - Add `isAnthropicModel()` helper to gate Anthropic-specific caching behavior - Add `createCacheBreakpointMiddleware` which places a `cache_control` breakpoint on the last block of the system message after all static middleware (todo, filesystem, subagent instructions, skills) have run - Add `addCacheControl` option to `createMemoryMiddleware` which tags the memory block with its own separate `cache_control` breakpoint - Wire both into `createDeepAgent` for Anthropic models, resulting in two breakpoints: one covering all stable static content, one covering memory ### Tests - Unit tests for `isAnthropicModel` covering string detection (claude-*, anthropic:*, non-Anthropic) and model object detection - Unit tests for `createCacheBreakpointMiddleware` covering single-block, multi-block (last-only), immutability, and empty passthrough - Unit tests for `createMemoryMiddleware` cache control covering `addCacheControl`: true/false, multi-block system messages, and block stability when memory changes - Integration test verifying both breakpoints are present and correctly placed when using an Anthropic model with system prompt and memory ``` Cache verification — model: claude-haiku-4-5-20251001 Scenario 1: system prompt only (no memory) Expected: cache write on first turn, cache read on second turn ──────────────────────────────────────────────────────────── Turn 1: input tokens: 7820 cache read tokens: 0 cache write tokens: 6622 output tokens: 13 Turn 2: input tokens: 7845 cache read tokens: 6622 cache write tokens: 0 output tokens: 13 Scenario 2: system prompt + memory Expected: two cache breakpoints — static content and memory block ──────────────────────────────────────────────────────────── Turn 1: input tokens: 8897 cache read tokens: 6622 cache write tokens: 1950 output tokens: 22 Turn 2 (same memory): input tokens: 8927 cache read tokens: 8572 cache write tokens: 0 output tokens: 28 Scenario 3: multi-turn, cache reads should increase across turns Expected: cache_read grows with each turn as history accumulates ──────────────────────────────────────────────────────────── Turn 1: "What is the capital of France?" input tokens: 7818 cache read tokens: 6622 cache write tokens: 0 output tokens: 11 Turn 2: "What is the capital of Germany?" input tokens: 7839 cache read tokens: 6622 cache write tokens: 0 output tokens: 11 Turn 3: "What is the capital of Japan?" input tokens: 7860 cache read tokens: 6622 cache write tokens: 0 output tokens: 11 Scenario 4: main agent delegates to a subagent Expected: subagent response has cache writes on its own static content ──────────────────────────────────────────────────────────── Main agent cache usage (last AI message): input tokens: 8011 cache read tokens: 6635 cache write tokens: 0 output tokens: 53 Total AI messages in thread: 2 (subagent responses appear as tool result messages — its cache usage is on its own API call and not visible here, but no errors = it ran) ──────────────────────────────────────────────────────────── Done. ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 17:22:41 -04:00
yindo closed this issue 2026-06-05 17:22:41 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
help wanted
needs-response
pull-request
Mirrored from GitHub Pull Request
 stale
triage: high-impact
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/deepagentsjs#326
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?