- Updated reasoning handling in the Anthropic and OpenAI clients to support real-time streaming of reasoning tokens.
- Refactored reasoning structures to improve integration with existing models, ensuring consistent behavior across different LLM implementations.
- Added comprehensive tests for reasoning and caching functionalities, validating the expected behavior in various scenarios.
- Removed deprecated files related to previous caching implementations to streamline the codebase.
This update improves the overall performance and reliability of reasoning capabilities in LLM interactions.
* llms: add prompt caching and reasoning token support
Add comprehensive infrastructure for prompt caching and reasoning/thinking tokens:
- Add ReasoningModel interface for models supporting extended reasoning
- Add CacheControl and CachedContent types for prompt caching
- Add ThinkingConfig with modes (none, low, medium, high, auto)
- Add token budget calculation and usage tracking for thinking tokens
- Add WithPromptCaching, WithThinking, and related call options
- Add model detection for reasoning-capable models (OpenAI o1/o3, Claude 3.7+, DeepSeek)
- Update options documentation for better provider compatibility notes
- Remove testify imports tracking file
Supports provider-specific implementations while maintaining consistent API.
* testing: add comprehensive LLM testing framework
Add standardized testing infrastructure for all LLM providers following Go's testing/fstest design philosophy:
- Add llmtest package with automatic capability discovery and parallel testing
- Add TestLLM function that probes and tests all supported LLM features
- Add provider-specific test files for all LLM implementations (anthropic, bedrock, cloudflare, cohere, ernie, fake, googleai, huggingface, llamafile, local, maritaca, mistral, ollama, openai, watsonx)
- Add comprehensive tests for prompt caching, reasoning tokens, and token utilization
- Add MockLLM implementation for testing without API calls
- Add ValidateLLM function for basic model validation
- Support automatic detection of streaming, tool calls, reasoning, and caching capabilities
The framework provides a simple API (TestLLM) that automatically discovers and tests all supported capabilities, enabling consistent testing across all provider implementations.
* llms: implement provider-specific prompt caching and reasoning support
Add comprehensive provider-specific implementations for prompt caching and reasoning tokens:
**Anthropic (Claude)**:
- Add prompt caching with ephemeral cache controls and beta headers
- Add extended thinking support for Claude 3.7+ with budget token configuration
- Add interleaved thinking for tool calls and 128K output support
- Add thinking content extraction from <thinking> tags
- Add cache token tracking (creation/read input tokens)
**Google AI (Gemini)**:
- Add CachingHelper for pre-created cached content management
- Add reasoning detection for Gemini 2.0+ models
- Add cached content support via metadata
- Add standardized token usage reporting with cache information
**Ollama**:
- Add ContextCache for in-memory conversation context caching
- Add reasoning support detection for DeepSeek R1, QwQ, and thinking models
- Add Think parameter support for reasoning-capable models
- Add cache statistics and hit/miss tracking
**OpenAI**:
- Add reasoning support detection for o1/o3 series and future GPT-5
- Add system message handling for models without system support (o1/o3)
- Add thinking content extraction and standardized token reporting
- Add metadata filtering to prevent internal fields from reaching API
- Add ReasoningEffort parameter preparation for future models
**Cross-provider features**:
- Implement ReasoningModel interface across all providers
- Add standardized GenerationInfo fields (ThinkingContent, ThinkingTokens, etc.)
- Add provider-specific options (WithPromptCaching, WithExtendedOutput, etc.)
- Add comprehensive test coverage for caching and reasoning features
- Add model capability detection and validation
All providers now support the unified prompt caching and reasoning interfaces while maintaining their specific implementation details and capabilities.
* examples: add comprehensive prompt caching and reasoning examples
Add practical examples demonstrating prompt caching and reasoning token features across multiple LLM providers:
**Anthropic Examples**:
- Add anthropic-extended-capabilities example showing combined extended thinking + 128K output
- Add anthropic-interleaved-thinking example demonstrating thinking between tool calls
- Show token budget management and thinking content extraction
- Include comprehensive README documentation for each example
**Multi-Provider Examples**:
- Add googleai-reasoning-caching example with Gemini 2.0+ reasoning and cached content
- Add ollama-reasoning-caching example with DeepSeek R1/QwQ models and context caching
- Add prompt-caching example showing Anthropic's 90% cost reduction on cached tokens
- Add reasoning-tokens example comparing o1-mini, Claude 3.7, and standard models
**Key Features Demonstrated**:
- Prompt caching for cost optimization and faster responses
- Reasoning/thinking tokens for improved response quality
- Token usage analysis and cache hit/miss tracking
- Provider-specific capabilities and configuration options
- Real-world use cases like data analysis, Q&A systems, and multi-step reasoning
All examples include detailed documentation, error handling, and clear output formatting to help developers understand and implement these advanced LLM features.
* gofmt: fix formatting in messages.go
Dmitry Ng