mirror of
https://github.com/mudler/cogito.git
synced 2026-07-23 10:25:44 -04:00
LocalAI [bot]
37b69f1aec
* feat: add automatic conversation compaction based on token threshold This commit adds automatic conversation compaction to prevent context overflow during long-running tool execution sessions. Key changes: - Added LLMUsage struct to track token usage from LLM responses - Modified LLM interface to return token usage alongside Fragment - Added WithCompactionThreshold option to set token count threshold - Added WithCompactionKeepMessages option to configure recent messages to keep - Added compaction logic in ExecuteTools after LLM calls - Added helper functions: compactFragment, checkAndCompact, estimateTokens - Added PromptConversationCompaction for generating conversation summaries - Updated OpenAI and LocalAI clients to return token usage - Updated mock client for testing When compactionThreshold is set (> 0), the conversation will be automatically compacted when estimated token count exceeds the threshold. The compaction generates a summary of the conversation history using an LLM call while preserving recent messages. Signed-off-by: Autonomous Coding Agent <agent@autonomous> * fix: use actual usage tokens from LLM response for compaction - Store LastUsage in Status struct from LLM responses - checkAndCompact now uses actual TotalTokens from LLM response - Removed estimateTokens function (no longer needed) - Fallback estimate only used on first iteration when no usage data available * fix: capture usage tokens after sink state LLM call for compaction The sink state handling was not capturing usage tokens from the LLM response, which meant the compaction check would use the rough estimate instead of actual usage tokens. This change ensures LastUsage is stored after the llm.Ask call in the hasSinkState block, allowing proper token-based compaction. * fix: move compaction check to beginning of tool loop - Removed compaction check after max iterations (not needed) - Removed compaction check after sink state (not needed) - Added compaction check at beginning of tool loop (after totalIterations++) - Uses actual usage tokens from LLM response * fix: update Ask to return usage tokens from LocalAIClient * fix: set LastUsage in Ask function return fragment This addresses reviewer feedback that Ask() should automatically update the Fragment's LastUsage, not have callers do it. The OpenAIClient and LocalAIClient Ask functions now set Status.LastUsage before returning. * refactor: Ask() updates Fragment.Status.LastUsage directly Instead of returning LLMUsage from Ask(), the LLM clients now update the Fragment's Status.LastUsage directly. This simplifies the interface and ensures usage is always tracked in the fragment. Changes: - LLM.Ask() now returns (Fragment, error) instead of (Fragment, LLMUsage, error) - Clients (openai_client.go, localai_client.go) set LastUsage on the returned fragment - Mock client also updated to set usage in Status - All callers updated to use new 2-value return signature This addresses reviewer feedback on PR #41. * Apply suggestion from @mudler * Apply suggestion from @mudler * Apply suggestion from @mudler * Apply suggestion from @mudler * Apply suggestions from code review * Apply suggestion from @mudler * test: add mocked tests for compaction functionality - Add DefaultPrompts() function to prompt package for tests - Export CompactFragment and CheckAndCompact functions for testing - Add comprehensive unit tests for compaction logic using mocks - Remove duplicate Ginkgo compaction tests that have import issues * test: add compaction tests to tools_test.go suite Add Ginkgo tests for compaction functionality within the existing tools_test.go suite. Tests cover: - No compaction when threshold is disabled (0) - No compaction when tokens below threshold - Compaction when token threshold is exceeded - Parent fragment preservation after compaction - Status preservation after compaction - Rough token estimate usage when LastUsage is not set This addresses the reviewer's request to keep tests consistent with other ginkgo tests in tools_test.go. * chore: run go fmt and add compaction docs to README * Make CompactFragment and CheckAndCompact private Per reviewer request: - Changed CompactFragment to compactFragment (private) - Changed CheckAndCompact to checkAndCompact (private) - Removed tools_compaction_test.go (tests should be in tools_test.go) The compaction functionality is still available internally via ExecuteTools with WithCompactionThreshold option. * chore: remove exported functions from README, keep them private * chore: verify all changes applied - build passes * chore: verify build and vet pass --------- Signed-off-by: Autonomous Coding Agent <agent@autonomous> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>