* feat: add automatic conversation compaction based on token threshold
This commit adds automatic conversation compaction to prevent context overflow
during long-running tool execution sessions.
Key changes:
- Added LLMUsage struct to track token usage from LLM responses
- Modified LLM interface to return token usage alongside Fragment
- Added WithCompactionThreshold option to set token count threshold
- Added WithCompactionKeepMessages option to configure recent messages to keep
- Added compaction logic in ExecuteTools after LLM calls
- Added helper functions: compactFragment, checkAndCompact, estimateTokens
- Added PromptConversationCompaction for generating conversation summaries
- Updated OpenAI and LocalAI clients to return token usage
- Updated mock client for testing
When compactionThreshold is set (> 0), the conversation will be automatically
compacted when estimated token count exceeds the threshold. The compaction
generates a summary of the conversation history using an LLM call while
preserving recent messages.
Signed-off-by: Autonomous Coding Agent <agent@autonomous>
* fix: use actual usage tokens from LLM response for compaction
- Store LastUsage in Status struct from LLM responses
- checkAndCompact now uses actual TotalTokens from LLM response
- Removed estimateTokens function (no longer needed)
- Fallback estimate only used on first iteration when no usage data available
* fix: capture usage tokens after sink state LLM call for compaction
The sink state handling was not capturing usage tokens from the LLM response,
which meant the compaction check would use the rough estimate instead of
actual usage tokens. This change ensures LastUsage is stored after the
llm.Ask call in the hasSinkState block, allowing proper token-based compaction.
* fix: move compaction check to beginning of tool loop
- Removed compaction check after max iterations (not needed)
- Removed compaction check after sink state (not needed)
- Added compaction check at beginning of tool loop (after totalIterations++)
- Uses actual usage tokens from LLM response
* fix: update Ask to return usage tokens from LocalAIClient
* fix: set LastUsage in Ask function return fragment
This addresses reviewer feedback that Ask() should automatically update
the Fragment's LastUsage, not have callers do it. The OpenAIClient and
LocalAIClient Ask functions now set Status.LastUsage before returning.
* refactor: Ask() updates Fragment.Status.LastUsage directly
Instead of returning LLMUsage from Ask(), the LLM clients now update
the Fragment's Status.LastUsage directly. This simplifies the interface
and ensures usage is always tracked in the fragment.
Changes:
- LLM.Ask() now returns (Fragment, error) instead of (Fragment, LLMUsage, error)
- Clients (openai_client.go, localai_client.go) set LastUsage on the returned fragment
- Mock client also updated to set usage in Status
- All callers updated to use new 2-value return signature
This addresses reviewer feedback on PR #41.
* Apply suggestion from @mudler
* Apply suggestion from @mudler
* Apply suggestion from @mudler
* Apply suggestion from @mudler
* Apply suggestions from code review
* Apply suggestion from @mudler
* test: add mocked tests for compaction functionality
- Add DefaultPrompts() function to prompt package for tests
- Export CompactFragment and CheckAndCompact functions for testing
- Add comprehensive unit tests for compaction logic using mocks
- Remove duplicate Ginkgo compaction tests that have import issues
* test: add compaction tests to tools_test.go suite
Add Ginkgo tests for compaction functionality within the existing
tools_test.go suite. Tests cover:
- No compaction when threshold is disabled (0)
- No compaction when tokens below threshold
- Compaction when token threshold is exceeded
- Parent fragment preservation after compaction
- Status preservation after compaction
- Rough token estimate usage when LastUsage is not set
This addresses the reviewer's request to keep tests consistent with
other ginkgo tests in tools_test.go.
* chore: run go fmt and add compaction docs to README
* Make CompactFragment and CheckAndCompact private
Per reviewer request:
- Changed CompactFragment to compactFragment (private)
- Changed CheckAndCompact to checkAndCompact (private)
- Removed tools_compaction_test.go (tests should be in tools_test.go)
The compaction functionality is still available internally via
ExecuteTools with WithCompactionThreshold option.
* chore: remove exported functions from README, keep them private
* chore: verify all changes applied - build passes
* chore: verify build and vet pass
---------
Signed-off-by: Autonomous Coding Agent <agent@autonomous>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
This allows to put arbitrary data as results of any tool and be later
picked up when inspecting the execution results of the agent.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat: declare tool args via struct
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* simplify even more the API
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat: enhance tool selection
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* try to adapt tests
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Make tests more stable
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add autoplan test
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Don't load MCP prompts if we don't have any
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Reorder things
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* TEST
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Revert "TEST"
This reverts commit b08bbc56d5.
* TEST
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* TEST2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Try to fix tests
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add back guidelines
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Improve next action logic
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* tests
* tests2
* forcereasoning
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* maxiterations
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fix tests
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Better error handling
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test
* error handling
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Force text reply
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Two-step param generation approach
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Code cleanup
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add a small note/comment
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Improve tests
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Small refinements
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Split e2e and unit tests
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Abstract away LLM to an interface
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Update README
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Enhance content improve generation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add unit test for reviewer
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Better handling of result status of tool calls
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add tools test file
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
LocalAI [bot]