Compare commits

...
Author SHA1 Message Date
MervinPraisonandClaude Opus 5.5 0b8cd48c16 fix(gateway): stop GitGuardian flagging the DeviceAuth lazy-import tuple
GitGuardian reads ("praisonaiagents.gateway.protocols", "DeviceAuth") as an
'Authentication Tuple'. Every PR that merges main carries that line and fails
the GitGuardian check, which the merge gate treats as failed CI. Use a
module-path constant instead; DeviceAuth still resolves lazily.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-11-05 09:03:46 +00:00
praisonai-triage-agent[bot] 20d023a864 Merge pull request #6247 from MervinPraison/docs/readme-exam-wording
docs(readme): say what the handover test tests
2026-11-05 08:57:04 +00:00
praisonai-triage-agent[bot] b05518baed Merge pull request #6246 from MervinPraison/claude/issue-6245-20261105
fix: fail loudly on unknown provider prefix and dropped safety YAML fields
2026-11-05 08:49:45 +00:00
MervinPraisonandClaude Opus 5.5 740a887ade docs(readme): say what the handover test tests
Step 2 now says the model takes an exam made from real past work it never
studied, matching the website and docs.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-11-05 08:43:59 +00:00
MervinPraisonandClaude Opus 5.5 6980ec13e2 fix: only fail closed on active safety gates (keep --trust working on crewai/autogen)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-05 08:38:13 +00:00
praisonai-triage-agent[bot] a362afc899 Merge pull request #6239 from MervinPraison/claude/issue-6238-20261105
fix: confine caller-supplied names to storage dirs and use shared async bridge
2026-11-05 08:31:38 +00:00
praisonai-triage-agent[bot] 4d9b514106 Merge pull request #6243 from MervinPraison/claude/issue-6241-20261105
fix: accept OpenAI-style 'messages' chat datasets in praisonai-train llm
2026-11-05 08:27:30 +00:00
praisonai-triage-agent[bot] d3ecf23533 Merge pull request #6244 from MervinPraison/claude/issue-6242-20261105
fix: add --no-push / ollama_push: false to keep Ollama exports local
2026-11-05 08:27:13 +00:00
MervinPraisonandClaude Opus 5.5 85922f349d fix: fail loudly on unknown provider prefix and dropped safety fields (fixes #6245)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-05 08:17:57 +00:00
MervinPraisonandClaude Opus 5.5 50a0949063 fix: bound safe_join filename length and keep non-ASCII names unescaped (#6238)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-05 08:16:24 +00:00
MervinPraisonandClaude Opus 5.5 8da80d1890 fix: let explicit --push override config ollama_push and test caller paths (#6242)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-05 08:09:17 +00:00
MervinPraisonandClaude Opus 5.5 861584bab4 fix: mention 'messages' chat shape in empty-dataset error
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-05 08:09:02 +00:00
MervinPraisonandClaude Opus 5.5 79669e7a7d fix: reversible safe_join encoding and stop tests leaking a global ConsoleBackend (#6238)
- safe_join now percent-encodes names (bijective), so distinct names such as
  'a/b' and 'a_b~c14cddc033f6' can no longer share one file; list_todos()
  decodes filenames so listed names round-trip through load/delete_todo.
- Legacy <name>.json fallback skips names containing '%' to avoid aliasing.
- Tests restoring the approval backend snapshotted get_backend(), whose
  ConsoleBackend fallback was then installed globally on restore and leaked
  into tests/unit/skills/test_approval_scoping.py under xdist. Snapshot
  _global_backend directly instead.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-05 08:08:46 +00:00
25f8d93a4c docs(readme): say what "your own AI" is in the handover section (#6240)
* docs(readme): say what "your own AI" is in the handover section

The five handover steps never said where the own model comes from. Add one
line: an open model you run yourself, used as it is or trained on your own
past work with PraisonAI Train, and only after it passes the test.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix: match handover wording to the Test step ("its tests")

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Co-authored-by: MervinPraison <[email protected]>
2026-11-05 08:05:23 +00:00
MervinPraisonandClaude Opus 5.5 ffa3068203 fix: add --no-push / ollama_push: false to keep Ollama exports local (fixes #6242)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-05 08:03:53 +00:00
MervinPraisonandClaude Opus 5.5 82d066299b fix: accept OpenAI-style 'messages' chat datasets in praisonai-train llm (fixes #6241)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-05 08:03:21 +00:00
Elon-Jiang 87bf746dba fix(ts): release hook deadline timers after settlement (#6195) 2026-11-05 08:59:33 +01:00
Wu Shuwen 2b78ce8804 fix: discover compute backends on first placement lookup (#6237) 2026-11-05 08:59:26 +01:00
Wu Shuwen c244c4db7e test: read Python source consistently on Windows (#6235)
* test: read Python source using its declared encoding

* test: read Python source using its declared encoding

* test: read Python source using its declared encoding
2026-11-05 08:59:21 +01:00
MervinPraisonandClaude Opus 5.5 42e0695bbe fix: keep listed/legacy todo names and custom plan filenames loadable (#6238)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-05 07:46:30 +00:00
MervinPraisonandClaude Opus 5.5 f024f6b3a1 fix: confine caller-supplied names to storage dirs and use shared async bridge (fixes #6238)
- utils/atomic_io.safe_join: injective, separator-free filename stems
  (same scheme as session store); legacy safe names are unchanged.
- PlanStorage save/load/delete_todo and save_plan use safe_join, so
  names like '../../x' can no longer escape todos_dir/plans_dir.
- ContextAgent creates PRPs/ and writes PRPs atomically via safe_join.
- generate_title and crawl4ai_*_sync use run_coroutine_from_any_context,
  so they work inside a running event loop instead of crashing or
  silently returning the fallback.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-05 07:29:53 +00:00
7d26272fa0 fix(ts): honor the configured FileCache TTL (#6180)
Co-authored-by: Elon-Jiang <[email protected]>
Co-authored-by: MervinPraison <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
2026-11-04 21:33:53 +01:00
Elon-Jiang 85c01743c8 fix(ts): re-enter downstream tool middleware on retry (#6196)
* fix(ts): re-enter downstream tool middleware on retry

* fix(tools): stop middleware retries after cancellation
2026-11-04 21:04:36 +01:00
Elon-Jiang 8420d1f93f fix(ts): preserve sentence chunk content and source offsets (#6188) 2026-11-04 21:04:31 +01:00
Elon-JiangandElon-Jiang 4c59b795d5 fix(ts): settle MCP requests when WebSocket connections close (#6184)
Co-authored-by: Elon-Jiang <[email protected]>
2026-11-04 21:04:25 +01:00
Wu Shuwen 790878b092 test: include agent root regression tests in core shard (#6233) 2026-11-04 20:58:23 +01:00
MervinPraison 9a84e99ecf docs: auto-update feature parity trackers [skip ci] 2026-11-04 17:40:37 +00:00
praisonai-triage-agent[bot] fa738028d1 Merge pull request #6232 from MervinPraison/fix/edit-hint-nested-tool-dicts
fix: count only tool dicts in nested lists for the edit hint
2026-11-04 17:39:50 +00:00
MervinPraisonandClaude Opus 5.5 71de7217fd fix: hide permission-denied tools from bridge tool_search/tool_describe
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 16:36:23 +00:00
MervinPraisonandClaude Opus 5.5 3d7ed12a3c fix: cache unpruned formatted tools so lifted denials take effect
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 15:34:37 +00:00
MervinPraisonandClaude Opus 5.5 45019589cf fix: re-prune permission-denied tools on formatted-tools cache hits
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 14:36:43 +00:00
praisonai-triage-agent[bot] e3074e5ac1 Merge pull request #6229 from MervinPraison/claude/issue-6228-20261104
fix: CLI keyless explicit ollama/ models and tolerant local probe
2026-11-04 13:53:48 +00:00
MervinPraisonandClaude Opus 5.5 09c443a71f fix: count only function-schema dicts in nested lists for the edit hint
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 13:48:09 +00:00
MervinPraisonandClaude Opus 5.5 a0ad968af0 fix: skip the edit hint for permission-denied edit tools
_prune_denied_tools drops denied tools from the payload, so the hint must
not name them. Tests now record the formatted tool payload, which also
checks that a nested edit schema is actually sent.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 13:39:58 +00:00
Mervin PraisonandClaude Opus 5.5 70e3b7cc70 docs(ts): finish the repositioning — same main line as the README, praison.ai/docs link (#6231)
* docs(ts): match the README's main line; link praison.ai/docs

Keeps the hosted-model privacy qualifier from the review fix.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs(ts): 'never has to leave' to match README and docs

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
2026-11-04 13:19:29 +00:00
1b3f576e00 docs(readme): finish the repositioning — regulated teams and everyone, pip install praisonai, praison.ai/docs links (#6230)
* docs(readme): pitch to regulated teams and everyone; pip install praisonai; praison.ai/docs links

- Top: one main line, then one line for highly regulated teams (the handover,
  early access) and one for everyone else (free, open source, no API key with
  a local model). Numbers now link to their public sources.
- Install line is pip install praisonai (praisonaiagents noted as the light core).
  The offline section no longer needs the [llm] extra: praisonai ships litellm.
- How the handover works: adds why now and what it is built on today.
- New 'The company' section for investors and partners: who, what, how the open
  framework and the licensed handover fit, traction, contact.
- All 128 docs.praison.ai links moved to praison.ai/docs after checking each
  new URL returns 200; every other link checked too.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs(readme): 'never has to leave' (true for hosted models too)

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs(readme): investor review: label early access first, cite the OpenAI notice, add why now, co-founders and contact email; no unproven partner or release-pace claims

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix: note praisonaiagents[llm] extra for core-only offline use in both READMEs

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Co-authored-by: MervinPraison <[email protected]>
2026-11-04 13:19:02 +00:00
MervinPraison 69ca4f9808 fix: count only tool dicts in nested lists for the edit hint
_format_tools_for_completion flattens a nested tool list but keeps only
its tool dicts, so tools=[["edit_file"]] sends no edit tool. Counting the
nested string told the model to use a tool it did not have.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
(cherry picked from commit 29e4d4e77f)
2026-11-04 13:18:13 +00:00
MervinPraison 8d9b32de91 docs: auto-update feature parity trackers [skip ci] 2026-11-04 13:04:25 +00:00
Elon-Jiang 318e15054c fix(ts): interrupt tool retry backoff on cancellation (#6199) 2026-11-04 14:03:52 +01:00
Elon-Jiang 6984acc862 fix(ts): enforce tool payload limits in UTF-8 bytes (#6198) 2026-11-04 14:03:47 +01:00
Elon-Jiang a563e08bf1 fix(ts): invalidate tool installation cache on registration changes (#6197) 2026-11-04 14:03:41 +01:00
Elon-Jiang 636586d7df fix(ts): keep hook inspection from freezing registration (#6194) 2026-11-04 14:03:35 +01:00
MervinPraisonandClaude Opus 5.5 5baea8dc57 fix: gate explicit Ollama agents on their declared base_url, not OLLAMA_HOST
Addresses review on #6229: run --agent with llm.base_url probed the env/default
host instead of the server the agent actually calls.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 13:03:30 +00:00
Elon-Jiang 26b93f9201 fix(ts): preserve concurrent FileCache list appends (#6193)
* fix(ts): preserve concurrent FileCache list appends

* test: cover FileCache appends queued behind a failed write
2026-11-04 14:03:30 +01:00
Elon-Jiang d6b8d4ff11 fix(ts): remove stale graph neighbors when an edge is rewired (#6191) 2026-11-04 14:03:25 +01:00
Elon-Jiang 534b6e3535 fix(ts): preserve session changes during file saves (#6186)
* fix(ts): preserve session changes during file saves

* fix(ts): avoid redundant autosave queue entries
2026-11-04 14:03:19 +01:00
yunyv 2e2081b38c fix(memory): enforce long-term capacity during promotion (#6185)
* fix(memory): enforce long-term capacity during promotion

* test(memory): cover bounded automatic promotion
2026-11-04 14:03:14 +01:00
Elon-JiangandElon-Jiang 0c5f2998f0 fix(ts): evict empty keys from bounded MemoryCache (#6183)
Co-authored-by: Elon-Jiang <[email protected]>
2026-11-04 14:03:08 +01:00
yunyv eca66b1efc fix(tools): match zero directories in grep fallback recursive globs (#6182)
* fix(tools): match zero directories in grep fallback recursive globs

* test(tools): cover zero-directory recursive grep globs
2026-11-04 14:03:02 +01:00
Elon-JiangandElon-Jiang bdd3d1655d fix(ts): resolve MCP WebSocket responses with zero request IDs (#6178)
Co-authored-by: Elon-Jiang <[email protected]>
2026-11-04 14:02:57 +01:00
sarankumar 82b776fa66 fix(standardise): escape braces in the tools basic example template (#6166)
* fix(standardise): escape braces in the tools basic example template

_get_feature_basic_structure builds every basic example structure as an
f-string. The tools template embeds a calculator whose code has literal
braces (the ops dict, {type(node)} and {e}), which were evaluated as
replacement fields when the dict was built. That raised TypeError
(unsupported format string passed to type.__format__) for every feature,
so basic example prompt generation always failed.

Double the braces so they render literally, and add regression tests.

Signed-off-by: sarankumar1325 <[email protected]>

* test(standardise): assert the tools example has no doubled braces

Signed-off-by: sarankumar1325 <[email protected]>

* fix(standardise): accept only numeric constants in the tools example calculator

A string constant such as 'x' * 1000000000 made the generated calculator
allocate a huge string. Reject non-numeric constants; the existing except
clause turns the TypeError into an error string. Tighten the calculator
test to check an operator entry and execute the generated calculator.

Signed-off-by: sarankumar1325 <[email protected]>

* fix(standardise): bound the tools example calculator's results

Catch OverflowError, reject integer powers whose result would exceed
4096 bits, and treat bools as non-numeric. The test also checks that the
example registers the calculator with tools=[calculator].

Signed-off-by: sarankumar1325 <[email protected]>

* fix(standardise): use an exact bound in the calculator power guard

The guard multiplied abs(left).bit_length() by the exponent, so it
rejected results that fit in 4096 bits (2 ** 3000, 1 ** 5000). Reject
only powers whose lower bound already exceeds the limit, then check the
real result size. The test now checks the Agent call's tools keyword
via the AST instead of a substring.

Signed-off-by: sarankumar1325 <[email protected]>

* fix(standardise): cap bare integer literals in calculator example

Apply the 4096-bit limit to integer constants too, so a huge literal is
rejected like an oversized operation result. Tighten the tools assertion
to require exactly [calculator] with no non-name entries.

Signed-off-by: sarankumar1325 <[email protected]>

---------

Signed-off-by: sarankumar1325 <[email protected]>
2026-11-04 14:02:52 +01:00
sarankumar 9122182753 fix(agent): generate final response after tool calls in _achat_completion (#6165)
* fix(agent): generate final response after tool calls in _achat_completion

The legacy async tool path referenced an undefined `stream` name and
`_get_llm_functions` (defined only in agent.py), and called
`self.console.print()` although `console` is None unless verbose. The
follow-up completion therefore always failed, the error was swallowed and
the raw tool output was returned instead of the model's final answer.

Request a streamed completion (the code consumes it with `async for`),
import the lazy LLM helper from .agent, and only print when a console is
available. Adds a regression test.

Signed-off-by: sarankumar1325 <[email protected]>

* fix(agent): keep the conversation in the post-tool follow-up request

_achat_completion sent only the system prompt and the tool results, so the
final answer never saw the user's question. Pass the original messages
through and append the tool results to them. Also skip stream chunks with
no choices (e.g. Azure content-filter annotations) instead of raising
IndexError, and add a live integration test (skipped without OPENAI_API_KEY).

Signed-off-by: sarankumar1325 <[email protected]>

* fix(agent): fall back to tool results when the follow-up stream has no answer

If every chunk of the post-tool follow-up stream has empty choices,
raise so the existing handler logs the error and returns the tool
results instead of an empty answer.

Tests: cover the legacy _achat_impl branch forwarding the conversation
to _achat_completion, and the no-answer-chunk fallback. The live test
now disables unified dispatch so it actually reaches _achat_completion,
and asserts the answer is model-written rather than the raw tool output.

Signed-off-by: sarankumar1325 <[email protected]>

* fix(agent): fall back to tool results when the follow-up has no answer text

The guard only checked whether any chunk had choices, so a follow-up
stream whose choices carry no delta.content returned an empty string
instead of the tool results. Fall back whenever neither answer text nor
(enabled) reasoning content was produced, and cover the contentless
choice case in the unit test.

Signed-off-by: sarankumar1325 <[email protected]>

* fix(agent): treat a whitespace-only follow-up stream as no answer

A follow-up stream that only yields whitespace passed the empty-text check
and was returned as the final response. Strip the accumulated text and
reasoning before the check so the tool-result fallback runs.

Signed-off-by: sarankumar1325 <[email protected]>

---------

Signed-off-by: sarankumar1325 <[email protected]>
2026-11-04 14:02:46 +01:00
yi-coding 2f5e07092c fix(ts): honor explicit zero memory history limits (#6143) 2026-11-04 14:02:40 +01:00
Elon-JiangandElon-Jiang bd1f8773ce fix(ts): recover FileMemory writes after append failures (#6142)
* fix(ts): recover FileMemory writes after append failures

* fix(ts): isolate memory records after partial append failures

* fix(memory): separate unterminated records after restart

---------

Co-authored-by: Elon-Jiang <[email protected]>
2026-11-04 14:02:35 +01:00
Elon-JiangandElon-Jiang 250c463c4b fix(ts): preserve arbitrary session state keys (#6149)
* fix(ts): preserve arbitrary session state keys

* fix(ts): preserve plain-object session JSON exports

---------

Co-authored-by: Elon-Jiang <[email protected]>
2026-11-04 14:02:30 +01:00
MervinPraisonandClaude Opus 5.5 b284b67011 fix: let explicit ollama/ models run keyless and make local probe tolerant (fixes #6228)
- Gate explicit ollama/ and ollama_chat/ models on the server answering
  /api/tags (OLLAMA_API_BASE, OLLAMA_HOST, or 127.0.0.1:11434) instead of
  requiring OLLAMA_HOST as a credential.
- Unreachable Ollama now says to start it with `ollama serve` rather than
  asking for an API key.
- Raise the local probe timeout from 0.15s to 1s; a running Ollama can take
  ~0.35s on its first call.
- Only print the 'would be used automatically' hint when auto-detection
  actually ran; name the model when one was given.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 12:56:04 +00:00
praisonai-triage-agent[bot] 29dd51cde1 Merge pull request #6221 from MervinPraison/fix/benchmark-scripts-fair-runs
fix(benchmarks): warm up, interleave and drop failed runs in execution benchmark
2026-11-04 12:38:43 +00:00
01fbdf524e docs(ts): lead the npm README and description with the new focus (#6227)
* docs(ts): lead the npm README and description with the new focus

Text only: the README intro paragraph and the package.json description.
No code, exports or keywords change.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(ts): qualify README privacy claim for hosted models

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Co-authored-by: MervinPraison <[email protected]>
2026-11-04 12:35:24 +00:00
9f5a30790d docs(readme): lead with private, offline AI agents; keep every feature below (#6226)
* docs(readme): lead with private, offline AI agents; keep full reference below

New top: title, approved tagline and one-liner, approved numbers,
pip install praisonaiagents, "Using PraisonAI for something else?",
Quick start, Run fully offline (Ollama), How the handover works
(early access), Features, JavaScript SDK, Docs/community/licence.

All previous sections are kept under "Full feature reference" with
their headings unchanged; the longest are folded in <details>. The
hero badges (Elon, desktop downloads, Trendshift, docs) move lower;
every link target from the old README is still present.

pyproject description for praisonaiagents and praisonai updated to the
approved short line. No code, version or dependency changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs(readme): apply review fixes

- vLLM: use hosted_vllm/ (vllm/ runs in-process, not a server)
- Ollama block: don't put the blocking 'ollama serve' before 'ollama pull'
- Licence link absolute so it works on pypi.org
- New JS heading renamed so the existing #javascript-sdk anchor is unchanged
- Tracing links to the Observability overview; quick starts print the answer

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix: clarify praisonai CLI install and offline model pre-download in README

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Co-authored-by: MervinPraison <[email protected]>
2026-11-04 12:35:08 +00:00
MervinPraisonandClaude Opus 5.5 dc35781e2f fix(benchmarks): rotate framework order each timed round
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 12:24:53 +00:00
MervinPraisonandClaude Opus 5.5 6713492023 fix(benchmarks): reuse Agno's model object, validate rounds, test the rules
- Each Agno OpenAIChat opens its own HTTP client, so a fresh one per round
  put a new connection inside the timer, while PraisonAI and CrewAI agents
  share theirs. One model object per runner (new Agent per round as before).
  Measured: Agno median 0.96 s -> 0.63 s.
- --iterations must be at least 1 and --warmup 0 or more.
- tests/unit/test_execution_benchmark_rules.py checks the counting rules
  with fake runners (no API key): warm-ups untimed and rounds interleaved,
  errors and empty answers are failures, a failed warm-up keeps the
  framework, ranking by median, only the call is timed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-11-04 12:23:33 +00:00
MervinPraison 16f148d6c2 docs: auto-update feature parity trackers [skip ci] 2026-11-04 12:18:10 +00:00
praisonai-triage-agent[bot] 3e314cf09b Merge pull request #6220 from MervinPraison/fix/system-prompt-only-relevant-hints
fix(agent): add the file-edit hint only when an edit tool is attached
2026-11-04 12:17:29 +00:00
praisonai-triage-agent[bot] 46263b2d76 Merge pull request #6225 from MervinPraison/claude/issue-6223-20261104
fix: add model-isolated 'secret' gateway request kind
2026-11-04 12:10:08 +00:00
MervinPraison e05b16b366 docs: auto-update feature parity trackers [skip ci] 2026-11-04 12:01:04 +00:00
praisonai-triage-agent[bot] 81aace69ec Merge pull request #6219 from MervinPraison/fix/litellm-route-duplicate-prompt
fix(agent): send the user turn once on the custom-LLM path
2026-11-04 12:00:16 +00:00
MervinPraisonandClaude Opus 5.5 d67a39ec40 fix: mention secret kind in server_reply handler comments (#6223)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 11:47:45 +00:00
MervinPraisonandClaude Opus 5.5 bcba4110ff fix: add model-isolated 'secret' gateway request kind (fixes #6223)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 11:36:40 +00:00
MervinPraisonandClaude Opus 5.5 4c3eecfb8f fix(benchmarks): time only the execution call and keep frameworks after a failed warm-up
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 11:34:01 +00:00
MervinPraisonandClaude Opus 5.5 d41f427f59 fix: detect edit tools inside nested tool lists for the edit hint
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-04 11:18:20 +00:00
MervinPraisonandClaude Opus 5.5 d1071a2ed9 fix(agent): add the file-edit hint only when an edit tool is attached
The model-family harness fragment ("When editing files, prefer targeted
string-replacement edits: use edit_file ..." for GPT, an apply_patch
hint for Claude) was prepended to every agent's system prompt, tools or
not. It now applies only when the tools for the call include edit_file
or apply_patch (by name: strings, OpenAI-style dicts, @tool(name=...),
BaseTool, functions). An explicit harness_base_prompt is still always
used.

- chat()/achat()/streamed start() on the native OpenAI path now pass the
  per-call tools to _build_messages, as the custom-LLM path already did,
  so the prompt follows the tools actually sent
- the system-prompt cache key includes the edit-tool flag and backstory,
  so a cached prefix follows changes to agent.tools / agent.backstory

Agent(instructions=X) sets backstory and goal to X, so the prompt said X
twice ("X / Your Role: Assistant / Your Goal: X"); the Goal line is now
left out when it repeats the backstory, in _build_system_prompt and in
the agent.system_prompt attribute.

A tool-less gpt-4o-mini agent's request drops from 149 to 123 prompt
tokens.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-11-04 10:45:10 +00:00
MervinPraisonandClaude Opus 5.5 c02b02454f fix(agent): send the user turn once on the custom-LLM path
chat(), achat() and streamed start() store the user turn in chat_history
before the model call, then pass that history as chat_history= and the
same turn as prompt= to LLM.get_response*. LLM._build_messages appends
prompt after chat_history, so every litellm route (ollama/, anthropic/,
groq/, openai/, base_url=) sent the prompt twice: ['system', 'user',
'user']. Measured on gpt-4o-mini via openai/: 177 billed prompt tokens
vs 149 on the native route for the same prompt; 149 on both with this fix.
Present since at least 1.2.4.

_history_before_turn() drops that trailing stored copy from the history
forwarded to the LLM only; the local histories (written back to
chat_history on the async path and handed to guardrail retries) are
unchanged, and messages a durable replay restores after the turn keep
their order.

test_stream_context_management_custom_llm counted the duplicate as part
of the history; it now expects the six earlier turns plus the prompt.

With Anthropic prompt caching, the history cache breakpoint now lands
where the other callers put it (it used to count the duplicate).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-11-04 10:45:01 +00:00
MervinPraisonandClaude Opus 5.5 dde474cc91 fix(benchmarks): warm up, interleave and drop failed runs in execution benchmark
execution_benchmark.py timed a framework's first call (cold imports and
the first connection), ran each framework's runs back to back so network
drift landed on one framework, averaged by mean over 3 runs, and timed a
run that returned None as if it had answered: with litellm broken, every
LiteLLM run returned None and the script reported a 15 s "average".

- one untimed warm-up call per framework (--warmup, default 1)
- timed runs interleaved round by round; default 10 rounds
- a run that raises or returns an empty answer is a failure, not a
  timing; Agno reports a failed run in RunOutput.status with the error
  text as content, so that status counts as a failure too
- summary and saved table report median, mean and ok/failed counts
- docstring no longer advertises a --no-save flag that never existed

simple_benchmark.py's "without tools" run built LangGraph and CrewAI
with a tool; they are now tool-free like every other framework there.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-11-04 10:35:31 +00:00
85 changed files with 4183 additions and 791 deletions

No files matched your search

+2
View File
@@ -158,6 +158,8 @@ jobs:
tests/unit/test_adapter_registry.py tests/unit/test_token_budget.py
tests/unit/test_streaming_events.py tests/unit/test_tool_decorator_approval.py
tests/unit/test_task_execution_config.py
tests/unit/test_agent_unknown_kwarg_guidance.py
tests/unit/test_agent_instantiation_lazy.py tests/unit/test_agent_os_exports.py
# Both tests previously deselected here now pass in combination.
# test_lazy_litellm_import was looking the loader module up by
# name and could get a different object than the one get_litellm()
+340 -177
View File
@@ -14,27 +14,145 @@
<a href="https://github.com/MervinPraison/PraisonAI"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License" /></a>
</p>
<div align="center">
# PraisonAI
# PraisonAI 🦞
**Private AI that earns trust one task at a time.**
<a href="https://trendshift.io/repositories/9130" target="_blank"><img src="https://trendshift.io/api/badge/repositories/9130" alt="MervinPraison%2FPraisonAI | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
PraisonAI runs AI agents on your own computers, even fully offline, so private data never has to leave the building.
</div>
- **For highly regulated teams (early access):** each task moves to your own AI only after it passes a test. If quality slips, it moves back automatically, and every step is recorded for regulators. [How it works](#how-the-handover-works-early-access)
- **For everyone else:** a free, open-source Python agent framework. With a local model you need no API key and pay no cloud bill, and your files stay yours.
PraisonAI 🦞 — **Hire a 24/7 AI Workforce.** Stop writing boilerplate and start shipping autonomous, self-improving agents that research, plan, and execute tasks across your apps. From one agent to an entire organization, deployed in 5 lines of code.
[2M+ downloads](https://pepy.tech/projects/praisonai) · [9.3K stars](https://github.com/MervinPraison/PraisonAI) · [600+ releases in the last year](https://github.com/MervinPraison/PraisonAI/releases) · [80+ contributors](https://github.com/MervinPraison/PraisonAI/graphs/contributors) · Python
```bash
curl -fsSL https://praison.ai/install.sh | bash
pip install praisonai
```
Only need the Python library? `pip install praisonaiagents` installs just the lightweight core.
## Using PraisonAI for something else?
PraisonAI is still a general-purpose, open-source agent framework, and everything works as before: the same packages, imports, commands and examples. Go straight to what you came for:
- **Agents and multi-agent teams:** [Agents](https://praison.ai/docs/concepts/agents) · [Handoffs](https://praison.ai/docs/concepts/handoffs)
- **Workflows:** [AgentFlow: route, parallel, loop, repeat](https://praison.ai/docs/concepts/agentflow)
- **Memory and knowledge:** [Memory](https://praison.ai/docs/concepts/memory) · [Knowledge](https://praison.ai/docs/concepts/knowledge) · [RAG](https://praison.ai/docs/concepts/rag)
- **Tools and MCP:** [Tools](https://praison.ai/docs/tools/tools) · [MCP](https://praison.ai/docs/mcp/transports)
- **Bots (Telegram, Discord, Slack, WhatsApp):** [Bot CLI](https://praison.ai/docs/cli/bot) · [Bot gateway](https://praison.ai/docs/features/bot-gateway)
- **Terminal CLI:** [`praisonai code`](https://praison.ai/docs/cli/code) · [`praisonai run`](https://praison.ai/docs/cli/run) · [`praisonai chat`](https://praison.ai/docs/cli/chat) · [CLI reference](https://praison.ai/docs/cli/cli-reference)
- **Training and fine-tuning:** [PraisonAI Train](https://praison.ai/docs/train)
- **Browser automation:** [Browser agent](https://praison.ai/docs/features/browser-agent) · [Browser CLI](https://praison.ai/docs/cli/browser)
- **Sandboxes:** [Sandboxed code execution](https://praison.ai/docs/features/sandbox)
- **Deploy:** [Deploy guide](https://praison.ai/docs/deploy)
- **Dashboard and UI:** [Claw dashboard](https://praison.ai/docs/concepts/claw) · [Chat UI](https://praison.ai/docs/ui)
- **YAML, no code:** [YAML agents](#-using-yaml-no-code) · [YAML reference](https://praison.ai/docs/features/yaml-configuration-reference)
- **JavaScript and TypeScript:** [JavaScript SDK](https://praison.ai/docs/js)
- **Desktop app:** [PraisonAI Desktop](https://praison.ai/docs/features/desktop/index) · [Downloads](#desktop-app)
- **Everything else:** the [full feature reference](#full-feature-reference) further down this page
## Quick start
Set a key for any supported provider (OpenAI shown here), or skip the key and use a local model as in the next section.
```bash
export OPENAI_API_KEY="your-api-key"
```
```python
from praisonaiagents import Agent
agent = Agent(instructions="You are a helpful assistant.")
print(agent.start("In two sentences, why do some companies keep their data on their own computers?"))
```
## Run fully offline
Run the same agent on a model on your own computer with [Ollama](https://ollama.com). You need no API key, and your prompts and answers stay on the machine.
Install the package and pull the model while you still have internet; after that, the agent runs with no network at all. If you installed only the core library, use `pip install "praisonaiagents[llm]"`, which adds the local-model support.
```bash
# Start Ollama first (the app, or "ollama serve" in another terminal)
ollama pull llama3.2
export DO_NOT_TRACK=true # keeps telemetry off (PraisonAI sends none unless you turn it on)
```
```python
from praisonaiagents import Agent
agent = Agent(instructions="You are a helpful assistant.", model="ollama/llama3.2")
print(agent.start("Why is the sky blue?"))
```
If Ollama runs on another computer on your network, pass `base_url` (or set `OLLAMA_API_BASE`):
```python
agent = Agent(instructions="...", model="ollama/llama3.2", base_url="http://192.168.1.10:11434")
```
vLLM and LM Studio work the same way: use `hosted_vllm/` (vLLM) or `lm_studio/` with your server's `base_url`. More in the [Ollama guide](https://praison.ai/docs/models/ollama) and [Models docs](https://praison.ai/docs/models).
## How the handover works (early access)
1. **Watch:** records how each task is done today.
2. **Test:** your own AI takes an exam made from real past work it never studied.
3. **Hand over:** a task moves to your own AI only when it passes.
4. **Roll back:** if quality drops, the task moves back automatically.
5. **Record:** every handover leaves a record your regulator can check.
**What is "your own AI"?** An open model, such as Llama, Qwen or Gemma, running on your own computers. Use it as it is, or train it on your own past work with [PraisonAI Train](https://praison.ai/docs/train) on your own GPU servers. Your team decides when to train; nothing trains automatically. Either way, a task moves to it only after it passes that exam.
Highly regulated teams, such as banks, insurers and hospitals, want AI but can't send customer data to outside models. From 6 January 2027, OpenAI also takes no new self-serve fine-tuning jobs ([notice](https://developers.openai.com/api/docs/deprecations)), so teams that customise models need models they own. The handover lets these teams move work to AI they own, one task at a time, with a record that shows why each step was trusted.
It is built on parts of this framework that are live today: tracing, evaluation, fine-tuning, model routing and local models. The handover itself is in early access, and we are taking on our first design partners. To take part, see [praison.ai/enterprise](https://praison.ai/enterprise/).
## Features
- **Agents and multi-agent teams:** [Agents](https://praison.ai/docs/concepts/agents) · [Handoffs](https://praison.ai/docs/concepts/handoffs) · [Workflows](https://praison.ai/docs/concepts/agentflow)
- **Memory:** [Memory](https://praison.ai/docs/concepts/memory) · [Knowledge and RAG](https://praison.ai/docs/concepts/rag) · [Sessions](https://praison.ai/docs/concepts/session-management)
- **Tools and MCP:** [Tools](https://praison.ai/docs/tools/tools) · [MCP](https://praison.ai/docs/mcp/transports)
- **Guardrails and approval:** [Guardrails](https://praison.ai/docs/concepts/guardrails) · [Human approval](https://praison.ai/docs/concepts/approval) · [Policy engine](https://praison.ai/docs/features/policy-engine)
- **Tracing and evaluation:** [Tracing](https://praison.ai/docs/observability/overview) · [Telemetry](https://praison.ai/docs/features/telemetry) · [Langfuse](https://praison.ai/docs/observability/langfuse) · [Evaluation](https://praison.ai/docs/concepts/evaluation)
- **Your choice of model:** [Models](https://praison.ai/docs/models) · [Model router](https://praison.ai/docs/features/model-router)
- **Fine-tuning:** [PraisonAI Train](https://praison.ai/docs/train)
- **Deploy:** [Deploy guide](https://praison.ai/docs/deploy)
## JavaScript and TypeScript
```bash
npm install praisonai
```
The TypeScript and JavaScript SDK has its own guide at [praison.ai/docs/js](https://praison.ai/docs/js). A short example is in [Using JavaScript Code](#-using-javascript-code) below.
## The company
- **Who:** Praison AI Limited in Cambridge, UK, building PraisonAI since September 2023. Co-founders Mervin Praison (CEO), Leebana Smith (COO) and Karthick Subramanian (CBO). [About us](https://praison.ai/about/)
- **What we're building:** the task handover above, as a self-hosted product on top of this framework. It is in early access, and we are taking on our first design partners.
- **Why now:** highly regulated teams want AI but can't send customer data to outside models, and from 6 January 2027 OpenAI takes no new self-serve fine-tuning jobs.
- **How it fits together:** the framework stays free and MIT-licensed, and brings developers in. Regulated teams license the handover to run inside their own walls.
- **Traction:** every number at the top of this page is public and linked to its source.
- **Get in touch:** design partners, investors and press: enterprise@praison.ai or [praison.ai/enterprise](https://praison.ai/enterprise/)
## Docs, community and licence
- **Documentation:** [praison.ai/docs](https://praison.ai/docs)
- **Questions and bugs:** [Discussions](https://github.com/MervinPraison/PraisonAI/discussions) · [Issues](https://github.com/MervinPraison/PraisonAI/issues)
- **Contributing:** [Contributing guide](https://github.com/MervinPraison/PraisonAI/blob/main/CONTRIBUTING.md)
- **Videos and news:** [YouTube](https://youtube.com/@MervinPraison) · [X](https://x.com/MervinPraison) · [LinkedIn](https://linkedin.com/in/mervinpraison)
- **Desktop app:** macOS, Windows and Linux builds are on the [latest release](https://github.com/MervinPraison/PraisonAI/releases/latest)
- **Licence:** [MIT](https://github.com/MervinPraison/PraisonAI/blob/main/LICENSE)
---
## Full feature reference
Everything below is the full reference that was on this page before. Nothing has been removed; the longest parts are folded, so click a heading's summary to open it.
### Desktop app
<div align="center">
<br>
<a href="https://x.com/elonmusk/status/1893870468249141688" target="_blank">
<img src="https://img.shields.io/badge/Highlighted_by_Elon_Musk-000000?style=for-the-badge&logo=x&logoColor=white" alt="Highlighted by Elon Musk" />
</a>
<br>
<br>
<a href="https://github.com/MervinPraison/PraisonAI/releases/latest">
<img src="https://img.shields.io/badge/Download_for_macOS-000000?style=for-the-badge&logo=apple&logoColor=white" alt="Download for macOS" />
</a>
@@ -46,12 +164,17 @@ curl -fsSL https://praison.ai/install.sh | bash
<a href="https://github.com/MervinPraison/PraisonAI/releases/latest">
<img src="https://img.shields.io/badge/Download_for_Linux-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download for Linux" />
</a>
<br>
</div>
<p align="center">
<img src=".github/images/dashboard.png" alt="PraisonAI Dashboard" width="800" />
</p>
### Other ways to install
One-line installer:
```bash
curl -fsSL https://praison.ai/install.sh | bash
```
The full `praisonai` package, with the CLI, bots, dashboard and integrations:
```
██████╗ ██████╗ █████╗ ██╗███████╗ ██████╗ ███╗ ██╗ █████╗ ██╗
@@ -64,10 +187,20 @@ curl -fsSL https://praison.ai/install.sh | bash
pip install praisonai
```
<p align="center">
<img src=".github/images/dashboard.png" alt="PraisonAI Dashboard" width="800" />
</p>
<div align="center">
<a href="https://docs.praison.ai">
<a href="https://trendshift.io/repositories/9130" target="_blank"><img src="https://trendshift.io/api/badge/repositories/9130" alt="MervinPraison%2FPraisonAI | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</div>
<div align="center">
<a href="https://praison.ai/docs">
<p align="center">
<img src="https://img.shields.io/badge/📚_Documentation-Visit_docs.praison.ai-blue?style=for-the-badge&logo=bookstack&logoColor=white" alt="Documentation" />
<img src="https://img.shields.io/badge/📚_Documentation-Visit_praison.ai%2Fdocs-blue?style=for-the-badge&logo=bookstack&logoColor=white" alt="Documentation" />
</p>
</a>
</div>
@@ -114,6 +247,9 @@ Most frameworks hand you one or two layers and leave the rest as homework. Prais
Each layer wraps the one inside it. When an agent misbehaves, the layer tells you where to look.
<details>
<summary><strong>Show the five layers, examples and Managed Agents</strong></summary>
```
┌─────────────────────────────────────────────────────────────────┐
│ ⬡ MANAGED AGENTS — Where does it actually run? │
@@ -174,7 +310,7 @@ agent = Agent(
)
```
> **Isolate** is `handoffs=[specialist]` — a sub-agent inherits the last few messages and the intersection of your tools, not your whole transcript. [📖 Handoffs](https://docs.praison.ai/docs/concepts/handoffs)
> **Isolate** is `handoffs=[specialist]` — a sub-agent inherits the last few messages and the intersection of your tools, not your whole transcript. [📖 Handoffs](https://praison.ai/docs/concepts/handoffs)
### Layer 3 · Harness — *Can it act, and be checked?*
@@ -217,7 +353,7 @@ print(result.completion_reason)
# surfaced here as completion_reason="error")
```
> **Doom-loop detection is on by default.** Repeated identical tool calls and A→B→A→B oscillation get caught — while a poller whose output keeps changing does not. [📖 Doom Loop Detection](https://docs.praison.ai/docs/features/doom-loop-detection)
> **Doom-loop detection is on by default.** Repeated identical tool calls and A→B→A→B oscillation get caught — while a poller whose output keeps changing does not. [📖 Doom Loop Detection](https://praison.ai/docs/features/doom-loop-detection)
### Layer 5 · Graph — *Who runs when, and who checks whom?*
@@ -237,7 +373,7 @@ flow = AgentFlow(steps=[
flow.run("Ticket #123: login fails on Safari")
```
> The same graph is expressible in YAML with no Python at all. [📖 AgentFlow](https://docs.praison.ai/docs/concepts/agentflow)
> The same graph is expressible in YAML with no Python at all. [📖 AgentFlow](https://praison.ai/docs/concepts/agentflow)
### ⬡ Outside the stack: Managed Agents — *Where does it actually run?*
@@ -332,6 +468,8 @@ Sandboxes shut themselves down when idle (`auto_shutdown`, `idle_timeout_s`), an
<sub>Stack framing adapted from [The Five-Layer Agent Stack](https://mer.vin/2026/07/five-layer-agent-stack-match-bug-to-right-layer/) and [Agent Harnesses vs Orbs](https://mer.vin/2026/08/agent-harnesses-vs-orbs-why-remote-sandboxes-beat-local-agent-loops/).</sub>
</details>
---
## 🌌 The PraisonAI Ecosystem
@@ -411,6 +549,11 @@ Powered by 100+ LLMs (OpenAI, Anthropic, Gemini & local models).
</details>
<div align="center">
<a href="https://x.com/elonmusk/status/1893870468249141688" target="_blank">
<img src="https://img.shields.io/badge/Highlighted_by_Elon_Musk-000000?style=for-the-badge&logo=x&logoColor=white" alt="Highlighted by Elon Musk" />
</a>
<br>
<br>
<a href="https://x.com/elonmusk/status/1893870468249141688" target="_blank">
<img src=".github/images/elon_musk_praisonai.png" alt="Highlighted by Elon Musk" width="600" />
</a>
@@ -426,13 +569,13 @@ Powered by 100+ LLMs (OpenAI, Anthropic, Gemini & local models).
|--|---------|-----|
| 🔌 | **MCP Protocol** — stdio, HTTP, WebSocket, SSE | `tools=MCP("npx ...")` |
| 🧠 | **Planning Mode** — plan → execute → reason | `planning=True` |
| 🔍 | **Deep Research** — multi-step autonomous research | [Docs](https://docs.praison.ai/docs/agents/deep-research) |
| 🤖 | **External Agents** — orchestrate Claude Code, Gemini CLI, Codex | [Docs](https://docs.praison.ai/docs/code/external-agents) |
| 🔍 | **Deep Research** — multi-step autonomous research | [Docs](https://praison.ai/docs/agents/deep-research) |
| 🤖 | **External Agents** — orchestrate Claude Code, Gemini CLI, Codex | [Docs](https://praison.ai/docs/code/external-agents) |
| 🔄 | **Agent Handoffs** — seamless conversation passing | `handoffs=[other_agent]` |
| 🛡️ | **Guardrails** — input/output validation | [Docs](https://docs.praison.ai/docs/concepts/guardrails) |
| 🛡️ | **Guardrails** — input/output validation | [Docs](https://praison.ai/docs/concepts/guardrails) |
| | **Web Search + Fetch** — native browsing | `web=True` |
| 🪞 | **Self Reflection** — agent reviews its own output | [Docs](https://docs.praison.ai/docs/concepts/reflection) |
| 🔀 | **Workflow Patterns** — route, parallel, loop, repeat | [Docs](https://docs.praison.ai/docs/concepts/agentflow) |
| 🪞 | **Self Reflection** — agent reviews its own output | [Docs](https://praison.ai/docs/concepts/reflection) |
| 🔀 | **Workflow Patterns** — route, parallel, loop, repeat | [Docs](https://praison.ai/docs/concepts/agentflow) |
| 🧠 | **Memory (zero deps)** — works out of the box | `memory=True` |
<details>
@@ -443,18 +586,18 @@ Powered by 100+ LLMs (OpenAI, Anthropic, Gemini & local models).
| 💡 | **Prompt Caching** — reduce latency + cost | `caching=True` |
| 💾 | **Sessions + Auto-Save** — persistent state across restarts | `auto_save="my-project"` |
| 💭 | **Thinking Budgets** — control reasoning depth | `agent.thinking_budget = 1024` |
| 📚 | **RAG + Quality-Based RAG** — auto quality scoring retrieval | [Docs](https://docs.praison.ai/docs/concepts/rag) |
| 📊 | **Model Router** — auto-routes to cheapest capable model | [Docs](https://docs.praison.ai/docs/features/model-router) |
| 🧊 | **Shadow Git Checkpoints** — auto-rollback on failure | [Docs](https://docs.praison.ai/docs/features/checkpoints) |
| 📡 | **A2A Protocol** — agent-to-agent interop | [Docs](https://docs.praison.ai/docs/features/a2a) |
| 📏 | **Context Compaction** — never hit token limits | [Docs](https://docs.praison.ai/docs/features/context-compaction) |
| 📡 | **Telemetry** — OpenTelemetry traces, spans, metrics | [Docs](https://docs.praison.ai/docs/features/telemetry) |
| 📜 | **Policy Engine** — declarative agent behavior control | [Docs](https://docs.praison.ai/docs/features/policy-engine) |
| 🔄 | **Background Tasks** — fire-and-forget agents | [Docs](https://docs.praison.ai/docs/features/background-tasks) |
| 🔁 | **Doom Loop Detection** — auto-recovery from stuck agents | [Docs](https://docs.praison.ai/docs/features/doom-loop-detection) |
| 🕸️ | **Graph Memory** — Neo4j-style relationship tracking | [Docs](https://docs.praison.ai/docs/features/graph-memory) |
| 🏖️ | **Sandbox Execution** — isolated code execution | [Docs](https://docs.praison.ai/docs/features/sandbox) |
| 🖥️ | **Bot Gateway** — multi-agent routing across channels | [Docs](https://docs.praison.ai/docs/features/bot-gateway) |
| 📚 | **RAG + Quality-Based RAG** — auto quality scoring retrieval | [Docs](https://praison.ai/docs/concepts/rag) |
| 📊 | **Model Router** — auto-routes to cheapest capable model | [Docs](https://praison.ai/docs/features/model-router) |
| 🧊 | **Shadow Git Checkpoints** — auto-rollback on failure | [Docs](https://praison.ai/docs/features/checkpoints) |
| 📡 | **A2A Protocol** — agent-to-agent interop | [Docs](https://praison.ai/docs/features/a2a) |
| 📏 | **Context Compaction** — never hit token limits | [Docs](https://praison.ai/docs/features/context-compaction) |
| 📡 | **Telemetry** — OpenTelemetry traces, spans, metrics | [Docs](https://praison.ai/docs/features/telemetry) |
| 📜 | **Policy Engine** — declarative agent behavior control | [Docs](https://praison.ai/docs/features/policy-engine) |
| 🔄 | **Background Tasks** — fire-and-forget agents | [Docs](https://praison.ai/docs/features/background-tasks) |
| 🔁 | **Doom Loop Detection** — auto-recovery from stuck agents | [Docs](https://praison.ai/docs/features/doom-loop-detection) |
| 🕸️ | **Graph Memory** — Neo4j-style relationship tracking | [Docs](https://praison.ai/docs/features/graph-memory) |
| 🏖️ | **Sandbox Execution** — isolated code execution | [Docs](https://praison.ai/docs/features/sandbox) |
| 🖥️ | **Bot Gateway** — multi-agent routing across channels | [Docs](https://praison.ai/docs/features/bot-gateway) |
</details>
@@ -465,6 +608,9 @@ Powered by 100+ LLMs (OpenAI, Anthropic, Gemini & local models).
## 📘 Using Python Code
<details>
<summary><strong>Show Python examples: single agent, multi-agent, MCP, tools, databases, Claw, Langflow, UI</strong></summary>
> Unsure whether a result is a bug or model randomness? Check the [SDK Behavior Matrix](https://github.com/MervinPraison/PraisonAI/blob/main/src/praisonai-agents/docs/sdk-behavior-matrix.md) for `stream`, memory prefetch, structured output, teams and AG-UI.
### 1. Single Agent
@@ -510,7 +656,7 @@ agent = Agent(
)
```
> 📖 [Full MCP docs](https://docs.praison.ai/docs/mcp/transports) — stdio, HTTP, WebSocket, SSE transports
> 📖 [Full MCP docs](https://praison.ai/docs/mcp/transports) — stdio, HTTP, WebSocket, SSE transports
### 4. Custom Tools
@@ -562,7 +708,7 @@ agent.start("Search for AI news and calculate 15*4")
```
> ⚠️ **Security Note:** Never use `eval()`, `exec()`, or `subprocess` in tool functions that process LLM-generated or user-supplied input. Always validate and sanitize inputs to prevent code injection attacks.
> 📖 [Full tools docs](https://docs.praison.ai/docs/tools/tools) — BaseTool, tool packages, 100+ built-in tools
> 📖 [Full tools docs](https://praison.ai/docs/tools/tools) — BaseTool, tool packages, 100+ built-in tools
### 5. Persistence (Databases)
@@ -579,7 +725,7 @@ agent = Agent(
agent.chat("Hello!") # Auto-persists messages, runs, traces
```
> 📖 [Full persistence docs](https://docs.praison.ai/docs/databases/overview) — PostgreSQL, MySQL, SQLite, MongoDB, Redis, and 20+ more
> 📖 [Full persistence docs](https://praison.ai/docs/databases/overview) — PostgreSQL, MySQL, SQLite, MongoDB, Redis, and 20+ more
### 6. PraisonAI Claw 🦞 (Dashboard UI)
@@ -601,7 +747,7 @@ Copy `.env.example` to `.env` and configure the following variables:
Open **http://localhost:8082** — the dashboard comes with 13 built-in pages: Chat, Agents, Memory, Knowledge, Channels, Guardrails, Cron, and more. Add messaging channels directly from the UI.
> 📖 [Full Claw docs](https://docs.praison.ai/docs/concepts/claw) — platform tokens, CLI options, Docker, and YAML agent mode
> 📖 [Full Claw docs](https://praison.ai/docs/concepts/claw) — platform tokens, CLI options, Docker, and YAML agent mode
### 7. Langflow Integration 🔗 (Visual Flow Builder)
@@ -614,7 +760,7 @@ praisonai flow
Open **http://localhost:7861** — use the **Agent** and **Agent Team** components to create sequential or parallel workflows. Connect Chat Input → Agent Team → Chat Output for instant multi-agent pipelines.
> 📖 [Full Flow docs](https://docs.praison.ai/docs/concepts/agentflow) — visual agent building, component reference, and deployment
> 📖 [Full Flow docs](https://praison.ai/docs/concepts/agentflow) — visual agent building, component reference, and deployment
### 8. PraisonAI UI 🤖 (Clean Chat)
@@ -625,10 +771,15 @@ pip install "praisonai[ui]"
praisonai ui
```
</details>
---
## 📄 Using YAML (No Code)
<details>
<summary><strong>Show YAML examples</strong></summary>
### Example 1: Two Agents Working Together
Create `agents.yaml`:
@@ -700,6 +851,8 @@ praisonai agents.yaml
> - Tools in `tools.py` are automatically discovered
> - The function's docstring helps the AI understand how to use it
</details>
---
## 🎯 CLI Quick Reference
@@ -718,23 +871,26 @@ praisonai agents.yaml
| **Development** | `commit`, `docs`, `checkpoint`, `hooks` |
| **Scheduling** | `schedule start`, `schedule list`, `schedule stop` |
> 📖 [Full CLI reference](https://docs.praison.ai/docs/cli/cli-reference)
> 📖 [Full CLI reference](https://praison.ai/docs/cli/cli-reference)
---
## ✨ Key Features
<details>
<summary><strong>Show all features by category</strong></summary>
<details open>
<summary><strong>🤖 Core Agents</strong></summary>
| Feature | Code | Docs |
|---------|:----:|:----:|
| Single Agent | [Example](examples/python/agents/single-agent.py) | [📖](https://docs.praison.ai/docs/agents/single) |
| Multi Agents | [Example](examples/python/general/mini_agents_example.py) | [📖](https://docs.praison.ai/docs/concepts/agents) |
| Auto Agents | [Example](examples/python/general/auto_agents_example.py) | [📖](https://docs.praison.ai/docs/features/autoagents) |
| Self Reflection AI Agents | [Example](examples/python/concepts/self-reflection-details.py) | [📖](https://docs.praison.ai/docs/concepts/reflection) |
| Reasoning AI Agents | [Example](examples/python/concepts/reasoning-extraction.py) | [📖](https://docs.praison.ai/docs/features/reasoning) |
| Multi Modal AI Agents | [Example](examples/python/general/multimodal.py) | [📖](https://docs.praison.ai/docs/features/multimodal) |
| Single Agent | [Example](examples/python/agents/single-agent.py) | [📖](https://praison.ai/docs/agents/single) |
| Multi Agents | [Example](examples/python/general/mini_agents_example.py) | [📖](https://praison.ai/docs/concepts/agents) |
| Auto Agents | [Example](examples/python/general/auto_agents_example.py) | [📖](https://praison.ai/docs/features/autoagents) |
| Self Reflection AI Agents | [Example](examples/python/concepts/self-reflection-details.py) | [📖](https://praison.ai/docs/concepts/reflection) |
| Reasoning AI Agents | [Example](examples/python/concepts/reasoning-extraction.py) | [📖](https://praison.ai/docs/features/reasoning) |
| Multi Modal AI Agents | [Example](examples/python/general/multimodal.py) | [📖](https://praison.ai/docs/features/multimodal) |
</details>
@@ -743,16 +899,16 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Simple Workflow | [Example](examples/python/workflows/simple_workflow.py) | [📖](https://docs.praison.ai/docs/concepts/agentflow) |
| Workflow with Agents | [Example](examples/python/workflows/workflow_with_agents.py) | [📖](https://docs.praison.ai/docs/concepts/agentflow) |
| Agentic Routing (`route()`) | [Example](examples/python/workflows/workflow_routing.py) | [📖](https://docs.praison.ai/docs/features/routing) |
| Parallel Execution (`parallel()`) | [Example](examples/python/workflows/workflow_parallel.py) | [📖](https://docs.praison.ai/docs/features/parallelisation) |
| Loop over List/CSV (`loop()`) | [Example](examples/python/workflows/workflow_loop_csv.py) | [📖](https://docs.praison.ai/docs/features/repetitive) |
| Evaluator-Optimizer (`repeat()`) | [Example](examples/python/workflows/workflow_repeat.py) | [📖](https://docs.praison.ai/docs/concepts/evaluation) |
| Conditional Steps | [Example](examples/python/workflows/workflow_conditional.py) | [📖](https://docs.praison.ai/docs/concepts/agentflow) |
| Workflow Branching | [Example](examples/python/workflows/workflow_branching.py) | [📖](https://docs.praison.ai/docs/concepts/agentflow) |
| Workflow Early Stop | [Example](examples/python/workflows/workflow_early_stop.py) | [📖](https://docs.praison.ai/docs/concepts/agentflow) |
| Workflow Checkpoints | [Example](examples/python/workflows/workflow_checkpoints.py) | [📖](https://docs.praison.ai/docs/concepts/agentflow) |
| Simple Workflow | [Example](examples/python/workflows/simple_workflow.py) | [📖](https://praison.ai/docs/concepts/agentflow) |
| Workflow with Agents | [Example](examples/python/workflows/workflow_with_agents.py) | [📖](https://praison.ai/docs/concepts/agentflow) |
| Agentic Routing (`route()`) | [Example](examples/python/workflows/workflow_routing.py) | [📖](https://praison.ai/docs/features/routing) |
| Parallel Execution (`parallel()`) | [Example](examples/python/workflows/workflow_parallel.py) | [📖](https://praison.ai/docs/features/parallelisation) |
| Loop over List/CSV (`loop()`) | [Example](examples/python/workflows/workflow_loop_csv.py) | [📖](https://praison.ai/docs/features/repetitive) |
| Evaluator-Optimizer (`repeat()`) | [Example](examples/python/workflows/workflow_repeat.py) | [📖](https://praison.ai/docs/concepts/evaluation) |
| Conditional Steps | [Example](examples/python/workflows/workflow_conditional.py) | [📖](https://praison.ai/docs/concepts/agentflow) |
| Workflow Branching | [Example](examples/python/workflows/workflow_branching.py) | [📖](https://praison.ai/docs/concepts/agentflow) |
| Workflow Early Stop | [Example](examples/python/workflows/workflow_early_stop.py) | [📖](https://praison.ai/docs/concepts/agentflow) |
| Workflow Checkpoints | [Example](examples/python/workflows/workflow_checkpoints.py) | [📖](https://praison.ai/docs/concepts/agentflow) |
</details>
@@ -761,13 +917,13 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Code Interpreter Agents | [Example](examples/python/agents/code-agent.py) | [📖](https://docs.praison.ai/docs/features/codeagent) |
| AI Code Editing Tools | [Example](examples/python/code/code_editing_example.py) | [📖](https://docs.praison.ai/docs/code/editing) |
| External Agents (All) | [Example](examples/python/code/external_agents_example.py) | [📖](https://docs.praison.ai/docs/code/external-agents) |
| Claude Code CLI | [Example](examples/python/code/claude_code_example.py) | [📖](https://docs.praison.ai/docs/code/claude-code) |
| Gemini CLI | [Example](examples/python/code/gemini_cli_example.py) | [📖](https://docs.praison.ai/docs/code/gemini-cli) |
| Codex CLI | [Example](examples/python/code/codex_cli_example.py) | [📖](https://docs.praison.ai/docs/code/codex-cli) |
| Cursor CLI | [Example](examples/python/code/cursor_cli_example.py) | [📖](https://docs.praison.ai/docs/code/cursor-cli) |
| Code Interpreter Agents | [Example](examples/python/agents/code-agent.py) | [📖](https://praison.ai/docs/features/codeagent) |
| AI Code Editing Tools | [Example](examples/python/code/code_editing_example.py) | [📖](https://praison.ai/docs/code/editing) |
| External Agents (All) | [Example](examples/python/code/external_agents_example.py) | [📖](https://praison.ai/docs/code/external-agents) |
| Claude Code CLI | [Example](examples/python/code/claude_code_example.py) | [📖](https://praison.ai/docs/code/claude-code) |
| Gemini CLI | [Example](examples/python/code/gemini_cli_example.py) | [📖](https://praison.ai/docs/code/gemini-cli) |
| Codex CLI | [Example](examples/python/code/codex_cli_example.py) | [📖](https://praison.ai/docs/code/codex-cli) |
| Cursor CLI | [Example](examples/python/code/cursor_cli_example.py) | [📖](https://praison.ai/docs/code/cursor-cli) |
</details>
@@ -776,18 +932,18 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Memory (Short & Long Term) | [Example](examples/python/general/memory_example.py) | [📖](https://docs.praison.ai/docs/concepts/memory) |
| File-Based Memory | [Example](examples/python/general/memory_example.py) | [📖](https://docs.praison.ai/docs/concepts/memory) |
| Claude Memory Tool | [Example](examples/python/memory/claude_memory_example.py) | [📖](https://docs.praison.ai/docs/features/claude-memory-tool) |
| Add Custom Knowledge | [Example](examples/python/concepts/knowledge-agents.py) | [📖](https://docs.praison.ai/docs/concepts/knowledge) |
| RAG Agents | [Example](examples/python/concepts/rag-agents.py) | [📖](https://docs.praison.ai/docs/concepts/rag) |
| Chat with PDF Agents | [Example](examples/python/concepts/chat-with-pdf.py) | [📖](https://docs.praison.ai/docs/features/chat-with-pdf) |
| Data Readers (PDF, DOCX, etc.) | [CLI](https://docs.praison.ai/docs/cli/knowledge) | [📖](https://docs.praison.ai/docs/features/chunking-strategies) |
| Vector Store Selection | [CLI](https://docs.praison.ai/docs/cli/knowledge) | [📖](https://docs.praison.ai/docs/features/knowledge-backends) |
| Retrieval Strategies | [CLI](https://docs.praison.ai/docs/cli/knowledge) | [📖](https://docs.praison.ai/docs/features/retrieval-strategies) |
| Rerankers | [CLI](https://docs.praison.ai/docs/cli/knowledge) | [📖](https://docs.praison.ai/docs/features/smart-retrieval) |
| Index Types (Vector/Keyword/Hybrid) | [CLI](https://docs.praison.ai/docs/cli/knowledge) | [📖](https://docs.praison.ai/docs/features/incremental-indexing) |
| Query Engines (Sub-Question, etc.) | [CLI](https://docs.praison.ai/docs/cli/knowledge) | [📖](https://docs.praison.ai/docs/features/retrieval) |
| Memory (Short & Long Term) | [Example](examples/python/general/memory_example.py) | [📖](https://praison.ai/docs/concepts/memory) |
| File-Based Memory | [Example](examples/python/general/memory_example.py) | [📖](https://praison.ai/docs/concepts/memory) |
| Claude Memory Tool | [Example](examples/python/memory/claude_memory_example.py) | [📖](https://praison.ai/docs/features/claude-memory-tool) |
| Add Custom Knowledge | [Example](examples/python/concepts/knowledge-agents.py) | [📖](https://praison.ai/docs/concepts/knowledge) |
| RAG Agents | [Example](examples/python/concepts/rag-agents.py) | [📖](https://praison.ai/docs/concepts/rag) |
| Chat with PDF Agents | [Example](examples/python/concepts/chat-with-pdf.py) | [📖](https://praison.ai/docs/features/chat-with-pdf) |
| Data Readers (PDF, DOCX, etc.) | [CLI](https://praison.ai/docs/cli/knowledge) | [📖](https://praison.ai/docs/features/chunking-strategies) |
| Vector Store Selection | [CLI](https://praison.ai/docs/cli/knowledge) | [📖](https://praison.ai/docs/features/knowledge-backends) |
| Retrieval Strategies | [CLI](https://praison.ai/docs/cli/knowledge) | [📖](https://praison.ai/docs/features/retrieval-strategies) |
| Rerankers | [CLI](https://praison.ai/docs/cli/knowledge) | [📖](https://praison.ai/docs/features/smart-retrieval) |
| Index Types (Vector/Keyword/Hybrid) | [CLI](https://praison.ai/docs/cli/knowledge) | [📖](https://praison.ai/docs/features/incremental-indexing) |
| Query Engines (Sub-Question, etc.) | [CLI](https://praison.ai/docs/cli/knowledge) | [📖](https://praison.ai/docs/features/retrieval) |
</details>
@@ -796,12 +952,12 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Deep Research Agents | [Example](examples/python/agents/research-agent.py) | [📖](https://docs.praison.ai/docs/agents/deep-research) |
| Query Rewriter Agent | [Example](examples/python/agents/query-rewriter-agent.py) | [📖](https://docs.praison.ai/docs/agents/query-rewriter) |
| Native Web Search | [Example](examples/python/agents/websearch-agent.py) | [📖](https://docs.praison.ai/docs/agents/websearch) |
| Built-in Search Tools | [Example](examples/python/agents/websearch-agent.py) | [📖](https://docs.praison.ai/docs/tools/tavily) |
| Unified Web Search | [Example](examples/python/web_search_example.py) | [📖](https://docs.praison.ai/docs/tools/web-search) |
| Web Fetch (Anthropic) | [Example](examples/python/agents/web-fetch-agent.py) | [📖](https://docs.praison.ai/docs/features/model-capabilities) |
| Deep Research Agents | [Example](examples/python/agents/research-agent.py) | [📖](https://praison.ai/docs/agents/deep-research) |
| Query Rewriter Agent | [Example](examples/python/agents/query-rewriter-agent.py) | [📖](https://praison.ai/docs/agents/query-rewriter) |
| Native Web Search | [Example](examples/python/agents/websearch-agent.py) | [📖](https://praison.ai/docs/agents/websearch) |
| Built-in Search Tools | [Example](examples/python/agents/websearch-agent.py) | [📖](https://praison.ai/docs/tools/tavily) |
| Unified Web Search | [Example](examples/python/web_search_example.py) | [📖](https://praison.ai/docs/tools/web-search) |
| Web Fetch (Anthropic) | [Example](examples/python/agents/web-fetch-agent.py) | [📖](https://praison.ai/docs/features/model-capabilities) |
</details>
@@ -810,12 +966,12 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Planning Mode | [Example](examples/python/agents/planning-agent.py) | [📖](https://docs.praison.ai/docs/concepts/planning) |
| Planning Tools | [Example](examples/python/agents/planning-agent.py) | [📖](https://docs.praison.ai/docs/concepts/planning) |
| Planning Reasoning | [Example](examples/python/agents/planning-agent.py) | [📖](https://docs.praison.ai/docs/concepts/planning) |
| Prompt Chaining | [Example](examples/python/general/prompt_chaining.py) | [📖](https://docs.praison.ai/docs/features/promptchaining) |
| Evaluator Optimiser | [Example](examples/python/general/evaluator-optimiser.py) | [📖](https://docs.praison.ai/docs/concepts/evaluation) |
| Orchestrator Workers | [Example](examples/python/general/orchestrator-workers.py) | [📖](https://docs.praison.ai/docs/concepts/orchestration) |
| Planning Mode | [Example](examples/python/agents/planning-agent.py) | [📖](https://praison.ai/docs/concepts/planning) |
| Planning Tools | [Example](examples/python/agents/planning-agent.py) | [📖](https://praison.ai/docs/concepts/planning) |
| Planning Reasoning | [Example](examples/python/agents/planning-agent.py) | [📖](https://praison.ai/docs/concepts/planning) |
| Prompt Chaining | [Example](examples/python/general/prompt_chaining.py) | [📖](https://praison.ai/docs/features/promptchaining) |
| Evaluator Optimiser | [Example](examples/python/general/evaluator-optimiser.py) | [📖](https://praison.ai/docs/concepts/evaluation) |
| Orchestrator Workers | [Example](examples/python/general/orchestrator-workers.py) | [📖](https://praison.ai/docs/concepts/orchestration) |
</details>
@@ -824,15 +980,15 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Data Analyst Agent | [Example](examples/python/agents/data-analyst-agent.py) | [📖](https://docs.praison.ai/docs/agents/data-analyst) |
| Finance Agent | [Example](examples/python/agents/finance-agent.py) | [📖](https://docs.praison.ai/docs/agents/finance) |
| Shopping Agent | [Example](examples/python/agents/shopping-agent.py) | [📖](https://docs.praison.ai/docs/agents/shopping) |
| Recommendation Agent | [Example](examples/python/agents/recommendation-agent.py) | [📖](https://docs.praison.ai/docs/agents/recommendation) |
| Wikipedia Agent | [Example](examples/python/agents/wikipedia-agent.py) | [📖](https://docs.praison.ai/docs/agents/wikipedia) |
| Programming Agent | [Example](examples/python/agents/programming-agent.py) | [📖](https://docs.praison.ai/docs/agents/programming) |
| Math Agents | [Example](examples/python/agents/math-agent.py) | [📖](https://docs.praison.ai/docs/features/mathagent) |
| Markdown Agent | [Example](examples/python/agents/markdown-agent.py) | [📖](https://docs.praison.ai/docs/agents/markdown) |
| Prompt Expander Agent | [Example](examples/python/agents/prompt-expander-agent.py) | [📖](https://docs.praison.ai/docs/agents/prompt-expander) |
| Data Analyst Agent | [Example](examples/python/agents/data-analyst-agent.py) | [📖](https://praison.ai/docs/agents/data-analyst) |
| Finance Agent | [Example](examples/python/agents/finance-agent.py) | [📖](https://praison.ai/docs/agents/finance) |
| Shopping Agent | [Example](examples/python/agents/shopping-agent.py) | [📖](https://praison.ai/docs/agents/shopping) |
| Recommendation Agent | [Example](examples/python/agents/recommendation-agent.py) | [📖](https://praison.ai/docs/agents/recommendation) |
| Wikipedia Agent | [Example](examples/python/agents/wikipedia-agent.py) | [📖](https://praison.ai/docs/agents/wikipedia) |
| Programming Agent | [Example](examples/python/agents/programming-agent.py) | [📖](https://praison.ai/docs/agents/programming) |
| Math Agents | [Example](examples/python/agents/math-agent.py) | [📖](https://praison.ai/docs/features/mathagent) |
| Markdown Agent | [Example](examples/python/agents/markdown-agent.py) | [📖](https://praison.ai/docs/agents/markdown) |
| Prompt Expander Agent | [Example](examples/python/agents/prompt-expander-agent.py) | [📖](https://praison.ai/docs/agents/prompt-expander) |
</details>
@@ -841,10 +997,10 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Image Generation Agent | [Example](examples/python/image/image-agent.py) | [📖](https://docs.praison.ai/docs/features/image-generation) |
| Image to Text Agent | [Example](examples/python/agents/image-to-text-agent.py) | [📖](https://docs.praison.ai/docs/agents/image-to-text) |
| Video Agent | [Example](examples/python/agents/video-agent.py) | [📖](https://docs.praison.ai/docs/agents/video) |
| Camera Integration | [Example](examples/python/camera/) | [📖](https://docs.praison.ai/docs/features/camera-integration) |
| Image Generation Agent | [Example](examples/python/image/image-agent.py) | [📖](https://praison.ai/docs/features/image-generation) |
| Image to Text Agent | [Example](examples/python/agents/image-to-text-agent.py) | [📖](https://praison.ai/docs/agents/image-to-text) |
| Video Agent | [Example](examples/python/agents/video-agent.py) | [📖](https://praison.ai/docs/agents/video) |
| Camera Integration | [Example](examples/python/camera/) | [📖](https://praison.ai/docs/features/camera-integration) |
</details>
@@ -853,12 +1009,12 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| MCP Transports | [Example](examples/python/mcp/mcp-transports-overview.py) | [📖](https://docs.praison.ai/docs/mcp/transports) |
| WebSocket MCP | [Example](examples/python/mcp/websocket-mcp.py) | [📖](https://docs.praison.ai/docs/mcp/sse-transport) |
| MCP Security | [Example](examples/python/mcp/mcp-security.py) | [📖](https://docs.praison.ai/docs/mcp/transports) |
| MCP Resumability | [Example](examples/python/mcp/mcp-resumability.py) | [📖](https://docs.praison.ai/docs/mcp/sse-transport) |
| MCP Config Management | [Docs](https://docs.praison.ai/docs/cli/mcp) | [📖](https://docs.praison.ai/docs/cli/mcp) |
| LangChain Integrated Agents | [Example](examples/python/general/langchain_example.py) | [📖](https://docs.praison.ai/docs/features/langchain) |
| MCP Transports | [Example](examples/python/mcp/mcp-transports-overview.py) | [📖](https://praison.ai/docs/mcp/transports) |
| WebSocket MCP | [Example](examples/python/mcp/websocket-mcp.py) | [📖](https://praison.ai/docs/mcp/sse-transport) |
| MCP Security | [Example](examples/python/mcp/mcp-security.py) | [📖](https://praison.ai/docs/mcp/transports) |
| MCP Resumability | [Example](examples/python/mcp/mcp-resumability.py) | [📖](https://praison.ai/docs/mcp/sse-transport) |
| MCP Config Management | [Docs](https://praison.ai/docs/cli/mcp) | [📖](https://praison.ai/docs/cli/mcp) |
| LangChain Integrated Agents | [Example](examples/python/general/langchain_example.py) | [📖](https://praison.ai/docs/features/langchain) |
</details>
@@ -867,9 +1023,9 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Guardrails | [Example](examples/python/guardrails/comprehensive-guardrails-example.py) | [📖](https://docs.praison.ai/docs/concepts/guardrails) |
| Human Approval | [Example](examples/python/general/human_approval_example.py) | [📖](https://docs.praison.ai/docs/concepts/approval) |
| Rules & Instructions | [Docs](https://docs.praison.ai/docs/features/rules) | [📖](https://docs.praison.ai/docs/features/rules) |
| Guardrails | [Example](examples/python/guardrails/comprehensive-guardrails-example.py) | [📖](https://praison.ai/docs/concepts/guardrails) |
| Human Approval | [Example](examples/python/general/human_approval_example.py) | [📖](https://praison.ai/docs/concepts/approval) |
| Rules & Instructions | [Docs](https://praison.ai/docs/features/rules) | [📖](https://praison.ai/docs/features/rules) |
</details>
@@ -878,16 +1034,16 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Async & Parallel Processing | [Example](examples/python/general/async_example.py) | [📖](https://docs.praison.ai/docs/features/async) |
| Parallelisation | [Example](examples/python/general/parallelisation.py) | [📖](https://docs.praison.ai/docs/features/parallelisation) |
| Repetitive Agents | [Example](examples/python/concepts/repetitive-agents.py) | [📖](https://docs.praison.ai/docs/features/repetitive) |
| Agent Handoffs | [Example](examples/python/handoff/handoff_basic.py) | [📖](https://docs.praison.ai/docs/concepts/handoffs) |
| Stateful Agents | [Example](examples/python/stateful/workflow-state-example.py) | [📖](https://docs.praison.ai/docs/features/stateful-agents) |
| Autonomous Workflow | [Example](examples/python/general/autonomous-agent.py) | [📖](https://docs.praison.ai/docs/concepts/autonomy) |
| Structured Output Agents | [Example](examples/python/general/structured_agents_example.py) | [📖](https://docs.praison.ai/docs/features/structured) |
| Model Router | [Example](examples/python/agents/router-agent-cost-optimization.py) | [📖](https://docs.praison.ai/docs/features/model-router) |
| Prompt Caching | [Example](examples/python/agents/prompt-caching-agent.py) | [📖](https://docs.praison.ai/docs/features/model-capabilities) |
| Fast Context | [Example](examples/context/00_agent_fast_context_basic.py) | [📖](https://docs.praison.ai/docs/features/fast-context) |
| Async & Parallel Processing | [Example](examples/python/general/async_example.py) | [📖](https://praison.ai/docs/features/async) |
| Parallelisation | [Example](examples/python/general/parallelisation.py) | [📖](https://praison.ai/docs/features/parallelisation) |
| Repetitive Agents | [Example](examples/python/concepts/repetitive-agents.py) | [📖](https://praison.ai/docs/features/repetitive) |
| Agent Handoffs | [Example](examples/python/handoff/handoff_basic.py) | [📖](https://praison.ai/docs/concepts/handoffs) |
| Stateful Agents | [Example](examples/python/stateful/workflow-state-example.py) | [📖](https://praison.ai/docs/features/stateful-agents) |
| Autonomous Workflow | [Example](examples/python/general/autonomous-agent.py) | [📖](https://praison.ai/docs/concepts/autonomy) |
| Structured Output Agents | [Example](examples/python/general/structured_agents_example.py) | [📖](https://praison.ai/docs/features/structured) |
| Model Router | [Example](examples/python/agents/router-agent-cost-optimization.py) | [📖](https://praison.ai/docs/features/model-router) |
| Prompt Caching | [Example](examples/python/agents/prompt-caching-agent.py) | [📖](https://praison.ai/docs/features/model-capabilities) |
| Fast Context | [Example](examples/context/00_agent_fast_context_basic.py) | [📖](https://praison.ai/docs/features/fast-context) |
</details>
@@ -896,21 +1052,21 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| 100+ Custom Tools | [Example](examples/python/general/tools_example.py) | [📖](https://docs.praison.ai/docs/tools/tools) |
| YAML Configuration | [Example](examples/cookbooks/yaml/secondary_market_research_agents.yaml) | [📖](https://docs.praison.ai/docs/developers/agents-playbook) |
| 100+ LLM Support | [Example](examples/python/providers/openai/openai_gpt4_example.py) | [📖](https://docs.praison.ai/docs/models) |
| Callback Agents | [Example](examples/python/general/advanced-callback-systems.py) | [📖](https://docs.praison.ai/docs/concepts/hooks) |
| Hooks | [Example](examples/python/hooks/hooks_example.py) | [📖](https://docs.praison.ai/docs/concepts/hooks) |
| Middleware System | [Example](examples/middleware/basic_middleware.py) | [📖](https://docs.praison.ai/docs/features/middleware) |
| Configurable Model | [Example](examples/middleware/configurable_model.py) | [📖](https://docs.praison.ai/docs/features/configurable-model) |
| Rate Limiter | [Example](examples/middleware/rate_limiter.py) | [📖](https://docs.praison.ai/docs/features/rate-limiter) |
| Injected Tool State | [Example](examples/middleware/injected_state.py) | [📖](https://docs.praison.ai/docs/features/injected-state) |
| Shadow Git Checkpoints | [Example](examples/checkpoints/basic_checkpoint.py) | [📖](https://docs.praison.ai/docs/features/checkpoints) |
| Background Tasks | [Example](examples/background/basic_background.py) | [📖](https://docs.praison.ai/docs/features/background-tasks) |
| Policy Engine | [Example](examples/policy/basic_policy.py) | [📖](https://docs.praison.ai/docs/features/policy-engine) |
| Thinking Budgets | [Example](examples/thinking/basic_thinking.py) | [📖](https://docs.praison.ai/docs/features/thinking-budgets) |
| Output Styles | [Example](examples/output/basic_output.py) | [📖](https://docs.praison.ai/docs/features/output-styles) |
| Context Compaction | [Example](examples/compaction/basic_compaction.py) | [📖](https://docs.praison.ai/docs/features/context-compaction) |
| 100+ Custom Tools | [Example](examples/python/general/tools_example.py) | [📖](https://praison.ai/docs/tools/tools) |
| YAML Configuration | [Example](examples/cookbooks/yaml/secondary_market_research_agents.yaml) | [📖](https://praison.ai/docs/developers/agents-playbook) |
| 100+ LLM Support | [Example](examples/python/providers/openai/openai_gpt4_example.py) | [📖](https://praison.ai/docs/models) |
| Callback Agents | [Example](examples/python/general/advanced-callback-systems.py) | [📖](https://praison.ai/docs/concepts/hooks) |
| Hooks | [Example](examples/python/hooks/hooks_example.py) | [📖](https://praison.ai/docs/concepts/hooks) |
| Middleware System | [Example](examples/middleware/basic_middleware.py) | [📖](https://praison.ai/docs/features/middleware) |
| Configurable Model | [Example](examples/middleware/configurable_model.py) | [📖](https://praison.ai/docs/features/configurable-model) |
| Rate Limiter | [Example](examples/middleware/rate_limiter.py) | [📖](https://praison.ai/docs/features/rate-limiter) |
| Injected Tool State | [Example](examples/middleware/injected_state.py) | [📖](https://praison.ai/docs/features/injected-state) |
| Shadow Git Checkpoints | [Example](examples/checkpoints/basic_checkpoint.py) | [📖](https://praison.ai/docs/features/checkpoints) |
| Background Tasks | [Example](examples/background/basic_background.py) | [📖](https://praison.ai/docs/features/background-tasks) |
| Policy Engine | [Example](examples/policy/basic_policy.py) | [📖](https://praison.ai/docs/features/policy-engine) |
| Thinking Budgets | [Example](examples/thinking/basic_thinking.py) | [📖](https://praison.ai/docs/features/thinking-budgets) |
| Output Styles | [Example](examples/output/basic_output.py) | [📖](https://praison.ai/docs/features/output-styles) |
| Context Compaction | [Example](examples/compaction/basic_compaction.py) | [📖](https://praison.ai/docs/features/context-compaction) |
</details>
@@ -919,14 +1075,14 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Sessions Management | [Example](examples/python/sessions/comprehensive-session-management.py) | [📖](https://docs.praison.ai/docs/concepts/session-management) |
| Auto-Save Sessions | [Docs](https://docs.praison.ai/docs/cli/session) | [📖](https://docs.praison.ai/docs/cli/session) |
| History in Context | [Docs](https://docs.praison.ai/docs/cli/session) | [📖](https://docs.praison.ai/docs/cli/session) |
| Telemetry | [Example](examples/python/telemetry/production-telemetry-example.py) | [📖](https://docs.praison.ai/docs/features/telemetry) |
| Langfuse Tracing | [Docs](https://docs.praison.ai/docs/observability/langfuse) | [📖](https://docs.praison.ai/docs/observability/langfuse) |
| Project Docs (.praison/docs/) | [Docs](https://docs.praison.ai/docs/cli/docs) | [📖](https://docs.praison.ai/docs/cli/docs) |
| AI Commit Messages | [Docs](https://docs.praison.ai/docs/cli/commit) | [📖](https://docs.praison.ai/docs/cli/commit) |
| @Mentions in Prompts | [Docs](https://docs.praison.ai/docs/cli/mentions) | [📖](https://docs.praison.ai/docs/cli/mentions) |
| Sessions Management | [Example](examples/python/sessions/comprehensive-session-management.py) | [📖](https://praison.ai/docs/concepts/session-management) |
| Auto-Save Sessions | [Docs](https://praison.ai/docs/cli/session) | [📖](https://praison.ai/docs/cli/session) |
| History in Context | [Docs](https://praison.ai/docs/cli/session) | [📖](https://praison.ai/docs/cli/session) |
| Telemetry | [Example](examples/python/telemetry/production-telemetry-example.py) | [📖](https://praison.ai/docs/features/telemetry) |
| Langfuse Tracing | [Docs](https://praison.ai/docs/observability/langfuse) | [📖](https://praison.ai/docs/observability/langfuse) |
| Project Docs (.praison/docs/) | [Docs](https://praison.ai/docs/cli/docs) | [📖](https://praison.ai/docs/cli/docs) |
| AI Commit Messages | [Docs](https://praison.ai/docs/cli/commit) | [📖](https://praison.ai/docs/cli/commit) |
| @Mentions in Prompts | [Docs](https://praison.ai/docs/cli/mentions) | [📖](https://praison.ai/docs/cli/mentions) |
</details>
@@ -935,20 +1091,20 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Slash Commands | [Example](examples/python/cli/slash_commands_example.py) | [📖](https://docs.praison.ai/docs/cli/slash-commands) |
| Autonomy Modes | [Example](examples/python/cli/autonomy_modes_example.py) | [📖](https://docs.praison.ai/docs/cli/autonomy-modes) |
| Cost Tracking | [Example](examples/python/cli/cost_tracking_example.py) | [📖](https://docs.praison.ai/docs/cli/cost-tracking) |
| Repository Map | [Example](examples/python/cli/repo_map_example.py) | [📖](https://docs.praison.ai/docs/cli/repo-map) |
| Interactive TUI | [Example](examples/python/cli/interactive_tui_example.py) | [📖](https://docs.praison.ai/docs/cli/interactive-tui) |
| Git Integration | [Example](examples/python/cli/git_integration_example.py) | [📖](https://docs.praison.ai/docs/cli/git-integration) |
| Sandbox Execution | [Example](examples/python/cli/sandbox_execution_example.py) | [📖](https://docs.praison.ai/docs/cli/sandbox-execution) |
| CLI Compare | [Example](examples/compare/cli_compare_basic.py) | [📖](https://docs.praison.ai/docs/cli/compare) |
| Profile/Benchmark | [Docs](https://docs.praison.ai/docs/cli/profile) | [📖](https://docs.praison.ai/docs/cli/profile) |
| Auto Mode | [Docs](https://docs.praison.ai/docs/cli/auto) | [📖](https://docs.praison.ai/docs/cli/auto) |
| Init | [Docs](https://docs.praison.ai/docs/cli/init) | [📖](https://docs.praison.ai/docs/cli/init) |
| File Input | [Docs](https://docs.praison.ai/docs/cli/file-input) | [📖](https://docs.praison.ai/docs/cli/file-input) |
| Final Agent | [Docs](https://docs.praison.ai/docs/cli/final-agent) | [📖](https://docs.praison.ai/docs/cli/final-agent) |
| Max Tokens | [Docs](https://docs.praison.ai/docs/cli/max-tokens) | [📖](https://docs.praison.ai/docs/cli/max-tokens) |
| Slash Commands | [Example](examples/python/cli/slash_commands_example.py) | [📖](https://praison.ai/docs/cli/slash-commands) |
| Autonomy Modes | [Example](examples/python/cli/autonomy_modes_example.py) | [📖](https://praison.ai/docs/cli/autonomy-modes) |
| Cost Tracking | [Example](examples/python/cli/cost_tracking_example.py) | [📖](https://praison.ai/docs/cli/cost-tracking) |
| Repository Map | [Example](examples/python/cli/repo_map_example.py) | [📖](https://praison.ai/docs/cli/repo-map) |
| Interactive TUI | [Example](examples/python/cli/interactive_tui_example.py) | [📖](https://praison.ai/docs/cli/interactive-tui) |
| Git Integration | [Example](examples/python/cli/git_integration_example.py) | [📖](https://praison.ai/docs/cli/git-integration) |
| Sandbox Execution | [Example](examples/python/cli/sandbox_execution_example.py) | [📖](https://praison.ai/docs/cli/sandbox-execution) |
| CLI Compare | [Example](examples/compare/cli_compare_basic.py) | [📖](https://praison.ai/docs/cli/compare) |
| Profile/Benchmark | [Docs](https://praison.ai/docs/cli/profile) | [📖](https://praison.ai/docs/cli/profile) |
| Auto Mode | [Docs](https://praison.ai/docs/cli/auto) | [📖](https://praison.ai/docs/cli/auto) |
| Init | [Docs](https://praison.ai/docs/cli/init) | [📖](https://praison.ai/docs/cli/init) |
| File Input | [Docs](https://praison.ai/docs/cli/file-input) | [📖](https://praison.ai/docs/cli/file-input) |
| Final Agent | [Docs](https://praison.ai/docs/cli/final-agent) | [📖](https://praison.ai/docs/cli/final-agent) |
| Max Tokens | [Docs](https://praison.ai/docs/cli/max-tokens) | [📖](https://praison.ai/docs/cli/max-tokens) |
</details>
@@ -957,10 +1113,10 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Accuracy Evaluation | [Example](examples/eval/accuracy_example.py) | [📖](https://docs.praison.ai/docs/cli/eval) |
| Performance Evaluation | [Example](examples/eval/performance_example.py) | [📖](https://docs.praison.ai/docs/cli/eval) |
| Reliability Evaluation | [Example](examples/eval/reliability_example.py) | [📖](https://docs.praison.ai/docs/cli/eval) |
| Criteria Evaluation | [Example](examples/eval/criteria_example.py) | [📖](https://docs.praison.ai/docs/cli/eval) |
| Accuracy Evaluation | [Example](examples/eval/accuracy_example.py) | [📖](https://praison.ai/docs/cli/eval) |
| Performance Evaluation | [Example](examples/eval/performance_example.py) | [📖](https://praison.ai/docs/cli/eval) |
| Reliability Evaluation | [Example](examples/eval/reliability_example.py) | [📖](https://praison.ai/docs/cli/eval) |
| Criteria Evaluation | [Example](examples/eval/criteria_example.py) | [📖](https://praison.ai/docs/cli/eval) |
</details>
@@ -969,8 +1125,8 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Skills Management | [Example](examples/skills/basic_skill_usage.py) | [📖](https://docs.praison.ai/docs/concepts/skills) |
| Custom Skills | [Example](examples/skills/custom_skill_example.py) | [📖](https://docs.praison.ai/docs/concepts/skills) |
| Skills Management | [Example](examples/skills/basic_skill_usage.py) | [📖](https://praison.ai/docs/concepts/skills) |
| Custom Skills | [Example](examples/skills/custom_skill_example.py) | [📖](https://praison.ai/docs/concepts/skills) |
</details>
@@ -979,7 +1135,9 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Agent Scheduler | [Example](examples/python/scheduled_agents/news_checker_live.py) | [📖](https://docs.praison.ai/docs/cli/scheduler) |
| Agent Scheduler | [Example](examples/python/scheduled_agents/news_checker_live.py) | [📖](https://praison.ai/docs/cli/scheduler) |
</details>
</details>
@@ -1008,7 +1166,7 @@ PraisonAI keeps framework overhead microsecond-class: creating an `Agent` (one t
|--------------------|-----------|----------|
| Avg Agent instantiation (1 tool, 1000 iterations) | **46 μs** | 2026-10-15 · praisonaiagents 1.7.12 |
Methodology, comparisons and how to reproduce: [benchmarks/BENCHMARK_RESULTS.md](https://github.com/MervinPraison/PraisonAI/blob/main/src/praisonai-agents/benchmarks/BENCHMARK_RESULTS.md) · [docs](https://docs.praison.ai/docs/developers/benchmarks)
Methodology, comparisons and how to reproduce: [benchmarks/BENCHMARK_RESULTS.md](https://github.com/MervinPraison/PraisonAI/blob/main/src/praisonai-agents/benchmarks/BENCHMARK_RESULTS.md) · [docs](https://praison.ai/docs/developers/benchmarks)
---
@@ -1018,7 +1176,7 @@ Methodology, comparisons and how to reproduce: [benchmarks/BENCHMARK_RESULTS.md]
## ⭐ Star History
[![Star History Chart](https://api.star-history.com/svg?repos=MervinPraison/PraisonAI&type=Date)](https://docs.praison.ai)
[![Star History Chart](https://api.star-history.com/svg?repos=MervinPraison/PraisonAI&type=Date)](https://praison.ai/docs)
---
@@ -1089,6 +1247,9 @@ We welcome contributions! Fork the repo, create a branch, and submit a PR → [C
## ❓ FAQ & Troubleshooting
<details>
<summary><strong>Show FAQ and troubleshooting</strong></summary>
<details>
<summary><strong>ModuleNotFoundError: No module named 'praisonaiagents'</strong></summary>
@@ -1107,7 +1268,7 @@ Ensure your API key is set:
export OPENAI_API_KEY=your_key_here
```
For other providers, see [Models docs](https://docs.praison.ai/docs/models).
For other providers, see [Models docs](https://praison.ai/docs/models).
</details>
@@ -1148,7 +1309,7 @@ agent = Agent(instructions="...", llm="ollama/llama3.2", base_url="http://192.16
The same shape works for other local runtimes — `lm_studio/`, `vllm/` and
`hosted_vllm/` with their server's `base_url`.
See [Models docs](https://docs.praison.ai/docs/models) for more details.
See [Models docs](https://praison.ai/docs/models) for more details.
</details>
@@ -1168,7 +1329,7 @@ agent = Agent(
)
```
See [Persistence docs](https://docs.praison.ai/docs/databases/overview) for supported databases.
See [Persistence docs](https://praison.ai/docs/databases/overview) for supported databases.
</details>
@@ -1185,7 +1346,7 @@ agent = Agent(
)
```
See [Memory docs](https://docs.praison.ai/docs/concepts/memory) for more options.
See [Memory docs](https://praison.ai/docs/concepts/memory) for more options.
</details>
@@ -1201,7 +1362,7 @@ agents = Agents(agents=[agent1, agent2])
agents.start()
```
See [Agents docs](https://docs.praison.ai/docs/concepts/agents) for more examples.
See [Agents docs](https://praison.ai/docs/concepts/agents) for more examples.
</details>
@@ -1216,13 +1377,15 @@ agent = Agent(
)
```
See [MCP docs](https://docs.praison.ai/docs/mcp/transports) for all transport options.
See [MCP docs](https://praison.ai/docs/mcp/transports) for all transport options.
</details>
</details>
### Getting Help
- 📚 [Full Documentation](https://docs.praison.ai)
- 📚 [Full Documentation](https://praison.ai/docs)
- 🐛 [Report Issues](https://github.com/MervinPraison/PraisonAI/issues)
- 💬 [Discussions](https://github.com/MervinPraison/PraisonAI/discussions)
@@ -1231,7 +1394,7 @@ See [MCP docs](https://docs.praison.ai/docs/mcp/transports) for all transport op
<div align="center">
<p><strong>Made with ❤️ by the PraisonAI Team</strong></p>
<p>
<a href="https://docs.praison.ai">📚 Documentation</a> •
<a href="https://praison.ai/docs">📚 Documentation</a> •
<a href="https://github.com/MervinPraison/PraisonAI">GitHub</a> •
<a href="https://youtube.com/@MervinPraison">▶️ YouTube</a> •
<a href="https://x.com/MervinPraison">𝕏 X</a> •
@@ -5,23 +5,39 @@ PraisonAI Agents - Real Execution Benchmark
Benchmarks ACTUAL agent execution with LLM API calls.
Tests single agent running a simple task across frameworks.
Each framework gets untimed warm-up calls first (imports, client set-up and
the first connection are not what this measures), then the timed runs are
interleaved round by round so network drift hits every framework alike.
A run that raises or returns an empty answer is counted as a failure, not
timed. The summary reports the median.
Usage:
export OPENAI_API_KEY=your_key
python benchmarks/execution_benchmark.py
python benchmarks/execution_benchmark.py --no-save
python benchmarks/execution_benchmark.py --iterations 5
python benchmarks/execution_benchmark.py --save
python benchmarks/execution_benchmark.py --iterations 10
python benchmarks/execution_benchmark.py --model gpt-4o --iterations 10
"""
import time
import os
import argparse
import os
import statistics
import time
from importlib.metadata import version as get_version
# Defaults (can be overridden via CLI)
DEFAULT_MODEL = "gpt-4o-mini"
DEFAULT_ITERATIONS = 3
DEFAULT_ITERATIONS = 10
DEFAULT_WARMUP = 1
DEFAULT_PROMPT = "What is 2+2? Reply with just the number."
INSTRUCTIONS = "You are a helpful assistant. Be very brief."
METHODS = {
'PraisonAI': 'agent.start()',
'PraisonAI (LiteLLM)': 'agent.start()',
'Agno': 'agent.run()',
'CrewAI': 'crew.kickoff()',
}
def get_package_versions():
@@ -35,260 +51,235 @@ def get_package_versions():
for name, pkg in packages.items():
try:
versions[name] = get_version(pkg)
except:
except Exception:
versions[name] = 'not installed'
return versions
def benchmark_praisonai(model, iterations, prompt):
"""Benchmark PraisonAI agent execution using agent.start()"""
def praisonai_runner(model, prompt):
"""PraisonAI: a new agent per run; only agent.start() is timed."""
from praisonaiagents import Agent
print("\n--- PraisonAI ---")
print("Method: agent.start()")
times = []
for i in range(iterations):
agent = Agent(
name="Calculator",
instructions="You are a helpful assistant. Be very brief.",
llm=model
)
start = time.perf_counter()
response = agent.start(prompt, output="silent")
elapsed = time.perf_counter() - start
times.append(elapsed)
print(f" Run {i+1}: {elapsed:.2f}s - Response: {str(response)[:50]}")
avg = sum(times) / len(times)
print(f" Average: {avg:.2f}s")
return avg
def build():
agent = Agent(name="Calculator", instructions=INSTRUCTIONS, llm=model)
return lambda: agent.start(prompt, output="silent")
return build
def benchmark_praisonai_litellm(model, iterations, prompt):
"""Benchmark PraisonAI with LiteLLM backend using agent.start()"""
from praisonaiagents import Agent
print("\n--- PraisonAI (LiteLLM) ---")
print("Method: agent.start()")
times = []
for i in range(iterations):
agent = Agent(
name="Calculator",
instructions="You are a helpful assistant. Be very brief.",
llm=f"openai/{model}"
)
start = time.perf_counter()
response = agent.start(prompt, output="silent")
elapsed = time.perf_counter() - start
times.append(elapsed)
print(f" Run {i+1}: {elapsed:.2f}s - Response: {str(response)[:50]}")
avg = sum(times) / len(times)
print(f" Average: {avg:.2f}s")
return avg
def benchmark_agno(model, iterations, prompt):
"""Benchmark Agno agent execution using agent.run()"""
def agno_runner(model, prompt):
"""Agno: a new agent per run; only agent.run() is timed."""
from agno.agent import Agent
from agno.models.openai import OpenAIChat
print("\n--- Agno ---")
print("Method: agent.run()")
times = []
# One model object per runner: each OpenAIChat opens its own HTTP client,
# so a fresh one per round would put a new connection inside the timer,
# while PraisonAI and CrewAI agents share theirs.
llm = OpenAIChat(id=model)
def build():
agent = Agent(model=llm, instructions=[INSTRUCTIONS])
def call():
response = agent.run(prompt)
# Agno reports a failed run in ``status`` and puts the error text in
# ``content``, so the content alone would pass for an answer.
if str(getattr(response, 'status', '')).lower().endswith('error'):
raise RuntimeError(f"Agno run failed: {str(response.content)[:200]}")
return getattr(response, 'content', response)
return call
return build
def crewai_runner(model, prompt):
"""CrewAI: a new one-agent crew per run; only crew.kickoff() is timed."""
from crewai import Agent, Crew, Task
def build():
agent = Agent(role="Calculator", goal="Answer math questions",
backstory=INSTRUCTIONS, llm=model, verbose=False)
task = Task(description=prompt, expected_output="A number", agent=agent)
return Crew(agents=[agent], tasks=[task], verbose=False).kickoff
return build
def build_runners(model, prompt):
"""Return {framework: build()} for every installed framework.
``build()`` creates fresh objects (untimed) and returns the call to time.
"""
factories = [
('PraisonAI', lambda: praisonai_runner(model, prompt)),
('PraisonAI (LiteLLM)', lambda: praisonai_runner(f"openai/{model}", prompt)),
('Agno', lambda: agno_runner(model, prompt)),
('CrewAI', lambda: crewai_runner(model, prompt)),
]
runners = {}
for name, factory in factories:
try:
runners[name] = factory()
except ImportError as e:
print(f" {name}: skipped ({e})")
return runners
def answer_text(response):
"""The answer as text, or '' when the run produced nothing usable."""
if response is None:
return ''
text = str(response).strip()
return '' if text == 'None' else text
def run_benchmark(runners, iterations, warmup):
"""Warm up each framework, then time interleaved rounds.
A failed warm-up is reported but the framework stays in the timed rounds,
so one transient error does not discard it. The order within a round
rotates every round. Object construction happens
before the timer starts; only the execution call is timed.
Returns {framework: {"times": [...], "failures": [...]}}.
"""
results = {name: {"times": [], "failures": []} for name in runners}
for name, build in runners.items():
for _ in range(warmup):
try:
if not answer_text(build()()):
raise RuntimeError("empty answer")
except Exception as e:
print(f" {name}: warm-up failed ({type(e).__name__}: {str(e)[:120]})")
# Rotate the start each round so no framework always holds the same slot.
names = list(runners)
for i in range(iterations):
agent = Agent(
model=OpenAIChat(id=model),
instructions=["You are a helpful assistant. Be very brief."],
)
start = time.perf_counter()
response = agent.run(prompt)
elapsed = time.perf_counter() - start
times.append(elapsed)
# Extract content from response
content = str(response.content)[:50] if hasattr(response, 'content') else str(response)[:50]
print(f" Run {i+1}: {elapsed:.2f}s - Response: {content}")
avg = sum(times) / len(times)
print(f" Average: {avg:.2f}s")
return avg
shift = i % len(names) if names else 0
for name in names[shift:] + names[:shift]:
build = runners[name]
try:
call = build()
start = time.perf_counter()
text = answer_text(call())
elapsed = time.perf_counter() - start
error = None if text else "empty answer"
except Exception as e:
text, error = '', f"{type(e).__name__}: {e}"
if error:
results[name]["failures"].append(error)
print(f" Round {i + 1} {name}: FAILED ({error[:120]})")
else:
results[name]["times"].append(elapsed)
print(f" Round {i + 1} {name}: {elapsed:.2f}s - {text[:40]}")
return results
def benchmark_crewai(model, iterations, prompt):
"""Benchmark CrewAI agent execution using crew.kickoff()"""
from crewai import Agent, Task, Crew
print("\n--- CrewAI ---")
print("Method: crew.kickoff()")
times = []
for i in range(iterations):
agent = Agent(
role="Calculator",
goal="Answer math questions",
backstory="You are a helpful assistant.",
llm=model,
verbose=False
)
task = Task(
description=prompt,
expected_output="A number",
agent=agent
)
crew = Crew(
agents=[agent],
tasks=[task],
verbose=False
)
start = time.perf_counter()
response = crew.kickoff()
elapsed = time.perf_counter() - start
times.append(elapsed)
print(f" Run {i+1}: {elapsed:.2f}s - Response: {str(response)[:50]}")
avg = sum(times) / len(times)
print(f" Average: {avg:.2f}s")
return avg
def summarise(results):
"""Rows of (framework, median, mean, ok, failed), fastest median first."""
rows = []
for name, r in results.items():
times = r["times"]
if times:
rows.append((name, statistics.median(times), statistics.fmean(times),
len(times), len(r["failures"])))
return sorted(rows, key=lambda row: row[1])
def save_results(results: dict, model: str, iterations: int, prompt: str):
def save_results(rows, results, model, iterations, warmup, prompt):
"""Save results to markdown file."""
import os
from datetime import datetime
filepath = os.path.join(os.path.dirname(__file__), 'EXECUTION_BENCHMARK_RESULTS.md')
versions = get_package_versions()
# Sort by time
sorted_results = sorted(results.items(), key=lambda x: x[1])
fastest = sorted_results[0][1]
fastest = rows[0][1]
with open(filepath, 'w') as f:
f.write('# PraisonAI Agents - Real Execution Benchmark\n\n')
f.write(f'**Generated:** {datetime.now().strftime("%Y-%m-%d %H:%M:%S")}\n')
f.write(f'**Model:** {model}\n')
f.write(f'**Iterations:** {iterations}\n')
f.write(f'**Iterations:** {iterations} interleaved rounds after {warmup} warm-up call(s) per framework\n')
f.write(f'**Prompt:** "{prompt}"\n\n')
f.write('## Results\n\n')
f.write('| Framework | Method | Avg Time | Relative |\n')
f.write('|-----------|--------|----------|----------|\n')
methods = {
'PraisonAI': 'agent.start()',
'PraisonAI (LiteLLM)': 'agent.start()',
'Agno': 'agent.run()',
'CrewAI': 'crew.kickoff()',
}
for name, avg in sorted_results:
ratio = avg / fastest
method = methods.get(name, 'unknown')
f.write('| Framework | Method | Median | Mean | Runs ok / failed | Relative |\n')
f.write('|-----------|--------|--------|------|------------------|----------|\n')
for name, median, mean, ok, failed in rows:
ratio = median / fastest
method = METHODS.get(name, 'unknown')
if ratio == 1.0:
f.write(f'| **{name}** | `{method}` | **{avg:.2f}s** | **1.00x (fastest)** |\n')
f.write(f'| **{name}** | `{method}` | **{median:.2f}s** | {mean:.2f}s | {ok} / {failed} | **1.00x (fastest)** |\n')
else:
f.write(f'| {name} | `{method}` | {avg:.2f}s | {ratio:.2f}x |\n')
f.write(f'| {name} | `{method}` | {median:.2f}s | {mean:.2f}s | {ok} / {failed} | {ratio:.2f}x |\n')
excluded = [name for name, r in results.items() if not r["times"]]
if excluded:
f.write(f'\nNo successful runs (not ranked): {", ".join(excluded)}\n')
f.write('\n## Package Versions\n\n')
f.write('| Package | Version |\n')
f.write('|---------|--------|\n')
for pkg, ver in versions.items():
f.write(f'| {pkg} | {ver} |\n')
f.write('\n## How to Reproduce\n\n')
f.write('```bash\n')
f.write('export OPENAI_API_KEY=your_key\n')
f.write('cd praisonai-agents\n')
f.write('python benchmarks/execution_benchmark.py\n')
f.write('cd src/praisonai-agents\n')
f.write('python benchmarks/execution_benchmark.py --save\n')
f.write('```\n')
print(f'\nResults saved to: {filepath}')
if __name__ == '__main__':
def main():
parser = argparse.ArgumentParser(description='PraisonAI Agents - Real Execution Benchmark')
parser.add_argument('--save', action='store_true', help='Save results to file')
parser.add_argument('--save', action='store_true', help='Save results to EXECUTION_BENCHMARK_RESULTS.md')
parser.add_argument('--model', '-m', type=str, default=DEFAULT_MODEL, help=f'Model to use (default: {DEFAULT_MODEL})')
parser.add_argument('--iterations', '-i', type=int, default=DEFAULT_ITERATIONS, help=f'Number of iterations (default: {DEFAULT_ITERATIONS})')
parser.add_argument('--iterations', '-i', type=int, default=DEFAULT_ITERATIONS, help=f'Timed rounds (default: {DEFAULT_ITERATIONS})')
parser.add_argument('--warmup', '-w', type=int, default=DEFAULT_WARMUP, help=f'Untimed warm-up calls per framework (default: {DEFAULT_WARMUP})')
parser.add_argument('--prompt', '-p', type=str, default=DEFAULT_PROMPT, help=f'Prompt to use (default: "{DEFAULT_PROMPT}")')
args = parser.parse_args()
# Check API key
if args.iterations < 1:
parser.error('--iterations must be at least 1')
if args.warmup < 0:
parser.error('--warmup must be 0 or more')
if not os.environ.get('OPENAI_API_KEY'):
print("ERROR: Set OPENAI_API_KEY environment variable")
exit(1)
model = args.model
iterations = args.iterations
prompt = args.prompt
return 1
print('=' * 60)
print('PraisonAI Agents - Real Execution Benchmark')
print('=' * 60)
print(f'\nModel: {model}')
print(f'Iterations: {iterations}')
print(f'Prompt: "{prompt}"')
versions = get_package_versions()
print(f'\nModel: {args.model}')
print(f'Rounds: {args.iterations} (after {args.warmup} warm-up call(s) per framework)')
print(f'Prompt: "{args.prompt}"')
print('\nPackage versions:')
for pkg, ver in versions.items():
for pkg, ver in get_package_versions().items():
print(f' {pkg}: {ver}')
results = {}
# Run benchmarks
try:
results['PraisonAI'] = benchmark_praisonai(model, iterations, prompt)
except Exception as e:
print(f"PraisonAI error: {e}")
try:
results['PraisonAI (LiteLLM)'] = benchmark_praisonai_litellm(model, iterations, prompt)
except Exception as e:
print(f"PraisonAI (LiteLLM) error: {e}")
try:
results['Agno'] = benchmark_agno(model, iterations, prompt)
except Exception as e:
print(f"Agno error: {e}")
try:
results['CrewAI'] = benchmark_crewai(model, iterations, prompt)
except Exception as e:
print(f"CrewAI error: {e}")
# Summary
print()
runners = build_runners(args.model, args.prompt)
results = run_benchmark(runners, args.iterations, args.warmup)
rows = summarise(results)
print('\n' + '=' * 60)
print('SUMMARY')
print('SUMMARY (median of successful runs)')
print('=' * 60)
if results:
sorted_results = sorted(results.items(), key=lambda x: x[1])
fastest = sorted_results[0][1]
print(f"\n{'Framework':<25} {'Avg Time':<12} {'Relative':<10}")
print('-' * 47)
for name, avg in sorted_results:
ratio = avg / fastest
print(f'{name:<25} {avg:<12.2f}s {ratio:.2f}x')
if rows:
fastest = rows[0][1]
print(f"\n{'Framework':<22} {'Median':>8} {'Mean':>8} {'ok/failed':>10} {'Relative':>9}")
print('-' * 61)
for name, median, mean, ok, failed in rows:
print(f'{name:<22} {median:>7.2f}s {mean:>7.2f}s {ok:>5}/{failed:<4} {median / fastest:>8.2f}x')
for name, r in results.items():
if not r["times"]:
print(f'{name:<22} no successful runs: {(r["failures"] or ["?"])[0][:80]}')
print('\n' + '=' * 60)
# Save results (if --save)
if results and args.save:
save_results(results, model, iterations, prompt)
if rows and args.save:
save_results(rows, results, args.model, args.iterations, args.warmup, args.prompt)
elif not args.save:
print('\nResults not saved (use --save flag to save results to file)')
return 0
if __name__ == '__main__':
raise SystemExit(main())
@@ -109,17 +109,11 @@ def run_benchmark(skip_crewai: bool = False, quiet: bool = False,
pass
try:
from langchain_core.tools import tool as langchain_tool
from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
@langchain_tool
def get_weather_lg(city: Literal['nyc', 'sf']):
"""Get weather info."""
return 'sunny' if city == 'sf' else 'cloudy'
results['LangGraph'] = measure(
lambda: create_react_agent(model=ChatOpenAI(model='gpt-4o-mini'), tools=[get_weather_lg])
lambda: create_react_agent(model=ChatOpenAI(model='gpt-4o-mini'), tools=[])
)
except ImportError:
pass
@@ -129,19 +123,12 @@ def run_benchmark(skip_crewai: bool = False, quiet: bool = False,
else:
try:
from crewai.agent import Agent as CrewAgent
from crewai.tools import tool as crewai_tool
@crewai_tool("Weather Tool")
def get_weather_crew(city: Literal['nyc', 'sf']):
"""Get weather info."""
return 'sunny' if city == 'sf' else 'cloudy'
results['CrewAI'] = measure(
lambda: CrewAgent(
role='Weather Agent',
goal='Provide weather info',
backstory='A weather expert',
tools=[get_weather_crew],
verbose=False
)
)
@@ -2698,6 +2698,8 @@ class Agent(GoalLoopMixin, SteeringMixin, SandboxMixin, SkillReviewMixin, Unifie
self.system_prompt = f"""{self.backstory}\n
Your Role: {self.role}\n
Your Goal: {self.goal}
""" if self.goal != self.backstory else f"""{self.backstory}\n
Your Role: {self.role}
"""
# Per-invocation system-prompt append (never persisted). Sourced from the
@@ -543,17 +543,53 @@ class ChatMixin:
return int(getattr(execution, "max_tool_calls_per_turn", default))
return default
def _resolve_harness_base_prompt(self):
_EDIT_TOOL_NAMES = frozenset({"edit_file", "apply_patch"})
def _offers_edit_tool(self, tools=None):
"""True when ``tools`` (default: the agent's tools) include a file-edit primitive."""
for tool in (tools if tools is not None else getattr(self, "tools", None)) or ():
if isinstance(tool, list):
# _format_tools_for_completion flattens a nested list but keeps
# only its function-schema dicts, so only those can offer an edit tool.
if self._offers_edit_tool([
t for t in tool
if isinstance(t, dict) and t.get("type") == "function"
and isinstance(t.get("function"), dict)
]):
return True
continue
if isinstance(tool, str):
name = tool
elif isinstance(tool, dict):
function = tool.get("function")
name = function.get("name") if isinstance(function, dict) else tool.get("name")
else:
# A tool's ``name`` is what the model is sent (``@tool(name=...)``
# keeps the function's own ``__name__``), so it wins.
name = getattr(tool, "name", None)
if not isinstance(name, str):
name = getattr(tool, "__name__", None)
# _prune_denied_tools drops denied tools from the payload, so a
# denied edit tool must not earn the hint either.
if name in self._EDIT_TOOL_NAMES and self._tool_name_allowed(name):
return True
return False
def _resolve_harness_base_prompt(self, tools=None):
"""Resolve the model-family harness base-prompt fragment, if any.
Honours an explicit ``harness_base_prompt`` override on the agent when
set; otherwise resolves lazily from the active model id. Returns an
empty string when no fragment applies (behaviour-preserving default).
Never raises: any resolution error collapses to no fragment.
set; otherwise resolves lazily from the active model id. The family
fragments are file-editing guidance, so they apply only when ``tools``
(default: the agent's tools) include ``edit_file`` or ``apply_patch``.
Returns an empty string when no fragment applies. Never raises: any
resolution error collapses to no fragment.
"""
override = getattr(self, "harness_base_prompt", None)
if override is not None:
return override
if not self._offers_edit_tool(tools):
return ""
try:
from ..model_harness import resolve_harness
# self.llm may be a string model id or an LLM object; resolve the
@@ -580,7 +616,10 @@ class ChatMixin:
# Check cache first (skip cache if memory is enabled since context is dynamic)
if not self._memory_instance:
tools_key = self._get_tools_cache_key(tools)
cache_key = f"{self.role}:{self.goal}:{tools_key}"
# tools=None keys as "none" whatever self.tools holds, so key the
# edit-hint decision too; otherwise a cached prefix would keep (or
# miss) the hint after the agent's tools change.
cache_key = f"{self.role}:{self.goal}:{hash(self.backstory)}:{tools_key}:{self._offers_edit_tool(tools)}"
cached_prompt = self._cache_get(self._system_prompt_cache, cache_key)
if cached_prompt is not None:
@@ -591,13 +630,16 @@ class ChatMixin:
cache_key = None # Don't cache when memory is enabled
system_prompt = f"""{self.backstory}\n
Your Role: {self.role}\n
Your Goal: {self.goal}"""
Your Role: {self.role}"""
# Agent(instructions=...) sets backstory and goal to the same text;
# state it once.
if self.goal != self.backstory:
system_prompt += f"\n\nYour Goal: {self.goal}"
# Prepend the model-family harness base prompt when one applies.
# Resolution is lazy and falls back to a behaviour-preserving default
# (no fragment) for unknown models, so output is unchanged by default.
harness_prompt = self._resolve_harness_base_prompt()
# (no fragment) for unknown models or agents without edit tools.
harness_prompt = self._resolve_harness_base_prompt(tools)
if harness_prompt:
system_prompt = f"{harness_prompt}\n\n{system_prompt}"
@@ -1204,7 +1246,8 @@ Your Goal: {self.goal}"""
if cached_entry is not None:
cached_tools, cached_metadata = cached_entry
self._tool_search_metadata = cached_metadata
return cached_tools
# Re-prune: permissions may have changed since the entry was cached.
return self._prune_denied_tools(cached_tools)
formatted_tools = []
# Provider-hosted tool specs have no local callable; the provider
@@ -1324,20 +1367,19 @@ Your Goal: {self.goal}"""
return sorted(tools_list, key=sort_key)
cleaned_tools = sort_formatted_tools(cleaned_tools)
# Prune permission-denied tools from the advertised set so the model is
# only ever offered tools it can actually call. Mirrors the execution-time
# enforcement in tool_execution.py (_perm_deny / _perm_allow). No-op when
# no deny set and no allow set are configured (backward compatible).
cleaned_tools = self._prune_denied_tools(cleaned_tools)
# Cache the formatted tools with LRU eviction, including tool search metadata
# Cache the unpruned tools with LRU eviction, including tool search
# metadata, so a later change to the deny/allow sets takes effect.
self._cache_put(
self._formatted_tools_cache,
cache_key,
(cleaned_tools, getattr(self, "_tool_search_metadata", None)),
)
return cleaned_tools
# Prune permission-denied tools from the advertised set so the model is
# only ever offered tools it can actually call. Mirrors the execution-time
# enforcement in tool_execution.py (_perm_deny / _perm_allow). No-op when
# no deny set and no allow set are configured (backward compatible).
return self._prune_denied_tools(cleaned_tools)
def _build_before_tool_definitions_input(self, formatted_tools):
"""Build the BEFORE_TOOL_DEFINITIONS hook input.
@@ -2042,6 +2084,21 @@ Your Goal: {self.goal}"""
return fn(*args, **kwargs)
return {**llm_kwargs, 'execute_tool_fn': tracked}, ran
@staticmethod
def _history_before_turn(history, content):
"""``history`` without the copy of this turn's user message stored before the call.
``LLM._build_messages`` appends ``prompt`` after ``chat_history``, so
forwarding the stored copy as well sends the prompt twice. Only a
trailing match is dropped: messages a durable replay restored after
the turn keep their order.
"""
if history:
last = history[-1]
if isinstance(last, dict) and last.get("role") == "user" and last.get("content") == content:
return list(history[:-1])
return history
def _custom_llm_hook_messages(self, llm_kwargs):
"""The custom-LLM request as the message list BEFORE_LLM / AFTER_LLM hooks see."""
messages = []
@@ -4090,7 +4147,7 @@ Your Goal: {self.goal}"""
llm_kwargs = dict(
prompt=llm_prompt,
system_prompt=system_prompt_for_llm,
chat_history=processed_history,
chat_history=self._history_before_turn(processed_history, normalized_content),
temperature=temperature,
tools=tool_param,
output_json=output_json,
@@ -4207,6 +4264,7 @@ Your Goal: {self.goal}"""
# Pass llm_prompt (which includes multimodal content if attachments present)
messages, original_prompt = self._build_messages(
llm_prompt, temperature, output_json, output_pydantic,
tools=tools,
use_native_format=use_native_format,
memory_prefetch_context=memory_prefetch_context,
)
@@ -4834,7 +4892,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
tools,
memory_prefetch_context=memory_prefetch_context,
),
'chat_history': effective_history,
'chat_history': self._history_before_turn(effective_history, normalized_content),
'temperature': temperature,
'tools': tools,
'output_json': output_json,
@@ -4928,6 +4986,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
temperature,
output_json,
output_pydantic,
tools=tools,
restore_durable=False,
memory_prefetch_context=memory_prefetch_context,
)
@@ -5266,7 +5325,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
response = await self._openai_client.async_client.chat.completions.create(
**tool_call_kwargs
)
result = await self._achat_completion(response, tools)
result = await self._achat_completion(response, tools, messages=messages)
if get_logger().getEffectiveLevel() == logging.DEBUG:
total_time = time.time() - start_time
logging.debug(f"Agent.achat completed in {total_time:.2f} seconds")
@@ -5446,8 +5505,12 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
# the except blocks above already read ownership before this runs.
self._clear_turn_tracking()
async def _achat_completion(self, response, tools, reasoning_steps=False):
"""Async version of _chat_completion method"""
async def _achat_completion(self, response, tools, reasoning_steps=False, messages=None):
"""Async version of _chat_completion method.
``messages`` is the conversation that produced ``response``; it is kept in
the follow-up request so the final answer still sees the user's question.
"""
try:
message = response.choices[0].message
if not hasattr(message, 'tool_calls') or not message.tool_calls:
@@ -5533,17 +5596,20 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
if results:
formatted_results = "\n".join([str(r) for r in results if r is not None])
if formatted_results:
messages = [
{"role": "system", "content": self.system_prompt},
if messages:
follow_up_messages = list(messages)
else:
follow_up_messages = [{"role": "system", "content": self.system_prompt}]
follow_up_messages.extend([
{"role": "assistant", "content": "Here are the tool results:"},
{"role": "user", "content": formatted_results + "\nPlease process these results and provide a final response."}
]
])
try:
final_response = await self._openai_client.async_client.chat.completions.create(
model=self.llm,
messages=messages,
messages=follow_up_messages,
temperature=1.0,
stream=stream
stream=True
)
full_response_text = ""
reasoning_content = ""
@@ -5553,6 +5619,9 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
# Process stream without display_generating since streaming is active
async for chunk in final_response:
chunks.append(chunk)
if not chunk.choices:
# e.g. Azure content-filter annotation chunks
continue
if chunk.choices[0].delta.content:
full_response_text += chunk.choices[0].delta.content
@@ -5560,9 +5629,18 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
rc = chunk.choices[0].delta.reasoning_content
if rc:
reasoning_content += rc
if not full_response_text.strip() and not reasoning_content.strip():
# No answer text (only annotation, contentless or
# whitespace-only chunks): fall back to the tool results via the
# except below.
raise ValueError("follow-up stream returned no answer text")
self.console.print()
console = self.console # None unless verbose
if console is not None:
console.print()
from .agent import _get_llm_functions
final_response = _get_llm_functions()['process_stream_chunks'](chunks)
# Return only reasoning content if reasoning_steps is True
if reasoning_steps and hasattr(final_response.choices[0].message, 'reasoning_content') and final_response.choices[0].message.reasoning_content:
@@ -5880,7 +5958,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
for chunk in self.llm_instance.get_response_stream(
prompt=actual_prompt,
system_prompt=stream_system_prompt,
chat_history=stream_history,
chat_history=self._history_before_turn(stream_history, normalized_content),
temperature=kwargs.get('temperature', 1.0),
tools=tool_param,
output_json=kwargs.get('output_json'),
@@ -5955,6 +6033,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
# Build messages using the helper method
messages, original_prompt = self._build_messages(actual_prompt, kwargs.get('temperature', 1.0),
kwargs.get('output_json'), kwargs.get('output_pydantic'),
tools=tool_param,
memory_prefetch_context=memory_prefetch_context)
# Apply context management so the streaming path compacts long
@@ -213,7 +213,8 @@ class ContextAgent(Agent):
self.output_dir / "agent_responses",
self.output_dir / "markdown_outputs",
self.output_dir / "debug_logs",
self.output_dir / "final_results"
self.output_dir / "final_results",
self.output_dir / "PRPs",
]
for directory in directories:
@@ -1270,11 +1271,12 @@ Every agent interaction has been saved for full audit trail and reproducibility.
return self._generate_combined_analysis(feature_request)
# Otherwise just save and return the PRP
from ..utils.atomic_io import atomic_write_text, safe_join
prp_dir = os.path.join(self.output_dir, "PRPs")
os.makedirs(prp_dir, exist_ok=True)
safe_filename = feature_request.lower().replace(' ', '_').replace('-', '_')[:50]
prp_path = os.path.join(self.output_dir, "PRPs", f"{safe_filename}.md")
with open(prp_path, "w", encoding="utf-8") as f:
f.write(prp)
prp_path = safe_join(prp_dir, safe_filename, ".md")
atomic_write_text(prp_path, prp)
print(f"📄 Comprehensive PRP following PRD template saved to: {prp_path}")
return prp
@@ -4362,8 +4362,13 @@ class ToolExecutionMixin:
return "Tool search not in bridge mode"
# Get deferrable tools from metadata
deferrable_tools = metadata.get("deferrable_tools", [])
# Hide permission-denied tools from search/describe; the cached
# metadata is built before pruning, so filter locally here.
deferrable_tools = [
t for t in metadata.get("deferrable_tools", [])
if self._tool_name_allowed(t.get("function", {}).get("name", ""))
]
if function_name == "tool_search":
# Handle tool_search bridge call
try:
@@ -18,6 +18,8 @@ from .._lazy import create_lazy_getattr
# 283 KB / 7,201-line ``protocols.py`` (and siblings) out of package-import
# time; each submodule is parsed only when one of its symbols is first
# touched. Mirrors the ``mcp``/``skills`` packages (AGENTS.md §4.2).
_PROTOCOLS_MODULE = "praisonaiagents.gateway.protocols"
_LAZY_IMPORTS = {
# protocols.py (parsed lazily, on first touch)
"GatewayProtocol": ("praisonaiagents.gateway.protocols", "GatewayProtocol"),
@@ -229,7 +231,9 @@ _LAZY_IMPORTS = {
# Schema-validated inbound frame codec (Issue #2831)
"HelloParams": ("praisonaiagents.gateway.protocols", "HelloParams"),
# Device-identity handshake contract (Issue #5998)
"DeviceAuth": ("praisonaiagents.gateway.protocols", "DeviceAuth"),
# Module path via a constant: GitGuardian reads a ("...", "DeviceAuth")
# string pair as an "authentication tuple" and fails every PR that merges main.
"DeviceAuth": (_PROTOCOLS_MODULE, "DeviceAuth"),
"DeviceIdentityProtocol": ("praisonaiagents.gateway.protocols", "DeviceIdentityProtocol"),
"device_id_from_public_key": ("praisonaiagents.gateway.protocols", "device_id_from_public_key"),
"HelloResult": ("praisonaiagents.gateway.protocols", "HelloResult"),
@@ -3841,12 +3841,19 @@ class ConversationRequestProtocol(Protocol):
# import lives in core.
# ---------------------------------------------------------------------------
GatewayRequestKind = Literal["approval", "choice", "input"]
GatewayRequestKind = Literal["approval", "choice", "input", "secret"]
"""Closed set of server→client interactive request kinds.
* ``approval`` — allow/deny; the reply ``value`` is ``"allow"`` or ``"deny"``.
* ``choice`` — select one of ``options``; the reply ``value`` is the option.
* ``input`` — free text; the reply ``value`` is the entered text.
* ``secret`` — a credential (password, API key, one-time code) captured
mid-turn (Issue #6223). Clients MUST render a masked input. Isolation
contract: the reply ``value`` is returned *only* to the caller of
:meth:`GatewayRequestChannelProtocol.request`; implementations MUST NOT log,
journal, echo, or persist it, and the caller (a tool/callback) MUST NOT
return it to the model or append it to history. A tool obtains one via
``get_request_channel().request(GatewayServerRequest(kind="secret", ...))``.
"""
@@ -3893,11 +3900,13 @@ class GatewayServerReply:
Attributes:
request_id: The id of the request being answered.
value: ``"allow"``/``"deny"`` for approval, the chosen option for
choice, or the entered text for input.
choice, the entered text for input, or the credential for secret.
Excluded from ``repr`` so a logged/traced reply never leaks a
secret.
"""
request_id: str
value: str
value: str = field(repr=False)
def as_dict(self) -> Dict[str, Any]:
"""Convert to a serializable dictionary."""
@@ -572,6 +572,14 @@ class FileMemory:
promoted.append(item.id)
if promoted:
# Promotion writes the same store as add_long_term, so retain the
# same highest-importance, newest-first items within its capacity.
limit = self.config["long_term_limit"]
if len(self._long_term) > limit:
self._long_term.sort(
key=lambda item: (item.importance, item.created_at), reverse=True
)
self._long_term = self._long_term[:limit]
self._save_long_term()
self._log(f"Auto-promoted {len(promoted)} items to long-term memory")
@@ -104,7 +104,12 @@ class PlanStorage:
if filename is None:
filename = self._get_plan_filename(plan)
path = os.path.join(self.plans_dir, filename)
from ..utils.atomic_io import safe_join
# Keep ".md" as the suffix so listing/lookup still find the file.
stem, ext = os.path.splitext(filename)
if ext.lower() != ".md":
stem, ext = filename, ".md"
path = safe_join(self.plans_dir, stem, ext)
# Convert plan to markdown
markdown = plan.to_markdown()
@@ -215,6 +220,29 @@ class PlanStorage:
return False
def _todo_path(self, name: str) -> str:
"""Confined path for a todo name, reusing a pre-existing legacy file.
Legacy files (``<name>.json`` written before Issue #6238) are reused
only when ``name`` is a plain filename, so upgrades keep working
without ever allowing traversal out of ``todos_dir``.
"""
from ..utils.atomic_io import safe_join
path = safe_join(self.todos_dir, name, ".json")
name = str(name)
if (
not os.path.exists(path)
and name not in ("", ".", "..")
and "\0" not in name
and "%" not in name
and os.path.basename(name) == name
and not (os.altsep and os.altsep in name)
):
legacy = os.path.join(self.todos_dir, f"{name}.json")
if os.path.isfile(legacy):
return legacy
return path
def save_todo(self, todo: TodoList, name: str = "current") -> str:
"""
Save a todo list to storage.
@@ -226,11 +254,10 @@ class PlanStorage:
Returns:
Path to the saved file
"""
filename = f"{name}.json"
path = os.path.join(self.todos_dir, filename)
from ..utils.atomic_io import atomic_write_text
path = self._todo_path(name)
# Atomic write so a crash mid-write never truncates the previous todos.
from ..utils.atomic_io import atomic_write_text
atomic_write_text(path, todo.to_json())
logger.debug(f"Saved todo list to {path}")
@@ -246,8 +273,7 @@ class PlanStorage:
Returns:
TodoList instance, or None if not found
"""
filename = f"{name}.json"
path = os.path.join(self.todos_dir, filename)
path = self._todo_path(name)
if not os.path.exists(path):
return None
@@ -265,12 +291,13 @@ class PlanStorage:
Returns:
List of todo list names
"""
from urllib.parse import unquote
todos = []
for filename in os.listdir(self.todos_dir):
if filename.endswith(".json"):
name = filename[:-5] # Remove .json extension
todos.append(name)
# Decode so listed names round-trip through load/delete_todo.
todos.append(unquote(filename[:-5]))
return todos
@@ -284,8 +311,7 @@ class PlanStorage:
Returns:
True if deleted, False if not found
"""
filename = f"{name}.json"
path = os.path.join(self.todos_dir, filename)
path = self._todo_path(name)
if os.path.exists(path):
os.remove(path)
@@ -73,28 +73,15 @@ def generate_title(
fallback_title = _create_fallback_title(user_msg, max_length)
try:
# Check if we're already in an event loop
asyncio.get_running_loop()
except RuntimeError:
# No running loop - safe to use asyncio.run
try:
return asyncio.run(generate_title_async(
user_msg, assistant_msg, llm_model, timeout, max_length,
primary_model=primary_model,
))
except Exception:
import logging
logger = logging.getLogger(__name__)
logger.debug("Title generation failed in sync context", exc_info=True)
return fallback_title
else:
# Already in event loop - cannot use asyncio.run
# Works with or without a running event loop (Issue #6238).
from ..utils.async_bridge import run_coroutine_from_any_context
return run_coroutine_from_any_context(generate_title_async(
user_msg, assistant_msg, llm_model, timeout, max_length,
primary_model=primary_model,
), timeout=None)
except Exception:
import logging
logger = logging.getLogger(__name__)
logger.warning(
"generate_title() called from running event loop; "
"use generate_title_async() instead. Returning fallback title."
)
logging.getLogger(__name__).debug("Title generation failed", exc_info=True)
return fallback_title
@@ -645,9 +645,8 @@ def crawl4ai_sync(
Returns:
Dict with crawl results
"""
return asyncio.run(
crawl4ai(url, css_selector, js_code, wait_for)
)
from ..utils.async_bridge import run_coroutine_from_any_context
return run_coroutine_from_any_context(crawl4ai(url, css_selector, js_code, wait_for), timeout=None)
def crawl4ai_extract_sync(
url: str,
@@ -664,9 +663,8 @@ def crawl4ai_extract_sync(
Returns:
Dict with extracted data
"""
return asyncio.run(
crawl4ai_extract(url, schema, js_code)
)
from ..utils.async_bridge import run_coroutine_from_any_context
return run_coroutine_from_any_context(crawl4ai_extract(url, schema, js_code), timeout=None)
if __name__ == "__main__":
# Example usage
@@ -126,11 +126,31 @@ def _matches_glob(fname: str, rel: str, glob: Optional[str]) -> bool:
if not glob:
return True
posix = rel.replace(os.sep, "/")
return (
if (
fnmatch.fnmatch(fname, glob)
or fnmatch.fnmatch(rel, glob)
or fnmatch.fnmatch(posix, glob)
)
):
return True
# fnmatch treats "**/" as requiring a slash. Recursive glob segments
# also match zero directories, as they do on the ripgrep path.
segments = glob.split("/")
if "**" not in segments:
return False
parts = posix.split("/")
positions = {0}
for segment in segments:
if not positions:
return False
if segment == "**":
positions = set(range(min(positions), len(parts) + 1))
else:
positions = {
i + 1 for i in positions
if i < len(parts) and fnmatch.fnmatch(parts[i], segment)
}
return len(parts) in positions
def _iter_files(root: str, glob: Optional[str], ignore_matcher):
@@ -9,8 +9,10 @@ the previous good file: the old bytes stay intact and ``os.replace`` swaps in th
new content atomically on both POSIX and Windows.
"""
import hashlib
import json
import os
import re
import tempfile
import threading
import time
@@ -21,9 +23,52 @@ __all__ = [
"atomic_write_json",
"atomic_write_text",
"quarantine_corrupt_file",
"safe_join",
"update_json",
]
# Max UTF-8 bytes for the final basename; leaves room under the common
# 255-byte limit for ``atomic_write_text``'s ``<basename>.XXXXXXXX.tmp`` file.
_MAX_NAME_BYTES = 200
_HASHED = re.compile(r"(.*)%~([0-9a-f]{16})", re.DOTALL)
_ESCAPE = re.compile(r'[%/\\:*?"<>|\x00-\x1f\x7f]')
def _escape(name: str) -> str:
stem = _ESCAPE.sub(lambda m: "%%%02X" % ord(m.group()), name)
return "%2E" + stem[1:] if stem.startswith(".") else stem
def safe_join(base: str, name: str, suffix: str = "") -> str:
"""Join a caller-supplied ``name`` under ``base`` without escaping it.
Only ``%``, path separators, control and Windows-reserved characters (and
a leading ``.``) are percent-encoded, so ordinary names - including spaces
and non-ASCII - keep their pre-#6238 filename, ``..`` and separators can
never leave ``base``, and ``urllib.parse.unquote`` recovers the name.
Names whose filename would exceed ``_MAX_NAME_BYTES`` are truncated and
tagged ``%~<sha256[:16]>``; that listed (unquoted) form maps back to the
same file, while no ordinary name can produce a raw ``%~`` (Issue #6238).
"""
name = str(name)
hashed = _HASHED.fullmatch(name)
if hashed:
stem = _escape(hashed.group(1)) + "%~" + hashed.group(2)
else:
stem = _escape(name)
budget = _MAX_NAME_BYTES - len(suffix.encode("utf-8"))
if len(stem.encode("utf-8", "surrogatepass")) > budget:
tag = "%~" + hashlib.sha256(name.encode("utf-8", "surrogatepass")).hexdigest()[:16]
kept, used = [], len(tag)
for tok in re.findall(r"%[0-9A-F]{2}|.", _escape(name), re.DOTALL):
used += len(tok.encode("utf-8", "surrogatepass"))
if used > budget:
break
kept.append(tok)
stem = "".join(kept) + tag
return os.path.join(base, stem + suffix)
# Per-path in-process mutexes so threads in one process serialise before they
# even reach the cross-process ``FileLock``. Keyed by absolute path.
_path_mutexes: "defaultdict[str, threading.RLock]" = defaultdict(threading.RLock)
+1 -1
View File
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "praisonaiagents"
version = "1.7.21"
description = "Praison AI agents for completing complex tasks with Self Reflection Agents"
description = "Open-source Python AI agent framework that runs on your own computers, even fully offline."
readme = "README.md"
requires-python = ">=3.10"
authors = [
@@ -0,0 +1,38 @@
"""Live check that async chat with a tool returns a model-written final answer.
Exercises the post-tool follow-up request in ``ChatMixin._achat_completion``
by turning off the default unified dispatch, so ``_achat_impl`` takes the
legacy OpenAI-client branch. Skipped unless ``OPENAI_API_KEY`` is set.
"""
import os
import pytest
from praisonaiagents import Agent
def get_secret_number() -> str:
"""Return the secret number."""
return "4217"
@pytest.mark.asyncio
@pytest.mark.skipif(not os.getenv("OPENAI_API_KEY"), reason="No API key")
async def test_achat_with_tool_returns_final_answer_live():
agent = Agent(
name="test",
instructions="You are a helpful assistant",
llm="gpt-4o-mini",
tools=[get_secret_number],
)
# The default unified path never calls _achat_completion.
agent._use_unified_llm_dispatch = False
result = await agent.astart(
"Use the get_secret_number tool, then reply with one sentence that "
"contains the secret number."
)
print(f"Agent output: {result}")
assert result
assert "4217" in str(result)
# Without the fix the raw tool output ("4217") is returned unchanged.
assert str(result).strip() != "4217"
@@ -169,3 +169,264 @@ async def test_execute_tool_async_resolves_aliased_function_name():
)
assert result == "aliased-ran"
@pytest.mark.asyncio
async def test_achat_completion_streams_final_response_after_tool_calls():
"""After running tools, the legacy async path must ask the model for a
final answer instead of failing and returning the raw tool output.
``_achat_completion`` used to reference an undefined ``stream`` name and
``_get_llm_functions`` (only defined in ``agent.py``), so the follow-up
completion always raised ``NameError``, which was swallowed and the raw
tool results were returned.
"""
agent = Agent(name="test", instructions="You are helpful", llm="gpt-4o-mini")
def lookup(query: str) -> str:
return "tool output"
tool_call = SimpleNamespace(
id="call-1",
function=SimpleNamespace(name="lookup", arguments='{"query": "x"}'),
)
response = SimpleNamespace(
choices=[SimpleNamespace(message=SimpleNamespace(tool_calls=[tool_call]))]
)
def _chunk(content):
return SimpleNamespace(
choices=[SimpleNamespace(delta=SimpleNamespace(content=content))]
)
async def _stream():
for part in ("final ", "answer"):
yield _chunk(part)
create = AsyncMock(return_value=_stream())
fake_client = SimpleNamespace(
async_client=SimpleNamespace(
chat=SimpleNamespace(completions=SimpleNamespace(create=create))
)
)
final = SimpleNamespace(
choices=[SimpleNamespace(message=SimpleNamespace(content="final answer"))]
)
process_stream_chunks = MagicMock(return_value=final)
with patch.object(
type(agent), "_openai_client", new=fake_client
), patch.object(
agent, "execute_tool_async", new_callable=AsyncMock, return_value="tool output"
), patch(
"praisonaiagents.agent.agent._get_llm_functions",
return_value={"process_stream_chunks": process_stream_chunks},
), patch(
"praisonaiagents.agent.chat_mixin._get_display_functions"
) as display:
result = await agent._achat_completion(response, tools=[lookup])
display.return_value.__getitem__.return_value.assert_not_called()
create.assert_awaited_once()
assert create.await_args.kwargs["stream"] is True
process_stream_chunks.assert_called_once()
assert result == "final answer"
@pytest.mark.asyncio
async def test_achat_completion_follow_up_keeps_conversation_and_skips_empty_chunks():
"""The post-tool follow-up request must keep the original conversation (so
the final answer still sees the user's question), and stream chunks with no
``choices`` (e.g. Azure content-filter annotations) must be skipped rather
than raising ``IndexError`` and falling back to the raw tool output.
"""
agent = Agent(name="test", instructions="You are helpful", llm="gpt-4o-mini")
def lookup(query: str) -> str:
return "tool output"
tool_call = SimpleNamespace(
id="call-1",
function=SimpleNamespace(name="lookup", arguments='{"query": "x"}'),
)
response = SimpleNamespace(
choices=[SimpleNamespace(message=SimpleNamespace(tool_calls=[tool_call]))]
)
conversation = [
{"role": "system", "content": "You are helpful"},
{"role": "user", "content": "Look up x and answer in French"},
]
async def _stream():
yield SimpleNamespace(choices=[])
yield SimpleNamespace(
choices=[SimpleNamespace(delta=SimpleNamespace(content="réponse"))]
)
create = AsyncMock(return_value=_stream())
fake_client = SimpleNamespace(
async_client=SimpleNamespace(
chat=SimpleNamespace(completions=SimpleNamespace(create=create))
)
)
final = SimpleNamespace(
choices=[SimpleNamespace(message=SimpleNamespace(content="réponse"))]
)
process_stream_chunks = MagicMock(return_value=final)
with patch.object(
type(agent), "_openai_client", new=fake_client
), patch.object(
agent, "execute_tool_async", new_callable=AsyncMock, return_value="tool output"
), patch(
"praisonaiagents.agent.agent._get_llm_functions",
return_value={"process_stream_chunks": process_stream_chunks},
), patch(
"praisonaiagents.agent.chat_mixin._get_display_functions"
) as display:
result = await agent._achat_completion(
response, tools=[lookup], messages=conversation
)
display.return_value.__getitem__.return_value.assert_not_called()
sent = create.await_args.kwargs["messages"]
assert sent[: len(conversation)] == conversation
assert "tool output" in sent[-1]["content"]
# The caller's list is not mutated.
assert len(conversation) == 2
assert result == "réponse"
@pytest.mark.asyncio
async def test_legacy_achat_impl_forwards_conversation_to_tool_follow_up():
"""The legacy ``_achat_impl`` tool branch must pass the conversation to
``_achat_completion`` so the follow-up request still has the user's prompt.
"""
agent = Agent(name="test", instructions="You are helpful", llm="gpt-4o-mini")
agent._using_custom_llm = False
agent._use_unified_llm_dispatch = False
def lookup(query: str) -> str:
return "tool output"
tool_call = SimpleNamespace(
id="call-1",
function=SimpleNamespace(name="lookup", arguments='{"query": "x"}'),
)
tool_response = SimpleNamespace(
choices=[SimpleNamespace(message=SimpleNamespace(tool_calls=[tool_call]))]
)
async def _stream():
yield SimpleNamespace(
choices=[SimpleNamespace(delta=SimpleNamespace(content="final answer"))]
)
create = AsyncMock(side_effect=[tool_response, _stream()])
fake_client = SimpleNamespace(
async_client=SimpleNamespace(
chat=SimpleNamespace(completions=SimpleNamespace(create=create))
)
)
final = SimpleNamespace(
choices=[SimpleNamespace(message=SimpleNamespace(content="final answer"))]
)
prompt_messages = [{"role": "user", "content": "Look up x for me"}]
with patch.object(type(agent), "_openai_client", new=fake_client), patch.object(
agent, "execute_tool_async", new_callable=AsyncMock, return_value="tool output"
), patch(
"praisonaiagents.agent.agent._get_llm_functions",
return_value={"process_stream_chunks": MagicMock(return_value=final)},
), patch.object(
agent, "_atrigger_after_agent_hook", new_callable=AsyncMock, side_effect=lambda p, r, t: r
), patch.object(agent, "_execute_callback_and_display"), patch.object(
agent, "_build_messages", return_value=(list(prompt_messages), "Look up x for me")
), patch.object(agent, "_persist_message"), patch.object(
agent, "_format_tools_for_completion", return_value=[{"type": "function"}]
):
result = await agent._achat_impl(
prompt="Look up x for me",
temperature=1.0,
tools=[lookup],
output_json=None,
output_pydantic=None,
reasoning_steps=False,
stream=False,
task_name=None,
task_description=None,
task_id=None,
config=None,
force_retrieval=False,
skip_retrieval=True,
attachments=None,
_trace_emitter=MagicMock(),
)
assert create.await_count == 2
follow_up = create.await_args_list[1].kwargs["messages"]
assert {"role": "user", "content": "Look up x for me"} in follow_up
assert result == "final answer"
@pytest.mark.asyncio
@pytest.mark.parametrize(
"follow_up_chunk",
[
pytest.param(SimpleNamespace(choices=[]), id="empty-choices"),
pytest.param(
SimpleNamespace(
choices=[SimpleNamespace(delta=SimpleNamespace(content=None))]
),
id="choice-without-content",
),
pytest.param(
SimpleNamespace(
choices=[SimpleNamespace(delta=SimpleNamespace(content="\n "))]
),
id="whitespace-only-content",
),
],
)
async def test_achat_completion_falls_back_when_follow_up_has_no_answer_chunks(follow_up_chunk):
"""A follow-up stream with no answer text (only empty-``choices`` chunks,
choices without ``delta.content`` or whitespace-only content) must log an error and return the tool
results, not an empty string.
"""
agent = Agent(name="test", instructions="You are helpful", llm="gpt-4o-mini")
def lookup(query: str) -> str:
return "tool output"
tool_call = SimpleNamespace(
id="call-1",
function=SimpleNamespace(name="lookup", arguments='{"query": "x"}'),
)
response = SimpleNamespace(
choices=[SimpleNamespace(message=SimpleNamespace(tool_calls=[tool_call]))]
)
async def _stream():
yield follow_up_chunk
create = AsyncMock(return_value=_stream())
fake_client = SimpleNamespace(
async_client=SimpleNamespace(
chat=SimpleNamespace(completions=SimpleNamespace(create=create))
)
)
process_stream_chunks = MagicMock()
with patch.object(type(agent), "_openai_client", new=fake_client), patch.object(
agent, "execute_tool_async", new_callable=AsyncMock, return_value="tool output"
), patch(
"praisonaiagents.agent.agent._get_llm_functions",
return_value={"process_stream_chunks": process_stream_chunks},
), patch(
"praisonaiagents.agent.chat_mixin._get_display_functions"
) as display:
result = await agent._achat_completion(response, tools=[lookup])
display.return_value.__getitem__.return_value.assert_called()
process_stream_chunks.assert_not_called()
assert result == "tool output"
@@ -0,0 +1,113 @@
"""The custom-LLM (``provider/model``) path sends each user turn once.
``chat()`` stores the user turn in ``chat_history`` before the model call, and
``LLM._build_messages`` appends ``prompt`` after ``chat_history`` itself, so
the stored copy must not be forwarded as history too. It was, on every
litellm route (``ollama/``, ``anthropic/``, ``openai/``, ``base_url=``), which
billed the prompt twice. ``litellm.completion`` is stubbed; nothing leaves the
machine.
"""
import asyncio
import pytest
pytest.importorskip("litellm")
from praisonaiagents import Agent # noqa: E402
MODEL = "anthropic/claude-3-5-sonnet-20240620"
def _litellm():
import litellm
return litellm
def _reply(model, text):
return _litellm().ModelResponse(model=model, usage={
"prompt_tokens": 10, "completion_tokens": 1, "total_tokens": 11},
choices=[{"index": 0, "finish_reason": "stop",
"message": {"role": "assistant", "content": text}}])
def _chunks(model, text):
litellm = _litellm()
yield litellm.ModelResponseStream(model=model, choices=[{
"index": 0, "delta": {"role": "assistant", "content": text}}])
yield litellm.ModelResponseStream(model=model, choices=[{
"index": 0, "delta": {}, "finish_reason": "stop"}])
@pytest.fixture
def sent(monkeypatch):
"""Record the ``messages`` of every model call; answer "A1", "A2", ..."""
monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-test")
monkeypatch.setattr(Agent, "_default_model", None)
monkeypatch.setattr(Agent, "_default_model_checked", False)
requests = []
def fake(*a, **k):
requests.append(k.get("messages"))
text = f"A{len(requests)}"
if k.get("stream"):
return _chunks(k.get("model"), text)
return _reply(k.get("model"), text)
async def afake(*a, **k):
return fake(*a, **k)
litellm = _litellm()
monkeypatch.setattr(litellm, "completion", fake)
monkeypatch.setattr(litellm, "acompletion", afake)
return requests
def _turns(messages):
return [(m["role"], m["content"]) for m in messages if m["role"] != "system"]
def _agent():
return Agent(name="p", instructions="Be brief.", llm=MODEL, output="silent")
def test_chat_sends_the_prompt_once(sent):
agent = _agent()
agent.chat("Q1")
agent.chat("Q2")
assert _turns(sent[0]) == [("user", "Q1")]
assert _turns(sent[1]) == [("user", "Q1"), ("assistant", "A1"), ("user", "Q2")]
def test_achat_sends_the_prompt_once(sent):
agent = _agent()
async def run():
await agent.achat("Q1")
await agent.achat("Q2")
asyncio.run(run())
assert _turns(sent[0]) == [("user", "Q1")]
assert _turns(sent[1]) == [("user", "Q1"), ("assistant", "A1"), ("user", "Q2")]
def test_streamed_turn_sends_the_prompt_once(sent):
agent = _agent()
assert "".join(agent.start("Q1", stream=True)) == "A1"
assert _turns(sent[0]) == [("user", "Q1")]
def test_history_still_records_each_turn_once(sent):
agent = _agent()
agent.chat("Q1")
agent.chat("Q2")
assert [(m["role"], m["content"]) for m in agent.chat_history] == [
("user", "Q1"), ("assistant", "A1"), ("user", "Q2"), ("assistant", "A2")]
def test_repeating_the_last_question_is_still_one_turn(sent):
# A user who asks the same thing twice in a row sends it once per request,
# with the earlier exchange kept as history.
agent = _agent()
agent.chat("Q1")
agent.chat("Q1")
assert _turns(sent[1]) == [("user", "Q1"), ("assistant", "A1"), ("user", "Q1")]
@@ -38,7 +38,7 @@ def test_cache_hit_refreshes_current_default_tools(agent, monkeypatch):
builds = []
monkeypatch.setattr(
agent, '_resolve_harness_base_prompt',
lambda: builds.append('build') or 'stable prefix',
lambda tools=None: builds.append('build') or 'stable prefix',
)
first = agent._build_system_prompt(tools=None)
assert 'sample_tool' in first
@@ -73,7 +73,7 @@ def test_cache_hit_refreshes_session_context(agent, monkeypatch):
def test_cache_hit_retains_prefix_reuse_and_appends_suffix_once(agent, monkeypatch):
"""Reuse the stable prefix while adding the suffix exactly once."""
calls = []
agent._resolve_harness_base_prompt = lambda: calls.append('build') or 'stable prefix'
agent._resolve_harness_base_prompt = lambda tools=None: calls.append('build') or 'stable prefix'
monkeypatch.setenv('PRAISONAI_APPEND_SYSTEM_PROMPT', 'per-turn suffix')
first = agent._build_system_prompt()
second = agent._build_system_prompt()
@@ -0,0 +1,230 @@
"""The system prompt carries only guidance that applies to the agent.
* The model-family file-editing hint ("use edit_file ..." / "use
apply_patch ...") is for agents that can edit files; every GPT or Claude
agent got it, tools or not.
* ``Agent(instructions=...)`` sets both backstory and goal to the
instructions, so the prompt repeated them under "Your Goal:".
"""
import pytest
from praisonaiagents import Agent
from praisonaiagents.tools.edit_tools import apply_patch, edit_file
OPENAI_HINT = "use edit_file to modify existing files precisely"
ANTHROPIC_HINT = "use apply_patch"
def lookup(query: str) -> str:
"""Look something up."""
return query
@pytest.fixture
def make(monkeypatch):
monkeypatch.setenv("OPENAI_API_KEY", "sk-test-not-used")
monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-test-not-used")
made = []
def _make(**kw):
kw.setdefault("instructions", "Be brief.")
agent = Agent(context=False, **kw)
made.append(agent)
return agent
yield _make
for agent in made:
agent.close()
@pytest.mark.parametrize("tools", [None, [lookup]])
def test_agent_without_edit_tools_gets_no_edit_hint(make, tools):
agent = make(llm="gpt-4o-mini", tools=tools)
assert OPENAI_HINT not in agent._build_system_prompt()
def test_agent_with_edit_file_gets_the_openai_hint(make):
agent = make(llm="gpt-4o-mini", tools=[edit_file, lookup])
assert OPENAI_HINT in agent._build_system_prompt()
def test_agent_with_apply_patch_gets_the_anthropic_hint(make):
agent = make(llm="anthropic/claude-sonnet-4-5", tools=[apply_patch])
assert ANTHROPIC_HINT in agent._build_system_prompt()
def test_edit_tool_named_by_string_counts(make):
agent = make(llm="gpt-4o-mini")
assert OPENAI_HINT in agent._build_system_prompt(tools=["edit_file"])
def test_per_call_tools_decide_the_hint(make):
agent = make(llm="gpt-4o-mini", tools=[edit_file])
assert OPENAI_HINT not in agent._build_system_prompt(tools=[lookup])
assert OPENAI_HINT in agent._build_system_prompt(tools=[edit_file])
def test_cached_prompt_follows_a_change_to_the_agents_tools(make):
agent = make(llm="gpt-4o-mini", tools=[lookup])
assert OPENAI_HINT not in agent._build_system_prompt()
agent.tools = [edit_file]
assert OPENAI_HINT in agent._build_system_prompt()
agent.tools = [lookup]
assert OPENAI_HINT not in agent._build_system_prompt()
def test_explicit_harness_override_is_always_used(make):
agent = make(llm="gpt-4o-mini")
agent.harness_base_prompt = "House rules."
assert agent._build_system_prompt().startswith("House rules.")
def test_instructions_appear_once(make):
agent = make(llm="gpt-4o-mini", instructions="Answer in French.")
prompt = agent._build_system_prompt()
assert prompt.count("Answer in French.") == 1
assert "Your Goal:" not in prompt
def test_a_distinct_goal_is_still_stated(make):
agent = make(llm="gpt-4o-mini", instructions=None, role="Editor",
goal="Tighten the copy", backstory="A veteran sub-editor")
prompt = agent._build_system_prompt()
assert "Your Goal: Tighten the copy" in prompt
assert "A veteran sub-editor" in prompt
assert "Your Role: Editor" in prompt
def test_a_tool_renamed_to_edit_file_counts(make):
from praisonaiagents import tool
@tool(name="edit_file")
def my_editor(filepath: str, old_string: str, new_string: str) -> str:
"""Edit a file."""
return "ok"
agent = make(llm="gpt-4o-mini", tools=[my_editor])
assert OPENAI_HINT in agent._build_system_prompt()
def test_nested_tool_list_offers_the_edit_hint(make):
# _format_tools_for_completion flattens nested lists of tool dicts.
agent = make(llm="gpt-4o-mini", tools=[[EDIT_SCHEMA]])
assert OPENAI_HINT in agent._build_system_prompt()
def test_nested_names_the_formatter_drops_get_no_hint(make):
# A nested string is not sent to the model, so neither is the hint.
agent = make(llm="gpt-4o-mini", tools=[["edit_file"]])
assert OPENAI_HINT not in agent._build_system_prompt()
def test_nested_bare_name_dict_gets_no_hint(make):
# A nested dict without a function schema is dropped by the formatter.
agent = make(llm="gpt-4o-mini", tools=[[{"name": "edit_file"}]])
assert agent._format_tools_for_completion() == []
assert OPENAI_HINT not in agent._build_system_prompt()
EDIT_SCHEMA = {"type": "function", "function": {
"name": "edit_file", "description": "Edit a file.",
"parameters": {"type": "object", "properties": {}}}}
def _system_sent(agent, monkeypatch, run):
"""Run ``run(agent)`` on the native path; return the system prompt and tool names sent."""
from types import SimpleNamespace
sent = []
sent_tools = []
def fake(messages=None, temperature=None, tools=None, *a, **k):
sent.append(messages if messages is not None else k.get("messages"))
# Record the payload the real request builds (inherit, flatten, prune).
formatted = agent._format_tools_for_completion(tools)
sent_tools.append([t.get("function", {}).get("name") for t in formatted])
message = SimpleNamespace(content="ok", tool_calls=None, role="assistant", reasoning_content=None)
return SimpleNamespace(choices=[SimpleNamespace(message=message, finish_reason="stop", index=0)],
usage=SimpleNamespace(prompt_tokens=1, completion_tokens=1, total_tokens=2),
model="gpt-4o-mini")
async def afake(*a, **k):
return fake(*a, **k)
monkeypatch.setattr(agent, "_chat_completion", fake)
monkeypatch.setattr(agent, "_achat_completion_with_retry", afake)
run(agent)
return next(m["content"] for m in sent[-1] if m["role"] == "system"), sent_tools[-1]
def test_native_chat_follows_per_call_edit_tools(make, monkeypatch):
agent = make(llm="gpt-4o-mini")
system, tool_names = _system_sent(agent, monkeypatch, lambda a: a.chat("hi", tools=[EDIT_SCHEMA]))
assert OPENAI_HINT in system
assert "edit_file" in tool_names
def test_nested_edit_schema_is_sent_with_the_hint(make, monkeypatch):
agent = make(llm="gpt-4o-mini", tools=[[EDIT_SCHEMA]])
system, tool_names = _system_sent(agent, monkeypatch, lambda a: a.chat("hi"))
assert OPENAI_HINT in system
assert "edit_file" in tool_names
def test_permission_denied_edit_tool_gets_no_hint(make, monkeypatch):
agent = make(llm="gpt-4o-mini", tools=[edit_file, lookup])
agent._perm_deny = frozenset({"edit_file"})
system, tool_names = _system_sent(agent, monkeypatch, lambda a: a.chat("hi"))
assert "edit_file" not in tool_names
assert OPENAI_HINT not in system
def test_deny_added_after_caching_prunes_cached_tools(make):
agent = make(llm="gpt-4o-mini", tools=[edit_file, lookup])
assert "edit_file" in [t["function"]["name"] for t in agent._format_tools_for_completion()]
agent._perm_deny = frozenset({"edit_file"})
assert "edit_file" not in [t["function"]["name"] for t in agent._format_tools_for_completion()]
assert OPENAI_HINT not in agent._build_system_prompt()
# Lifting the denial restores the tool from the same cache entry.
agent._perm_deny = frozenset()
assert "edit_file" in [t["function"]["name"] for t in agent._format_tools_for_completion()]
def test_deny_before_first_format_restores_after_lift(make):
agent = make(llm="gpt-4o-mini", tools=[edit_file, lookup])
agent._perm_deny = frozenset({"edit_file"})
assert "edit_file" not in [t["function"]["name"] for t in agent._format_tools_for_completion()]
agent._perm_deny = frozenset()
assert "edit_file" in [t["function"]["name"] for t in agent._format_tools_for_completion()]
def test_bridge_tool_search_hides_denied_tools(make):
agent = make(llm="gpt-4o-mini")
agent._tool_search_config = None
agent._tool_search_metadata = {"bridge_mode": True, "deferrable_tools": [EDIT_SCHEMA]}
agent._perm_deny = frozenset({"edit_file"})
out = agent._handle_bridge_tool_call("tool_describe", {"tool_name": "edit_file"})
assert "not found" in out
agent._perm_deny = frozenset()
assert "not found" not in agent._handle_bridge_tool_call("tool_describe", {"tool_name": "edit_file"})
def test_native_achat_follows_per_call_edit_tools(make, monkeypatch):
import asyncio
agent = make(llm="gpt-4o-mini")
system, tool_names = _system_sent(agent, monkeypatch, lambda a: asyncio.run(a.achat("hi", tools=[EDIT_SCHEMA])))
assert OPENAI_HINT in system
assert "edit_file" in tool_names
def test_system_prompt_attribute_states_instructions_once(make):
agent = make(llm="gpt-4o-mini", instructions="Answer in French.")
assert agent.system_prompt.count("Answer in French.") == 1
def test_cached_prompt_follows_a_backstory_change(make):
agent = make(llm="gpt-4o-mini", instructions="Answer in French.")
assert "Your Goal:" not in agent._build_system_prompt()
agent.backstory = "A translator"
prompt = agent._build_system_prompt()
assert "A translator" in prompt
assert "Your Goal: Answer in French." in prompt
@@ -32,6 +32,7 @@ def _agent_with_history(messages: int, words: int):
def fake_stream(**kwargs):
sent["history"] = list(kwargs.get("chat_history") or [])
sent["prompt"] = kwargs.get("prompt")
yield "ok"
agent._using_custom_llm = True
@@ -60,7 +61,10 @@ class TestCustomLlmStreamCompaction:
"""Compaction must not fire when it is not needed."""
agent, sent = _agent_with_history(6, 5)
history = _drain(agent, sent)
assert len(history) == 7, "a short history was compacted unnecessarily"
# The six earlier turns go as history; this turn goes once, as the prompt.
assert len(history) == 6, "a short history was compacted unnecessarily"
assert sent["prompt"] == "hello"
assert all(m["content"] != "hello" for m in history)
def test_the_context_manager_is_actually_configured_here(self):
"""Guards the premise: tools present auto-enable context management."""
@@ -0,0 +1,73 @@
"""Automatic promotion must honor the same capacity as explicit long-term writes."""
import json
import pytest
from praisonaiagents.memory.file_memory import FileMemory
@pytest.mark.parametrize("limit", [1, 2, 3])
def test_promotion_enforces_persisted_long_term_limit(tmp_path, limit):
memory = FileMemory(
base_path=str(tmp_path),
config={"short_term_limit": 2, "long_term_limit": limit},
)
for index in range(6):
memory.add_short_term(f"Important fact {index}", importance=0.8)
assert len(memory.get_long_term()) == limit
assert len(FileMemory(base_path=str(tmp_path)).get_long_term()) == limit
assert len(memory.get_short_term()) == 2
def test_promotion_keeps_highest_importance_and_newest_ties(tmp_path):
memory = FileMemory(
base_path=str(tmp_path),
config={"short_term_limit": 3, "long_term_limit": 2},
)
memory.add_long_term("Keep strongest", importance=1.0)
memory.add_long_term("Older tie", importance=0.8)
memory.add_short_term("Lower priority", importance=0.7)
memory.add_short_term("Newer tie", importance=0.8, metadata={"source": "fixture"})
memory.add_short_term("Below promotion threshold", importance=0.2)
for path, timestamp in [(memory.long_term_file, 100), (memory.short_term_file, 200)]:
records = json.loads(path.read_text())
for record in records:
record["created_at"] = timestamp
path.write_text(json.dumps(records))
promoted_before = next(
item for item in json.loads(memory.short_term_file.read_text())
if item["content"] == "Newer tie"
)
memory.add_short_term("Trigger overflow", importance=0.2)
contents = {item.content for item in memory.get_long_term()}
assert contents == {"Keep strongest", "Newer tie"}
promoted_after = next(item for item in memory.get_long_term() if item.content == "Newer tie")
assert promoted_after.to_dict() == promoted_before
assert {item.content for item in memory.get_short_term()} == {
"Newer tie", "Below promotion threshold", "Trigger overflow",
}
assert {item.content for item in FileMemory(base_path=str(tmp_path)).get_long_term()} == contents
def test_disabled_promotion_preserves_existing_long_term(tmp_path):
memory = FileMemory(
base_path=str(tmp_path),
config={"short_term_limit": 1, "long_term_limit": 1, "auto_promote": False},
)
memory.add_long_term("Existing fact")
for index in range(3):
memory.add_short_term(f"Unpromoted {index}", importance=1.0)
assert [item.content for item in memory.get_long_term()] == ["Existing fact"]
def test_promotion_zero_capacity_matches_explicit_long_term(tmp_path):
memory = FileMemory(
base_path=str(tmp_path),
config={"short_term_limit": 1, "long_term_limit": 0},
)
memory.add_long_term("Explicit fact", importance=1.0)
assert memory.get_long_term() == []
memory.add_short_term("Promoted one", importance=1.0)
memory.add_short_term("Promoted two", importance=1.0)
assert memory.get_long_term() == []
assert FileMemory(base_path=str(tmp_path)).get_long_term() == []
@@ -34,11 +34,11 @@ def _restore_approval_state():
from praisonaiagents.approval import get_approval_registry, clear_approval_context
registry = get_approval_registry()
previous = registry.get_backend()
previous = registry._global_backend # not get_backend(): its ConsoleBackend fallback would be installed globally on restore
try:
yield
finally:
registry.set_backend(previous)
registry._global_backend = previous
clear_approval_context()
@@ -154,7 +154,7 @@ def test_code_mode_changes_what_the_tool_can_do():
_probe_tool_name = "code_mode_probe"
get_registry().register(code_mode_probe)
previous = get_approval_registry().get_backend()
previous = get_approval_registry()._global_backend # not get_backend(): its ConsoleBackend fallback would be installed globally on restore
get_approval_registry().set_backend(AutoApproveBackend())
try:
safe = Agent(
@@ -182,7 +182,7 @@ def test_code_mode_changes_what_the_tool_can_do():
safe_result = safe_exec(code)
unsafe_result = unsafe_exec(code)
finally:
get_approval_registry().set_backend(previous)
get_approval_registry()._global_backend = previous
clear_approval_context()
# The tool registry is process-global. Leaving these probe tools in it
# changed what later tests saw when they enumerated or dispatched
@@ -221,7 +221,7 @@ def test_unsafe_code_mode_cannot_reach_tools_the_agent_was_not_granted():
_probe_tool_name = "ungranted_global_tool"
get_registry().register(ungranted_global_tool)
previous = get_approval_registry().get_backend()
previous = get_approval_registry()._global_backend # not get_backend(): its ConsoleBackend fallback would be installed globally on restore
get_approval_registry().set_backend(AutoApproveBackend())
try:
agent = Agent(
@@ -240,7 +240,7 @@ def test_unsafe_code_mode_cannot_reach_tools_the_agent_was_not_granted():
)
result = execute("print(ungranted_global_tool())")
finally:
get_approval_registry().set_backend(previous)
get_approval_registry()._global_backend = previous
clear_approval_context()
# The tool registry is process-global. Leaving these probe tools in it
# changed what later tests saw when they enumerated or dispatched
@@ -279,13 +279,13 @@ def test_code_execution_runs_a_trailing_expression_exactly_once():
from praisonaiagents.approval import clear_approval_context
registry = get_approval_registry()
previous = registry.get_backend()
previous = registry._global_backend # not get_backend(): its ConsoleBackend fallback would be installed globally on restore
registry.set_backend(AutoApproveBackend())
try:
assert execute_code("print(7)")["stdout"] == "7\n"
assert execute_code_with_tools("print(7)")["stdout"] == "7\n"
finally:
registry.set_backend(previous)
registry._global_backend = previous
clear_approval_context()
@@ -79,7 +79,7 @@ def test_no_handrolled_async_bridges_outside_helper():
for path in root.rglob("*.py"):
if path in allowed:
continue
tree = ast.parse(path.read_text(), filename=str(path))
tree = ast.parse(path.read_bytes(), filename=str(path))
for func in ast.walk(tree):
if not isinstance(func, (ast.FunctionDef, ast.AsyncFunctionDef)):
continue
@@ -93,6 +93,15 @@ def test_no_handrolled_async_bridges_outside_helper():
)
def test_scan_accepts_python_source_encoding_cookie(tmp_path, monkeypatch):
"""Scan the same Latin-1 source that Python's parser accepts."""
(tmp_path / "example.py").write_bytes(
b"# coding: latin-1; caf\xe9\nmessage = 'caf\xe9'\n"
)
monkeypatch.setitem(globals(), "_package_root", lambda: tmp_path)
test_no_handrolled_async_bridges_outside_helper()
def test_tool_timeout_uses_daemon_worker_and_does_not_leak():
before = {t.ident for t in threading.enumerate()}
@@ -0,0 +1,85 @@
"""Counting rules of benchmarks/execution_benchmark.py, with fake runners.
No framework is imported and nothing leaves the machine: each runner is a
``build()`` that returns the call to time, as the script's real runners do.
"""
import importlib.util
from pathlib import Path
import pytest
SCRIPT = Path(__file__).resolve().parents[2] / "benchmarks" / "execution_benchmark.py"
@pytest.fixture(scope="module")
def bench():
spec = importlib.util.spec_from_file_location("execution_benchmark", SCRIPT)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def runner(answers, log, name):
"""A build() whose calls return ``answers`` in order (an Exception is raised)."""
answers = iter(answers)
def build():
log.append(("build", name))
def call():
log.append(("call", name))
answer = next(answers)
if isinstance(answer, Exception):
raise answer
return answer
return call
return build
def test_warm_up_calls_are_not_timed_and_rounds_interleave(bench):
log = []
runners = {"A": runner(["w", "a1", "a2"], log, "A"), "B": runner(["w", "b1", "b2"], log, "B")}
results = bench.run_benchmark(runners, iterations=2, warmup=1)
assert [len(r["times"]) for r in results.values()] == [2, 2]
calls = [name for kind, name in log if kind == "call"]
assert calls == ["A", "B", "A", "B", "B", "A"] # warm-ups, round 1, round 2 rotated
def test_errors_and_empty_answers_are_failures_not_timings(bench):
log = []
runners = {"A": runner(["w", RuntimeError("401"), None, "None", " ", "4"], log, "A")}
results = bench.run_benchmark(runners, iterations=5, warmup=1)
assert len(results["A"]["times"]) == 1
assert len(results["A"]["failures"]) == 4
assert "RuntimeError: 401" in results["A"]["failures"][0]
def test_a_failed_warm_up_does_not_drop_the_framework(bench):
log = []
runners = {"A": runner([RuntimeError("timeout"), "a1", "a2"], log, "A")}
results = bench.run_benchmark(runners, iterations=2, warmup=1)
assert len(results["A"]["times"]) == 2
def test_summary_ranks_by_median_and_skips_frameworks_without_a_success(bench):
results = {
"slow": {"times": [1.0, 1.1, 9.0], "failures": []},
"fast": {"times": [0.5, 0.6, 0.7], "failures": ["x"]},
"broken": {"times": [], "failures": ["401"]},
}
rows = bench.summarise(results)
assert [r[0] for r in rows] == ["fast", "slow"]
name, median, mean, ok, failed = rows[1]
assert (median, ok, failed) == (1.1, 3, 0)
assert mean == pytest.approx(11.1 / 3)
def test_only_the_call_is_timed(bench, monkeypatch):
clock = iter([10.0, 10.5]) # start / end of the one timed call; build() reads no clock
def build():
return lambda: "4"
monkeypatch.setattr(bench.time, "perf_counter", lambda: next(clock))
results = bench.run_benchmark({"A": build}, iterations=1, warmup=0)
assert results["A"]["times"] == [0.5]
@@ -328,7 +328,7 @@ class TestMultiChannelGatewayConfig:
from praisonaiagents.gateway.config import ChannelRouteConfig, MultiChannelGatewayConfig
module = sys.modules["praisonaiagents.gateway.config"]
source_file = module.__file__
with open(source_file, "r") as f:
with open(source_file, "r", encoding="utf-8") as f:
content = f.read()
for dep in ["chromadb", "fastapi", "uvicorn", "pydantic", "litellm"]:
assert f"import {dep}" not in content, f"Found heavy dep '{dep}' in config.py"
@@ -376,7 +376,7 @@ class TestChatCommandInfo:
from praisonaiagents.bots.protocols import BotProtocol
module = sys.modules["praisonaiagents.bots.protocols"]
source_file = module.__file__
with open(source_file, "r") as f:
with open(source_file, "r", encoding="utf-8") as f:
content = f.read()
for dep in ["chromadb", "fastapi", "uvicorn", "litellm"]:
assert f"import {dep}" not in content, f"Found heavy dep '{dep}' in protocols.py"
@@ -0,0 +1,107 @@
"""Regression tests for Issue #6238.
* Caller-supplied names must never escape the storage directory.
* Sync entry points must work from inside a running event loop.
"""
import asyncio
import os
from praisonaiagents.planning.storage import PlanStorage
from praisonaiagents.planning.todo import TodoList
from praisonaiagents.utils.atomic_io import safe_join
def test_safe_join_keeps_legacy_names_and_confines_others(tmp_path):
base = str(tmp_path)
assert safe_join(base, "current", ".json") == os.path.join(base, "current.json")
for name in ("../../x", "/etc/passwd", "..", "", "a/b", "a\\b", ".hidden"):
path = safe_join(base, name, ".json")
assert os.path.dirname(path) == base, name
# Distinct inputs never collide on one file.
assert safe_join(base, "a/b") != safe_join(base, "a_b")
def test_plan_storage_todo_name_cannot_traverse(tmp_path):
storage = PlanStorage(base_path=str(tmp_path / "store"))
path = storage.save_todo(TodoList(), name="../../escaped")
assert os.path.dirname(path) == storage.todos_dir
assert not (tmp_path / "escaped.json").exists()
assert storage.load_todo("../../escaped") is not None
assert storage.delete_todo("../../escaped") is True
def test_generate_title_works_inside_running_loop(monkeypatch):
from praisonaiagents.session import title
async def fake(*args, **kwargs):
return "LLM Title"
monkeypatch.setattr(title, "generate_title_async", fake)
async def main():
return title.generate_title("hello", "world")
assert asyncio.run(main()) == "LLM Title"
def test_listed_todo_names_round_trip(tmp_path):
storage = PlanStorage(base_path=str(tmp_path / "store"))
storage.save_todo(TodoList(), name="my tasks")
(listed,) = storage.list_todos()
assert storage.load_todo(listed) is not None
assert storage.load_todo("my tasks") is not None
assert storage.delete_todo(listed) is True
assert storage.list_todos() == []
def test_legacy_todo_file_still_found(tmp_path):
storage = PlanStorage(base_path=str(tmp_path / "store"))
legacy = os.path.join(storage.todos_dir, "my tasks.json")
with open(legacy, "w", encoding="utf-8") as f:
f.write(TodoList().to_json())
assert storage.load_todo("my tasks") is not None
assert storage.save_todo(TodoList(), name="my tasks") == legacy
assert storage.list_todos() == ["my tasks"]
assert storage.delete_todo("my tasks") is True
def test_custom_plan_filename_keeps_md_suffix(tmp_path):
from praisonaiagents.planning.plan import Plan
storage = PlanStorage(base_path=str(tmp_path / "store"))
plan = Plan(name="demo")
path = storage.save_plan(plan, filename="my plan.md")
assert path.endswith(".md") and os.path.dirname(path) == storage.plans_dir
assert len(storage.list_plans()) == 1
def test_encoded_looking_names_do_not_share_a_file(tmp_path):
storage = PlanStorage(base_path=str(tmp_path / "store"))
names = ["a/b", "a_b~c14cddc033f6", "a%2Fb", "a b", "a%20b", "~x", ".."]
paths = {storage.save_todo(TodoList(name=n), name=n) for n in names}
assert len(paths) == len(names)
assert all(os.path.dirname(p) == storage.todos_dir for p in paths)
assert sorted(storage.list_todos()) == sorted(names)
for n in names:
assert storage.load_todo(n).name == n
assert storage.delete_todo("a/b") is True
assert storage.load_todo("a%2Fb") is not None
def test_long_unicode_names_are_bounded_and_round_trip(tmp_path):
storage = PlanStorage(base_path=str(tmp_path / "store"))
long_a, long_b = "我的任务" * 40, "我的任务" * 40 + "x"
pa = storage.save_todo(TodoList(name=long_a), name=long_a)
pb = storage.save_todo(TodoList(name=long_b), name=long_b)
assert pa != pb
for p in (pa, pb):
assert os.path.dirname(p) == storage.todos_dir
assert len(os.path.basename(p).encode("utf-8")) <= 200
# Short non-ASCII names keep their plain (pre-#6238) filename.
assert storage.save_todo(TodoList(), name="我的任务").endswith("我的任务.json")
for listed in storage.list_todos():
assert storage.load_todo(listed) is not None
assert storage.delete_todo(listed) is True
assert storage.list_todos() == []
assert storage.load_todo(long_a) is None
@@ -64,6 +64,35 @@ def test_grep_glob_filters_by_pattern(sample_tree):
assert "src" not in out
@pytest.mark.parametrize("pattern", ["**/*.py", "src/**/*.py", "src/**/**/test_*.py"])
def test_grep_recursive_glob_matches_zero_directories(sample_tree, pattern):
from praisonaiagents.tools import grep
direct = sample_tree / "src" / "test_direct.py"
direct.write_text("needle\n")
nested = sample_tree / "src" / "nested"
nested.mkdir()
(nested / "test_nested.py").write_text("needle\n")
with patch("shutil.which", return_value=None):
out = grep("needle", glob=pattern)
assert "src/test_direct.py:1:" in out.replace(os.sep, "/")
assert "src/nested/test_nested.py:1:" in out.replace(os.sep, "/")
if pattern == "**/*.py":
assert "a.py:2:" in out
else:
assert "a.py:2:" not in out
if "test_" in pattern:
assert "src/d.py:1:" not in out.replace(os.sep, "/")
def test_grep_recursive_glob_single_file(sample_tree):
from praisonaiagents.tools import grep
with patch("shutil.which", return_value=None):
out = grep("needle", path="a.py", glob="**/*.py")
assert "a.py:2:" in out
def test_grep_case_insensitive(sample_tree):
from praisonaiagents.tools import grep
with patch("shutil.which", return_value=None):
@@ -491,7 +491,7 @@ class TestTeamTasks:
return ApprovalDecision(approved=True, reason="looks good", approver="ana")
registry = get_approval_registry()
previous = registry.get_backend()
previous = registry._global_backend # not get_backend(): its ConsoleBackend fallback would be installed globally on restore
registry.set_backend(Reviewer())
try:
writer = Agent(name="Writer", instructions="write")
@@ -500,7 +500,7 @@ class TestTeamTasks:
out = TaskOutput(description="Draft", raw="api_key=sk-secret123456 text", agent="Writer")
team._apply_human_review(task, 0, out)
finally:
registry.set_backend(previous)
registry._global_backend = previous
requested, decided = _of(recorder, "approval_requested"), _of(recorder, "approval_decided")
assert len(requested) == len(decided) == 1
assert requested[0].data["kind"] == "task_output_review"
@@ -4646,7 +4646,7 @@ class WebSocketGateway:
elif msg_type == "server_reply":
# Issue #5351: a client's correlated answer to a server→client
# interactive request (approval / choice / input).
# interactive request (approval / choice / input / secret).
request_id = data.get("request_id")
value = data.get("value")
if not request_id or not isinstance(value, str):
@@ -4680,7 +4680,7 @@ class WebSocketGateway:
return True
# Answering an approval on behalf of the turn is an approval decision
# and requires the APPROVALS scope (parity with approvals.resolve);
# choice/input answers require WRITE like sending a message.
# choice/input/secret answers require WRITE like sending a message.
required_scope = (
OperatorScope.APPROVALS if req_kind == "approval" else OperatorScope.WRITE
)
@@ -5851,7 +5851,9 @@ class WebSocketGateway:
* ``approval`` → ``"allow"`` or ``"deny"``.
* ``choice`` → one of the offered ``options`` (any value if none given).
* ``input`` → any string.
* ``input`` / ``secret`` → any string. A ``secret`` value is handed
only to the waiting caller — never logged, echoed, or journalled
(Issue #6223).
"""
kind = getattr(req, "kind", "input")
if kind == "approval":
@@ -67,6 +67,37 @@ def test_server_reply_as_dict():
}
def test_server_reply_repr_hides_value():
reply = GatewayServerReply(request_id="r1", value="hunter2")
assert "hunter2" not in repr(reply)
assert reply.value == "hunter2"
def test_secret_request_resolves_only_to_caller():
"""Issue #6223: a secret reply reaches the waiting caller and nowhere else."""
async def go():
gw = WebSocketGateway()
sent = _connect(gw, "c1", "sess-1")
req = GatewayServerRequest(
request_id="req-s", kind="secret", prompt="sudo password?",
session_id="sess-1",
)
task = asyncio.create_task(gw.request_channel.request(req, timeout_s=5))
await asyncio.sleep(0)
assert sent[0]["request"]["kind"] == "secret"
await gw._handle_client_message(
"c1", {"type": "server_reply", "request_id": "req-s", "value": "hunter2"}
)
reply = await asyncio.wait_for(task, timeout=1)
assert reply.value == "hunter2"
# The value is never echoed back over the wire (ack carries no value).
assert all("hunter2" not in repr(frame) for frame in sent)
assert gw.request_channel.open_requests("sess-1") == []
asyncio.run(go())
# ---------------------------------------------------------------------------
# Test harness: a gateway with a connected, session-bound client that records
# every frame the server sends it.
@@ -1712,6 +1712,7 @@ def run_main(
# execution path does not discover and execute the same definitions twice.
preloaded_agent_config = None
custom_agent_model = None
custom_agent_base_url = None
if target or agent or command: # Any execution surface may need model/budget resolution
import sys
from praisonai_code.llm.credentials import ensure_configured_or_onboard
@@ -1731,6 +1732,10 @@ def run_main(
_preview_llm = (_preview_config or {}).get("llm")
if isinstance(_preview_llm, dict):
custom_agent_model = _preview_llm.get("model")
# Gate an Ollama agent on the server it will really call.
_base = _preview_llm.get("base_url")
if isinstance(_base, str) and _base.strip():
custom_agent_base_url = _base.strip()
else:
custom_agent_model = _preview_llm
if isinstance(custom_agent_model, str):
@@ -1747,7 +1752,9 @@ def run_main(
# endpoint, credentials, sampling, and retry options.
credential_model = model or custom_agent_model
resolved_model = ensure_configured_or_onboard(
model=credential_model, interactive=not _headless
model=credential_model,
interactive=not _headless,
base_url=None if model else custom_agent_base_url,
)
if not (agent and model is None and custom_agent_model):
model = resolved_model
@@ -123,6 +123,43 @@ def detect_local_endpoint():
return None
_OLLAMA_PREFIXES = ("ollama/", "ollama_chat/")
def _ollama_host_for_model(
model: Optional[str], base_url: Optional[str] = None
) -> Optional[str]:
"""Return the Ollama host to probe for an ``ollama/`` model, else ``None``.
Ollama needs no key, so an explicit Ollama model is gated on the server
being reachable rather than on a credential. The endpoint the caller will
actually use (e.g. an agent's ``llm.base_url``) wins, then
``OLLAMA_API_BASE`` (what the runtime honours), then ``OLLAMA_HOST``, then
the default port.
"""
import os
if not model or not model.lower().startswith(_OLLAMA_PREFIXES):
return None
from praisonai_code.llm.local_detect import _DEFAULT_OLLAMA_HOST
return (
base_url
or os.environ.get("OLLAMA_API_BASE")
or os.environ.get("OLLAMA_HOST")
or _DEFAULT_OLLAMA_HOST
)
def _ollama_reachable(host: str) -> bool:
try:
from praisonai_code.llm.local_detect import ollama_reachable
return ollama_reachable(host)
except Exception:
return False
def _provider_key_vars_for_model(model: str) -> tuple[str, ...]:
"""Map a model id to the environment variable(s) for its provider.
@@ -258,6 +295,10 @@ def is_configured(model: Optional[str] = None) -> bool:
if model is None:
model = default_model_for_available_provider()
ollama_host = _ollama_host_for_model(model) if explicit_model else None
if ollama_host is not None:
return _ollama_reachable(ollama_host)
required_vars = _provider_key_vars_for_model(model)
if required_vars:
@@ -303,6 +344,7 @@ def ensure_configured_or_onboard(
*,
model: Optional[str] = None,
interactive: bool = True,
base_url: Optional[str] = None,
) -> Optional[str]:
"""Shared first-run credential gate for LLM-bound interactive commands.
@@ -331,6 +373,9 @@ def ensure_configured_or_onboard(
keyless local-first fallback (its own provider gate applies).
interactive: Whether a wizard prompt may be shown. Callers pass ``False``
for headless (``-p``, piped, ``--output json``) paths.
base_url: The endpoint the caller will actually use (e.g. a custom
agent's ``llm.base_url``). For an explicit Ollama model this is the
server probed, instead of ``OLLAMA_API_BASE``/``OLLAMA_HOST``.
Returns:
The model id to proceed with. May be a detected local model id when the
@@ -343,6 +388,21 @@ def ensure_configured_or_onboard(
import typer
inject_credentials_into_env()
# An explicit Ollama model needs a running server, not a key: probe the
# endpoint the caller will really use and, if it is down, say so instead
# of sending the user to an API-key wizard.
ollama_host = _ollama_host_for_model(model, base_url)
if ollama_host is not None:
if _ollama_reachable(ollama_host):
return model
typer.echo(
f"Error: Ollama isn't reachable at {ollama_host} — start it with "
"`ollama serve` (or set OLLAMA_API_BASE).",
err=True,
)
raise typer.Exit(1)
if is_configured(model):
return model
@@ -360,11 +420,16 @@ def ensure_configured_or_onboard(
return local.model
if not interactive:
# Only mention auto-detection when it actually ran and found nothing.
hint = f" for model {model}" if model else ""
tail = (
"" if model else
"\n(no running local endpoint such as Ollama was found; one "
"would be used automatically)"
)
typer.echo(
"Error: No API key configured. Run: praisonai setup\n"
"or set environment variables like OPENAI_API_KEY\n"
"(a running local endpoint such as Ollama would be used "
"automatically)",
f"Error: No API key configured{hint}. Run: praisonai setup\n"
f"or set environment variables like OPENAI_API_KEY{tail}",
err=True,
)
raise typer.Exit(1)
@@ -19,9 +19,10 @@ from dataclasses import dataclass
from typing import Optional
from urllib.parse import urlsplit
# Total time budget for a probe so the credential/first-run hot path is never
# stalled when nothing is listening. Kept small deliberately.
_PROBE_TIMEOUT_S = 0.15
# Per-request probe timeout. Only paid on the keyless path; a closed port is
# refused instantly, while a running Ollama can take ~0.35 s on its first call,
# so a tighter budget made detection flaky (issue #6228).
_PROBE_TIMEOUT_S = 1.0
# How long a negative probe is remembered so repeated invocations in the same
# process (e.g. bare TUI then run) don't re-pay the connection latency.
@@ -187,6 +188,11 @@ def _probe_ollama_tags(host: str) -> Optional[str]:
return None
def ollama_reachable(host: str) -> bool:
"""Return ``True`` if an Ollama server answers ``/api/tags`` at ``host``."""
return _get_json(_root_host(_with_scheme(host)) + "/api/tags") is not None
def detect_local_model(*, use_cache: bool = True) -> Optional[LocalModel]:
"""Detect a reachable local OpenAI-compatible endpoint.
@@ -0,0 +1,144 @@
"""Tests for issue #6228 — explicit ``ollama/`` models need no API key.
An explicit Ollama model is gated on the server being reachable, not on
``OLLAMA_HOST`` being set; the probe tolerates a slow first response; and the
error text for an explicit model does not claim auto-detection would kick in.
"""
import io
import json
import pytest
import typer
from praisonai_code.llm import credentials, local_detect
_ENV = (
"OPENAI_API_KEY", "ANTHROPIC_API_KEY", "GEMINI_API_KEY", "GOOGLE_API_KEY",
"GROQ_API_KEY", "COHERE_API_KEY", "OPENROUTER_API_KEY", "OLLAMA_HOST",
"OLLAMA_API_BASE", "OPENAI_BASE_URL", "MODEL_NAME", "OPENAI_MODEL_NAME",
"PRAISONAI_LOCAL_ENDPOINTS",
)
@pytest.fixture
def clean_env(monkeypatch):
for var in _ENV:
monkeypatch.delenv(var, raising=False)
monkeypatch.setattr(credentials, "inject_credentials_into_env", lambda: False)
local_detect.reset_cache()
yield monkeypatch
local_detect.reset_cache()
class _Resp(io.BytesIO):
status = 200
def __enter__(self):
return self
def __exit__(self, *a):
return False
def _fake_urlopen(seen, delay=0.0):
def _urlopen(url, timeout):
seen.append((url, timeout))
if delay > timeout:
raise TimeoutError("probe timed out")
return _Resp(json.dumps({"models": [{"name": "llama3.2:latest"}]}).encode())
return _urlopen
def _refuse(url, timeout):
raise ConnectionRefusedError()
def test_explicit_ollama_configured_when_reachable(clean_env):
seen = []
clean_env.setattr(local_detect.urllib.request, "urlopen", _fake_urlopen(seen))
assert credentials.is_configured("ollama/llama3.2") is True
assert seen[0][0] == "http://127.0.0.1:11434/api/tags"
def test_explicit_ollama_honours_ollama_api_base(clean_env):
seen = []
clean_env.setenv("OLLAMA_API_BASE", "http://gpu-box:11434")
clean_env.setattr(local_detect.urllib.request, "urlopen", _fake_urlopen(seen))
assert credentials.is_configured("ollama_chat/llama3.2") is True
assert seen[0][0] == "http://gpu-box:11434/api/tags"
def test_explicit_ollama_not_configured_when_unreachable(clean_env):
clean_env.setattr(local_detect.urllib.request, "urlopen", _refuse)
assert credentials.is_configured("ollama/llama3.2") is False
def test_unreachable_ollama_error_is_actionable(clean_env, capsys):
clean_env.setattr(local_detect.urllib.request, "urlopen", _refuse)
with pytest.raises(typer.Exit):
credentials.ensure_configured_or_onboard(
model="ollama/llama3.2", interactive=False
)
err = capsys.readouterr().err
assert "Ollama isn't reachable at http://127.0.0.1:11434" in err
assert "ollama serve" in err
assert "API key" not in err
def test_explicit_ollama_passes_gate_when_reachable(clean_env):
clean_env.setattr(local_detect.urllib.request, "urlopen", _fake_urlopen([]))
assert credentials.ensure_configured_or_onboard(
model="ollama/llama3.2", interactive=False
) == "ollama/llama3.2"
def test_probe_tolerates_slow_first_response(clean_env):
"""A ~0.4 s first /api/tags response must still be detected."""
clean_env.setattr(
local_detect.urllib.request, "urlopen", _fake_urlopen([], delay=0.4)
)
result = local_detect.detect_local_model(use_cache=False)
assert result is not None
assert result.model == "ollama/llama3.2:latest"
def test_explicit_cloud_model_error_omits_auto_detect_hint(clean_env, capsys):
clean_env.setattr(credentials, "detect_local_endpoint", lambda: None)
clean_env.setattr(credentials, "is_configured", lambda model=None: False)
with pytest.raises(typer.Exit):
credentials.ensure_configured_or_onboard(
model="gpt-4o-mini", interactive=False
)
err = capsys.readouterr().err
assert "for model gpt-4o-mini" in err
assert "automatically" not in err
def test_keyless_no_model_error_mentions_auto_detect(clean_env, capsys):
clean_env.setattr(credentials, "detect_local_endpoint", lambda: None)
clean_env.setattr(credentials, "is_configured", lambda model=None: False)
with pytest.raises(typer.Exit):
credentials.ensure_configured_or_onboard(model=None, interactive=False)
assert "automatically" in capsys.readouterr().err
def test_gate_probes_agent_base_url_not_env_host(clean_env):
"""A custom agent's ``llm.base_url`` is the server gated, not OLLAMA_HOST."""
seen = []
clean_env.setenv("OLLAMA_HOST", "http://down-box:11434")
def _urlopen(url, timeout):
seen.append(url)
if "down-box" in url:
raise ConnectionRefusedError()
return _Resp(json.dumps({"models": []}).encode())
clean_env.setattr(local_detect.urllib.request, "urlopen", _urlopen)
assert credentials.ensure_configured_or_onboard(
model="ollama/llama3.2",
interactive=False,
base_url="http://gpu-box:11434/v1",
) == "ollama/llama3.2"
assert seen == ["http://gpu-box:11434/api/tags"]
@@ -717,6 +717,9 @@ def train_export(
mtp_draft: bool = typer.Option(
False, "--mtp-draft/--no-mtp-draft",
help="Also download the stock MTP drafter (Gemma-4 only) for fast inference"),
push: Optional[bool] = typer.Option(
None, "--push/--no-push",
help="ollama: push to ollama.com after `ollama create` (--no-push keeps it local)"),
):
"""
Export / publish an ALREADY-trained model without re-running training.
@@ -726,6 +729,7 @@ def train_export(
praisonai-train export gguf --model-dir lora_model --quant q4_k_m # local .gguf
praisonai-train export gguf --model-dir lora_model --hf me/my-model # local + push
praisonai-train export ollama --model-dir lora_model --ollama me/my-model --quant q4_k_m
praisonai-train export ollama --model-dir lora_model --ollama my-model --no-push # local only
"""
from ..output.console import get_output_controller
@@ -765,6 +769,9 @@ def train_export(
cfg["hf_model_name"] = hf
if ollama:
cfg["ollama_model"] = ollama
# Either explicit flag overrides the config; unset keeps config/default (push).
if isinstance(push, bool):
cfg["ollama_push"] = push
# Recover the model name (drives chat-template selection) from --base-model or
# the trained model's config.json (_name_or_path), falling back to the dir name.
@@ -1,4 +1,5 @@
ollama_save: "true"
ollama_push: "true" # set "false" to keep the Ollama model local (no upload to ollama.com)
huggingface_save: "true"
train: "true"
@@ -220,8 +220,9 @@ def create_and_push_ollama_model(
model_parameters: str,
modelfile_content: str,
quantization: Optional[str] = None,
push: bool = True,
) -> None:
"""Create and push an Ollama model with proper daemon management.
"""Create (and by default push) an Ollama model with proper daemon management.
Args:
ollama_model: Name of the Ollama model
@@ -229,6 +230,8 @@ def create_and_push_ollama_model(
modelfile_content: Content for the Modelfile
quantization: Optional quantization (e.g. "q4_k_m") passed to
``ollama create --quantize`` so the model isn't stored as huge f16.
push: When False, stop after a successful ``ollama create`` and keep the
model local-only (nothing is uploaded to ollama.com).
Raises:
RuntimeError: If ollama create/push operations fail (with actionable guidance)
@@ -259,6 +262,10 @@ def create_and_push_ollama_model(
f"{(create.stderr or create.stdout or '').strip()}"
)
if not push:
print(f"Created local Ollama model `{tag}` (not pushed). Run it with: ollama run {tag}")
return
# Push with captured output so a 401/403 becomes an actionable message rather
# than a raw non-zero exit.
result = subprocess.run(["ollama", "push", tag], capture_output=True, text=True)
@@ -545,6 +545,7 @@ class TrainModel:
"optim", "weight_decay", "lr_scheduler_type", "seed", "output_dir",
"assistant_only_loss", "train_on_responses_only", "save_steps",
"train", "huggingface_save", "huggingface_save_gguf", "ollama_save",
"ollama_push",
"method", "beta", "max_prompt_length", "desirable_weight", "undesirable_weight",
"reward_funcs", "num_generations", "max_completion_length",
"embedding_learning_rate",
@@ -951,6 +952,10 @@ class TrainModel:
if self._flag(dataset_info.get("shuffle"), default=False):
dataset = dataset.shuffle(
seed=int(dataset_info.get("seed", self.config.get("seed", 3407))))
# OpenAI-style chat rows ({"messages": [...]}, as written by
# praisonaiagents.eval.export_sft) are the ShareGPT shape under another name.
if "messages" in dataset.column_names and "conversations" not in dataset.column_names:
dataset = dataset.rename_column("messages", "conversations")
if "conversations" in dataset.column_names:
print("DEBUG: Standardizing dataset (ShareGPT style)...")
dataset = standardize_sharegpt(dataset)
@@ -970,8 +975,8 @@ class TrainModel:
if len(dataset) == 0:
raise ValueError(
"All examples formatted to empty text. The dataset columns are "
f"{columns_before}; this trainer understands ShareGPT "
"('conversations'), Alpaca ('instruction'/'input'/'output') and "
f"{columns_before}; this trainer understands chat "
"('conversations' or 'messages'), Alpaca ('instruction'/'input'/'output') and "
"raw text ('text'). Rename your columns to one of those shapes, "
"or set chat_template to match your data.")
return dataset
@@ -1784,6 +1789,7 @@ class TrainModel:
self.config.get('model_parameters', 'latest'),
modelfile_content,
quantization=self.config.get('quantization_method'),
push=self._flag(self.config.get('ollama_push'), default=True),
)
def run(self):
@@ -313,7 +313,8 @@ PARAMETER top_p 0.9
create_and_push_ollama_model(
self.config['ollama_model'],
self.config['model_parameters'],
modelfile_content
modelfile_content,
push=self._flag(self.config.get('ollama_push'), default=True),
)
def run(self):
@@ -0,0 +1,58 @@
"""An OpenAI-style `messages` chat dataset formatted to empty text.
`praisonai-train validate` and `praisonaiagents.eval.export_sft` both use the
`{"messages": [...]}` shape, but `process_dataset` only recognised
`conversations`, so every row fell into the Alpaca branch, formatted to "",
and the run died after the base model had already been loaded.
"""
from praisonai_train.train.llm import trainer as trainer_mod
class _FakeDataset:
def __init__(self, rows):
self.rows = rows
@property
def column_names(self):
return list(self.rows[0].keys()) if self.rows else []
def __len__(self):
return len(self.rows)
def rename_column(self, src, dst):
return _FakeDataset([{(dst if k == src else k): v for k, v in r.items()}
for r in self.rows])
def map(self, func, batched, remove_columns):
batch = {c: [r[c] for r in self.rows] for c in self.column_names}
out = func(batch)
return _FakeDataset([{"text": t} for t in out["text"]])
def filter(self, pred):
return _FakeDataset([r for r in self.rows if pred(r)])
class _Tokenizer:
def apply_chat_template(self, convo, tokenize, add_generation_prompt):
return "".join(f"<{m['role']}>{m['content']}" for m in convo)
def test_messages_column_is_treated_as_conversations(monkeypatch):
rows = [{"messages": [
{"role": "system", "content": "s"},
{"role": "user", "content": f"q{i}"},
{"role": "assistant", "content": f"a{i}"},
]} for i in range(3)]
monkeypatch.setattr(trainer_mod, "load_dataset",
lambda *a, **k: _FakeDataset(rows), raising=False)
monkeypatch.setattr(trainer_mod, "standardize_sharegpt", lambda d: d,
raising=False)
tm = trainer_mod.TrainModel.__new__(trainer_mod.TrainModel)
tm.config = {"method": "sft"}
tm.chat_tokenizer = _Tokenizer()
out = tm.process_dataset({"name": "hub/some-dataset"})
assert len(out) == 3
assert out.rows[0]["text"] == "<system>s<user>q0<assistant>a0"
@@ -412,3 +412,88 @@ def test_export_command_bad_target_exits(monkeypatch, tmp_path):
quant=None,
base_model=None,
)
def test_ollama_no_push_skips_push(monkeypatch, tmp_path):
"""push=False stops after `ollama create`; nothing is sent to ollama.com."""
monkeypatch.chdir(tmp_path)
monkeypatch.setattr(ollama_mod, "_check_ollama_disk", lambda *_a, **_k: None)
monkeypatch.setattr(ollama_mod, "ensure_ollama_running", lambda *_a, **_k: None)
calls = []
def _fake_run(cmd, *a, **k):
calls.append(cmd)
return types.SimpleNamespace(returncode=0, stdout="", stderr="")
monkeypatch.setattr(ollama_mod.subprocess, "run", _fake_run)
ollama_mod.create_and_push_ollama_model(
"my-model", "latest", "FROM ./model\n", push=False)
assert [c[1] for c in calls] == ["create"]
assert not any("push" in c for c in calls)
def _ollama_export_cfg(monkeypatch, tmp_path, push, config_push=None):
"""Run `export ollama` with a fake trainer; return the config it received."""
import praisonai_train.cli.commands.train as cli
seen = {}
class _FakeTrainer:
@classmethod
def for_export(cls, cfg):
obj = cls()
obj.config = cfg
obj.model = None
obj.hf_tokenizer = None
return obj
def load_model(self):
return ("MODEL", "TOK")
def create_and_push_ollama_model(self):
seen.update(self.config)
monkeypatch.setattr(trainer_mod, "TrainModel", _FakeTrainer)
config = None
if config_push is not None:
config = tmp_path / "cfg.yaml"
config.write_text(f"ollama_push: {config_push}\n")
cli.train_export(
target="ollama", model_dir=str(tmp_path), config=str(config) if config else None,
ollama="my-model", hf=None, quant=None, base_model="unsloth/gemma-2-2b",
mtp_draft=False, push=push,
)
return seen
def test_export_ollama_no_push_flag_sets_config(monkeypatch, tmp_path):
assert _ollama_export_cfg(monkeypatch, tmp_path, push=False)["ollama_push"] is False
def test_export_ollama_explicit_push_overrides_config(monkeypatch, tmp_path):
cfg = _ollama_export_cfg(monkeypatch, tmp_path, push=True, config_push="false")
assert cfg["ollama_push"] is True
def test_export_ollama_unset_push_keeps_config(monkeypatch, tmp_path):
cfg = _ollama_export_cfg(monkeypatch, tmp_path, push=None, config_push="false")
assert cfg["ollama_push"] is False # YAML bool preserved, not overridden
assert "ollama_push" not in _ollama_export_cfg(monkeypatch, tmp_path, push=None)
@pytest.mark.parametrize("value, expected", [
(False, False), ("false", False), (True, True), ("true", True), (None, True),
])
def test_llm_trainer_forwards_ollama_push(monkeypatch, value, expected):
"""TrainModel forwards `ollama_push` (bool or string) to the helper."""
captured = {}
monkeypatch.setattr(
ollama_mod, "create_and_push_ollama_model",
lambda *a, **k: captured.update(k))
fake = types.SimpleNamespace(
config={"ollama_model": "m", "ollama_push": value},
prepare_modelfile_content=lambda: "FROM ./model\n",
_flag=trainer_mod.TrainModel._flag,
)
trainer_mod.TrainModel.create_and_push_ollama_model(fake)
assert captured["push"] is expected
+3 -1
View File
@@ -1,6 +1,8 @@
# PraisonAI TypeScript Node.js AI Agents Framework
PraisonAI is a production-ready Multi AI Agents framework, designed to create AI Agents to automate and solve problems ranging from simple tasks to complex challenges. It provides a low-code solution to streamline the building and management of multi-agent LLM systems, emphasising simplicity, customisation, and effective human-agent collaboration.
PraisonAI runs AI agents on your own computers, even fully offline, so private data never has to leave the building. It is built for highly regulated teams, and free and open source for everyone.
This package is the TypeScript and JavaScript SDK of the open-source PraisonAI agent framework. Build single agents, multi-agent workflows, tools and memory in Node.js, on hosted models or on local models such as Ollama. Data stays on your own machines only when the model runs locally too; with a hosted model such as the default OpenAI one, prompts are sent to that provider. The Python framework and the full docs are at [github.com/MervinPraison/PraisonAI](https://github.com/MervinPraison/PraisonAI) and [praison.ai/docs](https://praison.ai/docs).
## Installation
+2 -2
View File
@@ -219,7 +219,7 @@ TS-only members: `previousResult`?, `onToken`?, `signal`?, `onEvent`?, `options`
### `Agent.chat`
- Python: `src/praisonai-agents/praisonaiagents/agent/chat_mixin.py:3657`
- Python: `src/praisonai-agents/praisonaiagents/agent/chat_mixin.py:3714`
- TypeScript: `src/praisonai-ts/src/agent/simple.ts:3592`
- Counts: 17 python params: 7 exact, 9 camelCase, 1 alias, 0 flattened, 0 missing; 0 mismatches; 0 waived; 2 TS-only of 19
@@ -316,7 +316,7 @@ TS-only members: `config`
### `Session.__init__`
- Python: `src/praisonai-agents/praisonaiagents/session/api.py:67`
- TypeScript: `src/praisonai-ts/src/session/session.ts:29` (ctor `src/praisonai-ts/src/session/session.ts:132`)
- TypeScript: `src/praisonai-ts/src/session/session.ts:29` (ctor `src/praisonai-ts/src/session/session.ts:133`)
- Counts: 7 python params: 1 exact, 6 camelCase, 0 alias, 0 flattened, 0 missing; 1 mismatches; 1 waived; 5 TS-only of 12
| Python param | Kind | Py default | Py type | Match | TS name | TS default | TS type | Status |
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "praisonai",
"version": "1.7.4",
"description": "PraisonAI TypeScript AI Agents Framework - Node.js, npm, and Javascript AI Agents Framework",
"description": "Open-source TypeScript and JavaScript AI agent framework for Node.js, on hosted or local models. The JavaScript SDK of PraisonAI.",
"main": "dist/index.js",
"module": "dist/esm/index.js",
"types": "dist/index.d.ts",
+2 -2
View File
@@ -1797,7 +1797,7 @@
}
],
"python": {
"location": "src/praisonai-agents/praisonaiagents/agent/chat_mixin.py:3657",
"location": "src/praisonai-agents/praisonaiagents/agent/chat_mixin.py:3714",
"resolved_class": "ChatMixin"
},
"ts_only": [
@@ -5142,7 +5142,7 @@
],
"ts_only_required": [],
"typescript": {
"ctor_location": "src/praisonai-ts/src/session/session.ts:132",
"ctor_location": "src/praisonai-ts/src/session/session.ts:133",
"location": "src/praisonai-ts/src/session/session.ts:29",
"options_parameters": [
"config"
+20 -5
View File
@@ -72,7 +72,7 @@ export class MemoryCache extends BaseCache {
if (this.maxSize && this.cache.size >= this.maxSize && !this.cache.has(key)) {
// Remove oldest entry
const firstKey = this.cache.keys().next().value;
if (firstKey) this.cache.delete(firstKey);
if (firstKey !== undefined) this.cache.delete(firstKey);
}
const effectiveTtl = ttl ?? this.ttl;
@@ -137,12 +137,15 @@ export class MemoryCache extends BaseCache {
*/
export class FileCache extends BaseCache {
private cacheDir: string;
private ttl?: number;
private fs: any;
private path: any;
private listQueues: Map<string, Promise<void>> = new Map();
constructor(config: CacheConfig & { cacheDir?: string } = {}) {
super(config);
this.cacheDir = config.cacheDir || '.cache';
this.ttl = config.ttl;
}
private async ensureDir(): Promise<void> {
@@ -177,10 +180,11 @@ export class FileCache extends BaseCache {
async set<T = any>(key: string, value: T, ttl?: number): Promise<void> {
await this.ensureDir();
const effectiveTtl = ttl ?? this.ttl;
const entry: CacheEntry<T> = {
value,
createdAt: Date.now(),
expiresAt: ttl ? Date.now() + ttl : undefined
expiresAt: effectiveTtl ? Date.now() + effectiveTtl : undefined
};
await this.fs.writeFile(this.getFilePath(key), JSON.stringify(entry));
}
@@ -227,9 +231,20 @@ export class FileCache extends BaseCache {
}
async listPush<T = any>(key: string, value: T): Promise<void> {
const list = (await this.get<T[]>(`list_${key}`)) || [];
list.push(value);
await this.set(`list_${key}`, list);
// Serialize the full read/modify/write operation for this instance and key.
// A failed append must not block subsequent appends.
const previous = this.listQueues.get(key) ?? Promise.resolve();
const next = previous.catch(() => {}).then(async () => {
const list = (await this.get<T[]>(`list_${key}`)) || [];
list.push(value);
await this.set(`list_${key}`, list);
});
this.listQueues.set(key, next);
try {
await next;
} finally {
if (this.listQueues.get(key) === next) this.listQueues.delete(key);
}
}
async listLength(key: string): Promise<number> {
+12 -7
View File
@@ -231,12 +231,17 @@ export class HooksManager {
return promise;
}
return Promise.race([
promise,
new Promise<T>((_, reject) =>
setTimeout(() => reject(new Error(`Hook timeout after ${timeout}ms`)), timeout)
),
]);
let timer: ReturnType<typeof setTimeout> | undefined;
try {
return await Promise.race([
promise,
new Promise<T>((_, reject) => {
timer = setTimeout(() => reject(new Error(`Hook timeout after ${timeout}ms`)), timeout);
}),
]);
} finally {
if (timer !== undefined) clearTimeout(timer);
}
}
/**
@@ -251,7 +256,7 @@ export class HooksManager {
* Get all registered hooks for an event
*/
getHooks(event: HookEvent): readonly HookConfig[] {
return Object.freeze(this.hooks.get(event) ?? []);
return Object.freeze([...(this.hooks.get(event) ?? [])]);
}
/**
+12 -8
View File
@@ -82,16 +82,20 @@ export class Chunking {
* Chunk by sentences
*/
chunkBySentence(text: string): Chunk[] {
const sentences = text.match(/[^.!?]+[.!?]+/g) || [text];
return sentences.map((content, index) => {
const startOffset = text.indexOf(content);
return {
content: content.trim(),
index,
const chunks: Chunk[] = [];
// Include the unterminated tail and punctuation-only spans as well.
for (const match of text.matchAll(/[^.!?]+[.!?]*|[.!?]+/g)) {
const content = match[0].trim();
if (!content) continue;
const startOffset = match.index + match[0].indexOf(content);
chunks.push({
content,
index: chunks.length,
startOffset,
endOffset: startOffset + content.length
};
});
});
}
return chunks;
}
/**
@@ -60,7 +60,23 @@ export class GraphStore {
* Add an edge to the graph
*/
addEdge(edge: GraphEdge): void {
const previous = this.edges.get(edge.id);
this.edges.set(edge.id, edge);
if (previous && (previous.source !== edge.source || previous.target !== edge.target)) {
// Another edge may still connect the old endpoints.
let hasParallelEdge = false;
for (const candidate of this.edges.values()) {
if (candidate.source === previous.source && candidate.target === previous.target) {
hasParallelEdge = true;
break;
}
}
if (!hasParallelEdge) {
this.adjacencyList.get(previous.source)?.delete(previous.target);
this.reverseAdjacencyList.get(previous.target)?.delete(previous.source);
}
}
// Update adjacency lists
if (!this.adjacencyList.has(edge.source)) {
+27 -12
View File
@@ -70,22 +70,24 @@ export class WebSocketTransport implements MCPTransport {
}
const WebSocket = wsModule.default;
this.ws = new WebSocket(this.url);
const socket = new WebSocket(this.url);
this.ws = socket;
await new Promise<void>((resolve, reject) => {
this.ws.on('open', () => {
socket.on('open', () => {
this.connected = true;
resolve();
});
this.ws.on('error', (err: Error) => reject(err));
socket.on('error', (err: Error) => reject(err));
});
this.ws.on('message', (data: Buffer | string) => {
socket.on('message', (data: Buffer | string) => {
if (this.ws !== socket) return;
try {
const message = JSON.parse(data.toString()) as TransportMessage;
// Handle response
if (message.id && this.pendingRequests.has(String(message.id))) {
if (message.id !== undefined && this.pendingRequests.has(String(message.id))) {
const { resolve, reject } = this.pendingRequests.get(String(message.id))!;
this.pendingRequests.delete(String(message.id));
if (message.error) {
@@ -102,16 +104,29 @@ export class WebSocketTransport implements MCPTransport {
}
});
this.ws.on('error', (err: Error) => this.errorHandler?.(err));
this.ws.on('close', () => { this.connected = false; });
socket.on('error', (err: Error) => {
if (this.ws === socket) this.errorHandler?.(err);
});
socket.on('close', () => {
if (this.ws !== socket) return;
this.connected = false;
this.ws = null;
this.rejectPendingRequests();
});
}
async disconnect(): Promise<void> {
if (this.ws) {
this.ws.close();
this.ws = null;
this.connected = false;
}
const socket = this.ws;
this.ws = null;
this.connected = false;
this.rejectPendingRequests();
socket?.close();
}
private rejectPendingRequests(): void {
const error = new Error('WebSocket connection closed');
for (const { reject } of this.pendingRequests.values()) reject(error);
this.pendingRequests.clear();
}
async send(message: TransportMessage): Promise<TransportMessage> {
+14 -4
View File
@@ -28,6 +28,7 @@ export class FileMemory {
private initialized: boolean = false;
private initPromise?: Promise<void>;
private writeQueue: Promise<void> = Promise.resolve();
private appendNeedsSeparator: boolean = false;
constructor(config: FileMemoryConfig) {
this.filePath = config.filePath;
@@ -67,6 +68,8 @@ export class FileMemory {
try {
const fs = await import('fs/promises');
const content = await fs.readFile(this.filePath, 'utf-8').catch(() => '');
// A previous instance may have left a record without its final newline.
this.appendNeedsSeparator = content.length > 0 && !content.endsWith('\n');
if (content) {
const lines = content.trim().split('\n').filter(Boolean);
@@ -155,6 +158,7 @@ export class FileMemory {
*/
async getRecent(count: number): Promise<MemoryEntry[]> {
const all = await this.getAll();
if (count === 0) return [];
return all.slice(-count);
}
@@ -265,7 +269,7 @@ export class FileMemory {
* Build context string from recent memory
*/
async buildContext(count?: number): Promise<string> {
const entries = count ? await this.getRecent(count) : await this.getAll();
const entries = count !== undefined ? await this.getRecent(count) : await this.getAll();
return entries
.map(e => `${e.role}: ${e.content}`)
.join('\n');
@@ -273,11 +277,17 @@ export class FileMemory {
private async appendEntry(entry: FileMemoryEntry): Promise<void> {
// Queue writes to ensure order
this.writeQueue = this.writeQueue.then(async () => {
const write = this.writeQueue.then(async () => {
const fs = await import('fs/promises');
await fs.appendFile(this.filePath, JSON.stringify(entry) + '\n');
// A failed append may have left an unfinished JSONL line.
const separator = this.appendNeedsSeparator ? '\n' : '';
this.appendNeedsSeparator = true;
await fs.appendFile(this.filePath, separator + JSON.stringify(entry) + '\n');
this.appendNeedsSeparator = false;
});
await this.writeQueue;
// Report this write's error to its caller without blocking later writes.
this.writeQueue = write.catch(() => {});
await write;
}
private textSearch(query: string, limit: number): SearchResult[] {
+2 -1
View File
@@ -87,6 +87,7 @@ export class Memory {
* Get recent entries
*/
getRecent(count: number): MemoryEntry[] {
if (count === 0) return [];
return this.getAll().slice(-count);
}
@@ -179,7 +180,7 @@ export class Memory {
* Build context string from recent memory
*/
buildContext(count?: number): string {
const entries = count ? this.getRecent(count) : this.getAll();
const entries = count !== undefined ? this.getRecent(count) : this.getAll();
return entries
.map(e => `${e.role}: ${e.content}`)
.join('\n');
+7 -6
View File
@@ -123,7 +123,8 @@ export class Session {
private readonly timeout: number;
private readonly isRemote: boolean;
private state: Record<string, any> = {};
// State keys are user data, including names inherited by ordinary objects.
private state: Record<string, any> = Object.create(null);
private messages: SessionMessage[] = [];
private createdAt: number;
private updatedAt: number;
@@ -264,7 +265,7 @@ export class Session {
* Set entire state object.
*/
setState(state: Record<string, any>): void {
this.state = { ...state };
this.state = Object.assign(Object.create(null), state);
this.updatedAt = Date.now();
}
@@ -272,7 +273,7 @@ export class Session {
* Clear all state.
*/
clearState(): void {
this.state = {};
this.state = Object.create(null);
this.updatedAt = Date.now();
}
@@ -364,7 +365,7 @@ export class Session {
const messages = await this.db.getMessages(this.id, 1);
if (messages.length > 0) {
const sessionState: SessionState = JSON.parse(messages[0].content || '{}');
this.state = sessionState.data || {};
this.setState(sessionState.data || {});
this.messages = sessionState.messages || [];
this.createdAt = sessionState.createdAt || Date.now();
this.updatedAt = sessionState.updatedAt || Date.now();
@@ -382,7 +383,7 @@ export class Session {
return {
id: this.id,
userId: this.userId,
data: this.state,
data: this.getState(),
messages: this.messages,
createdAt: this.createdAt,
updatedAt: this.updatedAt,
@@ -399,7 +400,7 @@ export class Session {
userId: json.userId,
...config,
});
session.state = json.data || {};
session.setState(json.data || {});
session.messages = json.messages || [];
session.createdAt = json.createdAt;
session.updatedAt = json.updatedAt;
+24 -5
View File
@@ -111,6 +111,8 @@ export class FileSessionStore implements ISessionStore {
private autoSaveInterval?: number;
private saveTimer?: NodeJS.Timeout;
private dirty: boolean = false;
private saveQueue: Promise<void> = Promise.resolve();
private pendingSaves: number = 0;
constructor(filePath: string, config?: { defaultTTL?: number; autoSaveInterval?: number }) {
this.filePath = filePath;
@@ -120,7 +122,7 @@ export class FileSessionStore implements ISessionStore {
if (this.autoSaveInterval) {
this.saveTimer = setInterval(() => {
if (this.dirty) this.save();
if (this.dirty && this.pendingSaves === 0) this.save();
}, this.autoSaveInterval);
}
}
@@ -146,10 +148,26 @@ export class FileSessionStore implements ISessionStore {
}
async save(): Promise<void> {
const fs = await import('fs').then(m => m.promises);
const sessions = Array.from(this.sessions.values());
await fs.writeFile(this.filePath, JSON.stringify(sessions, null, 2));
this.dirty = false;
this.pendingSaves++;
const saving = this.saveQueue.then(async () => {
const fs = await import('fs').then(m => m.promises);
const data = JSON.stringify(Array.from(this.sessions.values()), null, 2);
// Only this snapshot is clean; mutations during the write stay dirty.
this.dirty = false;
try {
await fs.writeFile(this.filePath, data);
} catch (error) {
this.dirty = true;
throw error;
}
});
// Keep the queue usable after a failure, while this caller still sees it.
this.saveQueue = saving.catch(() => {});
try {
await saving;
} finally {
this.pendingSaves--;
}
}
async get(id: string): Promise<StoredSession | null> {
@@ -205,6 +223,7 @@ export class FileSessionStore implements ISessionStore {
if (this.saveTimer) {
clearInterval(this.saveTimer);
}
await this.saveQueue;
if (this.dirty) {
await this.save();
}
+15 -1
View File
@@ -420,7 +420,21 @@ export class FunctionTool<TParams = any, TResult = any> {
}
const delay = retryDelayMs(policy, attempt - 1);
if (delay > 0) {
await new Promise<void>(resolve => setTimeout(resolve, delay));
const signal = context?.signal;
const completed = await new Promise<boolean>(resolve => {
let timer: ReturnType<typeof setTimeout> | undefined;
const finish = (value: boolean) => {
if (timer !== undefined) clearTimeout(timer);
signal?.removeEventListener('abort', onAbort);
resolve(value);
};
const onAbort = () => finish(false);
timer = setTimeout(() => finish(true), delay);
signal?.addEventListener('abort', onAbort, { once: true });
if (signal?.aborted) finish(false);
});
// Preserve the existing last-error behavior when cancellation stops retries.
if (!completed || signal?.aborted) throw error;
}
}
}
@@ -147,6 +147,9 @@ export function createRetryMiddleware(
let lastError: Error | undefined;
for (let attempt = 0; attempt <= maxRetries; attempt++) {
if (context.signal?.aborted) {
throw context.signal.reason ?? new Error('The operation was aborted');
}
try {
return await next();
} catch (error) {
@@ -212,7 +215,9 @@ export function createValidationMiddleware(): ToolMiddleware {
const limits = context.limits;
if (limits?.maxPayloadBytes) {
const size = JSON.stringify(input).length;
const serialized = JSON.stringify(input);
if (serialized === undefined) throw new TypeError('Input payload is not JSON serializable');
const size = new TextEncoder().encode(serialized).byteLength;
if (size > limits.maxPayloadBytes) {
throw new Error(`Input payload size (${size} bytes) exceeds limit (${limits.maxPayloadBytes} bytes)`);
}
@@ -227,17 +232,16 @@ export function createValidationMiddleware(): ToolMiddleware {
*/
export function composeMiddleware(...middlewares: ToolMiddleware[]): ToolMiddleware {
return async (input, context, next) => {
let index = 0;
const composedNext = async (): Promise<unknown> => {
// Each continuation keeps its own downstream position so retry middleware
// can invoke next() again without skipping the remaining layers.
const dispatch = async (index: number): Promise<unknown> => {
if (index < middlewares.length) {
const mw = middlewares[index++];
return mw(input, context, composedNext);
return middlewares[index](input, context, () => dispatch(index + 1));
}
return next();
};
return composedNext();
return dispatch(0);
};
}
+18 -25
View File
@@ -5,6 +5,8 @@
* Supports lazy loading, middleware, and multi-agent safety.
*/
import { composeMiddleware } from './middleware';
import type {
ToolMetadata,
ToolFactory,
@@ -61,6 +63,7 @@ export class ToolsRegistry {
*/
register(metadata: ToolMetadata, factory: ToolFactory): this {
this.tools.set(metadata.id, { metadata, factory });
this.installCache.delete(metadata.id);
return this;
}
@@ -68,6 +71,7 @@ export class ToolsRegistry {
* Unregister a tool
*/
unregister(id: string): boolean {
this.installCache.delete(id);
return this.tools.delete(id);
}
@@ -158,16 +162,7 @@ export class ToolsRegistry {
if (this.middleware.length === 0) {
result = await executeWithMiddleware();
} else {
// Chain middleware
let index = 0;
const next = async (): Promise<unknown> => {
if (index < this.middleware.length) {
const mw = this.middleware[index++];
return mw(input, ctx, next);
}
return executeWithMiddleware();
};
result = await next();
result = await composeMiddleware(...this.middleware)(input, ctx, executeWithMiddleware);
}
// Call afterToolCall hook
@@ -222,27 +217,25 @@ export class ToolsRegistry {
* Check if a tool's optional dependency is installed
*/
async checkInstalled(id: string): Promise<boolean> {
// Check cache first
const registered = this.tools.get(id);
if (!registered) return false;
if (this.installCache.has(id)) {
return this.installCache.get(id)!;
}
const registered = this.tools.get(id);
if (!registered) {
return false;
const { packageName } = registered.metadata;
let installed: boolean;
try {
await import(packageName);
installed = true;
} catch {
installed = false;
}
const { packageName } = registered.metadata;
try {
// Try to resolve the package
await import(packageName);
this.installCache.set(id, true);
return true;
} catch {
this.installCache.set(id, false);
return false;
}
// An earlier import must not repopulate the cache after replacement/removal.
if (this.tools.get(id) === registered) this.installCache.set(id, installed);
return installed;
}
/**
+86
View File
@@ -3,6 +3,9 @@
*/
import { MemoryCache, FileCache, createMemoryCache, createFileCache } from '../../../src/cache';
import { mkdtemp, rm } from 'fs/promises';
import { tmpdir } from 'os';
import { join } from 'path';
describe('MemoryCache', () => {
let cache: MemoryCache;
@@ -158,3 +161,86 @@ describe('Factory functions', () => {
expect(cache).toBeInstanceOf(FileCache);
});
});
describe('FileCache default TTL', () => {
let cacheDir: string;
let now: number;
beforeEach(async () => {
cacheDir = await mkdtemp(join(tmpdir(), 'praisonai-file-cache-ttl-'));
now = 1000;
jest.spyOn(Date, 'now').mockImplementation(() => now);
});
afterEach(async () => {
jest.restoreAllMocks();
await rm(cacheDir, { recursive: true, force: true });
});
it.each(['get', 'has'] as const)('honors the configured TTL through %s after reopening', async (method) => {
const cache = createFileCache({ cacheDir, ttl: 50 });
await cache.set('answer', 'old answer');
expect(await cache.get('answer')).toBe('old answer');
const reopened = new FileCache({ cacheDir });
now += 51;
if (method === 'get') {
expect(await reopened.get('answer')).toBeUndefined();
} else {
expect(await reopened.has('answer')).toBe(false);
}
expect(await reopened.keys()).not.toContain('answer');
});
it('allows a per-entry TTL to override the configured TTL', async () => {
const cache = new FileCache({ cacheDir, ttl: 50 });
await cache.set('answer', 'value', 100);
now += 51;
expect(await cache.get('answer')).toBe('value');
now += 50;
expect(await cache.get('answer')).toBeUndefined();
});
it('allows an explicit zero TTL to disable the configured expiry', async () => {
const cache = new FileCache({ cacheDir, ttl: 50 });
await cache.set('answer', 'value', 0);
now += 1000;
expect(await cache.get('answer')).toBe('value');
});
it('keeps entries without a default or explicit TTL', async () => {
const cache = new FileCache({ cacheDir });
await cache.set('answer', 'value');
now += 1000;
expect(await cache.get('answer')).toBe('value');
});
});
describe('MemoryCache capacity with empty keys', () => {
it.each([1, 2])('keeps capacity %i bounded when the oldest key is empty', async (maxSize) => {
const cache = new MemoryCache({ maxSize });
await cache.set('', 'oldest');
for (let i = 0; i <= maxSize; i++) {
await cache.set(`key-${i}`, i);
}
expect(await cache.size()).toBe(maxSize);
expect(await cache.has('')).toBe(false);
expect(await cache.keys()).toEqual(
Array.from({ length: maxSize }, (_, i) => `key-${i + 1}`)
);
expect(await cache.get(`key-${maxSize}`)).toBe(maxSize);
});
it('updates an existing empty key without evicting another entry', async () => {
const cache = new MemoryCache({ maxSize: 2 });
await cache.set('', 'old value');
await cache.set('other', 'retained');
await cache.set('', 'new value');
expect(await cache.size()).toBe(2);
expect(await cache.keys()).toEqual(['', 'other']);
expect(await cache.get('')).toBe('new value');
expect(await cache.get('other')).toBe('retained');
});
});
@@ -0,0 +1,70 @@
import { afterEach, beforeEach, describe, expect, it, jest } from '@jest/globals';
import { mkdtemp, mkdir, rm } from 'fs/promises';
import { tmpdir } from 'os';
import { join } from 'path';
import { createFileCache } from '../../../src/cache';
describe('FileCache concurrent list appends', () => {
let directory: string;
beforeEach(async () => { directory = await mkdtemp(join(tmpdir(), 'praison-list-')); });
afterEach(async () => { await rm(directory, { recursive: true, force: true }); });
it('persists every successful concurrent append in call order', async () => {
const cache = createFileCache({ cacheDir: directory });
await cache.listPush('events', 'existing');
const values = Array.from({ length: 25 }, (_, index) => `event-${index}`);
await Promise.all(values.map(value => cache.listPush('events', value)));
const reopened = createFileCache({ cacheDir: directory });
expect(await reopened.listRange('events', 0)).toEqual(['existing', ...values]);
expect(await reopened.listLength('events')).toBe(26);
});
it('retains each list when different keys receive concurrent appends', async () => {
const cache = createFileCache({ cacheDir: directory });
await cache.keys();
const values = Array.from({ length: 12 }, (_, index) => index + 1);
await Promise.all(values.flatMap(value => [cache.listPush('left', value), cache.listPush('right', -value)]));
const reopened = createFileCache({ cacheDir: directory });
expect(await reopened.listRange('left', 0)).toEqual(values);
expect(await reopened.listRange('right', 0)).toEqual(values.map(value => -value));
});
it('recovers after a failed append without persisting that value', async () => {
const cache = createFileCache({ cacheDir: directory });
const blockedFile = join(directory, 'list_events.json');
await mkdir(blockedFile);
await expect(cache.listPush('events', 'failed')).rejects.toThrow();
await rm(blockedFile, { recursive: true });
await cache.listPush('events', 'retry');
expect(await cache.listRange('events', 0)).toEqual(['retry']);
});
it('continues an already queued append after a transient write failure', async () => {
const cache = createFileCache({ cacheDir: directory });
let releaseFailure!: () => void;
let signalWrite!: () => void;
const blocked = new Promise<void>(resolve => { releaseFailure = resolve; });
const writing = new Promise<void>(resolve => { signalWrite = resolve; });
// Inject one failed public write; the queued retry uses the real filesystem.
jest.spyOn(cache, 'set').mockImplementationOnce(async () => {
signalWrite();
await blocked;
throw new Error('transient write failure');
});
const failed = expect(cache.listPush('events', 'failed')).rejects.toThrow('transient write failure');
const retry = cache.listPush('events', 'retry');
await writing;
releaseFailure();
await failed;
await retry;
const reopened = createFileCache({ cacheDir: directory });
expect(await reopened.listRange('events', 0)).toEqual(['retry']);
});
it('preserves sequential appends and list slicing', async () => {
const cache = createFileCache({ cacheDir: directory });
for (const value of [1, 2, 3]) await cache.listPush('events', value);
expect(await cache.listRange('events', 1, 2)).toEqual([2]);
expect(await cache.listRange('events', -1)).toEqual([3]);
});
});
@@ -0,0 +1,107 @@
import { promises as fs } from 'fs';
import * as os from 'os';
import * as path from 'path';
import { FileMemory } from '../../src/memory/file-memory';
describe('FileMemory write recovery', () => {
let tempDir: string;
beforeEach(async () => {
tempDir = await fs.mkdtemp(path.join(os.tmpdir(), 'praisonai-write-recovery-'));
});
afterEach(async () => {
jest.restoreAllMocks();
await fs.rm(tempDir, { recursive: true, force: true });
});
test('persists later writes after the filesystem becomes available again', async () => {
const directory = path.join(tempDir, 'store');
const unavailableDirectory = path.join(tempDir, 'unavailable');
await fs.mkdir(directory);
const filePath = path.join(directory, 'memory.jsonl');
const memory = new FileMemory({ filePath });
const initial = await memory.add('Before interruption', 'user');
await fs.rename(directory, unavailableDirectory);
await expect(memory.add('During interruption', 'user')).rejects.toMatchObject({ code: 'ENOENT' });
await fs.rename(unavailableDirectory, directory);
const recovered = await memory.add('After recovery', 'assistant');
const reopened = new FileMemory({ filePath });
expect(await reopened.getAll()).toEqual([initial, recovered]);
});
test('preserves each write result and continues queued writes after a rejection', async () => {
const filePath = path.join(tempDir, 'memory.jsonl');
const memory = new FileMemory({ filePath });
const initial = await memory.add('Before interruption', 'user');
const appendFile = fs.appendFile;
const failure = new Error('Temporary append failure');
let releaseFailure!: () => void;
let notifyAppend!: () => void;
const blocked = new Promise<void>(resolve => { releaseFailure = resolve; });
const started = new Promise<void>(resolve => { notifyAppend = resolve; });
jest.spyOn(fs, 'appendFile').mockImplementationOnce(async () => {
notifyAppend();
await blocked;
throw failure;
}).mockImplementation(appendFile);
const failed = memory.add('Failed write', 'user');
const queued = memory.add('Queued write', 'assistant');
const settled = Promise.allSettled([failed, queued]);
await started;
releaseFailure();
const [first, second] = await settled;
expect(first).toEqual({ status: 'rejected', reason: failure });
expect(second.status).toBe('fulfilled');
if (second.status !== 'fulfilled') throw second.reason;
const later = await memory.add('Later write', 'user');
const reopened = new FileMemory({ filePath });
expect(await reopened.getAll()).toEqual([initial, second.value, later]);
});
test.each([1, 2])('preserves later records after %i partial append failures', async (failures) => {
const filePath = path.join(tempDir, 'memory.jsonl');
const memory = new FileMemory({ filePath });
const initial = await memory.add('Before interruption', 'user');
const appendFile = fs.appendFile;
const failure = Object.assign(new Error('Disk full after a partial write'), { code: 'ENOSPC' });
let attempts = 0;
jest.spyOn(fs, 'appendFile').mockImplementation(async (file, data, options) => {
if (attempts++ < failures) {
// Write an actual unterminated JSON prefix before reporting failure.
await appendFile(file, String(data).slice(0, 20), options);
throw failure;
}
await appendFile(file, data, options);
});
for (let attempt = 0; attempt < failures; attempt++) {
await expect(memory.add('Interrupted write', 'user')).rejects.toBe(failure);
}
const recovered = await memory.add('Recovered write', 'assistant');
const later = await memory.add('Later write', 'user');
const reopened = new FileMemory({ filePath });
expect(await reopened.getAll()).toEqual([initial, recovered, later]);
});
test.each([
['incomplete JSON', false],
['complete JSON', true],
] as const)('preserves new records after loading an unterminated %s line', async (_, complete) => {
const filePath = path.join(tempDir, 'memory.jsonl');
const original = new FileMemory({ filePath });
const initial = await original.add('Before restart', 'user');
const last = { id: 'last-record', content: 'Last record', role: 'user', timestamp: initial.timestamp };
const tail = complete ? JSON.stringify(last) : '{"id":"interrupted","content":';
await fs.appendFile(filePath, tail);
const restarted = new FileMemory({ filePath });
const recovered = await restarted.add('After restart', 'assistant');
const reopened = new FileMemory({ filePath });
expect(await reopened.getAll()).toEqual(complete ? [initial, last, recovered] : [initial, recovered]);
});
});
@@ -0,0 +1,47 @@
import { describe, expect, it } from '@jest/globals';
import { HooksManager } from '../../src/hooks/manager';
describe('HooksManager inspection', () => {
it('allows registering and executing hooks after inspection', async () => {
const manager = new HooksManager();
const called: string[] = [];
const first = manager.register('pre_tool_call', () => { called.push('first'); });
const snapshot = manager.getHooks('pre_tool_call');
const second = manager.register('pre_tool_call', () => { called.push('second'); }, { priority: 1 });
expect(snapshot.map(hook => hook.id)).toEqual([first]);
expect(manager.getHooks('pre_tool_call').map(hook => hook.id)).toEqual([second, first]);
expect((await manager.execute('pre_tool_call', {})).success).toBe(true);
expect(called).toEqual(['second', 'first']);
});
it('allows unregistering a hook after inspection', async () => {
const manager = new HooksManager();
const called: string[] = [];
const first = manager.register('pre_tool_call', () => { called.push('first'); });
const second = manager.register('pre_tool_call', () => { called.push('second'); });
const snapshot = manager.getHooks('pre_tool_call');
expect(manager.unregister(first)).toBe(true);
expect(snapshot.map(hook => hook.id)).toEqual([first, second]);
expect(manager.getStats().totalHooks).toBe(1);
await manager.execute('pre_tool_call', {});
expect(called).toEqual(['second']);
});
it('keeps returned arrays frozen against caller mutations', () => {
const manager = new HooksManager();
const id = manager.register('pre_tool_call', () => {});
const snapshot = manager.getHooks('pre_tool_call');
expect(Object.isFrozen(snapshot)).toBe(true);
expect(() => (snapshot as unknown[]).pop()).toThrow(TypeError);
expect(manager.getHooks('pre_tool_call').map(hook => hook.id)).toEqual([id]);
});
it('allows first registration after inspecting an empty event', () => {
const manager = new HooksManager();
const snapshot = manager.getHooks('pre_tool_call');
const id = manager.register('pre_tool_call', () => {});
expect(snapshot).toEqual([]);
expect(Object.isFrozen(snapshot)).toBe(true);
expect(manager.getHooks('pre_tool_call').map(hook => hook.id)).toEqual([id]);
});
});
@@ -0,0 +1,59 @@
import { afterEach, beforeEach, describe, expect, it, jest } from '@jest/globals';
import { spawnSync } from 'child_process';
import { resolve } from 'path';
import { HooksManager } from '../../src/hooks/manager';
describe('HooksManager timeout lifecycle', () => {
beforeEach(() => { jest.useFakeTimers(); });
afterEach(() => { jest.clearAllTimers(); jest.useRealTimers(); });
it('releases the deadline after a successful asynchronous hook', async () => {
const hooks = new HooksManager({ timeout: 30000 });
hooks.register('pre_tool_call', async (context) => ({ ...context, checked: true }));
const result = await hooks.execute('pre_tool_call', { input: 'test' });
expect(result.modifiedContext).toEqual({ input: 'test', checked: true });
expect(result.blocked).toBe(false);
expect(jest.getTimerCount()).toBe(0);
});
it('releases the deadline after an asynchronous hook rejects', async () => {
const hooks = new HooksManager({ timeout: 30000 });
hooks.register('pre_tool_call', async () => { throw new Error('validation failed'); }, { blockOnFailure: true });
const result = await hooks.execute('pre_tool_call', {});
expect(result).toMatchObject({ success: false, blocked: true, error: 'validation failed' });
expect(jest.getTimerCount()).toBe(0);
});
it('still times out a pending hook and preserves blocking policy', async () => {
const hooks = new HooksManager({ timeout: 10 });
hooks.register('pre_tool_call', () => new Promise(() => {}), { blockOnFailure: true });
const execution = hooks.execute('pre_tool_call', {});
expect(jest.getTimerCount()).toBe(1);
await jest.advanceTimersByTimeAsync(10);
expect(await execution).toMatchObject({ success: false, blocked: true, error: 'Hook timeout after 10ms' });
expect(jest.getTimerCount()).toBe(0);
});
it('does not allocate a deadline for a synchronous hook', async () => {
const hooks = new HooksManager();
hooks.register('pre_tool_call', () => ({ checked: true }));
expect((await hooks.execute('pre_tool_call', {})).modifiedContext).toEqual({ checked: true });
expect(jest.getTimerCount()).toBe(0);
});
it('lets a real process exit naturally after its async hook settles', () => {
const script = `
const { HooksManager } = require(${JSON.stringify(resolve(__dirname, '../../src/hooks/manager'))});
const hooks = new HooksManager({ timeout: 30000 });
hooks.register('pre_tool_call', async context => context);
hooks.execute('pre_tool_call', {}).then(() => console.log('hook settled')).catch(error => { console.error(error); process.exitCode = 1; });
`;
const result = spawnSync(process.execPath, ['-r', 'ts-node/register/transpile-only', '-e', script], {
timeout: 10000,
encoding: 'utf-8',
});
expect(result.stdout).toContain('hook settled');
expect(result.error).toBeUndefined();
expect({ status: result.status, stderr: result.stderr }).toEqual({ status: 0, stderr: '' });
}, 15000);
});
@@ -30,6 +30,34 @@ describe('Chunking', () => {
expect(chunks.length).toBe(3);
});
it.each([
'First sentence. Final important fact without punctuation',
'Echo.Echo.Echo.',
' \nFirst. Second! \nTail \t',
'... Leading punctuation. Tail',
'No punctuation here',
'Really?! Next.',
'你好。完整段落',
])('preserves sentence content and its source span: %j', async text => {
const { Chunking } = await import('../../../src/knowledge/chunking');
const chunks = new Chunking({ strategy: 'sentence' }).chunk(text);
let cursor = 0;
for (const [index, chunk] of chunks.entries()) {
expect(chunk.index).toBe(index);
expect(chunk.content.length).toBeGreaterThan(0);
expect(chunk.startOffset).toBeGreaterThanOrEqual(cursor);
expect(text.slice(cursor, chunk.startOffset).trim()).toBe('');
expect(text.slice(chunk.startOffset, chunk.endOffset)).toBe(chunk.content);
cursor = chunk.endOffset;
}
expect(text.slice(cursor).trim()).toBe('');
});
it.each(['', ' \n\t '])('does not emit empty sentence chunks for %j', async text => {
const { Chunking } = await import('../../../src/knowledge/chunking');
expect(new Chunking({ strategy: 'sentence' }).chunk(text)).toEqual([]);
});
it('should chunk by paragraphs', async () => {
const { Chunking } = await import('../../../src/knowledge/chunking');
const chunker = new Chunking({ strategy: 'paragraph' });
@@ -0,0 +1,47 @@
import { describe, expect, it } from '@jest/globals';
import { GraphStore } from '../../../src/knowledge/graph-rag';
function graph(): GraphStore {
const store = new GraphStore();
for (const id of ['a', 'b', 'c', 'd']) store.addNode({ id, type: 'document', content: id });
return store;
}
describe('GraphStore edge replacement', () => {
it.each([
['a', 'c'], ['c', 'b'], ['c', 'd'], ['a', 'a'],
])('removes obsolete adjacency when an edge moves to %s -> %s', (source, target) => {
const store = graph();
store.addEdge({ id: 'edge', source: 'a', target: 'b', type: 'link' });
store.addEdge({ id: 'edge', source, target, type: 'link' });
expect(store.getNeighbors('a', 'outgoing').map(n => n.id)).not.toContain('b');
expect(store.getNeighbors('b', 'incoming').map(n => n.id)).not.toContain('a');
expect(store.getEdgesBetween('a', 'b')).toEqual([]);
expect(store.findPaths('a', 'b')).toEqual([]);
expect(store.getNeighbors(source, 'outgoing').map(n => n.id)).toContain(target);
expect(store.getNeighbors(target, 'incoming').map(n => n.id)).toContain(source);
expect(store.getStats().edgeCount).toBe(1);
});
it('retains adjacency backed by a parallel edge', () => {
const store = graph();
store.addEdge({ id: 'one', source: 'a', target: 'b', type: 'one' });
store.addEdge({ id: 'two', source: 'a', target: 'b', type: 'two' });
store.addEdge({ id: 'one', source: 'c', target: 'd', type: 'one' });
expect(store.getNeighbors('a', 'outgoing').map(n => n.id)).toEqual(['b']);
expect(store.getNeighbors('b', 'incoming').map(n => n.id)).toEqual(['a']);
expect(store.getEdgesBetween('a', 'b').map(e => e.id)).toEqual(['two']);
store.addEdge({ id: 'two', source: 'c', target: 'd', type: 'two' });
expect(store.getNeighbors('a', 'outgoing')).toEqual([]);
expect(store.getNeighbors('b', 'incoming')).toEqual([]);
expect(store.getStats().edgeCount).toBe(2);
});
it('preserves neighbors when only edge metadata changes', () => {
const store = graph();
store.addEdge({ id: 'edge', source: 'a', target: 'b', type: 'link', weight: 1 });
store.addEdge({ id: 'edge', source: 'a', target: 'b', type: 'link', weight: 2 });
expect(store.getNeighbors('a', 'outgoing').map(n => n.id)).toEqual(['b']);
expect(store.getEdgesBetween('a', 'b')[0].weight).toBe(2);
});
});
@@ -0,0 +1,93 @@
import { EventEmitter } from 'events';
import { WebSocketTransport } from '../../src/mcp/transports';
class MockWebSocket extends EventEmitter {
static last: MockWebSocket;
send = jest.fn();
// Real close handshakes finish asynchronously; tests emit close explicitly.
close = jest.fn();
constructor(_url: string) {
super();
MockWebSocket.last = this;
queueMicrotask(() => this.emit('open'));
}
}
jest.mock('ws', () => ({ __esModule: true, default: MockWebSocket }), { virtual: true });
async function boundedOutcome(pending: Promise<unknown>): Promise<unknown> {
let timer: ReturnType<typeof setTimeout> | undefined;
try {
return await Promise.race([
pending.then(() => 'resolved', error => error),
new Promise(resolve => { timer = setTimeout(() => resolve('pending'), 100); }),
]);
} finally {
if (timer) clearTimeout(timer);
}
}
const response = (source: string) => JSON.stringify({ jsonrpc: '2.0', id: 1, result: { source } });
describe('WebSocketTransport connection lifecycle', () => {
let transport: WebSocketTransport;
beforeEach(async () => {
transport = new WebSocketTransport('ws://localhost');
await transport.connect();
});
afterEach(async () => {
await transport.disconnect();
});
it.each(['remote', 'local'])('rejects every pending request on %s close', async (mode) => {
const outcomes = [1, 2].map(id => boundedOutcome(
transport.send({ jsonrpc: '2.0', id, method: 'tools/list' })
));
if (mode === 'remote') MockWebSocket.last.emit('close');
else await transport.disconnect();
expect(transport.isConnected()).toBe(false);
for (const outcome of await Promise.all(outcomes)) {
expect(outcome).toBeInstanceOf(Error);
expect(outcome).toEqual(expect.objectContaining({
message: expect.stringMatching(/closed|disconnect/i),
}));
}
});
it('keeps a replacement connection active when the old close event arrives', async () => {
const old = MockWebSocket.last;
await transport.disconnect();
await transport.connect();
old.emit('close');
expect(transport.isConnected()).toBe(true);
const pending = transport.send({ jsonrpc: '2.0', id: 1, method: 'tools/list' });
MockWebSocket.last.emit('message', response('new'));
await expect(pending).resolves.toMatchObject({ result: { source: 'new' } });
});
it('ignores a closing connection response when a new request reuses its ID', async () => {
const old = MockWebSocket.last;
await transport.disconnect();
await transport.connect();
const received = jest.fn();
transport.onMessage(received);
const pending = transport.send({ jsonrpc: '2.0', id: 1, method: 'tools/list' });
old.emit('message', response('old'));
MockWebSocket.last.emit('message', response('new'));
await expect(pending).resolves.toMatchObject({ result: { source: 'new' } });
expect(received).toHaveBeenCalledTimes(1);
});
it('keeps a completed response available after its connection closes', async () => {
const pending = transport.send({ jsonrpc: '2.0', id: 1, method: 'tools/list' });
MockWebSocket.last.emit('message', response('complete'));
MockWebSocket.last.emit('close');
await expect(pending).resolves.toMatchObject({ result: { source: 'complete' } });
});
});
@@ -0,0 +1,55 @@
import { EventEmitter } from 'events';
import { WebSocketTransport } from '../../src/mcp/transports';
class MockWebSocket extends EventEmitter {
static last: MockWebSocket;
send = jest.fn();
constructor(_url: string) {
super();
MockWebSocket.last = this;
queueMicrotask(() => this.emit('open'));
}
close(): void {
this.emit('close');
}
}
jest.mock('ws', () => ({ __esModule: true, default: MockWebSocket }), { virtual: true });
describe('WebSocketTransport response correlation', () => {
let transport: WebSocketTransport;
beforeEach(async () => {
transport = new WebSocketTransport('ws://localhost');
await transport.connect();
});
afterEach(async () => {
await transport.disconnect();
});
it.each([0, 1, 'request-1'])('resolves the request with id %p', async (id) => {
const response = { jsonrpc: '2.0' as const, id, result: { tools: [] } };
const received = jest.fn();
transport.onMessage(received);
const pending = transport.send({ jsonrpc: '2.0', id, method: 'tools/list' });
MockWebSocket.last.emit('message', Buffer.from(JSON.stringify(response)));
await expect(pending).resolves.toEqual(response);
expect(received).toHaveBeenCalledWith(response);
});
it('rejects an error response with id zero', async () => {
const pending = transport.send({ jsonrpc: '2.0', id: 0, method: 'tools/list' });
const rejected = expect(pending).rejects.toThrow('Tool listing failed');
MockWebSocket.last.emit('message', JSON.stringify({
jsonrpc: '2.0', id: 0, error: { code: -32603, message: 'Tool listing failed' },
}));
await rejected;
});
});
@@ -13,6 +13,31 @@ import {
import { promises as fs } from 'fs';
import * as path from 'path';
import * as os from 'os';
import { Memory } from '../../src/memory/memory';
describe('Memory history limits', () => {
test('an explicit zero returns no recent entries', async () => {
const memory = new Memory();
const first = await memory.add('First message', 'user');
const second = await memory.add('Second message', 'assistant');
expect(memory.getRecent(0)).toEqual([]);
expect(memory.getRecent(1)).toEqual([second]);
expect(memory.getRecent(3)).toEqual([first, second]);
expect(memory.getAll()).toEqual([first, second]);
});
test('zero context entries differs from an omitted limit', async () => {
const memory = new Memory();
await memory.add('First message', 'user');
await memory.add('Second message', 'assistant');
expect(memory.buildContext(0)).toBe('');
expect(memory.buildContext(1)).toBe('assistant: Second message');
expect(memory.buildContext()).toBe('user: First message\nassistant: Second message');
expect(memory.size).toBe(2);
});
});
describe('FileMemory', () => {
let tempDir: string;
@@ -76,6 +101,20 @@ describe('FileMemory', () => {
expect(recent[1].content).toBe('Message 3');
});
test('zero history limits survive a file-memory reload', async () => {
const memory = createFileMemory({ filePath: testFilePath });
const first = await memory.add('First message', 'user');
const second = await memory.add('Second message', 'assistant');
const reloaded = createFileMemory({ filePath: testFilePath });
expect(await reloaded.getRecent(0)).toEqual([]);
expect(await reloaded.buildContext(0)).toBe('');
expect(await reloaded.getRecent(1)).toEqual([second]);
expect(await reloaded.buildContext(1)).toBe('assistant: Second message');
expect(await reloaded.buildContext()).toBe('user: First message\nassistant: Second message');
expect(await reloaded.getAll()).toEqual([first, second]);
});
test('delete removes entry', async () => {
const memory = createFileMemory({ filePath: testFilePath });
const entry = await memory.add('Hello world', 'user');
@@ -268,6 +307,18 @@ describe('FileMemory', () => {
});
describe('AutoMemory', () => {
test('zero recent entries does not return stored automatic memories', async () => {
const memory = createAutoMemory({
policies: [{ name: 'store-all', condition: () => true, action: 'store', priority: 1000 }]
});
const first = await memory.add('First message', 'user');
const second = await memory.add('Second message', 'assistant');
expect(memory.getRecent(0)).toEqual([]);
expect(memory.getRecent(1)).toEqual([second]);
expect(memory.getAll()).toEqual([first, second]);
});
test('createAutoMemory creates instance', () => {
const memory = createAutoMemory();
expect(memory).toBeInstanceOf(AutoMemory);
@@ -0,0 +1,138 @@
import { afterEach, beforeEach, describe, expect, it, jest } from '@jest/globals';
import { promises as fs } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { FileSessionStore, createStoredSession } from '../../../src/session/store';
function gate(): { promise: Promise<void>; release: () => void } {
let release!: () => void;
const promise = new Promise<void>(resolve => { release = resolve; });
return { promise, release };
}
describe('FileSessionStore save ordering', () => {
let directory: string;
let filePath: string;
beforeEach(async () => {
directory = await fs.mkdtemp(join(tmpdir(), 'praison-session-save-'));
filePath = join(directory, 'sessions.json');
});
afterEach(async () => {
jest.restoreAllMocks();
jest.useRealTimers();
await fs.rm(directory, { recursive: true, force: true });
});
function pauseFirstWrite() {
const entered = gate();
const blocked = gate();
const writeFile = fs.writeFile.bind(fs);
let writes = 0;
jest.spyOn(fs, 'writeFile').mockImplementation(async (...args: Parameters<typeof fs.writeFile>) => {
if (++writes === 1) {
entered.release();
await blocked.promise;
}
return writeFile(...args);
});
return { entered: entered.promise, release: blocked.release, writes: () => writes };
}
it.each(['set', 'delete', 'clear'] as const)('persists %s during an in-flight save on close', async operation => {
const store = new FileSessionStore(filePath);
const first = createStoredSession({ turn: 1 });
const second = createStoredSession({ turn: 2 });
await store.set(first);
const paused = pauseFirstWrite();
const saving = store.save();
await paused.entered;
if (operation === 'set') await store.set(second);
else if (operation === 'delete') await store.delete(first.id);
else await store.clear();
paused.release();
await saving;
await store.close();
const reloaded = new FileSessionStore(filePath);
await reloaded.load();
expect((await reloaded.list()).map(session => session.id)).toEqual(operation === 'set' ? [first.id, second.id] : []);
});
it('serializes saves so an older snapshot cannot overwrite a newer one', async () => {
const store = new FileSessionStore(filePath);
const session = createStoredSession({ turn: 1 });
await store.set(session);
const paused = pauseFirstWrite();
const firstSave = store.save();
await paused.entered;
await store.set({ ...session, data: { turn: 2 } });
const secondSave = store.save();
await new Promise<void>(resolve => setImmediate(resolve));
const writesBeforeRelease = paused.writes();
paused.release();
await Promise.all([firstSave, secondSave]);
await store.close();
const reloaded = new FileSessionStore(filePath);
await reloaded.load();
expect(writesBeforeRelease).toBe(1);
expect((await reloaded.get(session.id))?.data).toEqual({ turn: 2 });
});
it('waits for an outstanding save before closing', async () => {
const store = new FileSessionStore(filePath);
await store.set(createStoredSession());
const paused = pauseFirstWrite();
const saving = store.save();
await paused.entered;
let closed = false;
const closing = store.close().then(() => { closed = true; });
await new Promise<void>(resolve => setImmediate(resolve));
const writesBeforeRelease = paused.writes();
const closedBeforeRelease = closed;
paused.release();
await Promise.all([saving, closing]);
expect(writesBeforeRelease).toBe(1);
expect(closedBeforeRelease).toBe(false);
});
it('does not queue repeated autosaves behind a blocked write', async () => {
jest.useFakeTimers();
const store = new FileSessionStore(filePath, { autoSaveInterval: 10 });
const save = jest.spyOn(store, 'save');
const first = createStoredSession({ turn: 1 });
const second = createStoredSession({ turn: 2 });
await store.set(first);
const paused = pauseFirstWrite();
const saving = store.save();
await paused.entered;
await store.set(second);
jest.advanceTimersByTime(100);
const saveCallsBeforeRelease = save.mock.calls.length;
paused.release();
await saving;
jest.advanceTimersByTime(10);
const saveCallsAfterRelease = save.mock.calls.length;
await store.close();
const reloaded = new FileSessionStore(filePath);
await reloaded.load();
expect(saveCallsBeforeRelease).toBe(1);
expect(saveCallsAfterRelease).toBe(2);
expect((await reloaded.list()).map(session => session.id)).toEqual([first.id, second.id]);
});
it('retries a failed save on close without poisoning later saves', async () => {
const store = new FileSessionStore(filePath);
const session = createStoredSession({ turn: 1 });
await store.set(session);
jest.spyOn(fs, 'writeFile').mockRejectedValueOnce(new Error('disk unavailable'));
await expect(store.save()).rejects.toThrow('disk unavailable');
await store.close();
const reloaded = new FileSessionStore(filePath);
await reloaded.load();
expect((await reloaded.get(session.id))?.data).toEqual({ turn: 1 });
});
});
@@ -0,0 +1,81 @@
import { describe, it, expect } from '@jest/globals';
import { Session } from '../../../src/session/session';
import { MemoryDbAdapter } from '../../../src/db/memory-adapter';
const factories: Array<[string, () => Session]> = [
['new', () => new Session()],
['setState', () => { const s = new Session(); s.setState({}); return s; }],
['clearState', () => { const s = new Session(); s.set('old', 1); s.clearState(); return s; }],
['fromJSON', () => Session.fromJSON(JSON.parse(JSON.stringify(new Session().toJSON())))],
];
describe.each(factories)('Session state keys after %s', (_name, create) => {
it.each(['toString', 'constructor', '__proto__'])('does not report an unstored %s key', key => {
const session = create();
expect(session.has(key)).toBe(false);
expect(session.get(key)).toBeUndefined();
expect(session.delete(key)).toBe(false);
});
it.each(['toString', 'constructor', '__proto__'])('stores and deletes %s as ordinary data', key => {
const session = create();
const value = { preference: 'window seat' };
session.set(key, value);
expect(session.has(key)).toBe(true);
expect(session.get(key)).toEqual(value);
expect(session.keys()).toEqual([key]);
expect(JSON.parse(JSON.stringify(session.toJSON())).data[key]).toEqual(value);
expect(session.delete(key)).toBe(true);
expect(session.has(key)).toBe(false);
expect(session.get(key)).toBeUndefined();
});
it('preserves an explicitly stored undefined value until deletion', () => {
const session = create();
session.set('optional', undefined);
expect(session.has('optional')).toBe(true);
expect(session.delete('optional')).toBe(true);
expect(session.has('optional')).toBe(false);
});
});
it('preserves special keys supplied through setState and a JSON round trip', () => {
const state = JSON.parse('{"__proto__":{"seat":"window"},"constructor":2,"toString":3}');
const session = new Session();
session.setState(state);
const restored = Session.fromJSON(JSON.parse(JSON.stringify(session.toJSON())));
expect(restored.getState()).toEqual(state);
expect(restored.delete('__proto__')).toBe(true);
expect(restored.has('__proto__')).toBe(false);
restored.set('__proto__', { seat: 'aisle' });
expect(restored.keys()).toContain('__proto__');
expect(restored.get('__proto__')).toEqual({ seat: 'aisle' });
});
it('preserves special keys across database save/load without inheriting unstored keys', async () => {
const db = new MemoryDbAdapter();
const session = new Session({ id: 'state-keys', db });
session.set('__proto__', { preference: 'window seat' });
await session.save();
const restored = new Session({ id: session.id, db });
await restored.load();
expect(restored.get('__proto__')).toEqual({ preference: 'window seat' });
expect(restored.keys()).toEqual(['__proto__']);
expect(restored.has('toString')).toBe(false);
restored.delete('__proto__');
restored.set('__proto__', { preference: 'aisle' });
expect(restored.keys()).toEqual(['__proto__']);
expect(restored.get('__proto__')).toEqual({ preference: 'aisle' });
});
it('exports state as an ordinary object while preserving special own keys', () => {
const session = new Session();
session.set('seat', 'window');
session.set('__proto__', { preference: 'quiet' });
const exported = session.toJSON().data;
expect(Object.getPrototypeOf(exported)).toBe(Object.prototype);
expect(exported.hasOwnProperty('seat')).toBe(true);
expect(exported.hasOwnProperty('__proto__')).toBe(true);
expect(JSON.parse(JSON.stringify(exported))['__proto__']).toEqual({ preference: 'quiet' });
expect(session.has('toString')).toBe(false);
});
@@ -0,0 +1,84 @@
import { describe, expect, it } from '@jest/globals';
import { ToolsRegistry } from '../../../src/tools/registry/registry';
import type { ToolMetadata } from '../../../src/tools/registry/types';
const missingPackage = '@praisonai-test/not-installed-registry-fixture';
function register(registry: ToolsRegistry, packageName: string) {
const metadata: ToolMetadata = {
id: 'dependency-test', displayName: 'Dependency test', description: 'Local status test',
tags: [], requiredEnv: [], optionalEnv: [], docsSlug: 'tools/dependency-test',
install: { npm: `npm install ${packageName}`, pnpm: '', yarn: '', bun: '' },
capabilities: {}, packageName,
};
registry.register(metadata, () => ({
name: 'dependency-test', description: 'Local status test',
parameters: { type: 'object', properties: {} }, execute: async () => 'ok',
}));
}
describe('ToolsRegistry installation cache lifecycle', () => {
it('does not report a removed cached tool as installed', async () => {
const registry = new ToolsRegistry();
register(registry, 'node:path');
expect(await registry.checkInstalled('dependency-test')).toBe(true);
expect(registry.unregister('dependency-test')).toBe(true);
expect(await registry.checkInstalled('dependency-test')).toBe(false);
expect(await registry.getInstallStatus('dependency-test')).toBeNull();
});
it('rechecks an installed dependency replaced by a missing one', async () => {
const registry = new ToolsRegistry();
register(registry, 'node:path');
expect(await registry.checkInstalled('dependency-test')).toBe(true);
register(registry, missingPackage);
expect(await registry.getInstallStatus('dependency-test')).toMatchObject({
installed: false, installCommand: `npm install ${missingPackage}`,
});
});
it('rechecks a missing dependency replaced by an installed one', async () => {
const registry = new ToolsRegistry();
register(registry, missingPackage);
expect(await registry.checkInstalled('dependency-test')).toBe(false);
register(registry, 'node:path');
expect(await registry.getInstallStatus('dependency-test')).toMatchObject({ installed: true, installCommand: undefined });
});
it('does not cache an old successful import after replacement', async () => {
const registry = new ToolsRegistry();
register(registry, 'node:path');
const oldCheck = registry.checkInstalled('dependency-test');
register(registry, missingPackage);
expect(await oldCheck).toBe(true);
expect(await registry.checkInstalled('dependency-test')).toBe(false);
});
it('does not cache an old failed import after replacement', async () => {
const registry = new ToolsRegistry();
register(registry, missingPackage);
const oldCheck = registry.checkInstalled('dependency-test');
register(registry, 'node:path');
expect(await oldCheck).toBe(false);
expect(await registry.checkInstalled('dependency-test')).toBe(true);
});
it('reports an unknown tool as unavailable', async () => {
const registry = new ToolsRegistry();
expect(await registry.checkInstalled('unknown')).toBe(false);
expect(await registry.getInstallStatus('unknown')).toBeNull();
});
it('keeps repeated installed checks consistent', async () => {
const registry = new ToolsRegistry();
register(registry, 'node:path');
expect(await registry.checkInstalled('dependency-test')).toBe(true);
expect(await registry.checkInstalled('dependency-test')).toBe(true);
});
it('keeps repeated missing checks consistent', async () => {
const registry = new ToolsRegistry();
register(registry, missingPackage);
expect(await registry.checkInstalled('dependency-test')).toBe(false);
expect(await registry.checkInstalled('dependency-test')).toBe(false);
});
});
@@ -0,0 +1,104 @@
import { describe, expect, it, jest } from '@jest/globals';
import { ToolsRegistry } from '../../../src/tools/registry/registry';
import { composeMiddleware, createRetryMiddleware } from '../../../src/tools/registry/middleware';
import type { ToolMetadata, ToolMiddleware, ToolExecutionContext } from '../../../src/tools/registry/types';
const metadata: ToolMetadata = {
id: 'retry-test', displayName: 'Retry test', description: 'Local retry test',
tags: [], requiredEnv: [], optionalEnv: [], docsSlug: 'tools/retry-test',
install: { npm: '', pnpm: '', yarn: '', bun: '' }, capabilities: {}, packageName: 'node:path',
};
function runPipeline(mode: 'composed' | 'registry', middleware: ToolMiddleware[], execute: () => Promise<unknown>, context: ToolExecutionContext = {}) {
if (mode === 'composed') return composeMiddleware(...middleware)({}, context, execute);
const registry = new ToolsRegistry();
registry.register(metadata, () => ({
name: 'retry-test', description: 'Local retry test',
parameters: { type: 'object', properties: {} }, execute,
}));
for (const layer of middleware) registry.use(layer);
return registry.create('retry-test').execute({}, context);
}
describe.each(['composed', 'registry'] as const)('%s middleware retry chain', mode => {
it('reapplies result shaping when the tool succeeds on retry', async () => {
let attempts = 0;
let shapingCalls = 0;
const shape: ToolMiddleware = async (_input, _context, next) => {
shapingCalls++;
return { wrapped: await next() };
};
const result = await runPipeline(mode, [createRetryMiddleware(1, 0), shape], async () => {
if (++attempts === 1) throw new Error('temporary tool failure');
return 'recovered';
});
expect(result).toEqual({ wrapped: 'recovered' });
expect(attempts).toBe(2);
expect(shapingCalls).toBe(2);
});
it('re-enters a downstream middleware that failed before the tool', async () => {
let layerAttempts = 0;
let toolCalls = 0;
const layer: ToolMiddleware = async (_input, _context, next) => {
if (++layerAttempts === 1) throw new Error('temporary middleware failure');
return { wrapped: await next() };
};
const result = await runPipeline(mode, [createRetryMiddleware(1, 0), layer], async () => {
toolCalls++;
return 'result';
});
expect(result).toEqual({ wrapped: 'result' });
expect(layerAttempts).toBe(2);
expect(toolCalls).toBe(1);
});
it('runs downstream failure handling on every exhausted attempt', async () => {
let attempts = 0;
let handled = 0;
const layer: ToolMiddleware = async (_input, _context, next) => {
try { return await next(); } finally { handled++; }
};
await expect(runPipeline(mode, [createRetryMiddleware(2, 0), layer], async () => {
throw new Error(`failure-${++attempts}`);
})).rejects.toThrow('failure-3');
expect(attempts).toBe(3);
expect(handled).toBe(3);
});
it('does not re-enter downstream middleware after cancellation during backoff', async () => {
jest.useFakeTimers();
try {
const controller = new AbortController();
let entries = 0;
let calls = 0;
const layer: ToolMiddleware = async (_input, _context, next) => { entries++; return next(); };
const done = runPipeline(mode, [createRetryMiddleware(1, 1000), layer], async () => {
if (++calls === 1) throw new Error('temporary');
return 'retried';
}, { signal: controller.signal }).then(value => ({ value, error: undefined }), error => ({ value: undefined, error }));
await jest.advanceTimersByTimeAsync(0);
expect(entries).toBe(1);
controller.abort();
await jest.advanceTimersByTimeAsync(1000);
expect(entries).toBe(1);
expect(calls).toBe(1);
expect((await done).error).toBe(controller.signal.reason);
} finally { jest.clearAllTimers(); jest.useRealTimers(); }
});
it('preserves middleware order for a successful first attempt', async () => {
const order: string[] = [];
const layer = (name: string): ToolMiddleware => async (_input, _context, next) => {
order.push(`${name}:before`);
const result = await next();
order.push(`${name}:after`);
return result;
};
expect(await runPipeline(mode, [createRetryMiddleware(1, 0), layer('a'), layer('b')], async () => {
order.push('tool');
return 'ok';
})).toBe('ok');
expect(order).toEqual(['a:before', 'b:before', 'tool', 'b:after', 'a:after']);
});
});
@@ -0,0 +1,54 @@
import { describe, expect, it, jest } from '@jest/globals';
import { createValidationMiddleware } from '../../../src/tools/registry/middleware';
const payloads: Array<[string, unknown]> = [
['CJK text', '你好'],
['emoji', '🙂'],
['nested JSON', { message: '中文🙂', values: ['é'] }],
];
describe('tool payload UTF-8 byte limits', () => {
it.each(payloads)('rejects %s above its actual byte budget', async (_label, input) => {
const bytes = Buffer.byteLength(JSON.stringify(input), 'utf8');
const next = jest.fn(async () => 'executed');
await expect(createValidationMiddleware()(input, { limits: { maxPayloadBytes: bytes - 1 } }, next))
.rejects.toThrow(`Input payload size (${bytes} bytes) exceeds limit (${bytes - 1} bytes)`);
expect(next).not.toHaveBeenCalled();
});
it.each(payloads)('accepts %s at the exact byte budget', async (_label, input) => {
const bytes = Buffer.byteLength(JSON.stringify(input), 'utf8');
const next = jest.fn(async () => 'executed');
expect(await createValidationMiddleware()(input, { limits: { maxPayloadBytes: bytes } }, next)).toBe('executed');
expect(next).toHaveBeenCalledTimes(1);
});
it('preserves ASCII boundaries', async () => {
const next = jest.fn(async () => 'executed');
const validate = createValidationMiddleware();
await expect(validate('hello', { limits: { maxPayloadBytes: 6 } }, next)).rejects.toThrow('7 bytes');
expect(await validate('hello', { limits: { maxPayloadBytes: 7 } }, next)).toBe('executed');
expect(next).toHaveBeenCalledTimes(1);
});
it('does not serialize inputs when no payload limit is configured', async () => {
const cyclic: { self?: unknown } = {};
cyclic.self = cyclic;
const next = jest.fn(async () => 'executed');
expect(await createValidationMiddleware()(cyclic, {}, next)).toBe('executed');
});
it('continues rejecting non-JSON undefined input when a limit is configured', async () => {
const next = jest.fn(async () => 'executed');
await expect(createValidationMiddleware()(undefined, { limits: { maxPayloadBytes: 10 } }, next)).rejects.toThrow(TypeError);
expect(next).not.toHaveBeenCalled();
});
it('preserves circular JSON errors when a limit is configured', async () => {
const cyclic: { self?: unknown } = {};
cyclic.self = cyclic;
const next = jest.fn(async () => 'executed');
await expect(createValidationMiddleware()(cyclic, { limits: { maxPayloadBytes: 100 } }, next)).rejects.toThrow(TypeError);
expect(next).not.toHaveBeenCalled();
});
});
@@ -0,0 +1,94 @@
import { afterEach, beforeEach, describe, expect, it, jest } from '@jest/globals';
import { getEventListeners } from 'events';
import { FunctionTool } from '../../../src/tools/decorator';
function makeTool(execute: () => Promise<string>) {
return new FunctionTool({ name: 'retry_test', execute, retryPolicy: { maxAttempts: 2, initialDelayMs: 1000, maxDelayMs: 1000 } });
}
describe('FunctionTool retry backoff cancellation', () => {
beforeEach(() => { jest.useFakeTimers(); });
afterEach(() => { jest.clearAllTimers(); jest.useRealTimers(); });
it.each(['execute', 'executeRaw'] as const)('%s settles promptly when cancelled during backoff', async method => {
const controller = new AbortController();
const failure = new Error('connection dropped');
let calls = 0;
let settled = false;
const tool = makeTool(async () => { if (++calls === 1) throw failure; return 'retried'; });
const done = tool[method]({}, { signal: controller.signal }).then(
value => { settled = true; return { value, error: undefined }; },
error => { settled = true; return { value: undefined, error }; },
);
await jest.advanceTimersByTimeAsync(0);
expect(calls).toBe(1);
controller.abort();
await jest.advanceTimersByTimeAsync(0);
expect(settled).toBe(true);
expect((await done).error).toBe(failure);
expect(jest.getTimerCount()).toBe(0);
expect(getEventListeners(controller.signal, 'abort')).toHaveLength(0);
await jest.advanceTimersByTimeAsync(2000);
expect(calls).toBe(1);
});
it.each(['execute', 'executeRaw'] as const)('%s does not start another call after the original delay', async method => {
const controller = new AbortController();
const failure = new Error('connection dropped');
let calls = 0;
const tool = makeTool(async () => { if (++calls === 1) throw failure; return 'retried'; });
const done = tool[method]({}, { signal: controller.signal }).then(value => ({ value, error: undefined }), error => ({ value: undefined, error }));
await jest.advanceTimersByTimeAsync(0);
controller.abort();
await jest.advanceTimersByTimeAsync(1000);
expect(calls).toBe(1);
expect((await done).error).toBe(failure);
});
it('retains the configured delay without cancellation and releases its listener', async () => {
const controller = new AbortController();
let calls = 0;
const tool = makeTool(async () => { if (++calls === 1) throw new Error('connection dropped'); return 'recovered'; });
const done = tool.execute({}, { signal: controller.signal });
await jest.advanceTimersByTimeAsync(999);
expect(calls).toBe(1);
await jest.advanceTimersByTimeAsync(1);
expect(await done).toBe('recovered');
expect(calls).toBe(2);
expect(jest.getTimerCount()).toBe(0);
expect(getEventListeners(controller.signal, 'abort')).toHaveLength(0);
});
it('retains retries without a signal', async () => {
let calls = 0;
const tool = makeTool(async () => { if (++calls === 1) throw new Error('connection dropped'); return 'recovered'; });
const done = tool.execute({});
await jest.advanceTimersByTimeAsync(1000);
expect(await done).toBe('recovered');
expect(calls).toBe(2);
});
it('preserves the last error when already cancelled at failure', async () => {
const controller = new AbortController();
const failure = new Error('connection dropped');
const tool = makeTool(async () => { controller.abort(); throw failure; });
await expect(tool.execute({}, { signal: controller.signal })).rejects.toBe(failure);
expect(jest.getTimerCount()).toBe(0);
});
it('does not retry a nonretryable failure', async () => {
const failure = new Error('invalid argument');
let calls = 0;
const tool = makeTool(async () => { calls++; throw failure; });
await expect(tool.execute({})).rejects.toBe(failure);
expect(calls).toBe(1);
expect(jest.getTimerCount()).toBe(0);
});
it('does not allocate a backoff on immediate success', async () => {
const controller = new AbortController();
expect(await makeTool(async () => 'ok').execute({}, { signal: controller.signal })).toBe('ok');
expect(jest.getTimerCount()).toBe(0);
expect(getEventListeners(controller.signal, 'abort')).toHaveLength(0);
});
});
+340 -177
View File
@@ -14,27 +14,145 @@
<a href="https://github.com/MervinPraison/PraisonAI"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License" /></a>
</p>
<div align="center">
# PraisonAI
# PraisonAI 🦞
**Private AI that earns trust one task at a time.**
<a href="https://trendshift.io/repositories/9130" target="_blank"><img src="https://trendshift.io/api/badge/repositories/9130" alt="MervinPraison%2FPraisonAI | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
PraisonAI runs AI agents on your own computers, even fully offline, so private data never has to leave the building.
</div>
- **For highly regulated teams (early access):** each task moves to your own AI only after it passes a test. If quality slips, it moves back automatically, and every step is recorded for regulators. [How it works](#how-the-handover-works-early-access)
- **For everyone else:** a free, open-source Python agent framework. With a local model you need no API key and pay no cloud bill, and your files stay yours.
PraisonAI 🦞 — **Hire a 24/7 AI Workforce.** Stop writing boilerplate and start shipping autonomous, self-improving agents that research, plan, and execute tasks across your apps. From one agent to an entire organization, deployed in 5 lines of code.
[2M+ downloads](https://pepy.tech/projects/praisonai) · [9.3K stars](https://github.com/MervinPraison/PraisonAI) · [600+ releases in the last year](https://github.com/MervinPraison/PraisonAI/releases) · [80+ contributors](https://github.com/MervinPraison/PraisonAI/graphs/contributors) · Python
```bash
curl -fsSL https://praison.ai/install.sh | bash
pip install praisonai
```
Only need the Python library? `pip install praisonaiagents` installs just the lightweight core.
## Using PraisonAI for something else?
PraisonAI is still a general-purpose, open-source agent framework, and everything works as before: the same packages, imports, commands and examples. Go straight to what you came for:
- **Agents and multi-agent teams:** [Agents](https://praison.ai/docs/concepts/agents) · [Handoffs](https://praison.ai/docs/concepts/handoffs)
- **Workflows:** [AgentFlow: route, parallel, loop, repeat](https://praison.ai/docs/concepts/agentflow)
- **Memory and knowledge:** [Memory](https://praison.ai/docs/concepts/memory) · [Knowledge](https://praison.ai/docs/concepts/knowledge) · [RAG](https://praison.ai/docs/concepts/rag)
- **Tools and MCP:** [Tools](https://praison.ai/docs/tools/tools) · [MCP](https://praison.ai/docs/mcp/transports)
- **Bots (Telegram, Discord, Slack, WhatsApp):** [Bot CLI](https://praison.ai/docs/cli/bot) · [Bot gateway](https://praison.ai/docs/features/bot-gateway)
- **Terminal CLI:** [`praisonai code`](https://praison.ai/docs/cli/code) · [`praisonai run`](https://praison.ai/docs/cli/run) · [`praisonai chat`](https://praison.ai/docs/cli/chat) · [CLI reference](https://praison.ai/docs/cli/cli-reference)
- **Training and fine-tuning:** [PraisonAI Train](https://praison.ai/docs/train)
- **Browser automation:** [Browser agent](https://praison.ai/docs/features/browser-agent) · [Browser CLI](https://praison.ai/docs/cli/browser)
- **Sandboxes:** [Sandboxed code execution](https://praison.ai/docs/features/sandbox)
- **Deploy:** [Deploy guide](https://praison.ai/docs/deploy)
- **Dashboard and UI:** [Claw dashboard](https://praison.ai/docs/concepts/claw) · [Chat UI](https://praison.ai/docs/ui)
- **YAML, no code:** [YAML agents](#-using-yaml-no-code) · [YAML reference](https://praison.ai/docs/features/yaml-configuration-reference)
- **JavaScript and TypeScript:** [JavaScript SDK](https://praison.ai/docs/js)
- **Desktop app:** [PraisonAI Desktop](https://praison.ai/docs/features/desktop/index) · [Downloads](#desktop-app)
- **Everything else:** the [full feature reference](#full-feature-reference) further down this page
## Quick start
Set a key for any supported provider (OpenAI shown here), or skip the key and use a local model as in the next section.
```bash
export OPENAI_API_KEY="your-api-key"
```
```python
from praisonaiagents import Agent
agent = Agent(instructions="You are a helpful assistant.")
print(agent.start("In two sentences, why do some companies keep their data on their own computers?"))
```
## Run fully offline
Run the same agent on a model on your own computer with [Ollama](https://ollama.com). You need no API key, and your prompts and answers stay on the machine.
Install the package and pull the model while you still have internet; after that, the agent runs with no network at all. If you installed only the core library, use `pip install "praisonaiagents[llm]"`, which adds the local-model support.
```bash
# Start Ollama first (the app, or "ollama serve" in another terminal)
ollama pull llama3.2
export DO_NOT_TRACK=true # keeps telemetry off (PraisonAI sends none unless you turn it on)
```
```python
from praisonaiagents import Agent
agent = Agent(instructions="You are a helpful assistant.", model="ollama/llama3.2")
print(agent.start("Why is the sky blue?"))
```
If Ollama runs on another computer on your network, pass `base_url` (or set `OLLAMA_API_BASE`):
```python
agent = Agent(instructions="...", model="ollama/llama3.2", base_url="http://192.168.1.10:11434")
```
vLLM and LM Studio work the same way: use `hosted_vllm/` (vLLM) or `lm_studio/` with your server's `base_url`. More in the [Ollama guide](https://praison.ai/docs/models/ollama) and [Models docs](https://praison.ai/docs/models).
## How the handover works (early access)
1. **Watch:** records how each task is done today.
2. **Test:** your own AI takes an exam made from real past work it never studied.
3. **Hand over:** a task moves to your own AI only when it passes.
4. **Roll back:** if quality drops, the task moves back automatically.
5. **Record:** every handover leaves a record your regulator can check.
**What is "your own AI"?** An open model, such as Llama, Qwen or Gemma, running on your own computers. Use it as it is, or train it on your own past work with [PraisonAI Train](https://praison.ai/docs/train) on your own GPU servers. Your team decides when to train; nothing trains automatically. Either way, a task moves to it only after it passes that exam.
Highly regulated teams, such as banks, insurers and hospitals, want AI but can't send customer data to outside models. From 6 January 2027, OpenAI also takes no new self-serve fine-tuning jobs ([notice](https://developers.openai.com/api/docs/deprecations)), so teams that customise models need models they own. The handover lets these teams move work to AI they own, one task at a time, with a record that shows why each step was trusted.
It is built on parts of this framework that are live today: tracing, evaluation, fine-tuning, model routing and local models. The handover itself is in early access, and we are taking on our first design partners. To take part, see [praison.ai/enterprise](https://praison.ai/enterprise/).
## Features
- **Agents and multi-agent teams:** [Agents](https://praison.ai/docs/concepts/agents) · [Handoffs](https://praison.ai/docs/concepts/handoffs) · [Workflows](https://praison.ai/docs/concepts/agentflow)
- **Memory:** [Memory](https://praison.ai/docs/concepts/memory) · [Knowledge and RAG](https://praison.ai/docs/concepts/rag) · [Sessions](https://praison.ai/docs/concepts/session-management)
- **Tools and MCP:** [Tools](https://praison.ai/docs/tools/tools) · [MCP](https://praison.ai/docs/mcp/transports)
- **Guardrails and approval:** [Guardrails](https://praison.ai/docs/concepts/guardrails) · [Human approval](https://praison.ai/docs/concepts/approval) · [Policy engine](https://praison.ai/docs/features/policy-engine)
- **Tracing and evaluation:** [Tracing](https://praison.ai/docs/observability/overview) · [Telemetry](https://praison.ai/docs/features/telemetry) · [Langfuse](https://praison.ai/docs/observability/langfuse) · [Evaluation](https://praison.ai/docs/concepts/evaluation)
- **Your choice of model:** [Models](https://praison.ai/docs/models) · [Model router](https://praison.ai/docs/features/model-router)
- **Fine-tuning:** [PraisonAI Train](https://praison.ai/docs/train)
- **Deploy:** [Deploy guide](https://praison.ai/docs/deploy)
## JavaScript and TypeScript
```bash
npm install praisonai
```
The TypeScript and JavaScript SDK has its own guide at [praison.ai/docs/js](https://praison.ai/docs/js). A short example is in [Using JavaScript Code](#-using-javascript-code) below.
## The company
- **Who:** Praison AI Limited in Cambridge, UK, building PraisonAI since September 2023. Co-founders Mervin Praison (CEO), Leebana Smith (COO) and Karthick Subramanian (CBO). [About us](https://praison.ai/about/)
- **What we're building:** the task handover above, as a self-hosted product on top of this framework. It is in early access, and we are taking on our first design partners.
- **Why now:** highly regulated teams want AI but can't send customer data to outside models, and from 6 January 2027 OpenAI takes no new self-serve fine-tuning jobs.
- **How it fits together:** the framework stays free and MIT-licensed, and brings developers in. Regulated teams license the handover to run inside their own walls.
- **Traction:** every number at the top of this page is public and linked to its source.
- **Get in touch:** design partners, investors and press: enterprise@praison.ai or [praison.ai/enterprise](https://praison.ai/enterprise/)
## Docs, community and licence
- **Documentation:** [praison.ai/docs](https://praison.ai/docs)
- **Questions and bugs:** [Discussions](https://github.com/MervinPraison/PraisonAI/discussions) · [Issues](https://github.com/MervinPraison/PraisonAI/issues)
- **Contributing:** [Contributing guide](https://github.com/MervinPraison/PraisonAI/blob/main/CONTRIBUTING.md)
- **Videos and news:** [YouTube](https://youtube.com/@MervinPraison) · [X](https://x.com/MervinPraison) · [LinkedIn](https://linkedin.com/in/mervinpraison)
- **Desktop app:** macOS, Windows and Linux builds are on the [latest release](https://github.com/MervinPraison/PraisonAI/releases/latest)
- **Licence:** [MIT](https://github.com/MervinPraison/PraisonAI/blob/main/LICENSE)
---
## Full feature reference
Everything below is the full reference that was on this page before. Nothing has been removed; the longest parts are folded, so click a heading's summary to open it.
### Desktop app
<div align="center">
<br>
<a href="https://x.com/elonmusk/status/1893870468249141688" target="_blank">
<img src="https://img.shields.io/badge/Highlighted_by_Elon_Musk-000000?style=for-the-badge&logo=x&logoColor=white" alt="Highlighted by Elon Musk" />
</a>
<br>
<br>
<a href="https://github.com/MervinPraison/PraisonAI/releases/latest">
<img src="https://img.shields.io/badge/Download_for_macOS-000000?style=for-the-badge&logo=apple&logoColor=white" alt="Download for macOS" />
</a>
@@ -46,12 +164,17 @@ curl -fsSL https://praison.ai/install.sh | bash
<a href="https://github.com/MervinPraison/PraisonAI/releases/latest">
<img src="https://img.shields.io/badge/Download_for_Linux-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download for Linux" />
</a>
<br>
</div>
<p align="center">
<img src=".github/images/dashboard.png" alt="PraisonAI Dashboard" width="800" />
</p>
### Other ways to install
One-line installer:
```bash
curl -fsSL https://praison.ai/install.sh | bash
```
The full `praisonai` package, with the CLI, bots, dashboard and integrations:
```
██████╗ ██████╗ █████╗ ██╗███████╗ ██████╗ ███╗ ██╗ █████╗ ██╗
@@ -64,10 +187,20 @@ curl -fsSL https://praison.ai/install.sh | bash
pip install praisonai
```
<p align="center">
<img src=".github/images/dashboard.png" alt="PraisonAI Dashboard" width="800" />
</p>
<div align="center">
<a href="https://docs.praison.ai">
<a href="https://trendshift.io/repositories/9130" target="_blank"><img src="https://trendshift.io/api/badge/repositories/9130" alt="MervinPraison%2FPraisonAI | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</div>
<div align="center">
<a href="https://praison.ai/docs">
<p align="center">
<img src="https://img.shields.io/badge/📚_Documentation-Visit_docs.praison.ai-blue?style=for-the-badge&logo=bookstack&logoColor=white" alt="Documentation" />
<img src="https://img.shields.io/badge/📚_Documentation-Visit_praison.ai%2Fdocs-blue?style=for-the-badge&logo=bookstack&logoColor=white" alt="Documentation" />
</p>
</a>
</div>
@@ -114,6 +247,9 @@ Most frameworks hand you one or two layers and leave the rest as homework. Prais
Each layer wraps the one inside it. When an agent misbehaves, the layer tells you where to look.
<details>
<summary><strong>Show the five layers, examples and Managed Agents</strong></summary>
```
┌─────────────────────────────────────────────────────────────────┐
│ ⬡ MANAGED AGENTS — Where does it actually run? │
@@ -174,7 +310,7 @@ agent = Agent(
)
```
> **Isolate** is `handoffs=[specialist]` — a sub-agent inherits the last few messages and the intersection of your tools, not your whole transcript. [📖 Handoffs](https://docs.praison.ai/docs/concepts/handoffs)
> **Isolate** is `handoffs=[specialist]` — a sub-agent inherits the last few messages and the intersection of your tools, not your whole transcript. [📖 Handoffs](https://praison.ai/docs/concepts/handoffs)
### Layer 3 · Harness — *Can it act, and be checked?*
@@ -217,7 +353,7 @@ print(result.completion_reason)
# surfaced here as completion_reason="error")
```
> **Doom-loop detection is on by default.** Repeated identical tool calls and A→B→A→B oscillation get caught — while a poller whose output keeps changing does not. [📖 Doom Loop Detection](https://docs.praison.ai/docs/features/doom-loop-detection)
> **Doom-loop detection is on by default.** Repeated identical tool calls and A→B→A→B oscillation get caught — while a poller whose output keeps changing does not. [📖 Doom Loop Detection](https://praison.ai/docs/features/doom-loop-detection)
### Layer 5 · Graph — *Who runs when, and who checks whom?*
@@ -237,7 +373,7 @@ flow = AgentFlow(steps=[
flow.run("Ticket #123: login fails on Safari")
```
> The same graph is expressible in YAML with no Python at all. [📖 AgentFlow](https://docs.praison.ai/docs/concepts/agentflow)
> The same graph is expressible in YAML with no Python at all. [📖 AgentFlow](https://praison.ai/docs/concepts/agentflow)
### ⬡ Outside the stack: Managed Agents — *Where does it actually run?*
@@ -332,6 +468,8 @@ Sandboxes shut themselves down when idle (`auto_shutdown`, `idle_timeout_s`), an
<sub>Stack framing adapted from [The Five-Layer Agent Stack](https://mer.vin/2026/07/five-layer-agent-stack-match-bug-to-right-layer/) and [Agent Harnesses vs Orbs](https://mer.vin/2026/08/agent-harnesses-vs-orbs-why-remote-sandboxes-beat-local-agent-loops/).</sub>
</details>
---
## 🌌 The PraisonAI Ecosystem
@@ -411,6 +549,11 @@ Powered by 100+ LLMs (OpenAI, Anthropic, Gemini & local models).
</details>
<div align="center">
<a href="https://x.com/elonmusk/status/1893870468249141688" target="_blank">
<img src="https://img.shields.io/badge/Highlighted_by_Elon_Musk-000000?style=for-the-badge&logo=x&logoColor=white" alt="Highlighted by Elon Musk" />
</a>
<br>
<br>
<a href="https://x.com/elonmusk/status/1893870468249141688" target="_blank">
<img src=".github/images/elon_musk_praisonai.png" alt="Highlighted by Elon Musk" width="600" />
</a>
@@ -426,13 +569,13 @@ Powered by 100+ LLMs (OpenAI, Anthropic, Gemini & local models).
|--|---------|-----|
| 🔌 | **MCP Protocol** — stdio, HTTP, WebSocket, SSE | `tools=MCP("npx ...")` |
| 🧠 | **Planning Mode** — plan → execute → reason | `planning=True` |
| 🔍 | **Deep Research** — multi-step autonomous research | [Docs](https://docs.praison.ai/docs/agents/deep-research) |
| 🤖 | **External Agents** — orchestrate Claude Code, Gemini CLI, Codex | [Docs](https://docs.praison.ai/docs/code/external-agents) |
| 🔍 | **Deep Research** — multi-step autonomous research | [Docs](https://praison.ai/docs/agents/deep-research) |
| 🤖 | **External Agents** — orchestrate Claude Code, Gemini CLI, Codex | [Docs](https://praison.ai/docs/code/external-agents) |
| 🔄 | **Agent Handoffs** — seamless conversation passing | `handoffs=[other_agent]` |
| 🛡️ | **Guardrails** — input/output validation | [Docs](https://docs.praison.ai/docs/concepts/guardrails) |
| 🛡️ | **Guardrails** — input/output validation | [Docs](https://praison.ai/docs/concepts/guardrails) |
| | **Web Search + Fetch** — native browsing | `web=True` |
| 🪞 | **Self Reflection** — agent reviews its own output | [Docs](https://docs.praison.ai/docs/concepts/reflection) |
| 🔀 | **Workflow Patterns** — route, parallel, loop, repeat | [Docs](https://docs.praison.ai/docs/concepts/agentflow) |
| 🪞 | **Self Reflection** — agent reviews its own output | [Docs](https://praison.ai/docs/concepts/reflection) |
| 🔀 | **Workflow Patterns** — route, parallel, loop, repeat | [Docs](https://praison.ai/docs/concepts/agentflow) |
| 🧠 | **Memory (zero deps)** — works out of the box | `memory=True` |
<details>
@@ -443,18 +586,18 @@ Powered by 100+ LLMs (OpenAI, Anthropic, Gemini & local models).
| 💡 | **Prompt Caching** — reduce latency + cost | `caching=True` |
| 💾 | **Sessions + Auto-Save** — persistent state across restarts | `auto_save="my-project"` |
| 💭 | **Thinking Budgets** — control reasoning depth | `agent.thinking_budget = 1024` |
| 📚 | **RAG + Quality-Based RAG** — auto quality scoring retrieval | [Docs](https://docs.praison.ai/docs/concepts/rag) |
| 📊 | **Model Router** — auto-routes to cheapest capable model | [Docs](https://docs.praison.ai/docs/features/model-router) |
| 🧊 | **Shadow Git Checkpoints** — auto-rollback on failure | [Docs](https://docs.praison.ai/docs/features/checkpoints) |
| 📡 | **A2A Protocol** — agent-to-agent interop | [Docs](https://docs.praison.ai/docs/features/a2a) |
| 📏 | **Context Compaction** — never hit token limits | [Docs](https://docs.praison.ai/docs/features/context-compaction) |
| 📡 | **Telemetry** — OpenTelemetry traces, spans, metrics | [Docs](https://docs.praison.ai/docs/features/telemetry) |
| 📜 | **Policy Engine** — declarative agent behavior control | [Docs](https://docs.praison.ai/docs/features/policy-engine) |
| 🔄 | **Background Tasks** — fire-and-forget agents | [Docs](https://docs.praison.ai/docs/features/background-tasks) |
| 🔁 | **Doom Loop Detection** — auto-recovery from stuck agents | [Docs](https://docs.praison.ai/docs/features/doom-loop-detection) |
| 🕸️ | **Graph Memory** — Neo4j-style relationship tracking | [Docs](https://docs.praison.ai/docs/features/graph-memory) |
| 🏖️ | **Sandbox Execution** — isolated code execution | [Docs](https://docs.praison.ai/docs/features/sandbox) |
| 🖥️ | **Bot Gateway** — multi-agent routing across channels | [Docs](https://docs.praison.ai/docs/features/bot-gateway) |
| 📚 | **RAG + Quality-Based RAG** — auto quality scoring retrieval | [Docs](https://praison.ai/docs/concepts/rag) |
| 📊 | **Model Router** — auto-routes to cheapest capable model | [Docs](https://praison.ai/docs/features/model-router) |
| 🧊 | **Shadow Git Checkpoints** — auto-rollback on failure | [Docs](https://praison.ai/docs/features/checkpoints) |
| 📡 | **A2A Protocol** — agent-to-agent interop | [Docs](https://praison.ai/docs/features/a2a) |
| 📏 | **Context Compaction** — never hit token limits | [Docs](https://praison.ai/docs/features/context-compaction) |
| 📡 | **Telemetry** — OpenTelemetry traces, spans, metrics | [Docs](https://praison.ai/docs/features/telemetry) |
| 📜 | **Policy Engine** — declarative agent behavior control | [Docs](https://praison.ai/docs/features/policy-engine) |
| 🔄 | **Background Tasks** — fire-and-forget agents | [Docs](https://praison.ai/docs/features/background-tasks) |
| 🔁 | **Doom Loop Detection** — auto-recovery from stuck agents | [Docs](https://praison.ai/docs/features/doom-loop-detection) |
| 🕸️ | **Graph Memory** — Neo4j-style relationship tracking | [Docs](https://praison.ai/docs/features/graph-memory) |
| 🏖️ | **Sandbox Execution** — isolated code execution | [Docs](https://praison.ai/docs/features/sandbox) |
| 🖥️ | **Bot Gateway** — multi-agent routing across channels | [Docs](https://praison.ai/docs/features/bot-gateway) |
</details>
@@ -465,6 +608,9 @@ Powered by 100+ LLMs (OpenAI, Anthropic, Gemini & local models).
## 📘 Using Python Code
<details>
<summary><strong>Show Python examples: single agent, multi-agent, MCP, tools, databases, Claw, Langflow, UI</strong></summary>
> Unsure whether a result is a bug or model randomness? Check the [SDK Behavior Matrix](https://github.com/MervinPraison/PraisonAI/blob/main/src/praisonai-agents/docs/sdk-behavior-matrix.md) for `stream`, memory prefetch, structured output, teams and AG-UI.
### 1. Single Agent
@@ -510,7 +656,7 @@ agent = Agent(
)
```
> 📖 [Full MCP docs](https://docs.praison.ai/docs/mcp/transports) — stdio, HTTP, WebSocket, SSE transports
> 📖 [Full MCP docs](https://praison.ai/docs/mcp/transports) — stdio, HTTP, WebSocket, SSE transports
### 4. Custom Tools
@@ -562,7 +708,7 @@ agent.start("Search for AI news and calculate 15*4")
```
> ⚠️ **Security Note:** Never use `eval()`, `exec()`, or `subprocess` in tool functions that process LLM-generated or user-supplied input. Always validate and sanitize inputs to prevent code injection attacks.
> 📖 [Full tools docs](https://docs.praison.ai/docs/tools/tools) — BaseTool, tool packages, 100+ built-in tools
> 📖 [Full tools docs](https://praison.ai/docs/tools/tools) — BaseTool, tool packages, 100+ built-in tools
### 5. Persistence (Databases)
@@ -579,7 +725,7 @@ agent = Agent(
agent.chat("Hello!") # Auto-persists messages, runs, traces
```
> 📖 [Full persistence docs](https://docs.praison.ai/docs/databases/overview) — PostgreSQL, MySQL, SQLite, MongoDB, Redis, and 20+ more
> 📖 [Full persistence docs](https://praison.ai/docs/databases/overview) — PostgreSQL, MySQL, SQLite, MongoDB, Redis, and 20+ more
### 6. PraisonAI Claw 🦞 (Dashboard UI)
@@ -601,7 +747,7 @@ Copy `.env.example` to `.env` and configure the following variables:
Open **http://localhost:8082** — the dashboard comes with 13 built-in pages: Chat, Agents, Memory, Knowledge, Channels, Guardrails, Cron, and more. Add messaging channels directly from the UI.
> 📖 [Full Claw docs](https://docs.praison.ai/docs/concepts/claw) — platform tokens, CLI options, Docker, and YAML agent mode
> 📖 [Full Claw docs](https://praison.ai/docs/concepts/claw) — platform tokens, CLI options, Docker, and YAML agent mode
### 7. Langflow Integration 🔗 (Visual Flow Builder)
@@ -614,7 +760,7 @@ praisonai flow
Open **http://localhost:7861** — use the **Agent** and **Agent Team** components to create sequential or parallel workflows. Connect Chat Input → Agent Team → Chat Output for instant multi-agent pipelines.
> 📖 [Full Flow docs](https://docs.praison.ai/docs/concepts/agentflow) — visual agent building, component reference, and deployment
> 📖 [Full Flow docs](https://praison.ai/docs/concepts/agentflow) — visual agent building, component reference, and deployment
### 8. PraisonAI UI 🤖 (Clean Chat)
@@ -625,10 +771,15 @@ pip install "praisonai[ui]"
praisonai ui
```
</details>
---
## 📄 Using YAML (No Code)
<details>
<summary><strong>Show YAML examples</strong></summary>
### Example 1: Two Agents Working Together
Create `agents.yaml`:
@@ -700,6 +851,8 @@ praisonai agents.yaml
> - Tools in `tools.py` are automatically discovered
> - The function's docstring helps the AI understand how to use it
</details>
---
## 🎯 CLI Quick Reference
@@ -718,23 +871,26 @@ praisonai agents.yaml
| **Development** | `commit`, `docs`, `checkpoint`, `hooks` |
| **Scheduling** | `schedule start`, `schedule list`, `schedule stop` |
> 📖 [Full CLI reference](https://docs.praison.ai/docs/cli/cli-reference)
> 📖 [Full CLI reference](https://praison.ai/docs/cli/cli-reference)
---
## ✨ Key Features
<details>
<summary><strong>Show all features by category</strong></summary>
<details open>
<summary><strong>🤖 Core Agents</strong></summary>
| Feature | Code | Docs |
|---------|:----:|:----:|
| Single Agent | [Example](examples/python/agents/single-agent.py) | [📖](https://docs.praison.ai/docs/agents/single) |
| Multi Agents | [Example](examples/python/general/mini_agents_example.py) | [📖](https://docs.praison.ai/docs/concepts/agents) |
| Auto Agents | [Example](examples/python/general/auto_agents_example.py) | [📖](https://docs.praison.ai/docs/features/autoagents) |
| Self Reflection AI Agents | [Example](examples/python/concepts/self-reflection-details.py) | [📖](https://docs.praison.ai/docs/concepts/reflection) |
| Reasoning AI Agents | [Example](examples/python/concepts/reasoning-extraction.py) | [📖](https://docs.praison.ai/docs/features/reasoning) |
| Multi Modal AI Agents | [Example](examples/python/general/multimodal.py) | [📖](https://docs.praison.ai/docs/features/multimodal) |
| Single Agent | [Example](examples/python/agents/single-agent.py) | [📖](https://praison.ai/docs/agents/single) |
| Multi Agents | [Example](examples/python/general/mini_agents_example.py) | [📖](https://praison.ai/docs/concepts/agents) |
| Auto Agents | [Example](examples/python/general/auto_agents_example.py) | [📖](https://praison.ai/docs/features/autoagents) |
| Self Reflection AI Agents | [Example](examples/python/concepts/self-reflection-details.py) | [📖](https://praison.ai/docs/concepts/reflection) |
| Reasoning AI Agents | [Example](examples/python/concepts/reasoning-extraction.py) | [📖](https://praison.ai/docs/features/reasoning) |
| Multi Modal AI Agents | [Example](examples/python/general/multimodal.py) | [📖](https://praison.ai/docs/features/multimodal) |
</details>
@@ -743,16 +899,16 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Simple Workflow | [Example](examples/python/workflows/simple_workflow.py) | [📖](https://docs.praison.ai/docs/concepts/agentflow) |
| Workflow with Agents | [Example](examples/python/workflows/workflow_with_agents.py) | [📖](https://docs.praison.ai/docs/concepts/agentflow) |
| Agentic Routing (`route()`) | [Example](examples/python/workflows/workflow_routing.py) | [📖](https://docs.praison.ai/docs/features/routing) |
| Parallel Execution (`parallel()`) | [Example](examples/python/workflows/workflow_parallel.py) | [📖](https://docs.praison.ai/docs/features/parallelisation) |
| Loop over List/CSV (`loop()`) | [Example](examples/python/workflows/workflow_loop_csv.py) | [📖](https://docs.praison.ai/docs/features/repetitive) |
| Evaluator-Optimizer (`repeat()`) | [Example](examples/python/workflows/workflow_repeat.py) | [📖](https://docs.praison.ai/docs/concepts/evaluation) |
| Conditional Steps | [Example](examples/python/workflows/workflow_conditional.py) | [📖](https://docs.praison.ai/docs/concepts/agentflow) |
| Workflow Branching | [Example](examples/python/workflows/workflow_branching.py) | [📖](https://docs.praison.ai/docs/concepts/agentflow) |
| Workflow Early Stop | [Example](examples/python/workflows/workflow_early_stop.py) | [📖](https://docs.praison.ai/docs/concepts/agentflow) |
| Workflow Checkpoints | [Example](examples/python/workflows/workflow_checkpoints.py) | [📖](https://docs.praison.ai/docs/concepts/agentflow) |
| Simple Workflow | [Example](examples/python/workflows/simple_workflow.py) | [📖](https://praison.ai/docs/concepts/agentflow) |
| Workflow with Agents | [Example](examples/python/workflows/workflow_with_agents.py) | [📖](https://praison.ai/docs/concepts/agentflow) |
| Agentic Routing (`route()`) | [Example](examples/python/workflows/workflow_routing.py) | [📖](https://praison.ai/docs/features/routing) |
| Parallel Execution (`parallel()`) | [Example](examples/python/workflows/workflow_parallel.py) | [📖](https://praison.ai/docs/features/parallelisation) |
| Loop over List/CSV (`loop()`) | [Example](examples/python/workflows/workflow_loop_csv.py) | [📖](https://praison.ai/docs/features/repetitive) |
| Evaluator-Optimizer (`repeat()`) | [Example](examples/python/workflows/workflow_repeat.py) | [📖](https://praison.ai/docs/concepts/evaluation) |
| Conditional Steps | [Example](examples/python/workflows/workflow_conditional.py) | [📖](https://praison.ai/docs/concepts/agentflow) |
| Workflow Branching | [Example](examples/python/workflows/workflow_branching.py) | [📖](https://praison.ai/docs/concepts/agentflow) |
| Workflow Early Stop | [Example](examples/python/workflows/workflow_early_stop.py) | [📖](https://praison.ai/docs/concepts/agentflow) |
| Workflow Checkpoints | [Example](examples/python/workflows/workflow_checkpoints.py) | [📖](https://praison.ai/docs/concepts/agentflow) |
</details>
@@ -761,13 +917,13 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Code Interpreter Agents | [Example](examples/python/agents/code-agent.py) | [📖](https://docs.praison.ai/docs/features/codeagent) |
| AI Code Editing Tools | [Example](examples/python/code/code_editing_example.py) | [📖](https://docs.praison.ai/docs/code/editing) |
| External Agents (All) | [Example](examples/python/code/external_agents_example.py) | [📖](https://docs.praison.ai/docs/code/external-agents) |
| Claude Code CLI | [Example](examples/python/code/claude_code_example.py) | [📖](https://docs.praison.ai/docs/code/claude-code) |
| Gemini CLI | [Example](examples/python/code/gemini_cli_example.py) | [📖](https://docs.praison.ai/docs/code/gemini-cli) |
| Codex CLI | [Example](examples/python/code/codex_cli_example.py) | [📖](https://docs.praison.ai/docs/code/codex-cli) |
| Cursor CLI | [Example](examples/python/code/cursor_cli_example.py) | [📖](https://docs.praison.ai/docs/code/cursor-cli) |
| Code Interpreter Agents | [Example](examples/python/agents/code-agent.py) | [📖](https://praison.ai/docs/features/codeagent) |
| AI Code Editing Tools | [Example](examples/python/code/code_editing_example.py) | [📖](https://praison.ai/docs/code/editing) |
| External Agents (All) | [Example](examples/python/code/external_agents_example.py) | [📖](https://praison.ai/docs/code/external-agents) |
| Claude Code CLI | [Example](examples/python/code/claude_code_example.py) | [📖](https://praison.ai/docs/code/claude-code) |
| Gemini CLI | [Example](examples/python/code/gemini_cli_example.py) | [📖](https://praison.ai/docs/code/gemini-cli) |
| Codex CLI | [Example](examples/python/code/codex_cli_example.py) | [📖](https://praison.ai/docs/code/codex-cli) |
| Cursor CLI | [Example](examples/python/code/cursor_cli_example.py) | [📖](https://praison.ai/docs/code/cursor-cli) |
</details>
@@ -776,18 +932,18 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Memory (Short & Long Term) | [Example](examples/python/general/memory_example.py) | [📖](https://docs.praison.ai/docs/concepts/memory) |
| File-Based Memory | [Example](examples/python/general/memory_example.py) | [📖](https://docs.praison.ai/docs/concepts/memory) |
| Claude Memory Tool | [Example](examples/python/memory/claude_memory_example.py) | [📖](https://docs.praison.ai/docs/features/claude-memory-tool) |
| Add Custom Knowledge | [Example](examples/python/concepts/knowledge-agents.py) | [📖](https://docs.praison.ai/docs/concepts/knowledge) |
| RAG Agents | [Example](examples/python/concepts/rag-agents.py) | [📖](https://docs.praison.ai/docs/concepts/rag) |
| Chat with PDF Agents | [Example](examples/python/concepts/chat-with-pdf.py) | [📖](https://docs.praison.ai/docs/features/chat-with-pdf) |
| Data Readers (PDF, DOCX, etc.) | [CLI](https://docs.praison.ai/docs/cli/knowledge) | [📖](https://docs.praison.ai/docs/features/chunking-strategies) |
| Vector Store Selection | [CLI](https://docs.praison.ai/docs/cli/knowledge) | [📖](https://docs.praison.ai/docs/features/knowledge-backends) |
| Retrieval Strategies | [CLI](https://docs.praison.ai/docs/cli/knowledge) | [📖](https://docs.praison.ai/docs/features/retrieval-strategies) |
| Rerankers | [CLI](https://docs.praison.ai/docs/cli/knowledge) | [📖](https://docs.praison.ai/docs/features/smart-retrieval) |
| Index Types (Vector/Keyword/Hybrid) | [CLI](https://docs.praison.ai/docs/cli/knowledge) | [📖](https://docs.praison.ai/docs/features/incremental-indexing) |
| Query Engines (Sub-Question, etc.) | [CLI](https://docs.praison.ai/docs/cli/knowledge) | [📖](https://docs.praison.ai/docs/features/retrieval) |
| Memory (Short & Long Term) | [Example](examples/python/general/memory_example.py) | [📖](https://praison.ai/docs/concepts/memory) |
| File-Based Memory | [Example](examples/python/general/memory_example.py) | [📖](https://praison.ai/docs/concepts/memory) |
| Claude Memory Tool | [Example](examples/python/memory/claude_memory_example.py) | [📖](https://praison.ai/docs/features/claude-memory-tool) |
| Add Custom Knowledge | [Example](examples/python/concepts/knowledge-agents.py) | [📖](https://praison.ai/docs/concepts/knowledge) |
| RAG Agents | [Example](examples/python/concepts/rag-agents.py) | [📖](https://praison.ai/docs/concepts/rag) |
| Chat with PDF Agents | [Example](examples/python/concepts/chat-with-pdf.py) | [📖](https://praison.ai/docs/features/chat-with-pdf) |
| Data Readers (PDF, DOCX, etc.) | [CLI](https://praison.ai/docs/cli/knowledge) | [📖](https://praison.ai/docs/features/chunking-strategies) |
| Vector Store Selection | [CLI](https://praison.ai/docs/cli/knowledge) | [📖](https://praison.ai/docs/features/knowledge-backends) |
| Retrieval Strategies | [CLI](https://praison.ai/docs/cli/knowledge) | [📖](https://praison.ai/docs/features/retrieval-strategies) |
| Rerankers | [CLI](https://praison.ai/docs/cli/knowledge) | [📖](https://praison.ai/docs/features/smart-retrieval) |
| Index Types (Vector/Keyword/Hybrid) | [CLI](https://praison.ai/docs/cli/knowledge) | [📖](https://praison.ai/docs/features/incremental-indexing) |
| Query Engines (Sub-Question, etc.) | [CLI](https://praison.ai/docs/cli/knowledge) | [📖](https://praison.ai/docs/features/retrieval) |
</details>
@@ -796,12 +952,12 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Deep Research Agents | [Example](examples/python/agents/research-agent.py) | [📖](https://docs.praison.ai/docs/agents/deep-research) |
| Query Rewriter Agent | [Example](examples/python/agents/query-rewriter-agent.py) | [📖](https://docs.praison.ai/docs/agents/query-rewriter) |
| Native Web Search | [Example](examples/python/agents/websearch-agent.py) | [📖](https://docs.praison.ai/docs/agents/websearch) |
| Built-in Search Tools | [Example](examples/python/agents/websearch-agent.py) | [📖](https://docs.praison.ai/docs/tools/tavily) |
| Unified Web Search | [Example](examples/python/web_search_example.py) | [📖](https://docs.praison.ai/docs/tools/web-search) |
| Web Fetch (Anthropic) | [Example](examples/python/agents/web-fetch-agent.py) | [📖](https://docs.praison.ai/docs/features/model-capabilities) |
| Deep Research Agents | [Example](examples/python/agents/research-agent.py) | [📖](https://praison.ai/docs/agents/deep-research) |
| Query Rewriter Agent | [Example](examples/python/agents/query-rewriter-agent.py) | [📖](https://praison.ai/docs/agents/query-rewriter) |
| Native Web Search | [Example](examples/python/agents/websearch-agent.py) | [📖](https://praison.ai/docs/agents/websearch) |
| Built-in Search Tools | [Example](examples/python/agents/websearch-agent.py) | [📖](https://praison.ai/docs/tools/tavily) |
| Unified Web Search | [Example](examples/python/web_search_example.py) | [📖](https://praison.ai/docs/tools/web-search) |
| Web Fetch (Anthropic) | [Example](examples/python/agents/web-fetch-agent.py) | [📖](https://praison.ai/docs/features/model-capabilities) |
</details>
@@ -810,12 +966,12 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Planning Mode | [Example](examples/python/agents/planning-agent.py) | [📖](https://docs.praison.ai/docs/concepts/planning) |
| Planning Tools | [Example](examples/python/agents/planning-agent.py) | [📖](https://docs.praison.ai/docs/concepts/planning) |
| Planning Reasoning | [Example](examples/python/agents/planning-agent.py) | [📖](https://docs.praison.ai/docs/concepts/planning) |
| Prompt Chaining | [Example](examples/python/general/prompt_chaining.py) | [📖](https://docs.praison.ai/docs/features/promptchaining) |
| Evaluator Optimiser | [Example](examples/python/general/evaluator-optimiser.py) | [📖](https://docs.praison.ai/docs/concepts/evaluation) |
| Orchestrator Workers | [Example](examples/python/general/orchestrator-workers.py) | [📖](https://docs.praison.ai/docs/concepts/orchestration) |
| Planning Mode | [Example](examples/python/agents/planning-agent.py) | [📖](https://praison.ai/docs/concepts/planning) |
| Planning Tools | [Example](examples/python/agents/planning-agent.py) | [📖](https://praison.ai/docs/concepts/planning) |
| Planning Reasoning | [Example](examples/python/agents/planning-agent.py) | [📖](https://praison.ai/docs/concepts/planning) |
| Prompt Chaining | [Example](examples/python/general/prompt_chaining.py) | [📖](https://praison.ai/docs/features/promptchaining) |
| Evaluator Optimiser | [Example](examples/python/general/evaluator-optimiser.py) | [📖](https://praison.ai/docs/concepts/evaluation) |
| Orchestrator Workers | [Example](examples/python/general/orchestrator-workers.py) | [📖](https://praison.ai/docs/concepts/orchestration) |
</details>
@@ -824,15 +980,15 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Data Analyst Agent | [Example](examples/python/agents/data-analyst-agent.py) | [📖](https://docs.praison.ai/docs/agents/data-analyst) |
| Finance Agent | [Example](examples/python/agents/finance-agent.py) | [📖](https://docs.praison.ai/docs/agents/finance) |
| Shopping Agent | [Example](examples/python/agents/shopping-agent.py) | [📖](https://docs.praison.ai/docs/agents/shopping) |
| Recommendation Agent | [Example](examples/python/agents/recommendation-agent.py) | [📖](https://docs.praison.ai/docs/agents/recommendation) |
| Wikipedia Agent | [Example](examples/python/agents/wikipedia-agent.py) | [📖](https://docs.praison.ai/docs/agents/wikipedia) |
| Programming Agent | [Example](examples/python/agents/programming-agent.py) | [📖](https://docs.praison.ai/docs/agents/programming) |
| Math Agents | [Example](examples/python/agents/math-agent.py) | [📖](https://docs.praison.ai/docs/features/mathagent) |
| Markdown Agent | [Example](examples/python/agents/markdown-agent.py) | [📖](https://docs.praison.ai/docs/agents/markdown) |
| Prompt Expander Agent | [Example](examples/python/agents/prompt-expander-agent.py) | [📖](https://docs.praison.ai/docs/agents/prompt-expander) |
| Data Analyst Agent | [Example](examples/python/agents/data-analyst-agent.py) | [📖](https://praison.ai/docs/agents/data-analyst) |
| Finance Agent | [Example](examples/python/agents/finance-agent.py) | [📖](https://praison.ai/docs/agents/finance) |
| Shopping Agent | [Example](examples/python/agents/shopping-agent.py) | [📖](https://praison.ai/docs/agents/shopping) |
| Recommendation Agent | [Example](examples/python/agents/recommendation-agent.py) | [📖](https://praison.ai/docs/agents/recommendation) |
| Wikipedia Agent | [Example](examples/python/agents/wikipedia-agent.py) | [📖](https://praison.ai/docs/agents/wikipedia) |
| Programming Agent | [Example](examples/python/agents/programming-agent.py) | [📖](https://praison.ai/docs/agents/programming) |
| Math Agents | [Example](examples/python/agents/math-agent.py) | [📖](https://praison.ai/docs/features/mathagent) |
| Markdown Agent | [Example](examples/python/agents/markdown-agent.py) | [📖](https://praison.ai/docs/agents/markdown) |
| Prompt Expander Agent | [Example](examples/python/agents/prompt-expander-agent.py) | [📖](https://praison.ai/docs/agents/prompt-expander) |
</details>
@@ -841,10 +997,10 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Image Generation Agent | [Example](examples/python/image/image-agent.py) | [📖](https://docs.praison.ai/docs/features/image-generation) |
| Image to Text Agent | [Example](examples/python/agents/image-to-text-agent.py) | [📖](https://docs.praison.ai/docs/agents/image-to-text) |
| Video Agent | [Example](examples/python/agents/video-agent.py) | [📖](https://docs.praison.ai/docs/agents/video) |
| Camera Integration | [Example](examples/python/camera/) | [📖](https://docs.praison.ai/docs/features/camera-integration) |
| Image Generation Agent | [Example](examples/python/image/image-agent.py) | [📖](https://praison.ai/docs/features/image-generation) |
| Image to Text Agent | [Example](examples/python/agents/image-to-text-agent.py) | [📖](https://praison.ai/docs/agents/image-to-text) |
| Video Agent | [Example](examples/python/agents/video-agent.py) | [📖](https://praison.ai/docs/agents/video) |
| Camera Integration | [Example](examples/python/camera/) | [📖](https://praison.ai/docs/features/camera-integration) |
</details>
@@ -853,12 +1009,12 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| MCP Transports | [Example](examples/python/mcp/mcp-transports-overview.py) | [📖](https://docs.praison.ai/docs/mcp/transports) |
| WebSocket MCP | [Example](examples/python/mcp/websocket-mcp.py) | [📖](https://docs.praison.ai/docs/mcp/sse-transport) |
| MCP Security | [Example](examples/python/mcp/mcp-security.py) | [📖](https://docs.praison.ai/docs/mcp/transports) |
| MCP Resumability | [Example](examples/python/mcp/mcp-resumability.py) | [📖](https://docs.praison.ai/docs/mcp/sse-transport) |
| MCP Config Management | [Docs](https://docs.praison.ai/docs/cli/mcp) | [📖](https://docs.praison.ai/docs/cli/mcp) |
| LangChain Integrated Agents | [Example](examples/python/general/langchain_example.py) | [📖](https://docs.praison.ai/docs/features/langchain) |
| MCP Transports | [Example](examples/python/mcp/mcp-transports-overview.py) | [📖](https://praison.ai/docs/mcp/transports) |
| WebSocket MCP | [Example](examples/python/mcp/websocket-mcp.py) | [📖](https://praison.ai/docs/mcp/sse-transport) |
| MCP Security | [Example](examples/python/mcp/mcp-security.py) | [📖](https://praison.ai/docs/mcp/transports) |
| MCP Resumability | [Example](examples/python/mcp/mcp-resumability.py) | [📖](https://praison.ai/docs/mcp/sse-transport) |
| MCP Config Management | [Docs](https://praison.ai/docs/cli/mcp) | [📖](https://praison.ai/docs/cli/mcp) |
| LangChain Integrated Agents | [Example](examples/python/general/langchain_example.py) | [📖](https://praison.ai/docs/features/langchain) |
</details>
@@ -867,9 +1023,9 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Guardrails | [Example](examples/python/guardrails/comprehensive-guardrails-example.py) | [📖](https://docs.praison.ai/docs/concepts/guardrails) |
| Human Approval | [Example](examples/python/general/human_approval_example.py) | [📖](https://docs.praison.ai/docs/concepts/approval) |
| Rules & Instructions | [Docs](https://docs.praison.ai/docs/features/rules) | [📖](https://docs.praison.ai/docs/features/rules) |
| Guardrails | [Example](examples/python/guardrails/comprehensive-guardrails-example.py) | [📖](https://praison.ai/docs/concepts/guardrails) |
| Human Approval | [Example](examples/python/general/human_approval_example.py) | [📖](https://praison.ai/docs/concepts/approval) |
| Rules & Instructions | [Docs](https://praison.ai/docs/features/rules) | [📖](https://praison.ai/docs/features/rules) |
</details>
@@ -878,16 +1034,16 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Async & Parallel Processing | [Example](examples/python/general/async_example.py) | [📖](https://docs.praison.ai/docs/features/async) |
| Parallelisation | [Example](examples/python/general/parallelisation.py) | [📖](https://docs.praison.ai/docs/features/parallelisation) |
| Repetitive Agents | [Example](examples/python/concepts/repetitive-agents.py) | [📖](https://docs.praison.ai/docs/features/repetitive) |
| Agent Handoffs | [Example](examples/python/handoff/handoff_basic.py) | [📖](https://docs.praison.ai/docs/concepts/handoffs) |
| Stateful Agents | [Example](examples/python/stateful/workflow-state-example.py) | [📖](https://docs.praison.ai/docs/features/stateful-agents) |
| Autonomous Workflow | [Example](examples/python/general/autonomous-agent.py) | [📖](https://docs.praison.ai/docs/concepts/autonomy) |
| Structured Output Agents | [Example](examples/python/general/structured_agents_example.py) | [📖](https://docs.praison.ai/docs/features/structured) |
| Model Router | [Example](examples/python/agents/router-agent-cost-optimization.py) | [📖](https://docs.praison.ai/docs/features/model-router) |
| Prompt Caching | [Example](examples/python/agents/prompt-caching-agent.py) | [📖](https://docs.praison.ai/docs/features/model-capabilities) |
| Fast Context | [Example](examples/context/00_agent_fast_context_basic.py) | [📖](https://docs.praison.ai/docs/features/fast-context) |
| Async & Parallel Processing | [Example](examples/python/general/async_example.py) | [📖](https://praison.ai/docs/features/async) |
| Parallelisation | [Example](examples/python/general/parallelisation.py) | [📖](https://praison.ai/docs/features/parallelisation) |
| Repetitive Agents | [Example](examples/python/concepts/repetitive-agents.py) | [📖](https://praison.ai/docs/features/repetitive) |
| Agent Handoffs | [Example](examples/python/handoff/handoff_basic.py) | [📖](https://praison.ai/docs/concepts/handoffs) |
| Stateful Agents | [Example](examples/python/stateful/workflow-state-example.py) | [📖](https://praison.ai/docs/features/stateful-agents) |
| Autonomous Workflow | [Example](examples/python/general/autonomous-agent.py) | [📖](https://praison.ai/docs/concepts/autonomy) |
| Structured Output Agents | [Example](examples/python/general/structured_agents_example.py) | [📖](https://praison.ai/docs/features/structured) |
| Model Router | [Example](examples/python/agents/router-agent-cost-optimization.py) | [📖](https://praison.ai/docs/features/model-router) |
| Prompt Caching | [Example](examples/python/agents/prompt-caching-agent.py) | [📖](https://praison.ai/docs/features/model-capabilities) |
| Fast Context | [Example](examples/context/00_agent_fast_context_basic.py) | [📖](https://praison.ai/docs/features/fast-context) |
</details>
@@ -896,21 +1052,21 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| 100+ Custom Tools | [Example](examples/python/general/tools_example.py) | [📖](https://docs.praison.ai/docs/tools/tools) |
| YAML Configuration | [Example](examples/cookbooks/yaml/secondary_market_research_agents.yaml) | [📖](https://docs.praison.ai/docs/developers/agents-playbook) |
| 100+ LLM Support | [Example](examples/python/providers/openai/openai_gpt4_example.py) | [📖](https://docs.praison.ai/docs/models) |
| Callback Agents | [Example](examples/python/general/advanced-callback-systems.py) | [📖](https://docs.praison.ai/docs/concepts/hooks) |
| Hooks | [Example](examples/python/hooks/hooks_example.py) | [📖](https://docs.praison.ai/docs/concepts/hooks) |
| Middleware System | [Example](examples/middleware/basic_middleware.py) | [📖](https://docs.praison.ai/docs/features/middleware) |
| Configurable Model | [Example](examples/middleware/configurable_model.py) | [📖](https://docs.praison.ai/docs/features/configurable-model) |
| Rate Limiter | [Example](examples/middleware/rate_limiter.py) | [📖](https://docs.praison.ai/docs/features/rate-limiter) |
| Injected Tool State | [Example](examples/middleware/injected_state.py) | [📖](https://docs.praison.ai/docs/features/injected-state) |
| Shadow Git Checkpoints | [Example](examples/checkpoints/basic_checkpoint.py) | [📖](https://docs.praison.ai/docs/features/checkpoints) |
| Background Tasks | [Example](examples/background/basic_background.py) | [📖](https://docs.praison.ai/docs/features/background-tasks) |
| Policy Engine | [Example](examples/policy/basic_policy.py) | [📖](https://docs.praison.ai/docs/features/policy-engine) |
| Thinking Budgets | [Example](examples/thinking/basic_thinking.py) | [📖](https://docs.praison.ai/docs/features/thinking-budgets) |
| Output Styles | [Example](examples/output/basic_output.py) | [📖](https://docs.praison.ai/docs/features/output-styles) |
| Context Compaction | [Example](examples/compaction/basic_compaction.py) | [📖](https://docs.praison.ai/docs/features/context-compaction) |
| 100+ Custom Tools | [Example](examples/python/general/tools_example.py) | [📖](https://praison.ai/docs/tools/tools) |
| YAML Configuration | [Example](examples/cookbooks/yaml/secondary_market_research_agents.yaml) | [📖](https://praison.ai/docs/developers/agents-playbook) |
| 100+ LLM Support | [Example](examples/python/providers/openai/openai_gpt4_example.py) | [📖](https://praison.ai/docs/models) |
| Callback Agents | [Example](examples/python/general/advanced-callback-systems.py) | [📖](https://praison.ai/docs/concepts/hooks) |
| Hooks | [Example](examples/python/hooks/hooks_example.py) | [📖](https://praison.ai/docs/concepts/hooks) |
| Middleware System | [Example](examples/middleware/basic_middleware.py) | [📖](https://praison.ai/docs/features/middleware) |
| Configurable Model | [Example](examples/middleware/configurable_model.py) | [📖](https://praison.ai/docs/features/configurable-model) |
| Rate Limiter | [Example](examples/middleware/rate_limiter.py) | [📖](https://praison.ai/docs/features/rate-limiter) |
| Injected Tool State | [Example](examples/middleware/injected_state.py) | [📖](https://praison.ai/docs/features/injected-state) |
| Shadow Git Checkpoints | [Example](examples/checkpoints/basic_checkpoint.py) | [📖](https://praison.ai/docs/features/checkpoints) |
| Background Tasks | [Example](examples/background/basic_background.py) | [📖](https://praison.ai/docs/features/background-tasks) |
| Policy Engine | [Example](examples/policy/basic_policy.py) | [📖](https://praison.ai/docs/features/policy-engine) |
| Thinking Budgets | [Example](examples/thinking/basic_thinking.py) | [📖](https://praison.ai/docs/features/thinking-budgets) |
| Output Styles | [Example](examples/output/basic_output.py) | [📖](https://praison.ai/docs/features/output-styles) |
| Context Compaction | [Example](examples/compaction/basic_compaction.py) | [📖](https://praison.ai/docs/features/context-compaction) |
</details>
@@ -919,14 +1075,14 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Sessions Management | [Example](examples/python/sessions/comprehensive-session-management.py) | [📖](https://docs.praison.ai/docs/concepts/session-management) |
| Auto-Save Sessions | [Docs](https://docs.praison.ai/docs/cli/session) | [📖](https://docs.praison.ai/docs/cli/session) |
| History in Context | [Docs](https://docs.praison.ai/docs/cli/session) | [📖](https://docs.praison.ai/docs/cli/session) |
| Telemetry | [Example](examples/python/telemetry/production-telemetry-example.py) | [📖](https://docs.praison.ai/docs/features/telemetry) |
| Langfuse Tracing | [Docs](https://docs.praison.ai/docs/observability/langfuse) | [📖](https://docs.praison.ai/docs/observability/langfuse) |
| Project Docs (.praison/docs/) | [Docs](https://docs.praison.ai/docs/cli/docs) | [📖](https://docs.praison.ai/docs/cli/docs) |
| AI Commit Messages | [Docs](https://docs.praison.ai/docs/cli/commit) | [📖](https://docs.praison.ai/docs/cli/commit) |
| @Mentions in Prompts | [Docs](https://docs.praison.ai/docs/cli/mentions) | [📖](https://docs.praison.ai/docs/cli/mentions) |
| Sessions Management | [Example](examples/python/sessions/comprehensive-session-management.py) | [📖](https://praison.ai/docs/concepts/session-management) |
| Auto-Save Sessions | [Docs](https://praison.ai/docs/cli/session) | [📖](https://praison.ai/docs/cli/session) |
| History in Context | [Docs](https://praison.ai/docs/cli/session) | [📖](https://praison.ai/docs/cli/session) |
| Telemetry | [Example](examples/python/telemetry/production-telemetry-example.py) | [📖](https://praison.ai/docs/features/telemetry) |
| Langfuse Tracing | [Docs](https://praison.ai/docs/observability/langfuse) | [📖](https://praison.ai/docs/observability/langfuse) |
| Project Docs (.praison/docs/) | [Docs](https://praison.ai/docs/cli/docs) | [📖](https://praison.ai/docs/cli/docs) |
| AI Commit Messages | [Docs](https://praison.ai/docs/cli/commit) | [📖](https://praison.ai/docs/cli/commit) |
| @Mentions in Prompts | [Docs](https://praison.ai/docs/cli/mentions) | [📖](https://praison.ai/docs/cli/mentions) |
</details>
@@ -935,20 +1091,20 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Slash Commands | [Example](examples/python/cli/slash_commands_example.py) | [📖](https://docs.praison.ai/docs/cli/slash-commands) |
| Autonomy Modes | [Example](examples/python/cli/autonomy_modes_example.py) | [📖](https://docs.praison.ai/docs/cli/autonomy-modes) |
| Cost Tracking | [Example](examples/python/cli/cost_tracking_example.py) | [📖](https://docs.praison.ai/docs/cli/cost-tracking) |
| Repository Map | [Example](examples/python/cli/repo_map_example.py) | [📖](https://docs.praison.ai/docs/cli/repo-map) |
| Interactive TUI | [Example](examples/python/cli/interactive_tui_example.py) | [📖](https://docs.praison.ai/docs/cli/interactive-tui) |
| Git Integration | [Example](examples/python/cli/git_integration_example.py) | [📖](https://docs.praison.ai/docs/cli/git-integration) |
| Sandbox Execution | [Example](examples/python/cli/sandbox_execution_example.py) | [📖](https://docs.praison.ai/docs/cli/sandbox-execution) |
| CLI Compare | [Example](examples/compare/cli_compare_basic.py) | [📖](https://docs.praison.ai/docs/cli/compare) |
| Profile/Benchmark | [Docs](https://docs.praison.ai/docs/cli/profile) | [📖](https://docs.praison.ai/docs/cli/profile) |
| Auto Mode | [Docs](https://docs.praison.ai/docs/cli/auto) | [📖](https://docs.praison.ai/docs/cli/auto) |
| Init | [Docs](https://docs.praison.ai/docs/cli/init) | [📖](https://docs.praison.ai/docs/cli/init) |
| File Input | [Docs](https://docs.praison.ai/docs/cli/file-input) | [📖](https://docs.praison.ai/docs/cli/file-input) |
| Final Agent | [Docs](https://docs.praison.ai/docs/cli/final-agent) | [📖](https://docs.praison.ai/docs/cli/final-agent) |
| Max Tokens | [Docs](https://docs.praison.ai/docs/cli/max-tokens) | [📖](https://docs.praison.ai/docs/cli/max-tokens) |
| Slash Commands | [Example](examples/python/cli/slash_commands_example.py) | [📖](https://praison.ai/docs/cli/slash-commands) |
| Autonomy Modes | [Example](examples/python/cli/autonomy_modes_example.py) | [📖](https://praison.ai/docs/cli/autonomy-modes) |
| Cost Tracking | [Example](examples/python/cli/cost_tracking_example.py) | [📖](https://praison.ai/docs/cli/cost-tracking) |
| Repository Map | [Example](examples/python/cli/repo_map_example.py) | [📖](https://praison.ai/docs/cli/repo-map) |
| Interactive TUI | [Example](examples/python/cli/interactive_tui_example.py) | [📖](https://praison.ai/docs/cli/interactive-tui) |
| Git Integration | [Example](examples/python/cli/git_integration_example.py) | [📖](https://praison.ai/docs/cli/git-integration) |
| Sandbox Execution | [Example](examples/python/cli/sandbox_execution_example.py) | [📖](https://praison.ai/docs/cli/sandbox-execution) |
| CLI Compare | [Example](examples/compare/cli_compare_basic.py) | [📖](https://praison.ai/docs/cli/compare) |
| Profile/Benchmark | [Docs](https://praison.ai/docs/cli/profile) | [📖](https://praison.ai/docs/cli/profile) |
| Auto Mode | [Docs](https://praison.ai/docs/cli/auto) | [📖](https://praison.ai/docs/cli/auto) |
| Init | [Docs](https://praison.ai/docs/cli/init) | [📖](https://praison.ai/docs/cli/init) |
| File Input | [Docs](https://praison.ai/docs/cli/file-input) | [📖](https://praison.ai/docs/cli/file-input) |
| Final Agent | [Docs](https://praison.ai/docs/cli/final-agent) | [📖](https://praison.ai/docs/cli/final-agent) |
| Max Tokens | [Docs](https://praison.ai/docs/cli/max-tokens) | [📖](https://praison.ai/docs/cli/max-tokens) |
</details>
@@ -957,10 +1113,10 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Accuracy Evaluation | [Example](examples/eval/accuracy_example.py) | [📖](https://docs.praison.ai/docs/cli/eval) |
| Performance Evaluation | [Example](examples/eval/performance_example.py) | [📖](https://docs.praison.ai/docs/cli/eval) |
| Reliability Evaluation | [Example](examples/eval/reliability_example.py) | [📖](https://docs.praison.ai/docs/cli/eval) |
| Criteria Evaluation | [Example](examples/eval/criteria_example.py) | [📖](https://docs.praison.ai/docs/cli/eval) |
| Accuracy Evaluation | [Example](examples/eval/accuracy_example.py) | [📖](https://praison.ai/docs/cli/eval) |
| Performance Evaluation | [Example](examples/eval/performance_example.py) | [📖](https://praison.ai/docs/cli/eval) |
| Reliability Evaluation | [Example](examples/eval/reliability_example.py) | [📖](https://praison.ai/docs/cli/eval) |
| Criteria Evaluation | [Example](examples/eval/criteria_example.py) | [📖](https://praison.ai/docs/cli/eval) |
</details>
@@ -969,8 +1125,8 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Skills Management | [Example](examples/skills/basic_skill_usage.py) | [📖](https://docs.praison.ai/docs/concepts/skills) |
| Custom Skills | [Example](examples/skills/custom_skill_example.py) | [📖](https://docs.praison.ai/docs/concepts/skills) |
| Skills Management | [Example](examples/skills/basic_skill_usage.py) | [📖](https://praison.ai/docs/concepts/skills) |
| Custom Skills | [Example](examples/skills/custom_skill_example.py) | [📖](https://praison.ai/docs/concepts/skills) |
</details>
@@ -979,7 +1135,9 @@ praisonai agents.yaml
| Feature | Code | Docs |
|---------|:----:|:----:|
| Agent Scheduler | [Example](examples/python/scheduled_agents/news_checker_live.py) | [📖](https://docs.praison.ai/docs/cli/scheduler) |
| Agent Scheduler | [Example](examples/python/scheduled_agents/news_checker_live.py) | [📖](https://praison.ai/docs/cli/scheduler) |
</details>
</details>
@@ -1008,7 +1166,7 @@ PraisonAI keeps framework overhead microsecond-class: creating an `Agent` (one t
|--------------------|-----------|----------|
| Avg Agent instantiation (1 tool, 1000 iterations) | **46 μs** | 2026-10-15 · praisonaiagents 1.7.12 |
Methodology, comparisons and how to reproduce: [benchmarks/BENCHMARK_RESULTS.md](https://github.com/MervinPraison/PraisonAI/blob/main/src/praisonai-agents/benchmarks/BENCHMARK_RESULTS.md) · [docs](https://docs.praison.ai/docs/developers/benchmarks)
Methodology, comparisons and how to reproduce: [benchmarks/BENCHMARK_RESULTS.md](https://github.com/MervinPraison/PraisonAI/blob/main/src/praisonai-agents/benchmarks/BENCHMARK_RESULTS.md) · [docs](https://praison.ai/docs/developers/benchmarks)
---
@@ -1018,7 +1176,7 @@ Methodology, comparisons and how to reproduce: [benchmarks/BENCHMARK_RESULTS.md]
## ⭐ Star History
[![Star History Chart](https://api.star-history.com/svg?repos=MervinPraison/PraisonAI&type=Date)](https://docs.praison.ai)
[![Star History Chart](https://api.star-history.com/svg?repos=MervinPraison/PraisonAI&type=Date)](https://praison.ai/docs)
---
@@ -1089,6 +1247,9 @@ We welcome contributions! Fork the repo, create a branch, and submit a PR → [C
## ❓ FAQ & Troubleshooting
<details>
<summary><strong>Show FAQ and troubleshooting</strong></summary>
<details>
<summary><strong>ModuleNotFoundError: No module named 'praisonaiagents'</strong></summary>
@@ -1107,7 +1268,7 @@ Ensure your API key is set:
export OPENAI_API_KEY=your_key_here
```
For other providers, see [Models docs](https://docs.praison.ai/docs/models).
For other providers, see [Models docs](https://praison.ai/docs/models).
</details>
@@ -1148,7 +1309,7 @@ agent = Agent(instructions="...", llm="ollama/llama3.2", base_url="http://192.16
The same shape works for other local runtimes — `lm_studio/`, `vllm/` and
`hosted_vllm/` with their server's `base_url`.
See [Models docs](https://docs.praison.ai/docs/models) for more details.
See [Models docs](https://praison.ai/docs/models) for more details.
</details>
@@ -1168,7 +1329,7 @@ agent = Agent(
)
```
See [Persistence docs](https://docs.praison.ai/docs/databases/overview) for supported databases.
See [Persistence docs](https://praison.ai/docs/databases/overview) for supported databases.
</details>
@@ -1185,7 +1346,7 @@ agent = Agent(
)
```
See [Memory docs](https://docs.praison.ai/docs/concepts/memory) for more options.
See [Memory docs](https://praison.ai/docs/concepts/memory) for more options.
</details>
@@ -1201,7 +1362,7 @@ agents = Agents(agents=[agent1, agent2])
agents.start()
```
See [Agents docs](https://docs.praison.ai/docs/concepts/agents) for more examples.
See [Agents docs](https://praison.ai/docs/concepts/agents) for more examples.
</details>
@@ -1216,13 +1377,15 @@ agent = Agent(
)
```
See [MCP docs](https://docs.praison.ai/docs/mcp/transports) for all transport options.
See [MCP docs](https://praison.ai/docs/mcp/transports) for all transport options.
</details>
</details>
### Getting Help
- 📚 [Full Documentation](https://docs.praison.ai)
- 📚 [Full Documentation](https://praison.ai/docs)
- 🐛 [Report Issues](https://github.com/MervinPraison/PraisonAI/issues)
- 💬 [Discussions](https://github.com/MervinPraison/PraisonAI/discussions)
@@ -1231,7 +1394,7 @@ See [MCP docs](https://docs.praison.ai/docs/mcp/transports) for all transport op
<div align="center">
<p><strong>Made with ❤️ by the PraisonAI Team</strong></p>
<p>
<a href="https://docs.praison.ai">📚 Documentation</a> •
<a href="https://praison.ai/docs">📚 Documentation</a> •
<a href="https://github.com/MervinPraison/PraisonAI">GitHub</a> •
<a href="https://youtube.com/@MervinPraison">▶️ YouTube</a> •
<a href="https://x.com/MervinPraison">𝕏 X</a> •
@@ -23,6 +23,26 @@ logger = logging.getLogger(__name__)
# Structural keys handled by the spec builder itself — never "unsupported".
_STRUCTURAL_FIELDS = {"tools", "tasks", "role", "goal", "backstory"}
# Explicit safety gates: an adapter that cannot honour them must not run.
_SAFETY_FIELDS = frozenset({"approval", "guardrails"})
def _is_active_gate(field: str, value: Any) -> bool:
"""True when dropping ``field`` would remove a real gate.
Disabled values (``approval: false``, ``guardrails: null``) and
auto-approve (``--trust`` -> backend ``auto``) gate nothing, so dropping
them is harmless. Unparseable approval config fails closed.
"""
if field != "approval":
return bool(value)
try:
from .._approval_spec import ApprovalSpec
spec = ApprovalSpec.from_yaml(value)
except Exception:
return True
return spec.enabled and spec.backend not in ("auto", "none")
def warn_unsupported_fields(adapter: Any, spec_extras: Dict[str, Any]) -> None:
"""Warn once per agent when a backend ignores declared YAML fields.
@@ -33,15 +53,23 @@ def warn_unsupported_fields(adapter: Any, spec_extras: Dict[str, Any]) -> None:
means "supports everything" — ``framework: praisonai`` honours the full
feature set, so no warning is emitted there.
Non-breaking: pure visibility. Anything a YAML declares that the target
adapter does not read is silently dropped otherwise, so warn rather than
let a safety-relevant field (e.g. ``approval``) be ignored without any
diagnostic.
Non-safety fields only warn. Explicit safety gates (``approval``,
``guardrails``) fail closed: running with the gate silently dropped would
execute tools the YAML author asked to be gated (safe by default).
"""
supported = getattr(adapter, "SUPPORTED_YAML_FIELDS", frozenset())
if not supported:
return
unhandled = set(spec_extras.keys()) - supported - _STRUCTURAL_FIELDS
dropped_safety = sorted(
f for f in unhandled & _SAFETY_FIELDS if _is_active_gate(f, spec_extras[f])
)
if dropped_safety:
raise ValueError(
f"framework={getattr(adapter, 'name', '?')!r} cannot enforce safety "
f"field(s) {dropped_safety} for agent {spec_extras.get('role')!r}. "
f"Use framework: praisonai, or remove the field(s) to run ungated."
)
if unhandled:
logger.warning(
"framework=%r ignores YAML field(s) %s for agent %r; "
+15 -3
View File
@@ -144,9 +144,21 @@ class PraisonAIModel:
self.api_key_var = "OPENROUTER_API_KEY"
self.base_url = base_url or "https://openrouter.ai/api/v1"
self.model_name = self.model.replace("openrouter/", "")
else:
self.api_key_var = api_key_var or "OPENAI_API_KEY"
self.base_url = base_url or os.environ.get("OPENAI_API_BASE") or os.environ.get("OPENAI_BASE_URL") or "https://api.openai.com/v1"
else:
custom_base = base_url or os.environ.get("OPENAI_API_BASE") or os.environ.get("OPENAI_BASE_URL")
# OpenAI model ids never contain "/". An unknown "provider/model"
# with no custom endpoint would be sent to api.openai.com and 404;
# fail loudly instead. Custom endpoints (gateways, vLLM, proxies)
# legitimately use slashed ids, so they pass through unchanged.
if "/" in self.model and not custom_base and not self._is_registered_provider():
raise ValueError(
f"Unknown LLM provider prefix {self.model.split('/', 1)[0]!r} in model "
f"{self.model!r}. Use one of {sorted(_BUILTIN_MODEL_PREFIXES)}, set "
f"base_url/OPENAI_API_BASE for an OpenAI-compatible endpoint, or "
f"register it via praisonai.llm.register_llm_provider()."
)
self.api_key_var = api_key_var or "OPENAI_API_KEY"
self.base_url = custom_base or "https://api.openai.com/v1"
self.model_name = self.model
logger.debug(f"Initialized PraisonAIModel with model {self.model_name}, api_key_var {self.api_key_var}, and base_url {self.base_url}")
@@ -119,6 +119,10 @@ class ManagedBackendRegistry(PluginRegistry):
self._ensure_compute_registered()
return super().list_names()
def list_all_names(self) -> list[str]:
self._ensure_compute_registered()
return super().list_all_names()
def has(self, name: str) -> bool:
self._ensure_compute_registered()
return super().has(name)
@@ -889,32 +889,45 @@ def calculator(expression: str) -> str:
import operator
# Safe operations mapping
ops = {
ops = {{
ast.Add: operator.add, ast.Sub: operator.sub,
ast.Mult: operator.mul, ast.Div: operator.truediv,
ast.Pow: operator.pow, ast.Mod: operator.mod,
ast.USub: operator.neg, ast.UAdd: operator.pos,
}
}}
def _eval_node(node):
if isinstance(node, ast.Constant):
if type(node.value) not in (int, float):
raise TypeError("Only numeric constants are supported")
if isinstance(node.value, int) and node.value.bit_length() > 4096:
raise ValueError("Result is too large")
return node.value
elif isinstance(node, ast.BinOp):
left = _eval_node(node.left)
right = _eval_node(node.right)
return ops[type(node.op)](left, right)
# Reject powers whose result has at least 4096 + 1 bits before
# computing them; |left| >= 2 ** (bit_length - 1).
if (isinstance(node.op, ast.Pow) and isinstance(left, int)
and isinstance(right, int)
and (abs(left).bit_length() - 1) * right > 4096):
raise ValueError("Result is too large")
result = ops[type(node.op)](left, right)
if isinstance(result, int) and result.bit_length() > 4096:
raise ValueError("Result is too large")
return result
elif isinstance(node, ast.UnaryOp):
operand = _eval_node(node.operand)
return ops[type(node.op)](operand)
else:
raise TypeError(f"Unsupported operation: {type(node)}")
raise TypeError(f"Unsupported operation: {{type(node)}}")
try:
tree = ast.parse(expression.strip(), mode='eval')
result = _eval_node(tree.body)
return str(result)
except (ValueError, SyntaxError, TypeError, ZeroDivisionError, KeyError) as e:
return f"Error: {e}"
except (ValueError, SyntaxError, TypeError, ZeroDivisionError, KeyError, OverflowError) as e:
return f"Error: {{e}}"
# Create agent with tool
agent = Agent(
+1 -1
View File
@@ -1,7 +1,7 @@
[project]
name = "PraisonAI"
dynamic = ["version"]
description = "PraisonAI is an AI Agents Framework with Self Reflection. PraisonAI application combines PraisonAI Agents, AutoGen, and CrewAI into a low-code solution for building and managing multi-agent LLM systems, focusing on simplicity, customisation, and efficient human-agent collaboration."
description = "Open-source Python AI agent framework that runs on your own computers, even fully offline."
readme = "README.md"
license = {text = "MIT"}
requires-python = ">=3.10,<3.15"
@@ -0,0 +1,85 @@
"""Basic example structures must render without evaluating the embedded code.
Regression: the ``tools`` template is an f-string whose example code contains
literal braces (the ``ops`` dict, ``{type(node)}`` and ``{e}``). They were not
escaped, so building the structures dict raised ``TypeError`` (and would hit
``NameError`` for ``node``/``e``) for every feature, breaking basic example
prompt generation.
"""
import ast
import pytest
from praisonai.standardise.ai_generator import AIGenerator
def _generator():
return AIGenerator(use_fast_context=False)
@pytest.mark.parametrize(
"slug",
["guardrails", "memory", "tools", "knowledge", "streaming", "handoffs"],
)
def test_basic_structure_renders_for_every_feature(slug):
structure = _generator()._get_feature_basic_structure(slug, "Demo")
assert structure.startswith("```python\n")
assert '"""Demo - Basic Example"""' in structure
def test_tools_structure_contains_runnable_calculator():
structure = _generator()._get_feature_basic_structure("tools", "Tools")
code = structure.split("```python\n", 1)[1].rsplit("```", 1)[0]
assert "{{" not in code and "}}" not in code
assert "ast.Add: operator.add" in code
assert 'f"Unsupported operation: {type(node)}"' in code
assert 'f"Error: {e}"' in code
# The rendered example must be valid Python with the calculator intact.
tree = ast.parse(code)
functions = {node.name for node in ast.walk(tree) if isinstance(node, ast.FunctionDef)}
assert {"calculator", "_eval_node"} <= functions
agent_calls = [
node for node in ast.walk(tree)
if isinstance(node, ast.Call) and getattr(node.func, "id", None) == "Agent"
]
assert any(
keyword.arg == "tools"
and isinstance(keyword.value, ast.List)
and all(isinstance(elt, ast.Name) for elt in keyword.value.elts)
and [elt.id for elt in keyword.value.elts] == ["calculator"]
for call in agent_calls
for keyword in call.keywords
)
def test_tools_calculator_rejects_non_numeric_constants():
structure = _generator()._get_feature_basic_structure("tools", "Tools")
code = structure.split("```python\n", 1)[1].rsplit("```", 1)[0]
calculator_def = next(
node for node in ast.parse(code).body
if isinstance(node, ast.FunctionDef) and node.name == "calculator"
)
namespace = {}
exec(compile(ast.Module(body=[calculator_def], type_ignores=[]), "<example>", "exec"), namespace)
calculator = namespace["calculator"]
assert calculator("2 + 3 * 4") == "14"
assert calculator("'x' * 3").startswith("Error:")
assert calculator("True + 1").startswith("Error:")
assert calculator("2 ** 10") == "1024"
assert calculator("9 ** 100000000").startswith("Error:")
assert calculator("2 ** 3000") == str(2 ** 3000)
assert calculator("1 ** 5000") == "1"
assert calculator("3 ** 5000").startswith("Error:")
assert calculator("2 ** 4096").startswith("Error:")
assert calculator("1e308 ** 2").startswith("Error:")
assert calculator(str(2 ** 4095)) == str(2 ** 4095)
assert calculator(str(2 ** 4096)).startswith("Error:")
assert calculator("-" + str(2 ** 4096)).startswith("Error:")
def test_basic_example_prompt_includes_feature_structure(tmp_path):
generator = AIGenerator(sdk_root=tmp_path, use_fast_context=False)
prompt = generator._build_basic_example_prompt("tools", "Tools", {})
assert "def calculator(expression: str) -> str:" in prompt
@@ -0,0 +1,67 @@
"""Regression tests for #6245: no silent OpenAI misroute, no silent safety drop."""
import pytest
from praisonai.framework_adapters.base import warn_unsupported_fields
from praisonai.inc.models import PraisonAIModel
@pytest.fixture
def no_openai_base(monkeypatch):
monkeypatch.delenv("OPENAI_API_BASE", raising=False)
monkeypatch.delenv("OPENAI_BASE_URL", raising=False)
monkeypatch.setenv("OPENAI_API_KEY", "sk-test")
def test_unknown_prefix_without_endpoint_fails_loudly(no_openai_base):
with pytest.raises(ValueError, match="Unknown LLM provider prefix 'mistral'"):
PraisonAIModel(model="mistral/mistral-small")
def test_unknown_prefix_with_custom_endpoint_passes_through(no_openai_base):
m = PraisonAIModel(model="meta-llama/Llama-3.1-8B", base_url="http://localhost:8000/v1")
assert m.model_name == "meta-llama/Llama-3.1-8B"
assert m.base_url == "http://localhost:8000/v1"
def test_unknown_prefix_with_env_endpoint_passes_through(no_openai_base, monkeypatch):
monkeypatch.setenv("OPENAI_API_BASE", "http://gateway:4000/v1")
m = PraisonAIModel(model="mistral/mistral-small")
assert m.base_url == "http://gateway:4000/v1"
def test_bare_model_still_defaults_to_openai(no_openai_base):
m = PraisonAIModel(model="gpt-4o-mini")
assert m.base_url == "https://api.openai.com/v1"
class _Adapter:
name = "crewai"
SUPPORTED_YAML_FIELDS = frozenset({"llm"})
@pytest.mark.parametrize("field", ["approval", "guardrails"])
def test_dropped_safety_field_fails_closed(field):
with pytest.raises(ValueError, match="cannot enforce safety"):
warn_unsupported_fields(_Adapter(), {"role": "r", field: {"backend": "console"}})
def test_dropped_non_safety_field_only_warns(caplog):
warn_unsupported_fields(_Adapter(), {"role": "r", "memory": True})
assert "ignores YAML field" in caplog.text
@pytest.mark.parametrize("extras", [
{"approval": False},
{"approval": None},
{"approval": {"enabled": True, "backend": "auto", "approve_all_tools": False}}, # --trust
{"guardrails": None},
{"guardrails": ""},
])
def test_inactive_safety_field_does_not_block(extras, caplog):
warn_unsupported_fields(_Adapter(), {"role": "r", **extras})
assert "ignores YAML field" in caplog.text
def test_unparseable_approval_fails_closed():
with pytest.raises(ValueError, match="cannot enforce safety"):
warn_unsupported_fields(_Adapter(), {"role": "r", "approval": {"typo": 1}})
@@ -0,0 +1,26 @@
"""The first runtime lookup must discover compute-backed placements."""
def test_first_all_names_lookup_registers_compute_and_keeps_aliases(monkeypatch):
from praisonai.integrations.backend_registry import ManagedBackendRegistry
from praisonaiagents.managed import _compute_bridge
monkeypatch.setattr(_compute_bridge, "available_providers", lambda: ["docker"])
registry = ManagedBackendRegistry()
registry.register("custom", object, aliases=["custom-alias"])
assert "docker" in registry.list_all_names()
assert "custom-alias" in registry.list_all_names()
def test_first_agent_placement_accepts_discovered_runtime(monkeypatch):
from praisonai.integrations import backend_registry
from praisonaiagents import Agent
from praisonaiagents.managed import _compute_bridge
monkeypatch.setattr(_compute_bridge, "available_providers", lambda: ["docker"])
registry = backend_registry.ManagedBackendRegistry()
monkeypatch.setattr(backend_registry, "get_backend_registry", lambda: registry)
agent = Agent(name="test", instructions="test", run_on="docker")
assert agent.backend.provider_name == "docker"