mirror of
https://github.com/crewAIInc/crewAI.git
synced 2026-10-12 02:12:31 +00:00
perf: async-path benchmark harness, results and report
Investigation only: measures crewai's akickoff path before/after the prototype commits on this branch. Not intended to merge as-is. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01DhtN2bvHsgQfKQmHNnVSFn
This commit is contained in:
Claude
@@ -0,0 +1,285 @@
|
||||
# Async execution path: is going thread-free worth it on CPython?
|
||||
|
||||
Throwaway investigation. Prototype commits are on this branch; nothing here is
|
||||
meant to merge as-is.
|
||||
|
||||
## TL;DR
|
||||
|
||||
**Recommendation: partial go.** The thread hops listed in the brief (file store,
|
||||
event bus, parallel tool pool, planning `to_thread`) are worth almost nothing
|
||||
on their own. But the investigation found a much bigger problem the brief
|
||||
didn't list: **`Crew.akickoff` with the default executor never calls
|
||||
`LLM.acall`.** `AgentExecutor` is a `Flow` whose steps are sync methods, and
|
||||
`Flow` runs every sync method via `asyncio.to_thread`. So every LLM call is a
|
||||
blocking `LLM.call` on the loop's *default executor*. The number of concurrent
|
||||
LLM calls across all `akickoff`s in a process is therefore capped at
|
||||
`min(32, cpu+4)` (8 on the 4-core benchmark box). Fixing that one thing gives:
|
||||
|
||||
| 200 concurrent `akickoff` (2 agents, 2 tasks) | baseline | prototype |
|
||||
|---|---|---|
|
||||
| LLM latency 200 ms, wall time | 18.5 s | **6.9 s** (2.7×) |
|
||||
| LLM latency 500 ms, wall time | 35.1 s | **7.2 s** (4.9×) |
|
||||
| + 4 parallel async tools, 200 ms | 27.9 s | **9.2 s** (3.0×) |
|
||||
| 50 crews with planning (3 parallel steps) | 16.5 s | **4.7 s** (3.5×) |
|
||||
| 20 crews, default executor shrunk to 4 workers | 2.69 s | **0.91 s** |
|
||||
| Cancel mid-run: LLM/tool calls that keep running | 1 LLM / 4 tools | **0 / 0** |
|
||||
| Threads started per kickoff | 7–11 | **0** |
|
||||
|
||||
Two things are **not** worth doing as proposed:
|
||||
|
||||
- **Event bus rework.** Measured no effect. On the `akickoff` path, `aemit` is
|
||||
never used; `emit()` hands off to a long-lived 10-thread pool and doesn't
|
||||
wait. The double hop the brief describes is only taken for handlers
|
||||
registered with `depends_on`, and it happens on the bus's private loop, not
|
||||
the caller's.
|
||||
- **Moving the task-output SQLite log off the loop (P6).** It improves loop lag
|
||||
but halves throughput at high concurrency.
|
||||
|
||||
The caveat: at small concurrency (1–10 crews) with real LLM latency the user
|
||||
won't notice anything except cancellation. The wins appear once the number of
|
||||
in-flight LLM calls exceeds the default executor size, i.e. ~8–32 concurrent
|
||||
calls per process. That threshold is easy to reach for a server that runs
|
||||
`akickoff` per request.
|
||||
|
||||
## Setup
|
||||
|
||||
- CPython 3.13.16, 4 vCPU (default executor = 8 workers), `uv sync --all-packages`.
|
||||
- Harness: `lib/crewai/tests/perf/bench_async_path.py`, one command:
|
||||
`uv run python lib/crewai/tests/perf/bench_async_path.py --label <name>`
|
||||
(`--quick` / `--only <substr>` available). Each scenario runs in its own
|
||||
subprocess. Raw JSON for every run is in `lib/crewai/tests/perf/results/`.
|
||||
- Mock `BaseLLM` subclass: `call` does `time.sleep(latency)`, `acall` does
|
||||
`asyncio.sleep(latency)`; both count invocations, so the harness reports which
|
||||
path crewai actually took. The "tools" variant returns 4 native tool calls to
|
||||
async `BaseTool`s (`_arun` = `asyncio.sleep(100 ms)`), then a final answer.
|
||||
The "planning" variant uses `planning=True` with a 3-step, dependency-free plan.
|
||||
- Metrics: wall time, throughput, peak `threading.active_count()` (2 ms
|
||||
sampler), `Thread.start` calls (patched), peak RSS (`ru_maxrss`), event-loop
|
||||
lag (drift of a 10 ms `asyncio.sleep` probe).
|
||||
- Telemetry/tracing disabled; `verbose=False`.
|
||||
|
||||
## Thread hops actually on the `akickoff` path (baseline)
|
||||
|
||||
From the brief, confirmed:
|
||||
|
||||
1. File store `_run_sync`: new `ThreadPoolExecutor` + `asyncio.run` per call.
|
||||
Not 2 calls per kickoff but **~7**: `Crew._prepare_tools`, `Task.prompt()`
|
||||
and `Task._aexecute_core`'s `finally` also hit it, every task, even when no
|
||||
files were ever stored.
|
||||
2. Event bus: see above. Measured as not mattering.
|
||||
3. Planning steps: `asyncio.to_thread(step_executor.execute)`, plus
|
||||
`execute_todo_sequential` as a sync Flow step.
|
||||
4. Parallel native tools: a fresh `ThreadPoolExecutor(≤8)` per batch. Inside
|
||||
it, every *async* tool is run via `BaseTool.run` → `asyncio.run(...)`: a new
|
||||
event loop per tool call.
|
||||
5. `Crew.kickoff_async` = `to_thread(kickoff)`: noted, not changed.
|
||||
|
||||
Not in the brief, and the ones that matter:
|
||||
|
||||
6. **Every `AgentExecutor` step is a sync Flow method → `asyncio.to_thread`**,
|
||||
including `call_llm_native_tools` / `call_llm_and_parse`, which call the
|
||||
*sync* `get_llm_response` → `LLM.call`. Every pure routing step
|
||||
(`check_max_iterations`, `route_by_answer_type`, …) also takes a hop, and
|
||||
every step takes a second hop for `_persist_method_completion` even when
|
||||
persistence is off. This is the default-executor cap.
|
||||
7. `_create_crew_output` calls the blocking `crewai_event_bus.flush()` **on the
|
||||
loop**, and it waits on handler futures from *all* concurrent crews.
|
||||
8. `_store_execution_log` (every task) and `TaskOutputStorageHandler.reset`
|
||||
(every kickoff) do a SQLite transaction behind a cross-process file lock
|
||||
(`store_lock`) **on the loop**. In a cProfile of the prototype this is
|
||||
~60% of per-kickoff wall time at 0 ms LLM latency.
|
||||
|
||||
## Prototype changes (one commit each)
|
||||
|
||||
| id | change | files |
|
||||
|---|---|---|
|
||||
| P1 | `akickoff` uses `aget_all_files` (prefetched into `_prepare_tools`) and `aclear_files` | `crew.py`, `crews/utils.py` |
|
||||
| P2 | `aflush()` on the bus; `akickoff` awaits it instead of the blocking `flush()`; `_emit_with_dependencies` awaits `wrap_future` (1 hop, not 2) | `event_bus.py`, `crew.py` |
|
||||
| P3+P4 | `Flow._async_step_override` hook. `AgentExecutor` (only when entered via `invoke_async`) runs async twins of the LLM steps (`acall`, falling back to `to_thread(call)` for LLMs without `acall`) and of `execute_native_tool` (async tools awaited and `asyncio.gather`ed, sync tools `to_thread`). Pure routing steps run inline. Persist hop skipped when persistence is off. Single-call lifecycle refactored into a generator that yields at the one blocking point, so sync and async share one body | `flow/runtime/__init__.py`, `experimental/agent_executor.py`, `utilities/agent_utils.py` |
|
||||
| P1b | File store remembers which ids ever stored files; lookups and clears for other ids return immediately (benefits the sync path too) | `utilities/file_store.py` |
|
||||
| P5 | `StepExecutor` converted to the same generator pattern: `execute()` drives it synchronously, `aexecute()` awaits `acall` / async tools. Sequential and parallel plan steps use it on the async path | `agents/step_executor.py`, `experimental/agent_executor.py`, `utilities/agent_utils.py` |
|
||||
| P6 | `akickoff` writes the per-task output log via `to_thread` | `crew.py` |
|
||||
|
||||
P3 (LLM steps) and P4 (parallel tools) were committed together because they
|
||||
share the `execute_native_tool` restructuring. The "plain" variant has no tool
|
||||
calls, so its column isolates P3; the "tools" variant shows P3+P4.
|
||||
|
||||
Cumulative: each column includes all previous ones. The P1b column's planning
|
||||
rows were measured on the commit before P5.
|
||||
|
||||
### Wall time (s)
|
||||
|
||||
| scenario | baseline | P1 | P2 | P3+P4 | P1b | P5 | P6 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| plain, 200 ms, N=1 | 0.45 | 0.44 | 0.46 | 0.44 | 0.43 | 0.44 | 0.44 |
|
||||
| plain, 200 ms, N=10 | 0.99 | 0.97 | 0.97 | 0.68 | 0.66 | 0.63 | 1.87 |
|
||||
| plain, 200 ms, N=50 | 4.23 | 4.08 | 4.10 | 1.80 | 1.57 | 1.65 | 2.24 |
|
||||
| plain, 200 ms, N=200 | 18.48 | 17.28 | 17.58 | 7.57 | 6.69 | 6.85 | 7.39 |
|
||||
| plain, 500 ms, N=200 | 35.10 | 32.61 | 32.81 | 7.70 | 7.10 | 7.17 | 8.10 |
|
||||
| tools, 200 ms, N=1 | 0.77 | 0.76 | 0.76 | 0.76 | 0.77 | 0.74 | 0.75 |
|
||||
| tools, 200 ms, N=10 | 1.25 | 1.25 | 1.21 | 1.02 | 0.97 | 0.98 | 1.69 |
|
||||
| tools, 200 ms, N=50 | 6.22 | 6.33 | 6.16 | 2.40 | 2.22 | 2.17 | 3.02 |
|
||||
| tools, 200 ms, N=200 | 27.89 | 27.07 | 26.96 | 9.34 | 8.90 | 9.22 | 17.21 |
|
||||
| planning, 200 ms, N=1 | 1.27 | – | – | – | 1.25 | 1.24 | 1.25 |
|
||||
| planning, 200 ms, N=50 | 16.52 | – | – | – | 15.23 | 4.70 | 5.30 |
|
||||
| default executor = 4, N=20 | 2.69 | 2.69 | 2.69 | 0.92 | 0.92 | 0.91 | 1.53 |
|
||||
|
||||
The ideal for "plain" (2 sequential LLM calls) is ~0.4 s at any N. Once P3 is
|
||||
in, the residual is crewai's own CPU work, not waiting (see "What's left").
|
||||
|
||||
### Per-kickoff overhead (LLM latency 0, sequential kickoffs, median)
|
||||
|
||||
| | baseline | P1 | P2 | P3+P4 | P1b | P5 | P6 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| plain: ms / kickoff | 26.1 | 26.2 | 25.6 | 21.0 | 19.7 | 19.6 | 17.0 |
|
||||
| plain: threads started / kickoff | 7.1 | 4.0 | 4.0 | 4.0 | 0.0 | 0.1 | 0.1 |
|
||||
| tools: ms / kickoff | 36.6 | 36.1 | 37.4 | 25.6 | 22.0 | 24.9 | 23.0 |
|
||||
| tools: threads started / kickoff | 11.1 | 8.1 | 8.0 | 4.2 | 0.2 | 0.2 | 0.2 |
|
||||
|
||||
### Threads and memory
|
||||
|
||||
| | baseline | after P5 |
|
||||
|---|---|---|
|
||||
| peak threads, tools N=200 | 49 | 20 |
|
||||
| threads started, tools N=200 | 2215 | 13 |
|
||||
| peak RSS, tools N=200 | 310 MB | 312 MB |
|
||||
|
||||
The pools are bounded, so thread counts never exploded, and RSS is unchanged.
|
||||
Memory is not an argument either way.
|
||||
|
||||
### Event-loop lag (ms, p99 / max)
|
||||
|
||||
| scenario | baseline | P3+P4 | P5 | P6 |
|
||||
|---|---|---|---|---|
|
||||
| plain, 200 ms, N=50 | 89 / 771 | 144 / 758 | 113 / 644 | 14 / 683 |
|
||||
| plain, 200 ms, N=200 | 72 / 3510 | 761 / 3163 | 499 / 2950 | 97 / 2920 |
|
||||
| tools, 200 ms, N=200 | 39 / 3829 | 1172 / 3477 | 599 / 3512 | 70 / 3556 |
|
||||
|
||||
The multi-second max lag is the same in every column. It comes from the burst
|
||||
of ~200 crew setups (`prepare_kickoff`, executor construction) running on the
|
||||
loop at start. That is pure CPU and not thread-related. p99 lag goes **up**
|
||||
once the executor is native: CPU work that used to be spread over worker threads
|
||||
(still serialised by the GIL, but interleaved with the loop) now runs on the
|
||||
loop itself. With P6 (SQLite off the loop) p99 falls back to baseline levels.
|
||||
|
||||
### Cancellation (`task.cancel()` on a running `akickoff`)
|
||||
|
||||
| | `CancelledError` surfaces after | calls still running to completion after cancel |
|
||||
|---|---|---|
|
||||
| baseline, cancel during LLM call | 2.8 ms | 1 LLM call (thread) |
|
||||
| baseline, cancel during 4 parallel tools | 1.9 ms | 4 tool calls (thread pool + private loops) |
|
||||
| P3+P4 onward | 0.2–1.5 ms | **0** (the `acall` / tool coroutines are cancelled) |
|
||||
|
||||
No new work starts after cancellation even on baseline. The fix is about work
|
||||
that is *already in flight*: on baseline it keeps running for its full duration.
|
||||
With real LLMs that means spending tokens on the result of a request the caller
|
||||
has abandoned. Sync tools are still not cancellable, because they stay in
|
||||
threads by design.
|
||||
|
||||
### Context propagation
|
||||
|
||||
`contextvars` set before `akickoff` are visible in the tool, the LLM call, a
|
||||
sync handler and an async handler, both before and after every prototype
|
||||
(`context/tools` row in each JSON).
|
||||
|
||||
## Which changes are worth it
|
||||
|
||||
| change | worth it alone? | risk | test fallout (vs main) |
|
||||
|---|---|---|---|
|
||||
| P1: async file store in `akickoff` | Marginal: −3 threads per kickoff, no wall-time change | Low | 0 |
|
||||
| **P1b: file store short-circuit** | **Yes**, cheap: removes the remaining ~4–7 `ThreadPoolExecutor` + `asyncio.run` per kickoff on *both* sync and async paths | Low. Stale ids after TTL only cost a real lookup. Cross-process stores would need a rethink, but the store is in-memory today | 0 |
|
||||
| P2: event bus `aflush` / single hop | **No measurable gain** at any concurrency. Keep `aflush` only as hygiene (it stops `akickoff` blocking the loop while slow handlers drain) | Low | 0 |
|
||||
| **P3: executor LLM steps on the loop** | **Yes, the main win**: 2.7–4.9× throughput at N≥50, cancellation, no default-executor coupling | Medium (see behaviour changes) | 1 (source-inspection test) + 1 fixed during the prototype |
|
||||
| **P4: gather async tools** | **Yes**, with P3: peak threads 49 → 20, tools N=200 27.9 → 9.2 s | Low/medium | shared with P3 |
|
||||
| **P5: planning steps native** | **Yes** if planning matters: 3.2× at N=50 | Medium: the generator refactor changes private method shapes | 5 (tests that call or mock `_execute_native` etc. directly) + 1 source-inspection test |
|
||||
| P6: SQLite log via `to_thread` | **No, as written**: p99 lag 499 → 97 ms but throughput 6.9 → 7.4 s (plain) and **9.2 → 17.2 s** (tools) at N=200, and N=10 gets 3× slower. Threads now contend on the file lock and on default-executor workers. The fix is to make the per-task replay log cheaper (batch per kickoff, async store, or opt-out), not to thread it | – | 0 |
|
||||
|
||||
All 7 failing tests are implementation-coupled (`inspect.getsource` checks or
|
||||
mocks of private methods); none is a behaviour regression. I fixed one real
|
||||
regression found along the way (`OutputParserError` raised by the LLM call
|
||||
itself was no longer routed to `parser_error`).
|
||||
|
||||
Test-suite note: in this container `pytest lib/crewai/tests` already has 407
|
||||
failures and 14 errors on `main`, from optional provider packages (litellm,
|
||||
anthropic, gemini, azure) not being installed. Fallout above is the set
|
||||
difference against that baseline: main 407 failed / 5198 passed; prototype 414
|
||||
failed / 5191 passed; nothing fixed or broken beyond the 7 listed.
|
||||
`ruff check` and `mypy` are clean on all 8 changed source files.
|
||||
|
||||
## Behaviour changes users could notice (P3–P5)
|
||||
|
||||
- **`LLM.acall` instead of `LLM.call` on `akickoff`.** Provider async paths
|
||||
differ from the sync ones (client objects, retry/timeout wiring, streaming).
|
||||
Users who customised `call` but not `acall` get the old behaviour via a
|
||||
`to_thread` fallback. Users who override both with different behaviour will
|
||||
see the `acall` one. The sync `kickoff()` is unchanged.
|
||||
- **Hooks and callbacks now run on the caller's event loop thread.** Before/after
|
||||
LLM-call hooks, tool hooks, `step_callback` and `enforce_rpm_limit` used to
|
||||
run in a worker thread. A blocking user callback now stalls the loop, and so
|
||||
does RPM throttling (`enforce_rpm_limit` sleeps). The RPM limiter needs an
|
||||
async variant before shipping.
|
||||
- **Async tools run on the caller's loop**, not a private `asyncio.run` loop in a
|
||||
thread. This is good for tools holding loop-bound clients (aiohttp sessions
|
||||
created on the app loop used to fail). An async tool that does blocking work
|
||||
inside `async def` now stalls the caller's loop.
|
||||
- **`contextvars` writes inside tools and hooks now persist** across steps
|
||||
within the same kickoff. Before, each `to_thread` ran on a context copy, so
|
||||
mutations were lost. Reads behave the same either way.
|
||||
- **Parallel tool batch aborts** (`ToolExecutionFailedError`) now cancel
|
||||
in-flight async siblings. Before, they could only drop not-yet-started ones.
|
||||
- **Cancellation** now reaches `acall` / tool coroutines; providers and tools see
|
||||
`CancelledError` mid-call.
|
||||
- **Exceptions**: same types and messages. Tracebacks are shorter (no thread
|
||||
boundary).
|
||||
- **Event handler ordering**: unchanged. `emit()` was not touched, and sync
|
||||
handlers still run on the bus pool.
|
||||
- **`CrewKickoffCompletedEvent` timing**: unchanged ordering. `akickoff` still
|
||||
drains memory writes and waits for *all* pending handlers (globally, across
|
||||
concurrent crews) before emitting. It now awaits instead of blocking the
|
||||
loop. The cross-crew coupling (one crew's completion waits on every other
|
||||
crew's handlers) is pre-existing and worth fixing separately.
|
||||
|
||||
## What's left after the prototype (the next bottleneck)
|
||||
|
||||
With latency out of the picture, a 2-task kickoff costs ~17–20 ms of CPU. In
|
||||
cProfile, the biggest items are:
|
||||
|
||||
- SQLite + file-lock task-output log: ~60% of wall time on the async path.
|
||||
- Flow listener resolution (`_find_triggered_methods` / `_condition_met`):
|
||||
~10 ms per kickoff, from 24k condition checks for 20 kickoffs.
|
||||
- `prepare_kickoff` per crew.
|
||||
|
||||
These bound throughput at roughly 25–30 kickoffs/s per process regardless of
|
||||
threads, and they are now what drives loop lag. Reducing them helps both
|
||||
`kickoff` and `akickoff`.
|
||||
|
||||
## Recommendation and PR breakdown
|
||||
|
||||
**Partial go.** Do the executor work (it is the real "async path"), skip the
|
||||
event-bus rework, and fix the storage write differently.
|
||||
|
||||
1. **File store short-circuit (P1b + P1).** Small, low risk, helps sync too.
|
||||
Ship first.
|
||||
2. **`Flow._async_step_override` + `AgentExecutor` async LLM steps (P3).**
|
||||
Include: `to_thread(call)` fallback for LLMs without `acall`, inline routing
|
||||
steps, skipping the persist hop when persistence is off, an async
|
||||
`enforce_rpm_limit`, and the `OutputParserError` routing fix. Update the one
|
||||
source-inspection test. Document the hooks-on-loop change.
|
||||
3. **Native tool batches on the loop (P4).** Generator-based single-call
|
||||
lifecycle shared by sync and async, gathered async tools, sync tools via
|
||||
`to_thread`. Could be folded into PR 2.
|
||||
4. **Planning `StepExecutor.aexecute` (P5).** Rewrite the 5 tests that mock
|
||||
private step methods to go through `execute()`. Then also convert the
|
||||
remaining sync planning LLM calls (plan generation, `PlannerObserver`,
|
||||
synthesis); 100 of the 450 planning LLM calls in the benchmark still go
|
||||
through threads.
|
||||
5. **Separately:** make the per-task replay log cheap (one write per kickoff, or
|
||||
async/opt-out), and reduce Flow listener resolution cost. These are the next
|
||||
throughput limit for both sync and async.
|
||||
6. **`aflush` (P2)**: optional hygiene, can ride along with PR 2. Not worth a
|
||||
PR by itself.
|
||||
|
||||
Out of scope and unchanged: `kickoff()`, `Task.execute_async`, the memory save
|
||||
pool, `Crew.kickoff_async`. That last one still wraps the whole sync
|
||||
`kickoff` in a thread; with PR 2 landed, pointing users at `akickoff` becomes
|
||||
a real recommendation rather than a cosmetic one.
|
||||
@@ -0,0 +1,491 @@
|
||||
"""Benchmarks for crewai's async execution path (``Crew.akickoff``).
|
||||
|
||||
Throwaway investigation harness, not a test module (no ``test_`` prefix, so
|
||||
pytest does not collect it). Every scenario runs in a fresh subprocess so
|
||||
thread counts, RSS and the global event bus start clean.
|
||||
|
||||
Run everything and write a JSON + markdown summary::
|
||||
|
||||
uv run python lib/crewai/tests/perf/bench_async_path.py --label baseline
|
||||
|
||||
Run a single scenario in-process (what the driver does per subprocess)::
|
||||
|
||||
uv run python lib/crewai/tests/perf/bench_async_path.py --one scaling \
|
||||
--n 50 --latency 0.2 --variant tools
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import contextvars
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import resource
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
|
||||
os.environ.setdefault("CREWAI_DISABLE_TELEMETRY", "true")
|
||||
os.environ.setdefault("CREWAI_DISABLE_TRACKING", "true")
|
||||
os.environ.setdefault("OTEL_SDK_DISABLED", "true")
|
||||
os.environ.setdefault("CREWAI_TRACING_ENABLED", "false")
|
||||
os.environ.setdefault("CREWAI_STORAGE_DIR", "/tmp/crewai-perf-storage")
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Instrumentation shared by all scenarios
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
_thread_starts = 0
|
||||
_orig_thread_start = threading.Thread.start
|
||||
|
||||
|
||||
def _counting_start(self: threading.Thread) -> None:
|
||||
global _thread_starts
|
||||
_thread_starts += 1
|
||||
_orig_thread_start(self)
|
||||
|
||||
|
||||
threading.Thread.start = _counting_start # type: ignore[method-assign]
|
||||
|
||||
|
||||
class Counters:
|
||||
lock = threading.Lock()
|
||||
sync_llm_calls = 0
|
||||
async_llm_calls = 0
|
||||
tool_calls = 0
|
||||
llm_call_times: list[float] = []
|
||||
tool_call_times: list[float] = []
|
||||
llm_done_times: list[float] = []
|
||||
tool_done_times: list[float] = []
|
||||
|
||||
@classmethod
|
||||
def done(cls, kind: str) -> None:
|
||||
with cls.lock:
|
||||
(cls.llm_done_times if kind == "llm" else cls.tool_done_times).append(
|
||||
time.perf_counter()
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def llm(cls, is_async: bool) -> None:
|
||||
with cls.lock:
|
||||
if is_async:
|
||||
cls.async_llm_calls += 1
|
||||
else:
|
||||
cls.sync_llm_calls += 1
|
||||
cls.llm_call_times.append(time.perf_counter())
|
||||
|
||||
@classmethod
|
||||
def tool(cls) -> None:
|
||||
with cls.lock:
|
||||
cls.tool_calls += 1
|
||||
cls.tool_call_times.append(time.perf_counter())
|
||||
|
||||
|
||||
CTX_VAR: contextvars.ContextVar[str] = contextvars.ContextVar(
|
||||
"bench_ctx", default="<unset>"
|
||||
)
|
||||
SEEN_CTX: dict[str, set[str]] = {"tool": set(), "handler": set(), "llm": set()}
|
||||
|
||||
|
||||
def _build_crew(variant: str, latency: float, tool_latency: float, n_tools: int):
|
||||
from crewai import Agent, Crew, Task
|
||||
from crewai.llms.base_llm import BaseLLM
|
||||
from crewai.tools import BaseTool
|
||||
|
||||
def _has_tool_result(messages: Any) -> bool:
|
||||
return isinstance(messages, list) and any(
|
||||
isinstance(m, dict) and m.get("role") == "tool" for m in messages
|
||||
)
|
||||
|
||||
def _respond(messages: Any, tools: Any, available_functions: Any = None) -> Any:
|
||||
SEEN_CTX["llm"].add(CTX_VAR.get())
|
||||
if available_functions and "create_reasoning_plan" in available_functions:
|
||||
return json.dumps(
|
||||
{
|
||||
"plan": "three independent steps",
|
||||
"steps": [
|
||||
{"step_number": k, "description": f"step {k}", "depends_on": []}
|
||||
for k in (1, 2, 3)
|
||||
],
|
||||
"ready": True,
|
||||
}
|
||||
)
|
||||
if tools and variant in ("tools", "planning") and not _has_tool_result(messages):
|
||||
return [
|
||||
{
|
||||
"id": f"call_{i}",
|
||||
"type": "function",
|
||||
"function": {"name": f"async_tool_{i}", "arguments": "{}"},
|
||||
}
|
||||
for i in range(n_tools)
|
||||
]
|
||||
if tools:
|
||||
return "done"
|
||||
return "Thought: I know the answer\nFinal Answer: done"
|
||||
|
||||
class MockLLM(BaseLLM):
|
||||
def call(self, messages, tools=None, callbacks=None, available_functions=None,
|
||||
from_task=None, from_agent=None, response_model=None): # noqa: D401
|
||||
Counters.llm(is_async=False)
|
||||
if latency:
|
||||
time.sleep(latency)
|
||||
Counters.done("llm")
|
||||
return _respond(messages, tools, available_functions)
|
||||
|
||||
async def acall(self, messages, tools=None, callbacks=None,
|
||||
available_functions=None, from_task=None, from_agent=None,
|
||||
response_model=None):
|
||||
Counters.llm(is_async=True)
|
||||
if latency:
|
||||
await asyncio.sleep(latency)
|
||||
Counters.done("llm")
|
||||
return _respond(messages, tools, available_functions)
|
||||
|
||||
def supports_function_calling(self) -> bool:
|
||||
return True
|
||||
|
||||
def supports_stop_words(self) -> bool:
|
||||
return False
|
||||
|
||||
def get_context_window_size(self) -> int:
|
||||
return 128000
|
||||
|
||||
def make_tool(i: int) -> BaseTool:
|
||||
class AsyncTool(BaseTool):
|
||||
name: str = f"async_tool_{i}"
|
||||
description: str = f"Async benchmark tool {i}"
|
||||
|
||||
async def _run(self) -> str: # type: ignore[override]
|
||||
return await self._arun()
|
||||
|
||||
async def _arun(self) -> str: # type: ignore[override]
|
||||
Counters.tool()
|
||||
SEEN_CTX["tool"].add(CTX_VAR.get())
|
||||
if tool_latency:
|
||||
await asyncio.sleep(tool_latency)
|
||||
Counters.done("tool")
|
||||
return f"result {i}"
|
||||
|
||||
return AsyncTool()
|
||||
|
||||
tools = (
|
||||
[make_tool(i) for i in range(n_tools)]
|
||||
if variant in ("tools", "planning")
|
||||
else []
|
||||
)
|
||||
agents = [
|
||||
Agent(
|
||||
role=f"Agent {k}",
|
||||
goal="Answer",
|
||||
backstory="Benchmark agent",
|
||||
llm=MockLLM(model="mock-model"),
|
||||
tools=tools if k == 0 else [],
|
||||
planning=variant == "planning" and k == 0,
|
||||
verbose=False,
|
||||
)
|
||||
for k in range(2)
|
||||
]
|
||||
tasks = [
|
||||
Task(description=f"Task {k}", expected_output="done", agent=agents[k])
|
||||
for k in range(2)
|
||||
]
|
||||
return Crew(agents=agents, tasks=tasks, verbose=False)
|
||||
|
||||
|
||||
class LoopProbe:
|
||||
"""Measure event-loop lag: how late a 10 ms sleep wakes up."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.lags: list[float] = []
|
||||
self._task: asyncio.Task[None] | None = None
|
||||
|
||||
async def _run(self) -> None:
|
||||
while True:
|
||||
t0 = time.perf_counter()
|
||||
await asyncio.sleep(0.01)
|
||||
self.lags.append(time.perf_counter() - t0 - 0.01)
|
||||
|
||||
def start(self) -> None:
|
||||
self._task = asyncio.create_task(self._run())
|
||||
|
||||
async def stop(self) -> dict[str, float]:
|
||||
assert self._task is not None
|
||||
self._task.cancel()
|
||||
try:
|
||||
await self._task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
lags = sorted(self.lags) or [0.0]
|
||||
return {
|
||||
"loop_lag_max_ms": round(lags[-1] * 1000, 1),
|
||||
"loop_lag_p99_ms": round(lags[int(len(lags) * 0.99) - 1 if len(lags) > 1 else 0] * 1000, 1),
|
||||
"loop_lag_mean_ms": round(statistics.fmean(lags) * 1000, 2),
|
||||
}
|
||||
|
||||
|
||||
class ThreadSampler:
|
||||
def __init__(self) -> None:
|
||||
self.peak = threading.active_count()
|
||||
self._stop = threading.Event()
|
||||
self._t = threading.Thread(target=self._run, daemon=True)
|
||||
|
||||
def _run(self) -> None:
|
||||
while not self._stop.is_set():
|
||||
self.peak = max(self.peak, threading.active_count() - 1)
|
||||
time.sleep(0.002)
|
||||
|
||||
def __enter__(self) -> ThreadSampler:
|
||||
_orig_thread_start(self._t)
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc: Any) -> None:
|
||||
self._stop.set()
|
||||
self._t.join()
|
||||
|
||||
|
||||
def _rss_mb() -> float:
|
||||
return round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, 1)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Scenarios (each runs in its own process)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def scenario_scaling(args: argparse.Namespace) -> dict[str, Any]:
|
||||
crews = [
|
||||
_build_crew(args.variant, args.latency, args.tool_latency, args.n_tools)
|
||||
for _ in range(args.n)
|
||||
]
|
||||
if args.small_executor:
|
||||
asyncio.get_running_loop().set_default_executor(
|
||||
ThreadPoolExecutor(args.small_executor)
|
||||
)
|
||||
# Warm-up kickoff so import/lazy-init costs are excluded.
|
||||
warm = _build_crew(args.variant, 0, 0, args.n_tools)
|
||||
await warm.akickoff()
|
||||
Counters.sync_llm_calls = Counters.async_llm_calls = Counters.tool_calls = 0
|
||||
starts_before = _thread_starts
|
||||
rss_before = _rss_mb()
|
||||
|
||||
probe = LoopProbe()
|
||||
probe.start()
|
||||
with ThreadSampler() as sampler:
|
||||
t0 = time.perf_counter()
|
||||
results = await asyncio.gather(*(c.akickoff() for c in crews))
|
||||
wall = time.perf_counter() - t0
|
||||
lag = await probe.stop()
|
||||
assert all(r.raw for r in results)
|
||||
return {
|
||||
"wall_s": round(wall, 3),
|
||||
"throughput_kickoffs_per_s": round(args.n / wall, 2),
|
||||
"peak_threads": sampler.peak,
|
||||
"thread_starts": _thread_starts - starts_before,
|
||||
"peak_rss_mb": _rss_mb(),
|
||||
"rss_growth_mb": round(_rss_mb() - rss_before, 1),
|
||||
"sync_llm_calls": Counters.sync_llm_calls,
|
||||
"async_llm_calls": Counters.async_llm_calls,
|
||||
"tool_calls": Counters.tool_calls,
|
||||
**lag,
|
||||
}
|
||||
|
||||
|
||||
async def scenario_overhead(args: argparse.Namespace) -> dict[str, Any]:
|
||||
crew = _build_crew(args.variant, 0, 0, args.n_tools)
|
||||
await crew.akickoff() # warm-up
|
||||
times = []
|
||||
starts_before = _thread_starts
|
||||
for _ in range(args.n):
|
||||
t0 = time.perf_counter()
|
||||
await crew.akickoff()
|
||||
times.append(time.perf_counter() - t0)
|
||||
return {
|
||||
"per_kickoff_ms_median": round(statistics.median(times) * 1000, 2),
|
||||
"per_kickoff_ms_mean": round(statistics.fmean(times) * 1000, 2),
|
||||
"thread_starts_per_kickoff": round((_thread_starts - starts_before) / args.n, 1),
|
||||
}
|
||||
|
||||
|
||||
async def scenario_cancel(args: argparse.Namespace) -> dict[str, Any]:
|
||||
crew = _build_crew(args.variant, args.latency, args.tool_latency, args.n_tools)
|
||||
await _build_crew(args.variant, 0, 0, args.n_tools).akickoff() # warm-up
|
||||
Counters.sync_llm_calls = Counters.async_llm_calls = Counters.tool_calls = 0
|
||||
for lst in (Counters.llm_call_times, Counters.tool_call_times,
|
||||
Counters.llm_done_times, Counters.tool_done_times):
|
||||
lst.clear()
|
||||
task = asyncio.create_task(crew.akickoff())
|
||||
await asyncio.sleep(args.cancel_after)
|
||||
t_cancel = time.perf_counter()
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
cancel_outcome = "completed (cancel ignored)"
|
||||
except asyncio.CancelledError:
|
||||
cancel_outcome = "CancelledError"
|
||||
except Exception as e: # noqa: BLE001
|
||||
cancel_outcome = f"{type(e).__name__}: {e}"
|
||||
t_returned = time.perf_counter() - t_cancel
|
||||
await asyncio.sleep(args.latency * 6 + args.tool_latency * 4 + 0.5)
|
||||
return {
|
||||
"cancel_outcome": cancel_outcome,
|
||||
"await_returned_after_ms": round(t_returned * 1000, 1),
|
||||
"llm_started_after_cancel": sum(1 for t in Counters.llm_call_times if t > t_cancel),
|
||||
"llm_finished_after_cancel": sum(1 for t in Counters.llm_done_times if t > t_cancel),
|
||||
"tool_started_after_cancel": sum(1 for t in Counters.tool_call_times if t > t_cancel),
|
||||
"tool_finished_after_cancel": sum(1 for t in Counters.tool_done_times if t > t_cancel),
|
||||
"llm_started_total": len(Counters.llm_call_times),
|
||||
"tool_started_total": len(Counters.tool_call_times),
|
||||
}
|
||||
|
||||
|
||||
async def scenario_context(args: argparse.Namespace) -> dict[str, Any]:
|
||||
from crewai.events.event_bus import crewai_event_bus
|
||||
from crewai.events.types.tool_usage_events import ToolUsageStartedEvent
|
||||
from crewai.events.types.llm_events import LLMCallStartedEvent
|
||||
|
||||
with crewai_event_bus.scoped_handlers():
|
||||
|
||||
@crewai_event_bus.on(ToolUsageStartedEvent)
|
||||
def _h(source: Any, event: Any) -> None:
|
||||
SEEN_CTX["handler"].add(CTX_VAR.get())
|
||||
|
||||
@crewai_event_bus.on(LLMCallStartedEvent)
|
||||
async def _ah(source: Any, event: Any) -> None:
|
||||
SEEN_CTX["handler"].add("async:" + CTX_VAR.get())
|
||||
|
||||
crew = _build_crew("tools", 0.0, 0.0, 2)
|
||||
CTX_VAR.set("set-before-akickoff")
|
||||
await crew.akickoff()
|
||||
crewai_event_bus.flush()
|
||||
return {k: sorted(v) for k, v in SEEN_CTX.items()}
|
||||
|
||||
|
||||
SCENARIOS = {
|
||||
"scaling": scenario_scaling,
|
||||
"overhead": scenario_overhead,
|
||||
"cancel": scenario_cancel,
|
||||
"context": scenario_context,
|
||||
}
|
||||
|
||||
|
||||
def run_one(args: argparse.Namespace) -> None:
|
||||
result = asyncio.run(SCENARIOS[args.one](args))
|
||||
print("RESULT_JSON " + json.dumps(result))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Driver
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
def _plan(quick: bool) -> list[tuple[str, list[str]]]:
|
||||
ns = [1, 10, 50] if quick else [1, 10, 50, 200]
|
||||
plan: list[tuple[str, list[str]]] = []
|
||||
for variant in ("plain", "tools"):
|
||||
for n in ns:
|
||||
plan.append(
|
||||
(
|
||||
f"scaling/{variant}/lat=200ms/N={n}",
|
||||
["--one", "scaling", "--variant", variant, "--n", str(n),
|
||||
"--latency", "0.2", "--tool-latency", "0.1"],
|
||||
)
|
||||
)
|
||||
for n in (1, 50):
|
||||
plan.append(
|
||||
(
|
||||
f"scaling/planning/lat=200ms/N={n}",
|
||||
["--one", "scaling", "--variant", "planning", "--n", str(n),
|
||||
"--latency", "0.2", "--tool-latency", "0.1", "--n-tools", "2"],
|
||||
)
|
||||
)
|
||||
plan.append(
|
||||
(
|
||||
"scaling/plain/lat=500ms/N=200" if not quick else "scaling/plain/lat=500ms/N=50",
|
||||
["--one", "scaling", "--variant", "plain", "--n", "50" if quick else "200",
|
||||
"--latency", "0.5"],
|
||||
)
|
||||
)
|
||||
for variant in ("plain", "tools"):
|
||||
plan.append(
|
||||
(
|
||||
f"overhead/{variant}/lat=0",
|
||||
["--one", "overhead", "--variant", variant, "--n", "30"],
|
||||
)
|
||||
)
|
||||
plan.append(
|
||||
(
|
||||
"small_executor(4)/plain/lat=200ms/N=20",
|
||||
["--one", "scaling", "--variant", "plain", "--n", "20", "--latency", "0.2",
|
||||
"--small-executor", "4"],
|
||||
)
|
||||
)
|
||||
for phase, after in (("during-llm", "0.1"), ("during-tools", "0.35")):
|
||||
plan.append(
|
||||
(
|
||||
f"cancel/{phase}/llm=200ms,tool=300ms",
|
||||
["--one", "cancel", "--variant", "tools", "--latency", "0.2",
|
||||
"--tool-latency", "0.3", "--cancel-after", after],
|
||||
)
|
||||
)
|
||||
plan.append(("context/tools", ["--one", "context"]))
|
||||
return plan
|
||||
|
||||
|
||||
def drive(args: argparse.Namespace) -> None:
|
||||
out_dir = HERE / "results"
|
||||
out_dir.mkdir(exist_ok=True)
|
||||
results: dict[str, Any] = {}
|
||||
for name, extra in _plan(args.quick):
|
||||
if args.only and not any(o in name for o in args.only.split(",")):
|
||||
continue
|
||||
t0 = time.perf_counter()
|
||||
proc = subprocess.run(
|
||||
[sys.executable, __file__, *extra],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=900,
|
||||
)
|
||||
line = next(
|
||||
(ln for ln in proc.stdout.splitlines() if ln.startswith("RESULT_JSON ")),
|
||||
None,
|
||||
)
|
||||
if line is None:
|
||||
results[name] = {"error": (proc.stderr or proc.stdout)[-2000:]}
|
||||
else:
|
||||
results[name] = json.loads(line[len("RESULT_JSON "):])
|
||||
print(f"{name:45s} {time.perf_counter() - t0:6.1f}s {results[name]}", flush=True)
|
||||
path = out_dir / f"{args.label}.json"
|
||||
path.write_text(json.dumps(results, indent=2))
|
||||
print(f"wrote {path}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--one", choices=sorted(SCENARIOS))
|
||||
p.add_argument("--label", default="run")
|
||||
p.add_argument("--quick", action="store_true")
|
||||
p.add_argument("--only", default="")
|
||||
p.add_argument("--variant", default="plain", choices=["plain", "tools", "planning"])
|
||||
p.add_argument("--n", type=int, default=1)
|
||||
p.add_argument("--latency", type=float, default=0.2)
|
||||
p.add_argument("--tool-latency", type=float, default=0.1)
|
||||
p.add_argument("--n-tools", type=int, default=4)
|
||||
p.add_argument("--small-executor", type=int, default=0)
|
||||
p.add_argument("--cancel-after", type=float, default=0.3)
|
||||
args = p.parse_args()
|
||||
if args.one:
|
||||
run_one(args)
|
||||
else:
|
||||
drive(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"cancel/during-llm/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 2.8,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 1,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 0,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 0
|
||||
},
|
||||
"cancel/during-tools/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 1.9,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 0,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 4,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 4
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"scaling/planning/lat=200ms/N=1": {
|
||||
"wall_s": 1.269,
|
||||
"throughput_kickoffs_per_s": 0.79,
|
||||
"peak_threads": 10,
|
||||
"thread_starts": 7,
|
||||
"peak_rss_mb": 179.4,
|
||||
"rss_growth_mb": 0.7,
|
||||
"sync_llm_calls": 9,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 6,
|
||||
"loop_lag_max_ms": 8.7,
|
||||
"loop_lag_p99_ms": 1.0,
|
||||
"loop_lag_mean_ms": 0.29
|
||||
},
|
||||
"scaling/planning/lat=200ms/N=50": {
|
||||
"wall_s": 16.524,
|
||||
"throughput_kickoffs_per_s": 3.03,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 362,
|
||||
"peak_rss_mb": 207.5,
|
||||
"rss_growth_mb": 16.1,
|
||||
"sync_llm_calls": 450,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 300,
|
||||
"loop_lag_max_ms": 839.6,
|
||||
"loop_lag_p99_ms": 9.9,
|
||||
"loop_lag_mean_ms": 1.3
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
{
|
||||
"scaling/plain/lat=200ms/N=1": {
|
||||
"wall_s": 0.45,
|
||||
"throughput_kickoffs_per_s": 2.22,
|
||||
"peak_threads": 6,
|
||||
"thread_starts": 8,
|
||||
"peak_rss_mb": 177.3,
|
||||
"rss_growth_mb": 0.3,
|
||||
"sync_llm_calls": 2,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 6.6,
|
||||
"loop_lag_p99_ms": 4.2,
|
||||
"loop_lag_mean_ms": 0.47
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=10": {
|
||||
"wall_s": 0.991,
|
||||
"throughput_kickoffs_per_s": 10.09,
|
||||
"peak_threads": 15,
|
||||
"thread_starts": 79,
|
||||
"peak_rss_mb": 181.7,
|
||||
"rss_growth_mb": 3.5,
|
||||
"sync_llm_calls": 20,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 166.4,
|
||||
"loop_lag_p99_ms": 14.7,
|
||||
"loop_lag_mean_ms": 3.4
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=50": {
|
||||
"wall_s": 4.226,
|
||||
"throughput_kickoffs_per_s": 11.83,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 365,
|
||||
"peak_rss_mb": 194.0,
|
||||
"rss_growth_mb": 12.2,
|
||||
"sync_llm_calls": 100,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 770.7,
|
||||
"loop_lag_p99_ms": 88.7,
|
||||
"loop_lag_mean_ms": 5.72
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=200": {
|
||||
"wall_s": 18.483,
|
||||
"throughput_kickoffs_per_s": 10.82,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 1415,
|
||||
"peak_rss_mb": 241.2,
|
||||
"rss_growth_mb": 43.1,
|
||||
"sync_llm_calls": 400,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 3510.1,
|
||||
"loop_lag_p99_ms": 72.3,
|
||||
"loop_lag_mean_ms": 8.66
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=1": {
|
||||
"wall_s": 0.766,
|
||||
"throughput_kickoffs_per_s": 1.31,
|
||||
"peak_threads": 10,
|
||||
"thread_starts": 11,
|
||||
"peak_rss_mb": 178.1,
|
||||
"rss_growth_mb": 0.6,
|
||||
"sync_llm_calls": 3,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 4,
|
||||
"loop_lag_max_ms": 16.9,
|
||||
"loop_lag_p99_ms": 4.1,
|
||||
"loop_lag_mean_ms": 0.53
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=10": {
|
||||
"wall_s": 1.254,
|
||||
"throughput_kickoffs_per_s": 7.97,
|
||||
"peak_threads": 37,
|
||||
"thread_starts": 119,
|
||||
"peak_rss_mb": 184.8,
|
||||
"rss_growth_mb": 3.7,
|
||||
"sync_llm_calls": 30,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 40,
|
||||
"loop_lag_max_ms": 178.8,
|
||||
"loop_lag_p99_ms": 21.8,
|
||||
"loop_lag_mean_ms": 3.15
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=50": {
|
||||
"wall_s": 6.221,
|
||||
"throughput_kickoffs_per_s": 8.04,
|
||||
"peak_threads": 47,
|
||||
"thread_starts": 565,
|
||||
"peak_rss_mb": 211.9,
|
||||
"rss_growth_mb": 14.2,
|
||||
"sync_llm_calls": 150,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 200,
|
||||
"loop_lag_max_ms": 1014.9,
|
||||
"loop_lag_p99_ms": 40.4,
|
||||
"loop_lag_mean_ms": 4.18
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=200": {
|
||||
"wall_s": 27.889,
|
||||
"throughput_kickoffs_per_s": 7.17,
|
||||
"peak_threads": 49,
|
||||
"thread_starts": 2215,
|
||||
"peak_rss_mb": 310.4,
|
||||
"rss_growth_mb": 53.0,
|
||||
"sync_llm_calls": 600,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 800,
|
||||
"loop_lag_max_ms": 3829.4,
|
||||
"loop_lag_p99_ms": 39.0,
|
||||
"loop_lag_mean_ms": 5.93
|
||||
},
|
||||
"scaling/plain/lat=500ms/N=200": {
|
||||
"wall_s": 35.096,
|
||||
"throughput_kickoffs_per_s": 5.7,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 1415,
|
||||
"peak_rss_mb": 241.0,
|
||||
"rss_growth_mb": 43.0,
|
||||
"sync_llm_calls": 400,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 5138.9,
|
||||
"loop_lag_p99_ms": 26.6,
|
||||
"loop_lag_mean_ms": 4.17
|
||||
},
|
||||
"overhead/plain/lat=0": {
|
||||
"per_kickoff_ms_median": 26.07,
|
||||
"per_kickoff_ms_mean": 27.3,
|
||||
"thread_starts_per_kickoff": 7.1
|
||||
},
|
||||
"overhead/tools/lat=0": {
|
||||
"per_kickoff_ms_median": 36.62,
|
||||
"per_kickoff_ms_mean": 36.67,
|
||||
"thread_starts_per_kickoff": 11.1
|
||||
},
|
||||
"small_executor(4)/plain/lat=200ms/N=20": {
|
||||
"wall_s": 2.685,
|
||||
"throughput_kickoffs_per_s": 7.45,
|
||||
"peak_threads": 15,
|
||||
"thread_starts": 149,
|
||||
"peak_rss_mb": 184.1,
|
||||
"rss_growth_mb": 4.9,
|
||||
"sync_llm_calls": 40,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 282.9,
|
||||
"loop_lag_p99_ms": 31.7,
|
||||
"loop_lag_mean_ms": 2.44
|
||||
},
|
||||
"cancel/tools/lat=200ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 2.2,
|
||||
"llm_calls_after_cancel": 0,
|
||||
"tool_calls_after_cancel": 0,
|
||||
"llm_calls_total": 1
|
||||
},
|
||||
"context/tools": {
|
||||
"tool": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"handler": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"llm": [
|
||||
"set-before-akickoff"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,183 @@
|
||||
{
|
||||
"scaling/plain/lat=200ms/N=1": {
|
||||
"wall_s": 0.442,
|
||||
"throughput_kickoffs_per_s": 2.26,
|
||||
"peak_threads": 6,
|
||||
"thread_starts": 4,
|
||||
"peak_rss_mb": 177.6,
|
||||
"rss_growth_mb": 0.3,
|
||||
"sync_llm_calls": 2,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 4.8,
|
||||
"loop_lag_p99_ms": 2.2,
|
||||
"loop_lag_mean_ms": 0.33
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=10": {
|
||||
"wall_s": 0.965,
|
||||
"throughput_kickoffs_per_s": 10.36,
|
||||
"peak_threads": 14,
|
||||
"thread_starts": 48,
|
||||
"peak_rss_mb": 181.5,
|
||||
"rss_growth_mb": 3.1,
|
||||
"sync_llm_calls": 20,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 143.4,
|
||||
"loop_lag_p99_ms": 13.6,
|
||||
"loop_lag_mean_ms": 2.62
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=50": {
|
||||
"wall_s": 4.078,
|
||||
"throughput_kickoffs_per_s": 12.26,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 215,
|
||||
"peak_rss_mb": 194.3,
|
||||
"rss_growth_mb": 12.2,
|
||||
"sync_llm_calls": 100,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 750.8,
|
||||
"loop_lag_p99_ms": 43.4,
|
||||
"loop_lag_mean_ms": 5.25
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=200": {
|
||||
"wall_s": 17.276,
|
||||
"throughput_kickoffs_per_s": 11.58,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 815,
|
||||
"peak_rss_mb": 240.4,
|
||||
"rss_growth_mb": 42.5,
|
||||
"sync_llm_calls": 400,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 2937.5,
|
||||
"loop_lag_p99_ms": 41.3,
|
||||
"loop_lag_mean_ms": 6.93
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=1": {
|
||||
"wall_s": 0.764,
|
||||
"throughput_kickoffs_per_s": 1.31,
|
||||
"peak_threads": 10,
|
||||
"thread_starts": 9,
|
||||
"peak_rss_mb": 178.3,
|
||||
"rss_growth_mb": 0.7,
|
||||
"sync_llm_calls": 3,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 4,
|
||||
"loop_lag_max_ms": 8.2,
|
||||
"loop_lag_p99_ms": 1.5,
|
||||
"loop_lag_mean_ms": 0.33
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=10": {
|
||||
"wall_s": 1.249,
|
||||
"throughput_kickoffs_per_s": 8.01,
|
||||
"peak_threads": 37,
|
||||
"thread_starts": 88,
|
||||
"peak_rss_mb": 185.1,
|
||||
"rss_growth_mb": 3.6,
|
||||
"sync_llm_calls": 30,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 40,
|
||||
"loop_lag_max_ms": 174.7,
|
||||
"loop_lag_p99_ms": 6.7,
|
||||
"loop_lag_mean_ms": 2.48
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=50": {
|
||||
"wall_s": 6.328,
|
||||
"throughput_kickoffs_per_s": 7.9,
|
||||
"peak_threads": 47,
|
||||
"thread_starts": 414,
|
||||
"peak_rss_mb": 212.2,
|
||||
"rss_growth_mb": 14.9,
|
||||
"sync_llm_calls": 150,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 200,
|
||||
"loop_lag_max_ms": 1019.6,
|
||||
"loop_lag_p99_ms": 63.6,
|
||||
"loop_lag_mean_ms": 4.48
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=200": {
|
||||
"wall_s": 27.07,
|
||||
"throughput_kickoffs_per_s": 7.39,
|
||||
"peak_threads": 48,
|
||||
"thread_starts": 1614,
|
||||
"peak_rss_mb": 310.5,
|
||||
"rss_growth_mb": 53.2,
|
||||
"sync_llm_calls": 600,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 800,
|
||||
"loop_lag_max_ms": 3733.0,
|
||||
"loop_lag_p99_ms": 37.8,
|
||||
"loop_lag_mean_ms": 5.8
|
||||
},
|
||||
"scaling/plain/lat=500ms/N=200": {
|
||||
"wall_s": 32.614,
|
||||
"throughput_kickoffs_per_s": 6.13,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 815,
|
||||
"peak_rss_mb": 240.5,
|
||||
"rss_growth_mb": 42.4,
|
||||
"sync_llm_calls": 400,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 3120.4,
|
||||
"loop_lag_p99_ms": 28.2,
|
||||
"loop_lag_mean_ms": 3.1
|
||||
},
|
||||
"overhead/plain/lat=0": {
|
||||
"per_kickoff_ms_median": 26.16,
|
||||
"per_kickoff_ms_mean": 27.46,
|
||||
"thread_starts_per_kickoff": 4.0
|
||||
},
|
||||
"overhead/tools/lat=0": {
|
||||
"per_kickoff_ms_median": 36.14,
|
||||
"per_kickoff_ms_mean": 36.74,
|
||||
"thread_starts_per_kickoff": 8.1
|
||||
},
|
||||
"small_executor(4)/plain/lat=200ms/N=20": {
|
||||
"wall_s": 2.694,
|
||||
"throughput_kickoffs_per_s": 7.43,
|
||||
"peak_threads": 14,
|
||||
"thread_starts": 88,
|
||||
"peak_rss_mb": 184.3,
|
||||
"rss_growth_mb": 5.2,
|
||||
"sync_llm_calls": 40,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 269.9,
|
||||
"loop_lag_p99_ms": 28.0,
|
||||
"loop_lag_mean_ms": 2.31
|
||||
},
|
||||
"cancel/during-llm/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 2.3,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 1,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 0,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 0
|
||||
},
|
||||
"cancel/during-tools/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 1.8,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 0,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 4,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 4
|
||||
},
|
||||
"context/tools": {
|
||||
"tool": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"handler": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"llm": [
|
||||
"set-before-akickoff"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,183 @@
|
||||
{
|
||||
"scaling/plain/lat=200ms/N=1": {
|
||||
"wall_s": 0.432,
|
||||
"throughput_kickoffs_per_s": 2.31,
|
||||
"peak_threads": 6,
|
||||
"thread_starts": 0,
|
||||
"peak_rss_mb": 177.6,
|
||||
"rss_growth_mb": 0.6,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 2,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 1.7,
|
||||
"loop_lag_p99_ms": 1.7,
|
||||
"loop_lag_mean_ms": 0.28
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=10": {
|
||||
"wall_s": 0.664,
|
||||
"throughput_kickoffs_per_s": 15.06,
|
||||
"peak_threads": 16,
|
||||
"thread_starts": 9,
|
||||
"peak_rss_mb": 181.4,
|
||||
"rss_growth_mb": 3.3,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 20,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 124.9,
|
||||
"loop_lag_p99_ms": 26.4,
|
||||
"loop_lag_mean_ms": 4.09
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=50": {
|
||||
"wall_s": 1.573,
|
||||
"throughput_kickoffs_per_s": 31.78,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 14,
|
||||
"peak_rss_mb": 194.0,
|
||||
"rss_growth_mb": 11.9,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 100,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 664.0,
|
||||
"loop_lag_p99_ms": 70.8,
|
||||
"loop_lag_mean_ms": 23.27
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=200": {
|
||||
"wall_s": 6.693,
|
||||
"throughput_kickoffs_per_s": 29.88,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 14,
|
||||
"peak_rss_mb": 241.4,
|
||||
"rss_growth_mb": 43.3,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 400,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 2846.5,
|
||||
"loop_lag_p99_ms": 498.7,
|
||||
"loop_lag_mean_ms": 212.83
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=1": {
|
||||
"wall_s": 0.765,
|
||||
"throughput_kickoffs_per_s": 1.31,
|
||||
"peak_threads": 8,
|
||||
"thread_starts": 1,
|
||||
"peak_rss_mb": 178.2,
|
||||
"rss_growth_mb": 0.6,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 3,
|
||||
"tool_calls": 4,
|
||||
"loop_lag_max_ms": 8.3,
|
||||
"loop_lag_p99_ms": 4.6,
|
||||
"loop_lag_mean_ms": 0.41
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=10": {
|
||||
"wall_s": 0.967,
|
||||
"throughput_kickoffs_per_s": 10.34,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 13,
|
||||
"peak_rss_mb": 184.7,
|
||||
"rss_growth_mb": 3.4,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 30,
|
||||
"tool_calls": 40,
|
||||
"loop_lag_max_ms": 140.2,
|
||||
"loop_lag_p99_ms": 15.8,
|
||||
"loop_lag_mean_ms": 2.61
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=50": {
|
||||
"wall_s": 2.218,
|
||||
"throughput_kickoffs_per_s": 22.54,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 13,
|
||||
"peak_rss_mb": 211.2,
|
||||
"rss_growth_mb": 13.6,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 150,
|
||||
"tool_calls": 200,
|
||||
"loop_lag_max_ms": 932.9,
|
||||
"loop_lag_p99_ms": 121.6,
|
||||
"loop_lag_mean_ms": 22.97
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=200": {
|
||||
"wall_s": 8.899,
|
||||
"throughput_kickoffs_per_s": 22.47,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 13,
|
||||
"peak_rss_mb": 310.1,
|
||||
"rss_growth_mb": 52.7,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 600,
|
||||
"tool_calls": 800,
|
||||
"loop_lag_max_ms": 3455.4,
|
||||
"loop_lag_p99_ms": 447.8,
|
||||
"loop_lag_mean_ms": 230.12
|
||||
},
|
||||
"scaling/plain/lat=500ms/N=200": {
|
||||
"wall_s": 7.101,
|
||||
"throughput_kickoffs_per_s": 28.17,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 14,
|
||||
"peak_rss_mb": 240.3,
|
||||
"rss_growth_mb": 42.0,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 400,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 2969.3,
|
||||
"loop_lag_p99_ms": 744.5,
|
||||
"loop_lag_mean_ms": 74.47
|
||||
},
|
||||
"overhead/plain/lat=0": {
|
||||
"per_kickoff_ms_median": 19.65,
|
||||
"per_kickoff_ms_mean": 19.99,
|
||||
"thread_starts_per_kickoff": 0.0
|
||||
},
|
||||
"overhead/tools/lat=0": {
|
||||
"per_kickoff_ms_median": 21.97,
|
||||
"per_kickoff_ms_mean": 22.09,
|
||||
"thread_starts_per_kickoff": 0.2
|
||||
},
|
||||
"small_executor(4)/plain/lat=200ms/N=20": {
|
||||
"wall_s": 0.922,
|
||||
"throughput_kickoffs_per_s": 21.68,
|
||||
"peak_threads": 16,
|
||||
"thread_starts": 10,
|
||||
"peak_rss_mb": 184.5,
|
||||
"rss_growth_mb": 5.8,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 40,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 262.7,
|
||||
"loop_lag_p99_ms": 53.0,
|
||||
"loop_lag_mean_ms": 9.1
|
||||
},
|
||||
"cancel/during-llm/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 0.2,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 0,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 0,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 0
|
||||
},
|
||||
"cancel/during-tools/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 0.4,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 0,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 0,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 4
|
||||
},
|
||||
"context/tools": {
|
||||
"tool": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"handler": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"llm": [
|
||||
"set-before-akickoff"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,183 @@
|
||||
{
|
||||
"scaling/plain/lat=200ms/N=1": {
|
||||
"wall_s": 0.459,
|
||||
"throughput_kickoffs_per_s": 2.18,
|
||||
"peak_threads": 6,
|
||||
"thread_starts": 5,
|
||||
"peak_rss_mb": 177.5,
|
||||
"rss_growth_mb": 0.5,
|
||||
"sync_llm_calls": 2,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 14.5,
|
||||
"loop_lag_p99_ms": 4.6,
|
||||
"loop_lag_mean_ms": 0.66
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=10": {
|
||||
"wall_s": 0.974,
|
||||
"throughput_kickoffs_per_s": 10.26,
|
||||
"peak_threads": 16,
|
||||
"thread_starts": 50,
|
||||
"peak_rss_mb": 181.6,
|
||||
"rss_growth_mb": 3.3,
|
||||
"sync_llm_calls": 20,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 164.9,
|
||||
"loop_lag_p99_ms": 12.9,
|
||||
"loop_lag_mean_ms": 3.03
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=50": {
|
||||
"wall_s": 4.103,
|
||||
"throughput_kickoffs_per_s": 12.19,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 215,
|
||||
"peak_rss_mb": 194.6,
|
||||
"rss_growth_mb": 12.5,
|
||||
"sync_llm_calls": 100,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 685.9,
|
||||
"loop_lag_p99_ms": 77.9,
|
||||
"loop_lag_mean_ms": 4.94
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=200": {
|
||||
"wall_s": 17.581,
|
||||
"throughput_kickoffs_per_s": 11.38,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 815,
|
||||
"peak_rss_mb": 242.5,
|
||||
"rss_growth_mb": 44.5,
|
||||
"sync_llm_calls": 400,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 3145.8,
|
||||
"loop_lag_p99_ms": 43.4,
|
||||
"loop_lag_mean_ms": 7.45
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=1": {
|
||||
"wall_s": 0.757,
|
||||
"throughput_kickoffs_per_s": 1.32,
|
||||
"peak_threads": 11,
|
||||
"thread_starts": 9,
|
||||
"peak_rss_mb": 178.2,
|
||||
"rss_growth_mb": 0.6,
|
||||
"sync_llm_calls": 3,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 4,
|
||||
"loop_lag_max_ms": 6.0,
|
||||
"loop_lag_p99_ms": 3.8,
|
||||
"loop_lag_mean_ms": 0.39
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=10": {
|
||||
"wall_s": 1.213,
|
||||
"throughput_kickoffs_per_s": 8.24,
|
||||
"peak_threads": 38,
|
||||
"thread_starts": 88,
|
||||
"peak_rss_mb": 185.2,
|
||||
"rss_growth_mb": 4.0,
|
||||
"sync_llm_calls": 30,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 40,
|
||||
"loop_lag_max_ms": 168.9,
|
||||
"loop_lag_p99_ms": 6.1,
|
||||
"loop_lag_mean_ms": 2.5
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=50": {
|
||||
"wall_s": 6.159,
|
||||
"throughput_kickoffs_per_s": 8.12,
|
||||
"peak_threads": 48,
|
||||
"thread_starts": 414,
|
||||
"peak_rss_mb": 211.9,
|
||||
"rss_growth_mb": 14.4,
|
||||
"sync_llm_calls": 150,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 200,
|
||||
"loop_lag_max_ms": 934.3,
|
||||
"loop_lag_p99_ms": 37.1,
|
||||
"loop_lag_mean_ms": 4.04
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=200": {
|
||||
"wall_s": 26.962,
|
||||
"throughput_kickoffs_per_s": 7.42,
|
||||
"peak_threads": 49,
|
||||
"thread_starts": 1614,
|
||||
"peak_rss_mb": 313.2,
|
||||
"rss_growth_mb": 55.8,
|
||||
"sync_llm_calls": 600,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 800,
|
||||
"loop_lag_max_ms": 3484.5,
|
||||
"loop_lag_p99_ms": 41.1,
|
||||
"loop_lag_mean_ms": 5.64
|
||||
},
|
||||
"scaling/plain/lat=500ms/N=200": {
|
||||
"wall_s": 32.808,
|
||||
"throughput_kickoffs_per_s": 6.1,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 815,
|
||||
"peak_rss_mb": 247.3,
|
||||
"rss_growth_mb": 49.4,
|
||||
"sync_llm_calls": 400,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 3058.2,
|
||||
"loop_lag_p99_ms": 31.7,
|
||||
"loop_lag_mean_ms": 3.22
|
||||
},
|
||||
"overhead/plain/lat=0": {
|
||||
"per_kickoff_ms_median": 25.57,
|
||||
"per_kickoff_ms_mean": 25.87,
|
||||
"thread_starts_per_kickoff": 4.0
|
||||
},
|
||||
"overhead/tools/lat=0": {
|
||||
"per_kickoff_ms_median": 37.35,
|
||||
"per_kickoff_ms_mean": 38.17,
|
||||
"thread_starts_per_kickoff": 8.0
|
||||
},
|
||||
"small_executor(4)/plain/lat=200ms/N=20": {
|
||||
"wall_s": 2.691,
|
||||
"throughput_kickoffs_per_s": 7.43,
|
||||
"peak_threads": 15,
|
||||
"thread_starts": 88,
|
||||
"peak_rss_mb": 184.7,
|
||||
"rss_growth_mb": 5.4,
|
||||
"sync_llm_calls": 40,
|
||||
"async_llm_calls": 0,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 258.8,
|
||||
"loop_lag_p99_ms": 21.4,
|
||||
"loop_lag_mean_ms": 2.28
|
||||
},
|
||||
"cancel/during-llm/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 1.3,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 1,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 0,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 0
|
||||
},
|
||||
"cancel/during-tools/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 1.3,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 0,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 4,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 4
|
||||
},
|
||||
"context/tools": {
|
||||
"tool": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"handler": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"llm": [
|
||||
"set-before-akickoff"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,183 @@
|
||||
{
|
||||
"scaling/plain/lat=200ms/N=1": {
|
||||
"wall_s": 0.438,
|
||||
"throughput_kickoffs_per_s": 2.28,
|
||||
"peak_threads": 6,
|
||||
"thread_starts": 4,
|
||||
"peak_rss_mb": 177.5,
|
||||
"rss_growth_mb": 0.7,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 2,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 4.2,
|
||||
"loop_lag_p99_ms": 4.0,
|
||||
"loop_lag_mean_ms": 0.43
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=10": {
|
||||
"wall_s": 0.681,
|
||||
"throughput_kickoffs_per_s": 14.69,
|
||||
"peak_threads": 18,
|
||||
"thread_starts": 52,
|
||||
"peak_rss_mb": 181.4,
|
||||
"rss_growth_mb": 3.0,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 20,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 131.0,
|
||||
"loop_lag_p99_ms": 24.9,
|
||||
"loop_lag_mean_ms": 4.44
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=50": {
|
||||
"wall_s": 1.8,
|
||||
"throughput_kickoffs_per_s": 27.77,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 215,
|
||||
"peak_rss_mb": 195.0,
|
||||
"rss_growth_mb": 12.8,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 100,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 758.3,
|
||||
"loop_lag_p99_ms": 144.1,
|
||||
"loop_lag_mean_ms": 31.7
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=200": {
|
||||
"wall_s": 7.568,
|
||||
"throughput_kickoffs_per_s": 26.43,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 815,
|
||||
"peak_rss_mb": 241.7,
|
||||
"rss_growth_mb": 43.4,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 400,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 3162.5,
|
||||
"loop_lag_p99_ms": 760.5,
|
||||
"loop_lag_mean_ms": 132.68
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=1": {
|
||||
"wall_s": 0.759,
|
||||
"throughput_kickoffs_per_s": 1.32,
|
||||
"peak_threads": 8,
|
||||
"thread_starts": 4,
|
||||
"peak_rss_mb": 178.2,
|
||||
"rss_growth_mb": 0.7,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 3,
|
||||
"tool_calls": 4,
|
||||
"loop_lag_max_ms": 7.8,
|
||||
"loop_lag_p99_ms": 6.0,
|
||||
"loop_lag_mean_ms": 0.53
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=10": {
|
||||
"wall_s": 1.023,
|
||||
"throughput_kickoffs_per_s": 9.78,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 51,
|
||||
"peak_rss_mb": 185.1,
|
||||
"rss_growth_mb": 3.8,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 30,
|
||||
"tool_calls": 40,
|
||||
"loop_lag_max_ms": 181.3,
|
||||
"loop_lag_p99_ms": 13.8,
|
||||
"loop_lag_mean_ms": 3.69
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=50": {
|
||||
"wall_s": 2.404,
|
||||
"throughput_kickoffs_per_s": 20.8,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 213,
|
||||
"peak_rss_mb": 211.8,
|
||||
"rss_growth_mb": 14.0,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 150,
|
||||
"tool_calls": 200,
|
||||
"loop_lag_max_ms": 972.4,
|
||||
"loop_lag_p99_ms": 159.1,
|
||||
"loop_lag_mean_ms": 24.28
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=200": {
|
||||
"wall_s": 9.344,
|
||||
"throughput_kickoffs_per_s": 21.4,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 813,
|
||||
"peak_rss_mb": 310.3,
|
||||
"rss_growth_mb": 53.0,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 600,
|
||||
"tool_calls": 800,
|
||||
"loop_lag_max_ms": 3477.1,
|
||||
"loop_lag_p99_ms": 1171.6,
|
||||
"loop_lag_mean_ms": 249.45
|
||||
},
|
||||
"scaling/plain/lat=500ms/N=200": {
|
||||
"wall_s": 7.704,
|
||||
"throughput_kickoffs_per_s": 25.96,
|
||||
"peak_threads": 21,
|
||||
"thread_starts": 814,
|
||||
"peak_rss_mb": 240.7,
|
||||
"rss_growth_mb": 42.6,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 400,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 3141.5,
|
||||
"loop_lag_p99_ms": 485.4,
|
||||
"loop_lag_mean_ms": 92.71
|
||||
},
|
||||
"overhead/plain/lat=0": {
|
||||
"per_kickoff_ms_median": 20.97,
|
||||
"per_kickoff_ms_mean": 21.28,
|
||||
"thread_starts_per_kickoff": 4.0
|
||||
},
|
||||
"overhead/tools/lat=0": {
|
||||
"per_kickoff_ms_median": 25.59,
|
||||
"per_kickoff_ms_mean": 26.21,
|
||||
"thread_starts_per_kickoff": 4.2
|
||||
},
|
||||
"small_executor(4)/plain/lat=200ms/N=20": {
|
||||
"wall_s": 0.924,
|
||||
"throughput_kickoffs_per_s": 21.64,
|
||||
"peak_threads": 15,
|
||||
"thread_starts": 89,
|
||||
"peak_rss_mb": 184.3,
|
||||
"rss_growth_mb": 5.4,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 40,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 275.3,
|
||||
"loop_lag_p99_ms": 41.5,
|
||||
"loop_lag_mean_ms": 9.25
|
||||
},
|
||||
"cancel/during-llm/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 1.3,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 0,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 0,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 0
|
||||
},
|
||||
"cancel/during-tools/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 1.5,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 0,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 0,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 4
|
||||
},
|
||||
"context/tools": {
|
||||
"tool": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"handler": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"llm": [
|
||||
"set-before-akickoff"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"scaling/planning/lat=200ms/N=1": {
|
||||
"wall_s": 1.247,
|
||||
"throughput_kickoffs_per_s": 0.8,
|
||||
"peak_threads": 8,
|
||||
"thread_starts": 0,
|
||||
"peak_rss_mb": 181.5,
|
||||
"rss_growth_mb": 0.3,
|
||||
"sync_llm_calls": 8,
|
||||
"async_llm_calls": 1,
|
||||
"tool_calls": 6,
|
||||
"loop_lag_max_ms": 4.5,
|
||||
"loop_lag_p99_ms": 1.5,
|
||||
"loop_lag_mean_ms": 0.26
|
||||
},
|
||||
"scaling/planning/lat=200ms/N=50": {
|
||||
"wall_s": 15.233,
|
||||
"throughput_kickoffs_per_s": 3.28,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 12,
|
||||
"peak_rss_mb": 207.4,
|
||||
"rss_growth_mb": 16.0,
|
||||
"sync_llm_calls": 400,
|
||||
"async_llm_calls": 50,
|
||||
"tool_calls": 300,
|
||||
"loop_lag_max_ms": 757.0,
|
||||
"loop_lag_p99_ms": 9.6,
|
||||
"loop_lag_mean_ms": 1.22
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,211 @@
|
||||
{
|
||||
"scaling/plain/lat=200ms/N=1": {
|
||||
"wall_s": 0.439,
|
||||
"throughput_kickoffs_per_s": 2.28,
|
||||
"peak_threads": 6,
|
||||
"thread_starts": 0,
|
||||
"peak_rss_mb": 179.1,
|
||||
"rss_growth_mb": 0.4,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 2,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 8.0,
|
||||
"loop_lag_p99_ms": 1.6,
|
||||
"loop_lag_mean_ms": 0.44
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=10": {
|
||||
"wall_s": 0.634,
|
||||
"throughput_kickoffs_per_s": 15.78,
|
||||
"peak_threads": 17,
|
||||
"thread_starts": 12,
|
||||
"peak_rss_mb": 182.5,
|
||||
"rss_growth_mb": 3.1,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 20,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 116.3,
|
||||
"loop_lag_p99_ms": 20.1,
|
||||
"loop_lag_mean_ms": 3.57
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=50": {
|
||||
"wall_s": 1.649,
|
||||
"throughput_kickoffs_per_s": 30.32,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 14,
|
||||
"peak_rss_mb": 195.2,
|
||||
"rss_growth_mb": 11.7,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 100,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 643.8,
|
||||
"loop_lag_p99_ms": 112.6,
|
||||
"loop_lag_mean_ms": 25.83
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=200": {
|
||||
"wall_s": 6.853,
|
||||
"throughput_kickoffs_per_s": 29.18,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 14,
|
||||
"peak_rss_mb": 243.0,
|
||||
"rss_growth_mb": 43.5,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 400,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 2950.2,
|
||||
"loop_lag_p99_ms": 498.9,
|
||||
"loop_lag_mean_ms": 152.99
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=1": {
|
||||
"wall_s": 0.741,
|
||||
"throughput_kickoffs_per_s": 1.35,
|
||||
"peak_threads": 8,
|
||||
"thread_starts": 0,
|
||||
"peak_rss_mb": 179.5,
|
||||
"rss_growth_mb": 0.8,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 3,
|
||||
"tool_calls": 4,
|
||||
"loop_lag_max_ms": 5.1,
|
||||
"loop_lag_p99_ms": 1.4,
|
||||
"loop_lag_mean_ms": 0.29
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=10": {
|
||||
"wall_s": 0.976,
|
||||
"throughput_kickoffs_per_s": 10.24,
|
||||
"peak_threads": 19,
|
||||
"thread_starts": 11,
|
||||
"peak_rss_mb": 186.1,
|
||||
"rss_growth_mb": 3.4,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 30,
|
||||
"tool_calls": 40,
|
||||
"loop_lag_max_ms": 140.5,
|
||||
"loop_lag_p99_ms": 8.2,
|
||||
"loop_lag_mean_ms": 2.67
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=50": {
|
||||
"wall_s": 2.167,
|
||||
"throughput_kickoffs_per_s": 23.07,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 13,
|
||||
"peak_rss_mb": 212.7,
|
||||
"rss_growth_mb": 13.9,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 150,
|
||||
"tool_calls": 200,
|
||||
"loop_lag_max_ms": 890.1,
|
||||
"loop_lag_p99_ms": 69.6,
|
||||
"loop_lag_mean_ms": 22.28
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=200": {
|
||||
"wall_s": 9.222,
|
||||
"throughput_kickoffs_per_s": 21.69,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 13,
|
||||
"peak_rss_mb": 311.7,
|
||||
"rss_growth_mb": 53.1,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 600,
|
||||
"tool_calls": 800,
|
||||
"loop_lag_max_ms": 3511.6,
|
||||
"loop_lag_p99_ms": 599.4,
|
||||
"loop_lag_mean_ms": 151.7
|
||||
},
|
||||
"scaling/planning/lat=200ms/N=1": {
|
||||
"wall_s": 1.243,
|
||||
"throughput_kickoffs_per_s": 0.8,
|
||||
"peak_threads": 10,
|
||||
"thread_starts": 2,
|
||||
"peak_rss_mb": 180.2,
|
||||
"rss_growth_mb": 0.0,
|
||||
"sync_llm_calls": 2,
|
||||
"async_llm_calls": 7,
|
||||
"tool_calls": 6,
|
||||
"loop_lag_max_ms": 6.1,
|
||||
"loop_lag_p99_ms": 1.0,
|
||||
"loop_lag_mean_ms": 0.27
|
||||
},
|
||||
"scaling/planning/lat=200ms/N=50": {
|
||||
"wall_s": 4.695,
|
||||
"throughput_kickoffs_per_s": 10.65,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 12,
|
||||
"peak_rss_mb": 206.3,
|
||||
"rss_growth_mb": 15.0,
|
||||
"sync_llm_calls": 100,
|
||||
"async_llm_calls": 350,
|
||||
"tool_calls": 300,
|
||||
"loop_lag_max_ms": 794.3,
|
||||
"loop_lag_p99_ms": 48.2,
|
||||
"loop_lag_mean_ms": 5.04
|
||||
},
|
||||
"scaling/plain/lat=500ms/N=200": {
|
||||
"wall_s": 7.172,
|
||||
"throughput_kickoffs_per_s": 27.89,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 14,
|
||||
"peak_rss_mb": 242.8,
|
||||
"rss_growth_mb": 43.3,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 400,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 2972.6,
|
||||
"loop_lag_p99_ms": 628.1,
|
||||
"loop_lag_mean_ms": 96.88
|
||||
},
|
||||
"overhead/plain/lat=0": {
|
||||
"per_kickoff_ms_median": 19.55,
|
||||
"per_kickoff_ms_mean": 20.42,
|
||||
"thread_starts_per_kickoff": 0.1
|
||||
},
|
||||
"overhead/tools/lat=0": {
|
||||
"per_kickoff_ms_median": 24.87,
|
||||
"per_kickoff_ms_mean": 25.16,
|
||||
"thread_starts_per_kickoff": 0.2
|
||||
},
|
||||
"small_executor(4)/plain/lat=200ms/N=20": {
|
||||
"wall_s": 0.912,
|
||||
"throughput_kickoffs_per_s": 21.93,
|
||||
"peak_threads": 16,
|
||||
"thread_starts": 9,
|
||||
"peak_rss_mb": 186.1,
|
||||
"rss_growth_mb": 5.9,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 40,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 270.3,
|
||||
"loop_lag_p99_ms": 43.8,
|
||||
"loop_lag_mean_ms": 9.76
|
||||
},
|
||||
"cancel/during-llm/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 0.2,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 0,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 0,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 0
|
||||
},
|
||||
"cancel/during-tools/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 0.3,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 0,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 0,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 4
|
||||
},
|
||||
"context/tools": {
|
||||
"tool": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"handler": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"llm": [
|
||||
"set-before-akickoff"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,211 @@
|
||||
{
|
||||
"scaling/plain/lat=200ms/N=1": {
|
||||
"wall_s": 0.439,
|
||||
"throughput_kickoffs_per_s": 2.28,
|
||||
"peak_threads": 5,
|
||||
"thread_starts": 0,
|
||||
"peak_rss_mb": 179.2,
|
||||
"rss_growth_mb": 0.7,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 2,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 8.7,
|
||||
"loop_lag_p99_ms": 1.3,
|
||||
"loop_lag_mean_ms": 0.44
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=10": {
|
||||
"wall_s": 1.867,
|
||||
"throughput_kickoffs_per_s": 5.36,
|
||||
"peak_threads": 15,
|
||||
"thread_starts": 10,
|
||||
"peak_rss_mb": 184.2,
|
||||
"rss_growth_mb": 4.5,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 20,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 141.7,
|
||||
"loop_lag_p99_ms": 4.3,
|
||||
"loop_lag_mean_ms": 1.24
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=50": {
|
||||
"wall_s": 2.238,
|
||||
"throughput_kickoffs_per_s": 22.34,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 14,
|
||||
"peak_rss_mb": 197.8,
|
||||
"rss_growth_mb": 14.2,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 100,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 683.4,
|
||||
"loop_lag_p99_ms": 13.9,
|
||||
"loop_lag_mean_ms": 5.96
|
||||
},
|
||||
"scaling/plain/lat=200ms/N=200": {
|
||||
"wall_s": 7.386,
|
||||
"throughput_kickoffs_per_s": 27.08,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 15,
|
||||
"peak_rss_mb": 252.5,
|
||||
"rss_growth_mb": 52.9,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 400,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 2919.9,
|
||||
"loop_lag_p99_ms": 97.2,
|
||||
"loop_lag_mean_ms": 14.05
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=1": {
|
||||
"wall_s": 0.745,
|
||||
"throughput_kickoffs_per_s": 1.34,
|
||||
"peak_threads": 8,
|
||||
"thread_starts": 1,
|
||||
"peak_rss_mb": 179.9,
|
||||
"rss_growth_mb": 1.0,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 3,
|
||||
"tool_calls": 4,
|
||||
"loop_lag_max_ms": 6.9,
|
||||
"loop_lag_p99_ms": 1.2,
|
||||
"loop_lag_mean_ms": 0.32
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=10": {
|
||||
"wall_s": 1.69,
|
||||
"throughput_kickoffs_per_s": 5.92,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 13,
|
||||
"peak_rss_mb": 187.8,
|
||||
"rss_growth_mb": 5.0,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 30,
|
||||
"tool_calls": 40,
|
||||
"loop_lag_max_ms": 152.1,
|
||||
"loop_lag_p99_ms": 2.9,
|
||||
"loop_lag_mean_ms": 1.41
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=50": {
|
||||
"wall_s": 3.022,
|
||||
"throughput_kickoffs_per_s": 16.54,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 13,
|
||||
"peak_rss_mb": 217.0,
|
||||
"rss_growth_mb": 18.1,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 150,
|
||||
"tool_calls": 200,
|
||||
"loop_lag_max_ms": 918.6,
|
||||
"loop_lag_p99_ms": 12.7,
|
||||
"loop_lag_mean_ms": 6.38
|
||||
},
|
||||
"scaling/tools/lat=200ms/N=200": {
|
||||
"wall_s": 17.212,
|
||||
"throughput_kickoffs_per_s": 11.62,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 13,
|
||||
"peak_rss_mb": 328.7,
|
||||
"rss_growth_mb": 69.9,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 600,
|
||||
"tool_calls": 800,
|
||||
"loop_lag_max_ms": 3556.0,
|
||||
"loop_lag_p99_ms": 70.1,
|
||||
"loop_lag_mean_ms": 6.48
|
||||
},
|
||||
"scaling/planning/lat=200ms/N=1": {
|
||||
"wall_s": 1.246,
|
||||
"throughput_kickoffs_per_s": 0.8,
|
||||
"peak_threads": 9,
|
||||
"thread_starts": 2,
|
||||
"peak_rss_mb": 179.8,
|
||||
"rss_growth_mb": 0.7,
|
||||
"sync_llm_calls": 2,
|
||||
"async_llm_calls": 7,
|
||||
"tool_calls": 6,
|
||||
"loop_lag_max_ms": 3.7,
|
||||
"loop_lag_p99_ms": 1.5,
|
||||
"loop_lag_mean_ms": 0.29
|
||||
},
|
||||
"scaling/planning/lat=200ms/N=50": {
|
||||
"wall_s": 5.296,
|
||||
"throughput_kickoffs_per_s": 9.44,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 12,
|
||||
"peak_rss_mb": 209.4,
|
||||
"rss_growth_mb": 17.8,
|
||||
"sync_llm_calls": 100,
|
||||
"async_llm_calls": 350,
|
||||
"tool_calls": 300,
|
||||
"loop_lag_max_ms": 788.8,
|
||||
"loop_lag_p99_ms": 12.7,
|
||||
"loop_lag_mean_ms": 3.12
|
||||
},
|
||||
"scaling/plain/lat=500ms/N=200": {
|
||||
"wall_s": 8.103,
|
||||
"throughput_kickoffs_per_s": 24.68,
|
||||
"peak_threads": 20,
|
||||
"thread_starts": 15,
|
||||
"peak_rss_mb": 254.8,
|
||||
"rss_growth_mb": 55.2,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 400,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 2883.4,
|
||||
"loop_lag_p99_ms": 121.9,
|
||||
"loop_lag_mean_ms": 12.23
|
||||
},
|
||||
"overhead/plain/lat=0": {
|
||||
"per_kickoff_ms_median": 16.96,
|
||||
"per_kickoff_ms_mean": 17.48,
|
||||
"thread_starts_per_kickoff": 0.1
|
||||
},
|
||||
"overhead/tools/lat=0": {
|
||||
"per_kickoff_ms_median": 23.0,
|
||||
"per_kickoff_ms_mean": 23.29,
|
||||
"thread_starts_per_kickoff": 0.2
|
||||
},
|
||||
"small_executor(4)/plain/lat=200ms/N=20": {
|
||||
"wall_s": 1.526,
|
||||
"throughput_kickoffs_per_s": 13.11,
|
||||
"peak_threads": 14,
|
||||
"thread_starts": 8,
|
||||
"peak_rss_mb": 186.7,
|
||||
"rss_growth_mb": 6.2,
|
||||
"sync_llm_calls": 0,
|
||||
"async_llm_calls": 40,
|
||||
"tool_calls": 0,
|
||||
"loop_lag_max_ms": 256.7,
|
||||
"loop_lag_p99_ms": 6.1,
|
||||
"loop_lag_mean_ms": 2.66
|
||||
},
|
||||
"cancel/during-llm/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 0.4,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 0,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 0,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 0
|
||||
},
|
||||
"cancel/during-tools/llm=200ms,tool=300ms": {
|
||||
"cancel_outcome": "CancelledError",
|
||||
"await_returned_after_ms": 0.4,
|
||||
"llm_started_after_cancel": 0,
|
||||
"llm_finished_after_cancel": 0,
|
||||
"tool_started_after_cancel": 0,
|
||||
"tool_finished_after_cancel": 0,
|
||||
"llm_started_total": 1,
|
||||
"tool_started_total": 4
|
||||
},
|
||||
"context/tools": {
|
||||
"tool": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"handler": [
|
||||
"set-before-akickoff"
|
||||
],
|
||||
"llm": [
|
||||
"set-before-akickoff"
|
||||
]
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user