diff --git a/ASYNC_PERF_REPORT.md b/ASYNC_PERF_REPORT.md new file mode 100644 index 000000000..1a58e47a8 --- /dev/null +++ b/ASYNC_PERF_REPORT.md @@ -0,0 +1,285 @@ +# Async execution path: is going thread-free worth it on CPython? + +Throwaway investigation. Prototype commits are on this branch; nothing here is +meant to merge as-is. + +## TL;DR + +**Recommendation: partial go.** The thread hops listed in the brief (file store, +event bus, parallel tool pool, planning `to_thread`) are worth almost nothing +on their own. But the investigation found a much bigger problem the brief +didn't list: **`Crew.akickoff` with the default executor never calls +`LLM.acall`.** `AgentExecutor` is a `Flow` whose steps are sync methods, and +`Flow` runs every sync method via `asyncio.to_thread`. So every LLM call is a +blocking `LLM.call` on the loop's *default executor*. The number of concurrent +LLM calls across all `akickoff`s in a process is therefore capped at +`min(32, cpu+4)` (8 on the 4-core benchmark box). Fixing that one thing gives: + +| 200 concurrent `akickoff` (2 agents, 2 tasks) | baseline | prototype | +|---|---|---| +| LLM latency 200 ms, wall time | 18.5 s | **6.9 s** (2.7×) | +| LLM latency 500 ms, wall time | 35.1 s | **7.2 s** (4.9×) | +| + 4 parallel async tools, 200 ms | 27.9 s | **9.2 s** (3.0×) | +| 50 crews with planning (3 parallel steps) | 16.5 s | **4.7 s** (3.5×) | +| 20 crews, default executor shrunk to 4 workers | 2.69 s | **0.91 s** | +| Cancel mid-run: LLM/tool calls that keep running | 1 LLM / 4 tools | **0 / 0** | +| Threads started per kickoff | 7–11 | **0** | + +Two things are **not** worth doing as proposed: + +- **Event bus rework.** Measured no effect. On the `akickoff` path, `aemit` is + never used; `emit()` hands off to a long-lived 10-thread pool and doesn't + wait. The double hop the brief describes is only taken for handlers + registered with `depends_on`, and it happens on the bus's private loop, not + the caller's. +- **Moving the task-output SQLite log off the loop (P6).** It improves loop lag + but halves throughput at high concurrency. + +The caveat: at small concurrency (1–10 crews) with real LLM latency the user +won't notice anything except cancellation. The wins appear once the number of +in-flight LLM calls exceeds the default executor size, i.e. ~8–32 concurrent +calls per process. That threshold is easy to reach for a server that runs +`akickoff` per request. + +## Setup + +- CPython 3.13.16, 4 vCPU (default executor = 8 workers), `uv sync --all-packages`. +- Harness: `lib/crewai/tests/perf/bench_async_path.py`, one command: + `uv run python lib/crewai/tests/perf/bench_async_path.py --label ` + (`--quick` / `--only ` available). Each scenario runs in its own + subprocess. Raw JSON for every run is in `lib/crewai/tests/perf/results/`. +- Mock `BaseLLM` subclass: `call` does `time.sleep(latency)`, `acall` does + `asyncio.sleep(latency)`; both count invocations, so the harness reports which + path crewai actually took. The "tools" variant returns 4 native tool calls to + async `BaseTool`s (`_arun` = `asyncio.sleep(100 ms)`), then a final answer. + The "planning" variant uses `planning=True` with a 3-step, dependency-free plan. +- Metrics: wall time, throughput, peak `threading.active_count()` (2 ms + sampler), `Thread.start` calls (patched), peak RSS (`ru_maxrss`), event-loop + lag (drift of a 10 ms `asyncio.sleep` probe). +- Telemetry/tracing disabled; `verbose=False`. + +## Thread hops actually on the `akickoff` path (baseline) + +From the brief, confirmed: + +1. File store `_run_sync`: new `ThreadPoolExecutor` + `asyncio.run` per call. + Not 2 calls per kickoff but **~7**: `Crew._prepare_tools`, `Task.prompt()` + and `Task._aexecute_core`'s `finally` also hit it, every task, even when no + files were ever stored. +2. Event bus: see above. Measured as not mattering. +3. Planning steps: `asyncio.to_thread(step_executor.execute)`, plus + `execute_todo_sequential` as a sync Flow step. +4. Parallel native tools: a fresh `ThreadPoolExecutor(≤8)` per batch. Inside + it, every *async* tool is run via `BaseTool.run` → `asyncio.run(...)`: a new + event loop per tool call. +5. `Crew.kickoff_async` = `to_thread(kickoff)`: noted, not changed. + +Not in the brief, and the ones that matter: + +6. **Every `AgentExecutor` step is a sync Flow method → `asyncio.to_thread`**, + including `call_llm_native_tools` / `call_llm_and_parse`, which call the + *sync* `get_llm_response` → `LLM.call`. Every pure routing step + (`check_max_iterations`, `route_by_answer_type`, …) also takes a hop, and + every step takes a second hop for `_persist_method_completion` even when + persistence is off. This is the default-executor cap. +7. `_create_crew_output` calls the blocking `crewai_event_bus.flush()` **on the + loop**, and it waits on handler futures from *all* concurrent crews. +8. `_store_execution_log` (every task) and `TaskOutputStorageHandler.reset` + (every kickoff) do a SQLite transaction behind a cross-process file lock + (`store_lock`) **on the loop**. In a cProfile of the prototype this is + ~60% of per-kickoff wall time at 0 ms LLM latency. + +## Prototype changes (one commit each) + +| id | change | files | +|---|---|---| +| P1 | `akickoff` uses `aget_all_files` (prefetched into `_prepare_tools`) and `aclear_files` | `crew.py`, `crews/utils.py` | +| P2 | `aflush()` on the bus; `akickoff` awaits it instead of the blocking `flush()`; `_emit_with_dependencies` awaits `wrap_future` (1 hop, not 2) | `event_bus.py`, `crew.py` | +| P3+P4 | `Flow._async_step_override` hook. `AgentExecutor` (only when entered via `invoke_async`) runs async twins of the LLM steps (`acall`, falling back to `to_thread(call)` for LLMs without `acall`) and of `execute_native_tool` (async tools awaited and `asyncio.gather`ed, sync tools `to_thread`). Pure routing steps run inline. Persist hop skipped when persistence is off. Single-call lifecycle refactored into a generator that yields at the one blocking point, so sync and async share one body | `flow/runtime/__init__.py`, `experimental/agent_executor.py`, `utilities/agent_utils.py` | +| P1b | File store remembers which ids ever stored files; lookups and clears for other ids return immediately (benefits the sync path too) | `utilities/file_store.py` | +| P5 | `StepExecutor` converted to the same generator pattern: `execute()` drives it synchronously, `aexecute()` awaits `acall` / async tools. Sequential and parallel plan steps use it on the async path | `agents/step_executor.py`, `experimental/agent_executor.py`, `utilities/agent_utils.py` | +| P6 | `akickoff` writes the per-task output log via `to_thread` | `crew.py` | + +P3 (LLM steps) and P4 (parallel tools) were committed together because they +share the `execute_native_tool` restructuring. The "plain" variant has no tool +calls, so its column isolates P3; the "tools" variant shows P3+P4. + +Cumulative: each column includes all previous ones. The P1b column's planning +rows were measured on the commit before P5. + +### Wall time (s) + +| scenario | baseline | P1 | P2 | P3+P4 | P1b | P5 | P6 | +|---|---|---|---|---|---|---|---| +| plain, 200 ms, N=1 | 0.45 | 0.44 | 0.46 | 0.44 | 0.43 | 0.44 | 0.44 | +| plain, 200 ms, N=10 | 0.99 | 0.97 | 0.97 | 0.68 | 0.66 | 0.63 | 1.87 | +| plain, 200 ms, N=50 | 4.23 | 4.08 | 4.10 | 1.80 | 1.57 | 1.65 | 2.24 | +| plain, 200 ms, N=200 | 18.48 | 17.28 | 17.58 | 7.57 | 6.69 | 6.85 | 7.39 | +| plain, 500 ms, N=200 | 35.10 | 32.61 | 32.81 | 7.70 | 7.10 | 7.17 | 8.10 | +| tools, 200 ms, N=1 | 0.77 | 0.76 | 0.76 | 0.76 | 0.77 | 0.74 | 0.75 | +| tools, 200 ms, N=10 | 1.25 | 1.25 | 1.21 | 1.02 | 0.97 | 0.98 | 1.69 | +| tools, 200 ms, N=50 | 6.22 | 6.33 | 6.16 | 2.40 | 2.22 | 2.17 | 3.02 | +| tools, 200 ms, N=200 | 27.89 | 27.07 | 26.96 | 9.34 | 8.90 | 9.22 | 17.21 | +| planning, 200 ms, N=1 | 1.27 | – | – | – | 1.25 | 1.24 | 1.25 | +| planning, 200 ms, N=50 | 16.52 | – | – | – | 15.23 | 4.70 | 5.30 | +| default executor = 4, N=20 | 2.69 | 2.69 | 2.69 | 0.92 | 0.92 | 0.91 | 1.53 | + +The ideal for "plain" (2 sequential LLM calls) is ~0.4 s at any N. Once P3 is +in, the residual is crewai's own CPU work, not waiting (see "What's left"). + +### Per-kickoff overhead (LLM latency 0, sequential kickoffs, median) + +| | baseline | P1 | P2 | P3+P4 | P1b | P5 | P6 | +|---|---|---|---|---|---|---|---| +| plain: ms / kickoff | 26.1 | 26.2 | 25.6 | 21.0 | 19.7 | 19.6 | 17.0 | +| plain: threads started / kickoff | 7.1 | 4.0 | 4.0 | 4.0 | 0.0 | 0.1 | 0.1 | +| tools: ms / kickoff | 36.6 | 36.1 | 37.4 | 25.6 | 22.0 | 24.9 | 23.0 | +| tools: threads started / kickoff | 11.1 | 8.1 | 8.0 | 4.2 | 0.2 | 0.2 | 0.2 | + +### Threads and memory + +| | baseline | after P5 | +|---|---|---| +| peak threads, tools N=200 | 49 | 20 | +| threads started, tools N=200 | 2215 | 13 | +| peak RSS, tools N=200 | 310 MB | 312 MB | + +The pools are bounded, so thread counts never exploded, and RSS is unchanged. +Memory is not an argument either way. + +### Event-loop lag (ms, p99 / max) + +| scenario | baseline | P3+P4 | P5 | P6 | +|---|---|---|---|---| +| plain, 200 ms, N=50 | 89 / 771 | 144 / 758 | 113 / 644 | 14 / 683 | +| plain, 200 ms, N=200 | 72 / 3510 | 761 / 3163 | 499 / 2950 | 97 / 2920 | +| tools, 200 ms, N=200 | 39 / 3829 | 1172 / 3477 | 599 / 3512 | 70 / 3556 | + +The multi-second max lag is the same in every column. It comes from the burst +of ~200 crew setups (`prepare_kickoff`, executor construction) running on the +loop at start. That is pure CPU and not thread-related. p99 lag goes **up** +once the executor is native: CPU work that used to be spread over worker threads +(still serialised by the GIL, but interleaved with the loop) now runs on the +loop itself. With P6 (SQLite off the loop) p99 falls back to baseline levels. + +### Cancellation (`task.cancel()` on a running `akickoff`) + +| | `CancelledError` surfaces after | calls still running to completion after cancel | +|---|---|---| +| baseline, cancel during LLM call | 2.8 ms | 1 LLM call (thread) | +| baseline, cancel during 4 parallel tools | 1.9 ms | 4 tool calls (thread pool + private loops) | +| P3+P4 onward | 0.2–1.5 ms | **0** (the `acall` / tool coroutines are cancelled) | + +No new work starts after cancellation even on baseline. The fix is about work +that is *already in flight*: on baseline it keeps running for its full duration. +With real LLMs that means spending tokens on the result of a request the caller +has abandoned. Sync tools are still not cancellable, because they stay in +threads by design. + +### Context propagation + +`contextvars` set before `akickoff` are visible in the tool, the LLM call, a +sync handler and an async handler, both before and after every prototype +(`context/tools` row in each JSON). + +## Which changes are worth it + +| change | worth it alone? | risk | test fallout (vs main) | +|---|---|---|---| +| P1: async file store in `akickoff` | Marginal: −3 threads per kickoff, no wall-time change | Low | 0 | +| **P1b: file store short-circuit** | **Yes**, cheap: removes the remaining ~4–7 `ThreadPoolExecutor` + `asyncio.run` per kickoff on *both* sync and async paths | Low. Stale ids after TTL only cost a real lookup. Cross-process stores would need a rethink, but the store is in-memory today | 0 | +| P2: event bus `aflush` / single hop | **No measurable gain** at any concurrency. Keep `aflush` only as hygiene (it stops `akickoff` blocking the loop while slow handlers drain) | Low | 0 | +| **P3: executor LLM steps on the loop** | **Yes, the main win**: 2.7–4.9× throughput at N≥50, cancellation, no default-executor coupling | Medium (see behaviour changes) | 1 (source-inspection test) + 1 fixed during the prototype | +| **P4: gather async tools** | **Yes**, with P3: peak threads 49 → 20, tools N=200 27.9 → 9.2 s | Low/medium | shared with P3 | +| **P5: planning steps native** | **Yes** if planning matters: 3.2× at N=50 | Medium: the generator refactor changes private method shapes | 5 (tests that call or mock `_execute_native` etc. directly) + 1 source-inspection test | +| P6: SQLite log via `to_thread` | **No, as written**: p99 lag 499 → 97 ms but throughput 6.9 → 7.4 s (plain) and **9.2 → 17.2 s** (tools) at N=200, and N=10 gets 3× slower. Threads now contend on the file lock and on default-executor workers. The fix is to make the per-task replay log cheaper (batch per kickoff, async store, or opt-out), not to thread it | – | 0 | + +All 7 failing tests are implementation-coupled (`inspect.getsource` checks or +mocks of private methods); none is a behaviour regression. I fixed one real +regression found along the way (`OutputParserError` raised by the LLM call +itself was no longer routed to `parser_error`). + +Test-suite note: in this container `pytest lib/crewai/tests` already has 407 +failures and 14 errors on `main`, from optional provider packages (litellm, +anthropic, gemini, azure) not being installed. Fallout above is the set +difference against that baseline: main 407 failed / 5198 passed; prototype 414 +failed / 5191 passed; nothing fixed or broken beyond the 7 listed. +`ruff check` and `mypy` are clean on all 8 changed source files. + +## Behaviour changes users could notice (P3–P5) + +- **`LLM.acall` instead of `LLM.call` on `akickoff`.** Provider async paths + differ from the sync ones (client objects, retry/timeout wiring, streaming). + Users who customised `call` but not `acall` get the old behaviour via a + `to_thread` fallback. Users who override both with different behaviour will + see the `acall` one. The sync `kickoff()` is unchanged. +- **Hooks and callbacks now run on the caller's event loop thread.** Before/after + LLM-call hooks, tool hooks, `step_callback` and `enforce_rpm_limit` used to + run in a worker thread. A blocking user callback now stalls the loop, and so + does RPM throttling (`enforce_rpm_limit` sleeps). The RPM limiter needs an + async variant before shipping. +- **Async tools run on the caller's loop**, not a private `asyncio.run` loop in a + thread. This is good for tools holding loop-bound clients (aiohttp sessions + created on the app loop used to fail). An async tool that does blocking work + inside `async def` now stalls the caller's loop. +- **`contextvars` writes inside tools and hooks now persist** across steps + within the same kickoff. Before, each `to_thread` ran on a context copy, so + mutations were lost. Reads behave the same either way. +- **Parallel tool batch aborts** (`ToolExecutionFailedError`) now cancel + in-flight async siblings. Before, they could only drop not-yet-started ones. +- **Cancellation** now reaches `acall` / tool coroutines; providers and tools see + `CancelledError` mid-call. +- **Exceptions**: same types and messages. Tracebacks are shorter (no thread + boundary). +- **Event handler ordering**: unchanged. `emit()` was not touched, and sync + handlers still run on the bus pool. +- **`CrewKickoffCompletedEvent` timing**: unchanged ordering. `akickoff` still + drains memory writes and waits for *all* pending handlers (globally, across + concurrent crews) before emitting. It now awaits instead of blocking the + loop. The cross-crew coupling (one crew's completion waits on every other + crew's handlers) is pre-existing and worth fixing separately. + +## What's left after the prototype (the next bottleneck) + +With latency out of the picture, a 2-task kickoff costs ~17–20 ms of CPU. In +cProfile, the biggest items are: + +- SQLite + file-lock task-output log: ~60% of wall time on the async path. +- Flow listener resolution (`_find_triggered_methods` / `_condition_met`): + ~10 ms per kickoff, from 24k condition checks for 20 kickoffs. +- `prepare_kickoff` per crew. + +These bound throughput at roughly 25–30 kickoffs/s per process regardless of +threads, and they are now what drives loop lag. Reducing them helps both +`kickoff` and `akickoff`. + +## Recommendation and PR breakdown + +**Partial go.** Do the executor work (it is the real "async path"), skip the +event-bus rework, and fix the storage write differently. + +1. **File store short-circuit (P1b + P1).** Small, low risk, helps sync too. + Ship first. +2. **`Flow._async_step_override` + `AgentExecutor` async LLM steps (P3).** + Include: `to_thread(call)` fallback for LLMs without `acall`, inline routing + steps, skipping the persist hop when persistence is off, an async + `enforce_rpm_limit`, and the `OutputParserError` routing fix. Update the one + source-inspection test. Document the hooks-on-loop change. +3. **Native tool batches on the loop (P4).** Generator-based single-call + lifecycle shared by sync and async, gathered async tools, sync tools via + `to_thread`. Could be folded into PR 2. +4. **Planning `StepExecutor.aexecute` (P5).** Rewrite the 5 tests that mock + private step methods to go through `execute()`. Then also convert the + remaining sync planning LLM calls (plan generation, `PlannerObserver`, + synthesis); 100 of the 450 planning LLM calls in the benchmark still go + through threads. +5. **Separately:** make the per-task replay log cheap (one write per kickoff, or + async/opt-out), and reduce Flow listener resolution cost. These are the next + throughput limit for both sync and async. +6. **`aflush` (P2)**: optional hygiene, can ride along with PR 2. Not worth a + PR by itself. + +Out of scope and unchanged: `kickoff()`, `Task.execute_async`, the memory save +pool, `Crew.kickoff_async`. That last one still wraps the whole sync +`kickoff` in a thread; with PR 2 landed, pointing users at `akickoff` becomes +a real recommendation rather than a cosmetic one. diff --git a/lib/crewai/tests/perf/bench_async_path.py b/lib/crewai/tests/perf/bench_async_path.py new file mode 100644 index 000000000..4e01b613e --- /dev/null +++ b/lib/crewai/tests/perf/bench_async_path.py @@ -0,0 +1,491 @@ +"""Benchmarks for crewai's async execution path (``Crew.akickoff``). + +Throwaway investigation harness, not a test module (no ``test_`` prefix, so +pytest does not collect it). Every scenario runs in a fresh subprocess so +thread counts, RSS and the global event bus start clean. + +Run everything and write a JSON + markdown summary:: + + uv run python lib/crewai/tests/perf/bench_async_path.py --label baseline + +Run a single scenario in-process (what the driver does per subprocess):: + + uv run python lib/crewai/tests/perf/bench_async_path.py --one scaling \ + --n 50 --latency 0.2 --variant tools +""" + +from __future__ import annotations + +import argparse +import asyncio +from concurrent.futures import ThreadPoolExecutor +import contextvars +import json +import os +from pathlib import Path +import resource +import statistics +import subprocess +import sys +import threading +import time +from typing import Any + + +os.environ.setdefault("CREWAI_DISABLE_TELEMETRY", "true") +os.environ.setdefault("CREWAI_DISABLE_TRACKING", "true") +os.environ.setdefault("OTEL_SDK_DISABLED", "true") +os.environ.setdefault("CREWAI_TRACING_ENABLED", "false") +os.environ.setdefault("CREWAI_STORAGE_DIR", "/tmp/crewai-perf-storage") + +HERE = Path(__file__).resolve().parent + +# -------------------------------------------------------------------------- +# Instrumentation shared by all scenarios +# -------------------------------------------------------------------------- + +_thread_starts = 0 +_orig_thread_start = threading.Thread.start + + +def _counting_start(self: threading.Thread) -> None: + global _thread_starts + _thread_starts += 1 + _orig_thread_start(self) + + +threading.Thread.start = _counting_start # type: ignore[method-assign] + + +class Counters: + lock = threading.Lock() + sync_llm_calls = 0 + async_llm_calls = 0 + tool_calls = 0 + llm_call_times: list[float] = [] + tool_call_times: list[float] = [] + llm_done_times: list[float] = [] + tool_done_times: list[float] = [] + + @classmethod + def done(cls, kind: str) -> None: + with cls.lock: + (cls.llm_done_times if kind == "llm" else cls.tool_done_times).append( + time.perf_counter() + ) + + @classmethod + def llm(cls, is_async: bool) -> None: + with cls.lock: + if is_async: + cls.async_llm_calls += 1 + else: + cls.sync_llm_calls += 1 + cls.llm_call_times.append(time.perf_counter()) + + @classmethod + def tool(cls) -> None: + with cls.lock: + cls.tool_calls += 1 + cls.tool_call_times.append(time.perf_counter()) + + +CTX_VAR: contextvars.ContextVar[str] = contextvars.ContextVar( + "bench_ctx", default="" +) +SEEN_CTX: dict[str, set[str]] = {"tool": set(), "handler": set(), "llm": set()} + + +def _build_crew(variant: str, latency: float, tool_latency: float, n_tools: int): + from crewai import Agent, Crew, Task + from crewai.llms.base_llm import BaseLLM + from crewai.tools import BaseTool + + def _has_tool_result(messages: Any) -> bool: + return isinstance(messages, list) and any( + isinstance(m, dict) and m.get("role") == "tool" for m in messages + ) + + def _respond(messages: Any, tools: Any, available_functions: Any = None) -> Any: + SEEN_CTX["llm"].add(CTX_VAR.get()) + if available_functions and "create_reasoning_plan" in available_functions: + return json.dumps( + { + "plan": "three independent steps", + "steps": [ + {"step_number": k, "description": f"step {k}", "depends_on": []} + for k in (1, 2, 3) + ], + "ready": True, + } + ) + if tools and variant in ("tools", "planning") and not _has_tool_result(messages): + return [ + { + "id": f"call_{i}", + "type": "function", + "function": {"name": f"async_tool_{i}", "arguments": "{}"}, + } + for i in range(n_tools) + ] + if tools: + return "done" + return "Thought: I know the answer\nFinal Answer: done" + + class MockLLM(BaseLLM): + def call(self, messages, tools=None, callbacks=None, available_functions=None, + from_task=None, from_agent=None, response_model=None): # noqa: D401 + Counters.llm(is_async=False) + if latency: + time.sleep(latency) + Counters.done("llm") + return _respond(messages, tools, available_functions) + + async def acall(self, messages, tools=None, callbacks=None, + available_functions=None, from_task=None, from_agent=None, + response_model=None): + Counters.llm(is_async=True) + if latency: + await asyncio.sleep(latency) + Counters.done("llm") + return _respond(messages, tools, available_functions) + + def supports_function_calling(self) -> bool: + return True + + def supports_stop_words(self) -> bool: + return False + + def get_context_window_size(self) -> int: + return 128000 + + def make_tool(i: int) -> BaseTool: + class AsyncTool(BaseTool): + name: str = f"async_tool_{i}" + description: str = f"Async benchmark tool {i}" + + async def _run(self) -> str: # type: ignore[override] + return await self._arun() + + async def _arun(self) -> str: # type: ignore[override] + Counters.tool() + SEEN_CTX["tool"].add(CTX_VAR.get()) + if tool_latency: + await asyncio.sleep(tool_latency) + Counters.done("tool") + return f"result {i}" + + return AsyncTool() + + tools = ( + [make_tool(i) for i in range(n_tools)] + if variant in ("tools", "planning") + else [] + ) + agents = [ + Agent( + role=f"Agent {k}", + goal="Answer", + backstory="Benchmark agent", + llm=MockLLM(model="mock-model"), + tools=tools if k == 0 else [], + planning=variant == "planning" and k == 0, + verbose=False, + ) + for k in range(2) + ] + tasks = [ + Task(description=f"Task {k}", expected_output="done", agent=agents[k]) + for k in range(2) + ] + return Crew(agents=agents, tasks=tasks, verbose=False) + + +class LoopProbe: + """Measure event-loop lag: how late a 10 ms sleep wakes up.""" + + def __init__(self) -> None: + self.lags: list[float] = [] + self._task: asyncio.Task[None] | None = None + + async def _run(self) -> None: + while True: + t0 = time.perf_counter() + await asyncio.sleep(0.01) + self.lags.append(time.perf_counter() - t0 - 0.01) + + def start(self) -> None: + self._task = asyncio.create_task(self._run()) + + async def stop(self) -> dict[str, float]: + assert self._task is not None + self._task.cancel() + try: + await self._task + except asyncio.CancelledError: + pass + lags = sorted(self.lags) or [0.0] + return { + "loop_lag_max_ms": round(lags[-1] * 1000, 1), + "loop_lag_p99_ms": round(lags[int(len(lags) * 0.99) - 1 if len(lags) > 1 else 0] * 1000, 1), + "loop_lag_mean_ms": round(statistics.fmean(lags) * 1000, 2), + } + + +class ThreadSampler: + def __init__(self) -> None: + self.peak = threading.active_count() + self._stop = threading.Event() + self._t = threading.Thread(target=self._run, daemon=True) + + def _run(self) -> None: + while not self._stop.is_set(): + self.peak = max(self.peak, threading.active_count() - 1) + time.sleep(0.002) + + def __enter__(self) -> ThreadSampler: + _orig_thread_start(self._t) + return self + + def __exit__(self, *exc: Any) -> None: + self._stop.set() + self._t.join() + + +def _rss_mb() -> float: + return round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, 1) + + +# -------------------------------------------------------------------------- +# Scenarios (each runs in its own process) +# -------------------------------------------------------------------------- + + +async def scenario_scaling(args: argparse.Namespace) -> dict[str, Any]: + crews = [ + _build_crew(args.variant, args.latency, args.tool_latency, args.n_tools) + for _ in range(args.n) + ] + if args.small_executor: + asyncio.get_running_loop().set_default_executor( + ThreadPoolExecutor(args.small_executor) + ) + # Warm-up kickoff so import/lazy-init costs are excluded. + warm = _build_crew(args.variant, 0, 0, args.n_tools) + await warm.akickoff() + Counters.sync_llm_calls = Counters.async_llm_calls = Counters.tool_calls = 0 + starts_before = _thread_starts + rss_before = _rss_mb() + + probe = LoopProbe() + probe.start() + with ThreadSampler() as sampler: + t0 = time.perf_counter() + results = await asyncio.gather(*(c.akickoff() for c in crews)) + wall = time.perf_counter() - t0 + lag = await probe.stop() + assert all(r.raw for r in results) + return { + "wall_s": round(wall, 3), + "throughput_kickoffs_per_s": round(args.n / wall, 2), + "peak_threads": sampler.peak, + "thread_starts": _thread_starts - starts_before, + "peak_rss_mb": _rss_mb(), + "rss_growth_mb": round(_rss_mb() - rss_before, 1), + "sync_llm_calls": Counters.sync_llm_calls, + "async_llm_calls": Counters.async_llm_calls, + "tool_calls": Counters.tool_calls, + **lag, + } + + +async def scenario_overhead(args: argparse.Namespace) -> dict[str, Any]: + crew = _build_crew(args.variant, 0, 0, args.n_tools) + await crew.akickoff() # warm-up + times = [] + starts_before = _thread_starts + for _ in range(args.n): + t0 = time.perf_counter() + await crew.akickoff() + times.append(time.perf_counter() - t0) + return { + "per_kickoff_ms_median": round(statistics.median(times) * 1000, 2), + "per_kickoff_ms_mean": round(statistics.fmean(times) * 1000, 2), + "thread_starts_per_kickoff": round((_thread_starts - starts_before) / args.n, 1), + } + + +async def scenario_cancel(args: argparse.Namespace) -> dict[str, Any]: + crew = _build_crew(args.variant, args.latency, args.tool_latency, args.n_tools) + await _build_crew(args.variant, 0, 0, args.n_tools).akickoff() # warm-up + Counters.sync_llm_calls = Counters.async_llm_calls = Counters.tool_calls = 0 + for lst in (Counters.llm_call_times, Counters.tool_call_times, + Counters.llm_done_times, Counters.tool_done_times): + lst.clear() + task = asyncio.create_task(crew.akickoff()) + await asyncio.sleep(args.cancel_after) + t_cancel = time.perf_counter() + task.cancel() + try: + await task + cancel_outcome = "completed (cancel ignored)" + except asyncio.CancelledError: + cancel_outcome = "CancelledError" + except Exception as e: # noqa: BLE001 + cancel_outcome = f"{type(e).__name__}: {e}" + t_returned = time.perf_counter() - t_cancel + await asyncio.sleep(args.latency * 6 + args.tool_latency * 4 + 0.5) + return { + "cancel_outcome": cancel_outcome, + "await_returned_after_ms": round(t_returned * 1000, 1), + "llm_started_after_cancel": sum(1 for t in Counters.llm_call_times if t > t_cancel), + "llm_finished_after_cancel": sum(1 for t in Counters.llm_done_times if t > t_cancel), + "tool_started_after_cancel": sum(1 for t in Counters.tool_call_times if t > t_cancel), + "tool_finished_after_cancel": sum(1 for t in Counters.tool_done_times if t > t_cancel), + "llm_started_total": len(Counters.llm_call_times), + "tool_started_total": len(Counters.tool_call_times), + } + + +async def scenario_context(args: argparse.Namespace) -> dict[str, Any]: + from crewai.events.event_bus import crewai_event_bus + from crewai.events.types.tool_usage_events import ToolUsageStartedEvent + from crewai.events.types.llm_events import LLMCallStartedEvent + + with crewai_event_bus.scoped_handlers(): + + @crewai_event_bus.on(ToolUsageStartedEvent) + def _h(source: Any, event: Any) -> None: + SEEN_CTX["handler"].add(CTX_VAR.get()) + + @crewai_event_bus.on(LLMCallStartedEvent) + async def _ah(source: Any, event: Any) -> None: + SEEN_CTX["handler"].add("async:" + CTX_VAR.get()) + + crew = _build_crew("tools", 0.0, 0.0, 2) + CTX_VAR.set("set-before-akickoff") + await crew.akickoff() + crewai_event_bus.flush() + return {k: sorted(v) for k, v in SEEN_CTX.items()} + + +SCENARIOS = { + "scaling": scenario_scaling, + "overhead": scenario_overhead, + "cancel": scenario_cancel, + "context": scenario_context, +} + + +def run_one(args: argparse.Namespace) -> None: + result = asyncio.run(SCENARIOS[args.one](args)) + print("RESULT_JSON " + json.dumps(result)) + + +# -------------------------------------------------------------------------- +# Driver +# -------------------------------------------------------------------------- + +def _plan(quick: bool) -> list[tuple[str, list[str]]]: + ns = [1, 10, 50] if quick else [1, 10, 50, 200] + plan: list[tuple[str, list[str]]] = [] + for variant in ("plain", "tools"): + for n in ns: + plan.append( + ( + f"scaling/{variant}/lat=200ms/N={n}", + ["--one", "scaling", "--variant", variant, "--n", str(n), + "--latency", "0.2", "--tool-latency", "0.1"], + ) + ) + for n in (1, 50): + plan.append( + ( + f"scaling/planning/lat=200ms/N={n}", + ["--one", "scaling", "--variant", "planning", "--n", str(n), + "--latency", "0.2", "--tool-latency", "0.1", "--n-tools", "2"], + ) + ) + plan.append( + ( + "scaling/plain/lat=500ms/N=200" if not quick else "scaling/plain/lat=500ms/N=50", + ["--one", "scaling", "--variant", "plain", "--n", "50" if quick else "200", + "--latency", "0.5"], + ) + ) + for variant in ("plain", "tools"): + plan.append( + ( + f"overhead/{variant}/lat=0", + ["--one", "overhead", "--variant", variant, "--n", "30"], + ) + ) + plan.append( + ( + "small_executor(4)/plain/lat=200ms/N=20", + ["--one", "scaling", "--variant", "plain", "--n", "20", "--latency", "0.2", + "--small-executor", "4"], + ) + ) + for phase, after in (("during-llm", "0.1"), ("during-tools", "0.35")): + plan.append( + ( + f"cancel/{phase}/llm=200ms,tool=300ms", + ["--one", "cancel", "--variant", "tools", "--latency", "0.2", + "--tool-latency", "0.3", "--cancel-after", after], + ) + ) + plan.append(("context/tools", ["--one", "context"])) + return plan + + +def drive(args: argparse.Namespace) -> None: + out_dir = HERE / "results" + out_dir.mkdir(exist_ok=True) + results: dict[str, Any] = {} + for name, extra in _plan(args.quick): + if args.only and not any(o in name for o in args.only.split(",")): + continue + t0 = time.perf_counter() + proc = subprocess.run( + [sys.executable, __file__, *extra], + capture_output=True, + text=True, + timeout=900, + ) + line = next( + (ln for ln in proc.stdout.splitlines() if ln.startswith("RESULT_JSON ")), + None, + ) + if line is None: + results[name] = {"error": (proc.stderr or proc.stdout)[-2000:]} + else: + results[name] = json.loads(line[len("RESULT_JSON "):]) + print(f"{name:45s} {time.perf_counter() - t0:6.1f}s {results[name]}", flush=True) + path = out_dir / f"{args.label}.json" + path.write_text(json.dumps(results, indent=2)) + print(f"wrote {path}") + + +def main() -> None: + p = argparse.ArgumentParser() + p.add_argument("--one", choices=sorted(SCENARIOS)) + p.add_argument("--label", default="run") + p.add_argument("--quick", action="store_true") + p.add_argument("--only", default="") + p.add_argument("--variant", default="plain", choices=["plain", "tools", "planning"]) + p.add_argument("--n", type=int, default=1) + p.add_argument("--latency", type=float, default=0.2) + p.add_argument("--tool-latency", type=float, default=0.1) + p.add_argument("--n-tools", type=int, default=4) + p.add_argument("--small-executor", type=int, default=0) + p.add_argument("--cancel-after", type=float, default=0.3) + args = p.parse_args() + if args.one: + run_one(args) + else: + drive(args) + + +if __name__ == "__main__": + main() diff --git a/lib/crewai/tests/perf/results/baseline-cancel.json b/lib/crewai/tests/perf/results/baseline-cancel.json new file mode 100644 index 000000000..60ee427d0 --- /dev/null +++ b/lib/crewai/tests/perf/results/baseline-cancel.json @@ -0,0 +1,22 @@ +{ + "cancel/during-llm/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 2.8, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 1, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 0, + "llm_started_total": 1, + "tool_started_total": 0 + }, + "cancel/during-tools/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 1.9, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 0, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 4, + "llm_started_total": 1, + "tool_started_total": 4 + } +} \ No newline at end of file diff --git a/lib/crewai/tests/perf/results/baseline-planning.json b/lib/crewai/tests/perf/results/baseline-planning.json new file mode 100644 index 000000000..f943f9890 --- /dev/null +++ b/lib/crewai/tests/perf/results/baseline-planning.json @@ -0,0 +1,30 @@ +{ + "scaling/planning/lat=200ms/N=1": { + "wall_s": 1.269, + "throughput_kickoffs_per_s": 0.79, + "peak_threads": 10, + "thread_starts": 7, + "peak_rss_mb": 179.4, + "rss_growth_mb": 0.7, + "sync_llm_calls": 9, + "async_llm_calls": 0, + "tool_calls": 6, + "loop_lag_max_ms": 8.7, + "loop_lag_p99_ms": 1.0, + "loop_lag_mean_ms": 0.29 + }, + "scaling/planning/lat=200ms/N=50": { + "wall_s": 16.524, + "throughput_kickoffs_per_s": 3.03, + "peak_threads": 21, + "thread_starts": 362, + "peak_rss_mb": 207.5, + "rss_growth_mb": 16.1, + "sync_llm_calls": 450, + "async_llm_calls": 0, + "tool_calls": 300, + "loop_lag_max_ms": 839.6, + "loop_lag_p99_ms": 9.9, + "loop_lag_mean_ms": 1.3 + } +} \ No newline at end of file diff --git a/lib/crewai/tests/perf/results/baseline.json b/lib/crewai/tests/perf/results/baseline.json new file mode 100644 index 000000000..e28006374 --- /dev/null +++ b/lib/crewai/tests/perf/results/baseline.json @@ -0,0 +1,170 @@ +{ + "scaling/plain/lat=200ms/N=1": { + "wall_s": 0.45, + "throughput_kickoffs_per_s": 2.22, + "peak_threads": 6, + "thread_starts": 8, + "peak_rss_mb": 177.3, + "rss_growth_mb": 0.3, + "sync_llm_calls": 2, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 6.6, + "loop_lag_p99_ms": 4.2, + "loop_lag_mean_ms": 0.47 + }, + "scaling/plain/lat=200ms/N=10": { + "wall_s": 0.991, + "throughput_kickoffs_per_s": 10.09, + "peak_threads": 15, + "thread_starts": 79, + "peak_rss_mb": 181.7, + "rss_growth_mb": 3.5, + "sync_llm_calls": 20, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 166.4, + "loop_lag_p99_ms": 14.7, + "loop_lag_mean_ms": 3.4 + }, + "scaling/plain/lat=200ms/N=50": { + "wall_s": 4.226, + "throughput_kickoffs_per_s": 11.83, + "peak_threads": 21, + "thread_starts": 365, + "peak_rss_mb": 194.0, + "rss_growth_mb": 12.2, + "sync_llm_calls": 100, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 770.7, + "loop_lag_p99_ms": 88.7, + "loop_lag_mean_ms": 5.72 + }, + "scaling/plain/lat=200ms/N=200": { + "wall_s": 18.483, + "throughput_kickoffs_per_s": 10.82, + "peak_threads": 21, + "thread_starts": 1415, + "peak_rss_mb": 241.2, + "rss_growth_mb": 43.1, + "sync_llm_calls": 400, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 3510.1, + "loop_lag_p99_ms": 72.3, + "loop_lag_mean_ms": 8.66 + }, + "scaling/tools/lat=200ms/N=1": { + "wall_s": 0.766, + "throughput_kickoffs_per_s": 1.31, + "peak_threads": 10, + "thread_starts": 11, + "peak_rss_mb": 178.1, + "rss_growth_mb": 0.6, + "sync_llm_calls": 3, + "async_llm_calls": 0, + "tool_calls": 4, + "loop_lag_max_ms": 16.9, + "loop_lag_p99_ms": 4.1, + "loop_lag_mean_ms": 0.53 + }, + "scaling/tools/lat=200ms/N=10": { + "wall_s": 1.254, + "throughput_kickoffs_per_s": 7.97, + "peak_threads": 37, + "thread_starts": 119, + "peak_rss_mb": 184.8, + "rss_growth_mb": 3.7, + "sync_llm_calls": 30, + "async_llm_calls": 0, + "tool_calls": 40, + "loop_lag_max_ms": 178.8, + "loop_lag_p99_ms": 21.8, + "loop_lag_mean_ms": 3.15 + }, + "scaling/tools/lat=200ms/N=50": { + "wall_s": 6.221, + "throughput_kickoffs_per_s": 8.04, + "peak_threads": 47, + "thread_starts": 565, + "peak_rss_mb": 211.9, + "rss_growth_mb": 14.2, + "sync_llm_calls": 150, + "async_llm_calls": 0, + "tool_calls": 200, + "loop_lag_max_ms": 1014.9, + "loop_lag_p99_ms": 40.4, + "loop_lag_mean_ms": 4.18 + }, + "scaling/tools/lat=200ms/N=200": { + "wall_s": 27.889, + "throughput_kickoffs_per_s": 7.17, + "peak_threads": 49, + "thread_starts": 2215, + "peak_rss_mb": 310.4, + "rss_growth_mb": 53.0, + "sync_llm_calls": 600, + "async_llm_calls": 0, + "tool_calls": 800, + "loop_lag_max_ms": 3829.4, + "loop_lag_p99_ms": 39.0, + "loop_lag_mean_ms": 5.93 + }, + "scaling/plain/lat=500ms/N=200": { + "wall_s": 35.096, + "throughput_kickoffs_per_s": 5.7, + "peak_threads": 21, + "thread_starts": 1415, + "peak_rss_mb": 241.0, + "rss_growth_mb": 43.0, + "sync_llm_calls": 400, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 5138.9, + "loop_lag_p99_ms": 26.6, + "loop_lag_mean_ms": 4.17 + }, + "overhead/plain/lat=0": { + "per_kickoff_ms_median": 26.07, + "per_kickoff_ms_mean": 27.3, + "thread_starts_per_kickoff": 7.1 + }, + "overhead/tools/lat=0": { + "per_kickoff_ms_median": 36.62, + "per_kickoff_ms_mean": 36.67, + "thread_starts_per_kickoff": 11.1 + }, + "small_executor(4)/plain/lat=200ms/N=20": { + "wall_s": 2.685, + "throughput_kickoffs_per_s": 7.45, + "peak_threads": 15, + "thread_starts": 149, + "peak_rss_mb": 184.1, + "rss_growth_mb": 4.9, + "sync_llm_calls": 40, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 282.9, + "loop_lag_p99_ms": 31.7, + "loop_lag_mean_ms": 2.44 + }, + "cancel/tools/lat=200ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 2.2, + "llm_calls_after_cancel": 0, + "tool_calls_after_cancel": 0, + "llm_calls_total": 1 + }, + "context/tools": { + "tool": [ + "set-before-akickoff" + ], + "handler": [ + "set-before-akickoff" + ], + "llm": [ + "set-before-akickoff" + ] + } +} \ No newline at end of file diff --git a/lib/crewai/tests/perf/results/p1-filestore.json b/lib/crewai/tests/perf/results/p1-filestore.json new file mode 100644 index 000000000..3e2508617 --- /dev/null +++ b/lib/crewai/tests/perf/results/p1-filestore.json @@ -0,0 +1,183 @@ +{ + "scaling/plain/lat=200ms/N=1": { + "wall_s": 0.442, + "throughput_kickoffs_per_s": 2.26, + "peak_threads": 6, + "thread_starts": 4, + "peak_rss_mb": 177.6, + "rss_growth_mb": 0.3, + "sync_llm_calls": 2, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 4.8, + "loop_lag_p99_ms": 2.2, + "loop_lag_mean_ms": 0.33 + }, + "scaling/plain/lat=200ms/N=10": { + "wall_s": 0.965, + "throughput_kickoffs_per_s": 10.36, + "peak_threads": 14, + "thread_starts": 48, + "peak_rss_mb": 181.5, + "rss_growth_mb": 3.1, + "sync_llm_calls": 20, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 143.4, + "loop_lag_p99_ms": 13.6, + "loop_lag_mean_ms": 2.62 + }, + "scaling/plain/lat=200ms/N=50": { + "wall_s": 4.078, + "throughput_kickoffs_per_s": 12.26, + "peak_threads": 21, + "thread_starts": 215, + "peak_rss_mb": 194.3, + "rss_growth_mb": 12.2, + "sync_llm_calls": 100, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 750.8, + "loop_lag_p99_ms": 43.4, + "loop_lag_mean_ms": 5.25 + }, + "scaling/plain/lat=200ms/N=200": { + "wall_s": 17.276, + "throughput_kickoffs_per_s": 11.58, + "peak_threads": 21, + "thread_starts": 815, + "peak_rss_mb": 240.4, + "rss_growth_mb": 42.5, + "sync_llm_calls": 400, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 2937.5, + "loop_lag_p99_ms": 41.3, + "loop_lag_mean_ms": 6.93 + }, + "scaling/tools/lat=200ms/N=1": { + "wall_s": 0.764, + "throughput_kickoffs_per_s": 1.31, + "peak_threads": 10, + "thread_starts": 9, + "peak_rss_mb": 178.3, + "rss_growth_mb": 0.7, + "sync_llm_calls": 3, + "async_llm_calls": 0, + "tool_calls": 4, + "loop_lag_max_ms": 8.2, + "loop_lag_p99_ms": 1.5, + "loop_lag_mean_ms": 0.33 + }, + "scaling/tools/lat=200ms/N=10": { + "wall_s": 1.249, + "throughput_kickoffs_per_s": 8.01, + "peak_threads": 37, + "thread_starts": 88, + "peak_rss_mb": 185.1, + "rss_growth_mb": 3.6, + "sync_llm_calls": 30, + "async_llm_calls": 0, + "tool_calls": 40, + "loop_lag_max_ms": 174.7, + "loop_lag_p99_ms": 6.7, + "loop_lag_mean_ms": 2.48 + }, + "scaling/tools/lat=200ms/N=50": { + "wall_s": 6.328, + "throughput_kickoffs_per_s": 7.9, + "peak_threads": 47, + "thread_starts": 414, + "peak_rss_mb": 212.2, + "rss_growth_mb": 14.9, + "sync_llm_calls": 150, + "async_llm_calls": 0, + "tool_calls": 200, + "loop_lag_max_ms": 1019.6, + "loop_lag_p99_ms": 63.6, + "loop_lag_mean_ms": 4.48 + }, + "scaling/tools/lat=200ms/N=200": { + "wall_s": 27.07, + "throughput_kickoffs_per_s": 7.39, + "peak_threads": 48, + "thread_starts": 1614, + "peak_rss_mb": 310.5, + "rss_growth_mb": 53.2, + "sync_llm_calls": 600, + "async_llm_calls": 0, + "tool_calls": 800, + "loop_lag_max_ms": 3733.0, + "loop_lag_p99_ms": 37.8, + "loop_lag_mean_ms": 5.8 + }, + "scaling/plain/lat=500ms/N=200": { + "wall_s": 32.614, + "throughput_kickoffs_per_s": 6.13, + "peak_threads": 21, + "thread_starts": 815, + "peak_rss_mb": 240.5, + "rss_growth_mb": 42.4, + "sync_llm_calls": 400, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 3120.4, + "loop_lag_p99_ms": 28.2, + "loop_lag_mean_ms": 3.1 + }, + "overhead/plain/lat=0": { + "per_kickoff_ms_median": 26.16, + "per_kickoff_ms_mean": 27.46, + "thread_starts_per_kickoff": 4.0 + }, + "overhead/tools/lat=0": { + "per_kickoff_ms_median": 36.14, + "per_kickoff_ms_mean": 36.74, + "thread_starts_per_kickoff": 8.1 + }, + "small_executor(4)/plain/lat=200ms/N=20": { + "wall_s": 2.694, + "throughput_kickoffs_per_s": 7.43, + "peak_threads": 14, + "thread_starts": 88, + "peak_rss_mb": 184.3, + "rss_growth_mb": 5.2, + "sync_llm_calls": 40, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 269.9, + "loop_lag_p99_ms": 28.0, + "loop_lag_mean_ms": 2.31 + }, + "cancel/during-llm/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 2.3, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 1, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 0, + "llm_started_total": 1, + "tool_started_total": 0 + }, + "cancel/during-tools/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 1.8, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 0, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 4, + "llm_started_total": 1, + "tool_started_total": 4 + }, + "context/tools": { + "tool": [ + "set-before-akickoff" + ], + "handler": [ + "set-before-akickoff" + ], + "llm": [ + "set-before-akickoff" + ] + } +} \ No newline at end of file diff --git a/lib/crewai/tests/perf/results/p1b-filestore-shortcircuit.json b/lib/crewai/tests/perf/results/p1b-filestore-shortcircuit.json new file mode 100644 index 000000000..fdcab0c04 --- /dev/null +++ b/lib/crewai/tests/perf/results/p1b-filestore-shortcircuit.json @@ -0,0 +1,183 @@ +{ + "scaling/plain/lat=200ms/N=1": { + "wall_s": 0.432, + "throughput_kickoffs_per_s": 2.31, + "peak_threads": 6, + "thread_starts": 0, + "peak_rss_mb": 177.6, + "rss_growth_mb": 0.6, + "sync_llm_calls": 0, + "async_llm_calls": 2, + "tool_calls": 0, + "loop_lag_max_ms": 1.7, + "loop_lag_p99_ms": 1.7, + "loop_lag_mean_ms": 0.28 + }, + "scaling/plain/lat=200ms/N=10": { + "wall_s": 0.664, + "throughput_kickoffs_per_s": 15.06, + "peak_threads": 16, + "thread_starts": 9, + "peak_rss_mb": 181.4, + "rss_growth_mb": 3.3, + "sync_llm_calls": 0, + "async_llm_calls": 20, + "tool_calls": 0, + "loop_lag_max_ms": 124.9, + "loop_lag_p99_ms": 26.4, + "loop_lag_mean_ms": 4.09 + }, + "scaling/plain/lat=200ms/N=50": { + "wall_s": 1.573, + "throughput_kickoffs_per_s": 31.78, + "peak_threads": 20, + "thread_starts": 14, + "peak_rss_mb": 194.0, + "rss_growth_mb": 11.9, + "sync_llm_calls": 0, + "async_llm_calls": 100, + "tool_calls": 0, + "loop_lag_max_ms": 664.0, + "loop_lag_p99_ms": 70.8, + "loop_lag_mean_ms": 23.27 + }, + "scaling/plain/lat=200ms/N=200": { + "wall_s": 6.693, + "throughput_kickoffs_per_s": 29.88, + "peak_threads": 20, + "thread_starts": 14, + "peak_rss_mb": 241.4, + "rss_growth_mb": 43.3, + "sync_llm_calls": 0, + "async_llm_calls": 400, + "tool_calls": 0, + "loop_lag_max_ms": 2846.5, + "loop_lag_p99_ms": 498.7, + "loop_lag_mean_ms": 212.83 + }, + "scaling/tools/lat=200ms/N=1": { + "wall_s": 0.765, + "throughput_kickoffs_per_s": 1.31, + "peak_threads": 8, + "thread_starts": 1, + "peak_rss_mb": 178.2, + "rss_growth_mb": 0.6, + "sync_llm_calls": 0, + "async_llm_calls": 3, + "tool_calls": 4, + "loop_lag_max_ms": 8.3, + "loop_lag_p99_ms": 4.6, + "loop_lag_mean_ms": 0.41 + }, + "scaling/tools/lat=200ms/N=10": { + "wall_s": 0.967, + "throughput_kickoffs_per_s": 10.34, + "peak_threads": 20, + "thread_starts": 13, + "peak_rss_mb": 184.7, + "rss_growth_mb": 3.4, + "sync_llm_calls": 0, + "async_llm_calls": 30, + "tool_calls": 40, + "loop_lag_max_ms": 140.2, + "loop_lag_p99_ms": 15.8, + "loop_lag_mean_ms": 2.61 + }, + "scaling/tools/lat=200ms/N=50": { + "wall_s": 2.218, + "throughput_kickoffs_per_s": 22.54, + "peak_threads": 20, + "thread_starts": 13, + "peak_rss_mb": 211.2, + "rss_growth_mb": 13.6, + "sync_llm_calls": 0, + "async_llm_calls": 150, + "tool_calls": 200, + "loop_lag_max_ms": 932.9, + "loop_lag_p99_ms": 121.6, + "loop_lag_mean_ms": 22.97 + }, + "scaling/tools/lat=200ms/N=200": { + "wall_s": 8.899, + "throughput_kickoffs_per_s": 22.47, + "peak_threads": 20, + "thread_starts": 13, + "peak_rss_mb": 310.1, + "rss_growth_mb": 52.7, + "sync_llm_calls": 0, + "async_llm_calls": 600, + "tool_calls": 800, + "loop_lag_max_ms": 3455.4, + "loop_lag_p99_ms": 447.8, + "loop_lag_mean_ms": 230.12 + }, + "scaling/plain/lat=500ms/N=200": { + "wall_s": 7.101, + "throughput_kickoffs_per_s": 28.17, + "peak_threads": 20, + "thread_starts": 14, + "peak_rss_mb": 240.3, + "rss_growth_mb": 42.0, + "sync_llm_calls": 0, + "async_llm_calls": 400, + "tool_calls": 0, + "loop_lag_max_ms": 2969.3, + "loop_lag_p99_ms": 744.5, + "loop_lag_mean_ms": 74.47 + }, + "overhead/plain/lat=0": { + "per_kickoff_ms_median": 19.65, + "per_kickoff_ms_mean": 19.99, + "thread_starts_per_kickoff": 0.0 + }, + "overhead/tools/lat=0": { + "per_kickoff_ms_median": 21.97, + "per_kickoff_ms_mean": 22.09, + "thread_starts_per_kickoff": 0.2 + }, + "small_executor(4)/plain/lat=200ms/N=20": { + "wall_s": 0.922, + "throughput_kickoffs_per_s": 21.68, + "peak_threads": 16, + "thread_starts": 10, + "peak_rss_mb": 184.5, + "rss_growth_mb": 5.8, + "sync_llm_calls": 0, + "async_llm_calls": 40, + "tool_calls": 0, + "loop_lag_max_ms": 262.7, + "loop_lag_p99_ms": 53.0, + "loop_lag_mean_ms": 9.1 + }, + "cancel/during-llm/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 0.2, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 0, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 0, + "llm_started_total": 1, + "tool_started_total": 0 + }, + "cancel/during-tools/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 0.4, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 0, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 0, + "llm_started_total": 1, + "tool_started_total": 4 + }, + "context/tools": { + "tool": [ + "set-before-akickoff" + ], + "handler": [ + "set-before-akickoff" + ], + "llm": [ + "set-before-akickoff" + ] + } +} \ No newline at end of file diff --git a/lib/crewai/tests/perf/results/p2-eventbus.json b/lib/crewai/tests/perf/results/p2-eventbus.json new file mode 100644 index 000000000..ee4ff42e3 --- /dev/null +++ b/lib/crewai/tests/perf/results/p2-eventbus.json @@ -0,0 +1,183 @@ +{ + "scaling/plain/lat=200ms/N=1": { + "wall_s": 0.459, + "throughput_kickoffs_per_s": 2.18, + "peak_threads": 6, + "thread_starts": 5, + "peak_rss_mb": 177.5, + "rss_growth_mb": 0.5, + "sync_llm_calls": 2, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 14.5, + "loop_lag_p99_ms": 4.6, + "loop_lag_mean_ms": 0.66 + }, + "scaling/plain/lat=200ms/N=10": { + "wall_s": 0.974, + "throughput_kickoffs_per_s": 10.26, + "peak_threads": 16, + "thread_starts": 50, + "peak_rss_mb": 181.6, + "rss_growth_mb": 3.3, + "sync_llm_calls": 20, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 164.9, + "loop_lag_p99_ms": 12.9, + "loop_lag_mean_ms": 3.03 + }, + "scaling/plain/lat=200ms/N=50": { + "wall_s": 4.103, + "throughput_kickoffs_per_s": 12.19, + "peak_threads": 21, + "thread_starts": 215, + "peak_rss_mb": 194.6, + "rss_growth_mb": 12.5, + "sync_llm_calls": 100, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 685.9, + "loop_lag_p99_ms": 77.9, + "loop_lag_mean_ms": 4.94 + }, + "scaling/plain/lat=200ms/N=200": { + "wall_s": 17.581, + "throughput_kickoffs_per_s": 11.38, + "peak_threads": 21, + "thread_starts": 815, + "peak_rss_mb": 242.5, + "rss_growth_mb": 44.5, + "sync_llm_calls": 400, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 3145.8, + "loop_lag_p99_ms": 43.4, + "loop_lag_mean_ms": 7.45 + }, + "scaling/tools/lat=200ms/N=1": { + "wall_s": 0.757, + "throughput_kickoffs_per_s": 1.32, + "peak_threads": 11, + "thread_starts": 9, + "peak_rss_mb": 178.2, + "rss_growth_mb": 0.6, + "sync_llm_calls": 3, + "async_llm_calls": 0, + "tool_calls": 4, + "loop_lag_max_ms": 6.0, + "loop_lag_p99_ms": 3.8, + "loop_lag_mean_ms": 0.39 + }, + "scaling/tools/lat=200ms/N=10": { + "wall_s": 1.213, + "throughput_kickoffs_per_s": 8.24, + "peak_threads": 38, + "thread_starts": 88, + "peak_rss_mb": 185.2, + "rss_growth_mb": 4.0, + "sync_llm_calls": 30, + "async_llm_calls": 0, + "tool_calls": 40, + "loop_lag_max_ms": 168.9, + "loop_lag_p99_ms": 6.1, + "loop_lag_mean_ms": 2.5 + }, + "scaling/tools/lat=200ms/N=50": { + "wall_s": 6.159, + "throughput_kickoffs_per_s": 8.12, + "peak_threads": 48, + "thread_starts": 414, + "peak_rss_mb": 211.9, + "rss_growth_mb": 14.4, + "sync_llm_calls": 150, + "async_llm_calls": 0, + "tool_calls": 200, + "loop_lag_max_ms": 934.3, + "loop_lag_p99_ms": 37.1, + "loop_lag_mean_ms": 4.04 + }, + "scaling/tools/lat=200ms/N=200": { + "wall_s": 26.962, + "throughput_kickoffs_per_s": 7.42, + "peak_threads": 49, + "thread_starts": 1614, + "peak_rss_mb": 313.2, + "rss_growth_mb": 55.8, + "sync_llm_calls": 600, + "async_llm_calls": 0, + "tool_calls": 800, + "loop_lag_max_ms": 3484.5, + "loop_lag_p99_ms": 41.1, + "loop_lag_mean_ms": 5.64 + }, + "scaling/plain/lat=500ms/N=200": { + "wall_s": 32.808, + "throughput_kickoffs_per_s": 6.1, + "peak_threads": 21, + "thread_starts": 815, + "peak_rss_mb": 247.3, + "rss_growth_mb": 49.4, + "sync_llm_calls": 400, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 3058.2, + "loop_lag_p99_ms": 31.7, + "loop_lag_mean_ms": 3.22 + }, + "overhead/plain/lat=0": { + "per_kickoff_ms_median": 25.57, + "per_kickoff_ms_mean": 25.87, + "thread_starts_per_kickoff": 4.0 + }, + "overhead/tools/lat=0": { + "per_kickoff_ms_median": 37.35, + "per_kickoff_ms_mean": 38.17, + "thread_starts_per_kickoff": 8.0 + }, + "small_executor(4)/plain/lat=200ms/N=20": { + "wall_s": 2.691, + "throughput_kickoffs_per_s": 7.43, + "peak_threads": 15, + "thread_starts": 88, + "peak_rss_mb": 184.7, + "rss_growth_mb": 5.4, + "sync_llm_calls": 40, + "async_llm_calls": 0, + "tool_calls": 0, + "loop_lag_max_ms": 258.8, + "loop_lag_p99_ms": 21.4, + "loop_lag_mean_ms": 2.28 + }, + "cancel/during-llm/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 1.3, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 1, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 0, + "llm_started_total": 1, + "tool_started_total": 0 + }, + "cancel/during-tools/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 1.3, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 0, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 4, + "llm_started_total": 1, + "tool_started_total": 4 + }, + "context/tools": { + "tool": [ + "set-before-akickoff" + ], + "handler": [ + "set-before-akickoff" + ], + "llm": [ + "set-before-akickoff" + ] + } +} \ No newline at end of file diff --git a/lib/crewai/tests/perf/results/p3-executor-native.json b/lib/crewai/tests/perf/results/p3-executor-native.json new file mode 100644 index 000000000..1be09f132 --- /dev/null +++ b/lib/crewai/tests/perf/results/p3-executor-native.json @@ -0,0 +1,183 @@ +{ + "scaling/plain/lat=200ms/N=1": { + "wall_s": 0.438, + "throughput_kickoffs_per_s": 2.28, + "peak_threads": 6, + "thread_starts": 4, + "peak_rss_mb": 177.5, + "rss_growth_mb": 0.7, + "sync_llm_calls": 0, + "async_llm_calls": 2, + "tool_calls": 0, + "loop_lag_max_ms": 4.2, + "loop_lag_p99_ms": 4.0, + "loop_lag_mean_ms": 0.43 + }, + "scaling/plain/lat=200ms/N=10": { + "wall_s": 0.681, + "throughput_kickoffs_per_s": 14.69, + "peak_threads": 18, + "thread_starts": 52, + "peak_rss_mb": 181.4, + "rss_growth_mb": 3.0, + "sync_llm_calls": 0, + "async_llm_calls": 20, + "tool_calls": 0, + "loop_lag_max_ms": 131.0, + "loop_lag_p99_ms": 24.9, + "loop_lag_mean_ms": 4.44 + }, + "scaling/plain/lat=200ms/N=50": { + "wall_s": 1.8, + "throughput_kickoffs_per_s": 27.77, + "peak_threads": 21, + "thread_starts": 215, + "peak_rss_mb": 195.0, + "rss_growth_mb": 12.8, + "sync_llm_calls": 0, + "async_llm_calls": 100, + "tool_calls": 0, + "loop_lag_max_ms": 758.3, + "loop_lag_p99_ms": 144.1, + "loop_lag_mean_ms": 31.7 + }, + "scaling/plain/lat=200ms/N=200": { + "wall_s": 7.568, + "throughput_kickoffs_per_s": 26.43, + "peak_threads": 21, + "thread_starts": 815, + "peak_rss_mb": 241.7, + "rss_growth_mb": 43.4, + "sync_llm_calls": 0, + "async_llm_calls": 400, + "tool_calls": 0, + "loop_lag_max_ms": 3162.5, + "loop_lag_p99_ms": 760.5, + "loop_lag_mean_ms": 132.68 + }, + "scaling/tools/lat=200ms/N=1": { + "wall_s": 0.759, + "throughput_kickoffs_per_s": 1.32, + "peak_threads": 8, + "thread_starts": 4, + "peak_rss_mb": 178.2, + "rss_growth_mb": 0.7, + "sync_llm_calls": 0, + "async_llm_calls": 3, + "tool_calls": 4, + "loop_lag_max_ms": 7.8, + "loop_lag_p99_ms": 6.0, + "loop_lag_mean_ms": 0.53 + }, + "scaling/tools/lat=200ms/N=10": { + "wall_s": 1.023, + "throughput_kickoffs_per_s": 9.78, + "peak_threads": 20, + "thread_starts": 51, + "peak_rss_mb": 185.1, + "rss_growth_mb": 3.8, + "sync_llm_calls": 0, + "async_llm_calls": 30, + "tool_calls": 40, + "loop_lag_max_ms": 181.3, + "loop_lag_p99_ms": 13.8, + "loop_lag_mean_ms": 3.69 + }, + "scaling/tools/lat=200ms/N=50": { + "wall_s": 2.404, + "throughput_kickoffs_per_s": 20.8, + "peak_threads": 21, + "thread_starts": 213, + "peak_rss_mb": 211.8, + "rss_growth_mb": 14.0, + "sync_llm_calls": 0, + "async_llm_calls": 150, + "tool_calls": 200, + "loop_lag_max_ms": 972.4, + "loop_lag_p99_ms": 159.1, + "loop_lag_mean_ms": 24.28 + }, + "scaling/tools/lat=200ms/N=200": { + "wall_s": 9.344, + "throughput_kickoffs_per_s": 21.4, + "peak_threads": 21, + "thread_starts": 813, + "peak_rss_mb": 310.3, + "rss_growth_mb": 53.0, + "sync_llm_calls": 0, + "async_llm_calls": 600, + "tool_calls": 800, + "loop_lag_max_ms": 3477.1, + "loop_lag_p99_ms": 1171.6, + "loop_lag_mean_ms": 249.45 + }, + "scaling/plain/lat=500ms/N=200": { + "wall_s": 7.704, + "throughput_kickoffs_per_s": 25.96, + "peak_threads": 21, + "thread_starts": 814, + "peak_rss_mb": 240.7, + "rss_growth_mb": 42.6, + "sync_llm_calls": 0, + "async_llm_calls": 400, + "tool_calls": 0, + "loop_lag_max_ms": 3141.5, + "loop_lag_p99_ms": 485.4, + "loop_lag_mean_ms": 92.71 + }, + "overhead/plain/lat=0": { + "per_kickoff_ms_median": 20.97, + "per_kickoff_ms_mean": 21.28, + "thread_starts_per_kickoff": 4.0 + }, + "overhead/tools/lat=0": { + "per_kickoff_ms_median": 25.59, + "per_kickoff_ms_mean": 26.21, + "thread_starts_per_kickoff": 4.2 + }, + "small_executor(4)/plain/lat=200ms/N=20": { + "wall_s": 0.924, + "throughput_kickoffs_per_s": 21.64, + "peak_threads": 15, + "thread_starts": 89, + "peak_rss_mb": 184.3, + "rss_growth_mb": 5.4, + "sync_llm_calls": 0, + "async_llm_calls": 40, + "tool_calls": 0, + "loop_lag_max_ms": 275.3, + "loop_lag_p99_ms": 41.5, + "loop_lag_mean_ms": 9.25 + }, + "cancel/during-llm/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 1.3, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 0, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 0, + "llm_started_total": 1, + "tool_started_total": 0 + }, + "cancel/during-tools/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 1.5, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 0, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 0, + "llm_started_total": 1, + "tool_started_total": 4 + }, + "context/tools": { + "tool": [ + "set-before-akickoff" + ], + "handler": [ + "set-before-akickoff" + ], + "llm": [ + "set-before-akickoff" + ] + } +} \ No newline at end of file diff --git a/lib/crewai/tests/perf/results/p4-planning.json b/lib/crewai/tests/perf/results/p4-planning.json new file mode 100644 index 000000000..1f70eab83 --- /dev/null +++ b/lib/crewai/tests/perf/results/p4-planning.json @@ -0,0 +1,30 @@ +{ + "scaling/planning/lat=200ms/N=1": { + "wall_s": 1.247, + "throughput_kickoffs_per_s": 0.8, + "peak_threads": 8, + "thread_starts": 0, + "peak_rss_mb": 181.5, + "rss_growth_mb": 0.3, + "sync_llm_calls": 8, + "async_llm_calls": 1, + "tool_calls": 6, + "loop_lag_max_ms": 4.5, + "loop_lag_p99_ms": 1.5, + "loop_lag_mean_ms": 0.26 + }, + "scaling/planning/lat=200ms/N=50": { + "wall_s": 15.233, + "throughput_kickoffs_per_s": 3.28, + "peak_threads": 20, + "thread_starts": 12, + "peak_rss_mb": 207.4, + "rss_growth_mb": 16.0, + "sync_llm_calls": 400, + "async_llm_calls": 50, + "tool_calls": 300, + "loop_lag_max_ms": 757.0, + "loop_lag_p99_ms": 9.6, + "loop_lag_mean_ms": 1.22 + } +} \ No newline at end of file diff --git a/lib/crewai/tests/perf/results/p5-planning.json b/lib/crewai/tests/perf/results/p5-planning.json new file mode 100644 index 000000000..c3e5e8e6c --- /dev/null +++ b/lib/crewai/tests/perf/results/p5-planning.json @@ -0,0 +1,211 @@ +{ + "scaling/plain/lat=200ms/N=1": { + "wall_s": 0.439, + "throughput_kickoffs_per_s": 2.28, + "peak_threads": 6, + "thread_starts": 0, + "peak_rss_mb": 179.1, + "rss_growth_mb": 0.4, + "sync_llm_calls": 0, + "async_llm_calls": 2, + "tool_calls": 0, + "loop_lag_max_ms": 8.0, + "loop_lag_p99_ms": 1.6, + "loop_lag_mean_ms": 0.44 + }, + "scaling/plain/lat=200ms/N=10": { + "wall_s": 0.634, + "throughput_kickoffs_per_s": 15.78, + "peak_threads": 17, + "thread_starts": 12, + "peak_rss_mb": 182.5, + "rss_growth_mb": 3.1, + "sync_llm_calls": 0, + "async_llm_calls": 20, + "tool_calls": 0, + "loop_lag_max_ms": 116.3, + "loop_lag_p99_ms": 20.1, + "loop_lag_mean_ms": 3.57 + }, + "scaling/plain/lat=200ms/N=50": { + "wall_s": 1.649, + "throughput_kickoffs_per_s": 30.32, + "peak_threads": 20, + "thread_starts": 14, + "peak_rss_mb": 195.2, + "rss_growth_mb": 11.7, + "sync_llm_calls": 0, + "async_llm_calls": 100, + "tool_calls": 0, + "loop_lag_max_ms": 643.8, + "loop_lag_p99_ms": 112.6, + "loop_lag_mean_ms": 25.83 + }, + "scaling/plain/lat=200ms/N=200": { + "wall_s": 6.853, + "throughput_kickoffs_per_s": 29.18, + "peak_threads": 20, + "thread_starts": 14, + "peak_rss_mb": 243.0, + "rss_growth_mb": 43.5, + "sync_llm_calls": 0, + "async_llm_calls": 400, + "tool_calls": 0, + "loop_lag_max_ms": 2950.2, + "loop_lag_p99_ms": 498.9, + "loop_lag_mean_ms": 152.99 + }, + "scaling/tools/lat=200ms/N=1": { + "wall_s": 0.741, + "throughput_kickoffs_per_s": 1.35, + "peak_threads": 8, + "thread_starts": 0, + "peak_rss_mb": 179.5, + "rss_growth_mb": 0.8, + "sync_llm_calls": 0, + "async_llm_calls": 3, + "tool_calls": 4, + "loop_lag_max_ms": 5.1, + "loop_lag_p99_ms": 1.4, + "loop_lag_mean_ms": 0.29 + }, + "scaling/tools/lat=200ms/N=10": { + "wall_s": 0.976, + "throughput_kickoffs_per_s": 10.24, + "peak_threads": 19, + "thread_starts": 11, + "peak_rss_mb": 186.1, + "rss_growth_mb": 3.4, + "sync_llm_calls": 0, + "async_llm_calls": 30, + "tool_calls": 40, + "loop_lag_max_ms": 140.5, + "loop_lag_p99_ms": 8.2, + "loop_lag_mean_ms": 2.67 + }, + "scaling/tools/lat=200ms/N=50": { + "wall_s": 2.167, + "throughput_kickoffs_per_s": 23.07, + "peak_threads": 20, + "thread_starts": 13, + "peak_rss_mb": 212.7, + "rss_growth_mb": 13.9, + "sync_llm_calls": 0, + "async_llm_calls": 150, + "tool_calls": 200, + "loop_lag_max_ms": 890.1, + "loop_lag_p99_ms": 69.6, + "loop_lag_mean_ms": 22.28 + }, + "scaling/tools/lat=200ms/N=200": { + "wall_s": 9.222, + "throughput_kickoffs_per_s": 21.69, + "peak_threads": 20, + "thread_starts": 13, + "peak_rss_mb": 311.7, + "rss_growth_mb": 53.1, + "sync_llm_calls": 0, + "async_llm_calls": 600, + "tool_calls": 800, + "loop_lag_max_ms": 3511.6, + "loop_lag_p99_ms": 599.4, + "loop_lag_mean_ms": 151.7 + }, + "scaling/planning/lat=200ms/N=1": { + "wall_s": 1.243, + "throughput_kickoffs_per_s": 0.8, + "peak_threads": 10, + "thread_starts": 2, + "peak_rss_mb": 180.2, + "rss_growth_mb": 0.0, + "sync_llm_calls": 2, + "async_llm_calls": 7, + "tool_calls": 6, + "loop_lag_max_ms": 6.1, + "loop_lag_p99_ms": 1.0, + "loop_lag_mean_ms": 0.27 + }, + "scaling/planning/lat=200ms/N=50": { + "wall_s": 4.695, + "throughput_kickoffs_per_s": 10.65, + "peak_threads": 20, + "thread_starts": 12, + "peak_rss_mb": 206.3, + "rss_growth_mb": 15.0, + "sync_llm_calls": 100, + "async_llm_calls": 350, + "tool_calls": 300, + "loop_lag_max_ms": 794.3, + "loop_lag_p99_ms": 48.2, + "loop_lag_mean_ms": 5.04 + }, + "scaling/plain/lat=500ms/N=200": { + "wall_s": 7.172, + "throughput_kickoffs_per_s": 27.89, + "peak_threads": 20, + "thread_starts": 14, + "peak_rss_mb": 242.8, + "rss_growth_mb": 43.3, + "sync_llm_calls": 0, + "async_llm_calls": 400, + "tool_calls": 0, + "loop_lag_max_ms": 2972.6, + "loop_lag_p99_ms": 628.1, + "loop_lag_mean_ms": 96.88 + }, + "overhead/plain/lat=0": { + "per_kickoff_ms_median": 19.55, + "per_kickoff_ms_mean": 20.42, + "thread_starts_per_kickoff": 0.1 + }, + "overhead/tools/lat=0": { + "per_kickoff_ms_median": 24.87, + "per_kickoff_ms_mean": 25.16, + "thread_starts_per_kickoff": 0.2 + }, + "small_executor(4)/plain/lat=200ms/N=20": { + "wall_s": 0.912, + "throughput_kickoffs_per_s": 21.93, + "peak_threads": 16, + "thread_starts": 9, + "peak_rss_mb": 186.1, + "rss_growth_mb": 5.9, + "sync_llm_calls": 0, + "async_llm_calls": 40, + "tool_calls": 0, + "loop_lag_max_ms": 270.3, + "loop_lag_p99_ms": 43.8, + "loop_lag_mean_ms": 9.76 + }, + "cancel/during-llm/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 0.2, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 0, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 0, + "llm_started_total": 1, + "tool_started_total": 0 + }, + "cancel/during-tools/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 0.3, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 0, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 0, + "llm_started_total": 1, + "tool_started_total": 4 + }, + "context/tools": { + "tool": [ + "set-before-akickoff" + ], + "handler": [ + "set-before-akickoff" + ], + "llm": [ + "set-before-akickoff" + ] + } +} \ No newline at end of file diff --git a/lib/crewai/tests/perf/results/p6-final.json b/lib/crewai/tests/perf/results/p6-final.json new file mode 100644 index 000000000..0f9728a73 --- /dev/null +++ b/lib/crewai/tests/perf/results/p6-final.json @@ -0,0 +1,211 @@ +{ + "scaling/plain/lat=200ms/N=1": { + "wall_s": 0.439, + "throughput_kickoffs_per_s": 2.28, + "peak_threads": 5, + "thread_starts": 0, + "peak_rss_mb": 179.2, + "rss_growth_mb": 0.7, + "sync_llm_calls": 0, + "async_llm_calls": 2, + "tool_calls": 0, + "loop_lag_max_ms": 8.7, + "loop_lag_p99_ms": 1.3, + "loop_lag_mean_ms": 0.44 + }, + "scaling/plain/lat=200ms/N=10": { + "wall_s": 1.867, + "throughput_kickoffs_per_s": 5.36, + "peak_threads": 15, + "thread_starts": 10, + "peak_rss_mb": 184.2, + "rss_growth_mb": 4.5, + "sync_llm_calls": 0, + "async_llm_calls": 20, + "tool_calls": 0, + "loop_lag_max_ms": 141.7, + "loop_lag_p99_ms": 4.3, + "loop_lag_mean_ms": 1.24 + }, + "scaling/plain/lat=200ms/N=50": { + "wall_s": 2.238, + "throughput_kickoffs_per_s": 22.34, + "peak_threads": 20, + "thread_starts": 14, + "peak_rss_mb": 197.8, + "rss_growth_mb": 14.2, + "sync_llm_calls": 0, + "async_llm_calls": 100, + "tool_calls": 0, + "loop_lag_max_ms": 683.4, + "loop_lag_p99_ms": 13.9, + "loop_lag_mean_ms": 5.96 + }, + "scaling/plain/lat=200ms/N=200": { + "wall_s": 7.386, + "throughput_kickoffs_per_s": 27.08, + "peak_threads": 20, + "thread_starts": 15, + "peak_rss_mb": 252.5, + "rss_growth_mb": 52.9, + "sync_llm_calls": 0, + "async_llm_calls": 400, + "tool_calls": 0, + "loop_lag_max_ms": 2919.9, + "loop_lag_p99_ms": 97.2, + "loop_lag_mean_ms": 14.05 + }, + "scaling/tools/lat=200ms/N=1": { + "wall_s": 0.745, + "throughput_kickoffs_per_s": 1.34, + "peak_threads": 8, + "thread_starts": 1, + "peak_rss_mb": 179.9, + "rss_growth_mb": 1.0, + "sync_llm_calls": 0, + "async_llm_calls": 3, + "tool_calls": 4, + "loop_lag_max_ms": 6.9, + "loop_lag_p99_ms": 1.2, + "loop_lag_mean_ms": 0.32 + }, + "scaling/tools/lat=200ms/N=10": { + "wall_s": 1.69, + "throughput_kickoffs_per_s": 5.92, + "peak_threads": 20, + "thread_starts": 13, + "peak_rss_mb": 187.8, + "rss_growth_mb": 5.0, + "sync_llm_calls": 0, + "async_llm_calls": 30, + "tool_calls": 40, + "loop_lag_max_ms": 152.1, + "loop_lag_p99_ms": 2.9, + "loop_lag_mean_ms": 1.41 + }, + "scaling/tools/lat=200ms/N=50": { + "wall_s": 3.022, + "throughput_kickoffs_per_s": 16.54, + "peak_threads": 20, + "thread_starts": 13, + "peak_rss_mb": 217.0, + "rss_growth_mb": 18.1, + "sync_llm_calls": 0, + "async_llm_calls": 150, + "tool_calls": 200, + "loop_lag_max_ms": 918.6, + "loop_lag_p99_ms": 12.7, + "loop_lag_mean_ms": 6.38 + }, + "scaling/tools/lat=200ms/N=200": { + "wall_s": 17.212, + "throughput_kickoffs_per_s": 11.62, + "peak_threads": 20, + "thread_starts": 13, + "peak_rss_mb": 328.7, + "rss_growth_mb": 69.9, + "sync_llm_calls": 0, + "async_llm_calls": 600, + "tool_calls": 800, + "loop_lag_max_ms": 3556.0, + "loop_lag_p99_ms": 70.1, + "loop_lag_mean_ms": 6.48 + }, + "scaling/planning/lat=200ms/N=1": { + "wall_s": 1.246, + "throughput_kickoffs_per_s": 0.8, + "peak_threads": 9, + "thread_starts": 2, + "peak_rss_mb": 179.8, + "rss_growth_mb": 0.7, + "sync_llm_calls": 2, + "async_llm_calls": 7, + "tool_calls": 6, + "loop_lag_max_ms": 3.7, + "loop_lag_p99_ms": 1.5, + "loop_lag_mean_ms": 0.29 + }, + "scaling/planning/lat=200ms/N=50": { + "wall_s": 5.296, + "throughput_kickoffs_per_s": 9.44, + "peak_threads": 20, + "thread_starts": 12, + "peak_rss_mb": 209.4, + "rss_growth_mb": 17.8, + "sync_llm_calls": 100, + "async_llm_calls": 350, + "tool_calls": 300, + "loop_lag_max_ms": 788.8, + "loop_lag_p99_ms": 12.7, + "loop_lag_mean_ms": 3.12 + }, + "scaling/plain/lat=500ms/N=200": { + "wall_s": 8.103, + "throughput_kickoffs_per_s": 24.68, + "peak_threads": 20, + "thread_starts": 15, + "peak_rss_mb": 254.8, + "rss_growth_mb": 55.2, + "sync_llm_calls": 0, + "async_llm_calls": 400, + "tool_calls": 0, + "loop_lag_max_ms": 2883.4, + "loop_lag_p99_ms": 121.9, + "loop_lag_mean_ms": 12.23 + }, + "overhead/plain/lat=0": { + "per_kickoff_ms_median": 16.96, + "per_kickoff_ms_mean": 17.48, + "thread_starts_per_kickoff": 0.1 + }, + "overhead/tools/lat=0": { + "per_kickoff_ms_median": 23.0, + "per_kickoff_ms_mean": 23.29, + "thread_starts_per_kickoff": 0.2 + }, + "small_executor(4)/plain/lat=200ms/N=20": { + "wall_s": 1.526, + "throughput_kickoffs_per_s": 13.11, + "peak_threads": 14, + "thread_starts": 8, + "peak_rss_mb": 186.7, + "rss_growth_mb": 6.2, + "sync_llm_calls": 0, + "async_llm_calls": 40, + "tool_calls": 0, + "loop_lag_max_ms": 256.7, + "loop_lag_p99_ms": 6.1, + "loop_lag_mean_ms": 2.66 + }, + "cancel/during-llm/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 0.4, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 0, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 0, + "llm_started_total": 1, + "tool_started_total": 0 + }, + "cancel/during-tools/llm=200ms,tool=300ms": { + "cancel_outcome": "CancelledError", + "await_returned_after_ms": 0.4, + "llm_started_after_cancel": 0, + "llm_finished_after_cancel": 0, + "tool_started_after_cancel": 0, + "tool_finished_after_cancel": 0, + "llm_started_total": 1, + "tool_started_total": 4 + }, + "context/tools": { + "tool": [ + "set-before-akickoff" + ], + "handler": [ + "set-before-akickoff" + ], + "llm": [ + "set-before-akickoff" + ] + } +} \ No newline at end of file