OpenAI models on Bedrock reject inferenceConfig.stopSequences with a 400,
and crewAI adds the "\nObservation:" stop word to every agent call, so
Agent.kickoff() and Crew.kickoff() failed on GPT-6 Sol/Luna, GPT-5.6 and
gpt-oss.
supports_stop_words() now returns False for "openai." model ids and
_get_inference_config() only sends stopSequences when it returns True,
mirroring the Azure provider. Stop words are still applied client-side
by _apply_stop_words, and executors fall back to use_stop_words=False.
Co-authored-by: Claude <[email protected]>
* feat(llms): add deepinfra as an openai-compatible provider
Register deepinfra in the OpenAI-compatible provider registry so
LLM(model="deepinfra/<org>/<model>") routes natively with DEEPINFRA_API_KEY
and no LiteLLM install. Model ids keep their org/model form,
DEEPINFRA_BASE_URL overrides the endpoint, and the docs accordion is added
in en, ar, ko and pt-BR, stating that the provider uses the native OpenAI
client and needs no LiteLLM install.
The model id check requires exactly one slash with both parts non-empty, and
it also runs on the explicit provider="deepinfra" path, where a malformed id
raises a ValueError that shows the expected form.
deepinfra/ model strings previously routed through LiteLLM; is_litellm=True
keeps that path.
* fix(llms): reject malformed deepinfra/ ids before the LiteLLM fallback
A deepinfra/<model> reference that is not org/model fell through to
LiteLLM, so the same bad id failed later and differently depending on
what was installed. The prefixed form now raises the same ValueError as
provider="deepinfra", with the example and the echoed value written the
way the caller wrote the id; both branches share one helper.
---------
Co-authored-by: João Moura <[email protected]>
* feat(cli): record why an evaluation stopped before its verdict
`cli_usage:evaluate` counts presses of Evaluate and `cli_usage:eval` counts
evaluations that started, but about four in five presses never become an
evaluation and nothing said why. Every stop now counts once as
`cli_usage:eval_stopped:<reason>` from a closed list: untraced, declined,
no_project, login_unreadable, network_error, amp_401/403/404/4xx/5xx,
invalid_response, unexpected. The reason rides on EvaluationStoppedError from
where it is raised; messages, exit codes and screens are unchanged.
A run that `crewai eval` hands to the run app and that turns out untraced is
counted by the command only, so one stop is never counted twice.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(cli): count each eval stop once, with the reason shown
- the run app answers for a delegated run it could not start, so the waiting crewai eval does not start it again and count the stop twice
- an unreadable saved login on the unattended path counts as login_unreadable, not untraced
- an evaluation AMP itself failed counts as evaluation_failed, in the CLI and the app; --models unchanged
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* crewai eval --models: per-run cost and time, and no repeated progress lines
The comparison table printed each setup's cost and time summed over all
its runs, while the brief and the report say them per run. It now prints
cost_per_run / seconds_per_run under 'cost / run' and 'time / run' (the
totals, headed 'cost' / 'time', only when the service sends no per-run
figure; a column never mixes the two).
Progress deduped only against the previous line, so parallel setups made
polls alternate and the same lines printed again and again. Each line is
now said once per wait.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* Keep the per-run unit when a model is unpriced, and explain its dash
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
About half the presses of Deploy in the run app never reach a deployment
attempt, and nothing said why. With no saved login the press raises before the
attempt is counted and prompts for one. That prompt now counts once as
`cli_usage:deploy_login_prompted`, and a press that still never reaches an
attempt counts as `cli_usage:deploy_stopped:<reason>`: login_still_required,
login_abandoned or login_failed. Counting re-raises; what the user sees is
unchanged.
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Move r_locked and w_locked acquire calls ahead of the try/finally
that releases them, so an interrupted wait does not release a lock
that was never acquired.
* fix(cli): show actual task output in log-tasks-outputs
* fix(cli): show task result instead of storage blob in log-tasks-outputs
---------
Co-authored-by: Vidit Ostwal <[email protected]>
* feat(cli): crewai eval prints a markdown brief when run by an agent
When stdout is not a terminal (or CREWAI_DMN is on), `crewai eval` and
`crewai eval --models` print the poll response's `brief_markdown` after the
verdict and the report link — written straight to stdout, never through
Rich markup — or "Brief (markdown): <brief_url>" when only the link came.
An older AMP sends neither and nothing extra is printed. In a terminal
nothing changes. Exit codes unchanged.
`crewai eval --help` and the `--models` help say so, and the AGENTS.md the
crew, flow and declarative-flow templates scaffold gains an "Evaluating"
section telling a coding agent to run `crewai eval` after a change and act
on the brief.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* docs(cli): an agent re-runs before it re-evaluates; each command's exit rule; the brief may be a link
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(cli): keep the eval guidance out of tool projects; the brief is conditional in --help
`crewai create tool` copies the shared AGENTS.md too, but a tool project has
no `crewai run` and no traced run for `crewai eval`. The evaluation blocks
are marked in the template and dropped for a tool (`evaluation=False`);
crew, flow and JSON-crew projects keep them, markers stripped.
--help now says the brief prints when AMP provides it, else its link, else
nothing more.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat(flow): introduce experimental turn and public reply identities
This update adds a new module for handling public replies within an existing conversational flow. It introduces the `TurnRunner`, which manages the lifecycle of turns and replies, allowing for experimental features such as tracking input revisions and reply identities. The changes include new classes for `TurnRecord`, `ReplyRecord`, and `TurnState`, enhancing the flow's capabilities while maintaining backward compatibility with existing APIs. Additionally, tests have been added to ensure the functionality of the new features.
* fix docs
* fix(flow): preserve live turn state and terminal events
* fix(flow): remove redundant failure terminal marker
---------
Co-authored-by: Vidit Ostwal <[email protected]>
The lock was still on versions flagged by GHSA-q43m-vhcp-mhvm, GHSA-27vj-qcqg-25rc, GHSA-54p9-h82j-f925, the pymongo 4.18.2 fixes, and GHSA-qqj6-54q6-cxv6.
* fix(tracing): keep tool and task outputs whole in exported spans
Every gen_ai.* content attribute was capped at 32 KiB, and a value over the
cap that was not a message array became an envelope holding a 4 KiB preview:
a 33 KB tool result reached the trace as 4 KB. Graders checking a summary
against the data a tool returned could not decide.
- DEFAULT_MAX_ATTR_BYTES is 384 KiB, sized to Wharf's 3,072,000-byte request
limit: seven attributes at the bound still fit one request.
- Over the bound, a value loses only what is over it (the envelope preview
fills the bound; a message's text is cut by the overshoot, not halved), with
the existing <attr>.truncated / <attr>.original_size_bytes markers.
- Every other string attribute (crewai.task.output, crewai.mcp.tool_result,
serialized flow state) gets the same bound and markers; before, one huge
value made the span too large for Wharf and the exporter dropped it whole.
- An OpenTelemetry SDK attribute length limit, when set, lowers the bound, so
the SDK (which cuts without a marker) never has anything left to cut.
CREWAI_OTEL_MAX_ATTR_BYTES still replaces the bound; no new setting.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(tracing): honour SDK length limits as the SDK reads them; cut plain attributes plainly
- The span attribute length limit is read like SpanLimits: the span setting
wins over the general one, an explicitly empty value means unlimited, and 0
is a limit (every string cut to nothing, with crewAI's marker).
- Plain attributes (crewai.task.output, crewai.mcp.tool_result, flow state)
keep their head up to the bound with the .truncated / .original_size_bytes
markers; only gen_ai.* messages get message-aware truncation, so a
role-bearing JSON list a task produced is never reshaped as a conversation.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(tracing): clamp the attribute bound at the Wharf-safe ceiling; shrink a span that does not fit instead of dropping it
- CREWAI_OTEL_MAX_ATTR_BYTES and the OTel SDK span attribute length limit
can lower the 384 KiB bound, never raise it. A higher
CREWAI_OTEL_MAX_ATTR_BYTES is clamped with one warning naming the value,
the ceiling and why (Wharf's 3,072,000-byte request). No new setting.
- GrantSpanExporter no longer drops a span whose attributes, each within its
bound, add up to more than one request: the largest string attributes are
cut by the overshoot, each with .truncated / .original_size_bytes, until it
fits. A span whose excess is in its resource, scope or events is still
skipped, as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(tracing): a refused trace grant runs the crew untraced instead of failing it
An authenticated run asks AMP for a trace grant as it starts, and a refusal
raised out of `begin_execution` — so when a `crewai login` expired or a token
was revoked, every traced crew and flow failed with "AMP trace grant request
failed (HTTP 401)" before it did any work. A trace is a record of the run, not
a condition of it.
Now the run goes on untraced and logs one warning naming the fix: for a 401/403,
"Run `crewai login` again to trace your runs"; otherwise that the run itself is
unaffected. It never falls back to an anonymous upload of a logged-in user's run
(the grant client's "never downgrade a supplied credential" still holds). The
test that pinned the raise now pins the run completing, the context restored,
nothing uploaded, and the warning.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(tracing): name the credential AMP refused, not always the saved login
`tracing_credential()` sends CREWAI_USER_PAT first, then the platform
integration token, then the saved `crewai login` — so "run `crewai login`
again" was the wrong fix for two of the three, and sent somebody to refresh a
credential that was never the problem. `tracing_credential_source()` follows
the same order (one private resolver behind both), and the warning names the
credential that was refused and the fix for that one.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(tracing): a blank credential runs the crew untraced too
The grant client refuses a blank credential in its constructor with the same
TraceGrantError a refused grant raises, and the constructor sat outside the
boundary — so that case still failed the run. It is inside now. The new tests
also import the grants module one way only.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(tracing): the refused credential is the one that was sent
Resolve the tracing credential and its source once, as a pair, and use that
pair for both the grant request and the warning. Reading the source again
after the request could name a credential AMP never saw.
Also assert a refused grant never builds a span exporter.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(deps): bump pyjwt to >=2.14.0 and defer oauthlib 4.0.0
PyJWT 2.14.0 fixes GHSA-w6j9-cwv2-h6wq and several other advisories in
PyJWKClient and JWK parsing, which crewai-core uses to verify CLI login
tokens. Raise the floor in crewai, crewai-cli and crewai-core so installs
cannot resolve a vulnerable release, and lock 2.14.0 (2.15.0 rejects
padded tokens until 2.15.1, which is still inside the uv cooldown).
oauthlib 4.0.0, the fix for GHSA-xpv3-w29h-x7cv, is under two days old
and inside the 3-day exclude-newer cooldown. The flaw is in the
server-side PKCE verifier check; oauthlib reaches CrewAI only as an OAuth
client (chromadb -> kubernetes -> requests-oauthlib). Ignore the advisory
in pip-audit until 4.0.0 clears the cooldown, with a dated TODO.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(deps): bump pyjwt to 2.15.0, urllib3 to 2.8.0 and virtualenv to 21.13.0
Advisories published 2026-09-30 against the versions #7823 locks:
- PyJWT 2.14.0: GHSA-42vr-xj54-vc7v (fixed in 2.15.0); floor raised to >=2.15.0 in crewai, crewai-core and crewai-cli
- urllib3 2.7.0: GHSA-8988-9cw3-xx77, GHSA-gh4c-6fx4-qh6g, GHSA-vxq7-64xx-v4gw (fixed in 2.8.0); override raised to >=2.8.0
- virtualenv 21.4.2: PYSEC-2026-4011..4014 (fixed by 21.7.13); override >=21.7.13, locked at 21.13.0
All three fixes are older than the 3-day exclude-newer cutoff, so no per-package override is needed.
Local pip-audit over the exported lock: no known vulnerabilities (8 ignored, the repo's list).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Co-authored-by: Craig Estep <[email protected]>
litellm <1.88.6 has GHSA-3cv6-jpf6-8222 (CVE-2026-84377): authenticated
SSRF and provider-credential exfiltration via unvalidated request-body
routing parameters in the LiteLLM proxy server. crewAI uses litellm as a
client SDK and never runs the proxy, but pip-audit flags the package and
fails the vulnerability scan on every PR.
- lib/crewai/pyproject.toml: crewai[litellm] floor litellm>=1.84.0 -> >=1.88.6
- pyproject.toml: advisory comment beside the other overrides
- uv.lock: litellm 1.84.8 -> 1.88.6; nothing else moved (same dependency set)
1.88.6 (2026-08-09) is older than the 3-day exclude-newer cutoff, so no
exclude-newer-package override is needed.
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat(llm): llm_overlay maps models too, not only agent roles
A key `model:<provider/model>` maps every LLM built from that model string
inside the block, and `model:*` every LLM built from any model string. That
reaches what a role cannot: a flow step's own `LLM(...).call()`. The key is
read in `LLM.__new__` (and so `create_llm` and an agent's `llm="..."`), and
on an agent's declared llm built outside the block; a role key still wins for
that agent. The caller's settings follow the mapped model by the rule a
declared llm's do. `MODEL_KEY_PREFIX` is public so another package can
feature-detect the form.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(telemetry): allow-list cli_usage:eval_models attributes
The span `crewai eval --models` records carries whether the caller was
logged in, the models compared (provider/model names) and how many, in both
telemetry classes. Nothing about the run, its output or the organization.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(cli): crewai eval --models compares models on the deployment
`crewai eval --models "a,b"` (and `--deployment UUID` when AMP cannot tell
the deployment from the project id) asks AMP for a models evaluation: the
project id, the models from one comma-separated list (stripped, repeats
dropped, provider/model required, 1-5) and the project's eval.jsonc. It
needs the login, prints and opens the report link, says where the
comparison is while it runs, then prints one row per model (goal, tasks,
agents, tools, cost, time; the deployed one marked, the best per column
starred) and the top three suggestions. Exit 0 when it finished, 1 when it
failed or could not start. Everything from the wire is printed as Text.
The comparison is counted as `cli_usage:eval_models` once AMP accepts it.
Mode 1 now sends the project id with `create_evaluation` too, so every
evaluation is filed under the project it is about.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(cli): crewai deploy create and push send the project id
Every create (git or ZIP) and every push (redeploy by name or uuid, or a
ZIP update) carries `[tool.crewai].project_id` when the project has one, so
AMP knows which project a deployment runs and `crewai eval --models` can find
it. The id is read, never minted: a deploy does not rewrite pyproject.toml.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(llm): a model key maps an LLM whose declared SDK is not installed
Nothing is built on the declared model, so a missing SDK for it is no
reason to fail the mapped build; its credentials cannot be matched to the
new provider, so none are carried.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* feat(cli): crewai eval --models leaves its criteria behind as eval.jsonc
When the comparison's answer carries `eval_config` and the project has no
eval.jsonc, it is written through `write_eval_config` and the same one line
Mode 1 prints says so. An existing file is never overwritten.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(llm): a model key compares providers by provider, not by class
`_same_route_provider` now follows `_same_provider`'s rule case for case: a
LiteLLM-routed `LLM(...)` is on the same provider as a native route when its
provider names that native class, so its key and endpoint follow a model
key to another model of the same provider. Another provider still gets none.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* chore(llm): say why mark_mapped may skip an object, and one import form in its tests
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(cli): usage stats name a compared model only when it is a public one
`cli_usage:eval_models` carried the `--models` strings as typed, and those
can identify a customer: a fine-tune id, an Azure deployment name, a
self-hosted model or host. A model is now sent by name only when it is an
exact entry of crewAI's model catalog (`LLM_CONTEXT_WINDOW_SIZES`), else as
`<provider>/other`, and a provider crewAI does not route as `other/other`.
An `ft:` id is never sent.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(cli): a refused comparison's test no longer keeps a None return
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* test(llms): re-importing OpenAI's completion module restores its package attribute
`test_openai_completion_module_is_imported` deletes the module from
sys.modules and lets `LLM(...)` import it again, which binds the new module
on `crewai.llms.providers.openai` as well. monkeypatch restored only the
sys.modules entry, so every later test on that worker saw two module
objects for one name: `patch("crewai.llms.providers.openai.completion.X")`
patched the attribute's module while `from ... import X` read sys.modules'.
On Python 3.10 that made tests/llms/azure/test_azure_responses.py fail
whenever it shared a worker with test_openai.py and test_crew_loader.py,
which the new tests' even split now did. The attribute is restored too.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(llm): model keys strip only a model's own prefix; a role builds from the declaration
Two review findings on the model-for-model overlay:
- `overlay_model_for_model` took the text after the first slash as another
name for the model, so an aggregator's route (`openrouter/openai/gpt-4o`)
matched a `model:openai/gpt-4o` or `model:gpt-4o` key meant for the
native model. A prefix is now stripped only when it is a native provider's
own and what remains has no slash; an instance on an aggregator is compared
as `<provider>/<its model id>`.
- With a role key and a model key both active, the model key mapped the
declared llm first and the role's model was built like that mapped
instance, losing the caller's key and endpoint. The overlay now remembers
what the caller declared for every llm it built, and a role key is built
from that declaration.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(llm): model keys read a provider's other names; a LiteLLM choice survives a missing SDK
- The router's provider aliases are now one table, `crewai.llm.PROVIDER_ALIASES`,
and `llm_overlay` compares model names through it: `google/x` and
`gemini/x` are one model whether the llm is built in the block or was
built before it (a prebuilt instance records `gemini`). An exact key still
wins, and an aggregator's route is still compared whole.
- When the declared model's SDK is missing, a caller's explicit
`is_litellm=True` is kept for the mapped model instead of dropping to a
native route.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(deploy): a git create sends the project id once, at the top level
AMP reads project_id at the top level on every deploy route, so the git
create no longer repeats it inside `deploy`. The redeploy docstring records
why its optional body is safe on an AMP that predates project ids.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* refactor: move message summarization into SummarizeMessages
Keep context compaction in one class so agent_utils only starts the summary when the context window is exceeded.
* refactor: keep SummarizeMessages inside agent_utils
Reuse one summarizer instance and drop the separate module now that compaction lives next to summarize_messages.
* fix: retry summarizer chunks with tighter token estimates on overflow
When a chunk hits the context window during summarization, re-chunk at 3 and 2.5 chars per token before failing.
* refactor: drive summarizer retries with char_level_index on summarize
Chunk at levels 4/3/2.5 chars per token and recurse through summarize until index 3 raises.
* refactor: colocate summarizer token levels on SummarizeMessages
Keep chars-per-token retry levels as a class constant instead of module scope.
* test: cover summarizer retries across all char-per-token levels
Assert chunking uses indices 0–2 when context overflows twice before a successful summary.
* fix: use a new SummarizeMessages instance per compaction
Avoid sharing one summarizer across concurrent context-window recoveries.
* fix: keep summarizer context retries inside async _summarize_one
Re-chunk failed chunks at tighter token levels with gather instead of calling summarize recursively.
* refactor: nest _summarize_one inside _summarize_all
Per-chunk summarization is only reachable from _summarize_all or its own retry path; tests exercise _summarize_all instead of calling _summarize_one directly.
Since 1.15.22 every crew and flow kickoff owns an execution uuid, so the
legacy TraceBatchManager handlers never finalize a batch and the
`tracing:ephemeral_sent` / `tracing:authenticated_sent` Feature Usage
events stopped. Emit them from GrantSpanExporter.record_export, the point
both upload tiers reach once every span has arrived.
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* feat(llms): centralize context window definitions
* refactor(llm): use shared context window resolver
* refactor(llms): share native context window lookup
* test(llms): cover context window parity
* fix(llms): preserve regional Bedrock context window
* chore(llms): refresh provider context window catalogs
* fix(llms): restore affected model context windows
* fix(llms): add active Azure and Anthropic context windows
Record catalog limits for current Azure OpenAI and Claude model IDs so lookups match the providers' active context windows.
* fix(llms): sync Gemini and Bedrock context windows
Drop retired catalog IDs and record the active Gemini and Bedrock text-model limits from the current provider docs.
* fix(llms): add missing OpenAI context windows
Record active OpenAI IDs that do not match an existing prefix, including gpt-5.6-cyber so it keeps its 400k window.
* fix(llms): keep Gemini thinking snapshot window in sync
Restore the retired thinking snapshot so native Gemini and LiteLLM resolve the same 1M context window.
* test(llms): drop retired Gemini thinking snapshot from parity
The model is no longer in the Gemini catalog, so native and LiteLLM no longer need to agree on its context window.
* test(llms): expect Azure GPT-4 and GPT-4o to share a 128k window
Azure GPT-4 is no longer smaller than GPT-4o, so the comparison asserted equal usable windows.
* fix(llms): address context-window review findings
Keep provider catalogs from overwriting each other, restore Gemma and legacy Bedrock Claude windows, and match the global Kimi inference id.
* fix(llms): drop unused context-window re-exports from LLM
Import the usage ratio from context_window in tests so llm.py no longer re-exports unused constants.
* fix(cli): `crewai eval` exits 1 unless the gate passed, and says to log in when nothing was traced unattended
The exit code is what a CI job reads, and it said only whether the evaluation
finished: a run whose goal gate FAILED exited 0, so a pipeline gating on
`crewai eval` waved it through. It is the gate's now — 0 only for PASSED; a
failed gate, one without a verdict, and an evaluation that stopped are 1. The
command's help says so.
With no terminal and no login, an anonymous run's trace stays on the machine
even with tracing on (nobody is there to approve the upload). `crewai eval` then
told the user to turn tracing on, which they had — the same loop again. When
tracing is on and nobody is logged in, it now says what traces an unattended
run: `crewai login`.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(cli): an unreadable login is the reason given when nothing was traced unattended
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* fix(cli): print the unattended reason as text, never markup
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
* fix(tracing): turning tracing on is the answer; stop asking again
Two things a user who ran `crewai eval` sees today.
`crewai eval` offers to turn tracing on and run the crew. The run finishes —
and the ephemeral buffer asks "Share this execution trace with CrewAI?", which
is the same question again. Turning tracing on IS consent: `tracing_asked_for()`
is true when `CREWAI_TRACING_ENABLED` says so or when `tracing=True` was passed
in this context, and the buffer takes it as the answer. First-time
auto-collection still asks, because nobody asked for that one.
And the offer named the mechanism rather than the effect — "Turn tracing on
(CREWAI_TRACING_ENABLED=true stays in .env) and run the crew now?", then
"Tracing is on for this project (CREWAI_TRACING_ENABLED=true in .env)." The
variable is how it works, not what the user is agreeing to; both lines now say
what happens and leave the plumbing to the .env file it is written in.
491 telemetry tests and 62 eval-CLI tests green.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* fix(tracing): the TUI's modal is a prompt too, and say where traces go
Two follow-ups from running it.
The terminal UI installs its own consent callback, so turning tracing on still
bought a modal at the end of the run — the callback is consulted before the
default prompt, and the check only guarded the prompt. A host's callback stays
authoritative, because an embedder may answer no for reasons of its own; what
changed is that the TUI's callback, which exists to ASK, now answers for itself
when tracing was asked for.
And having taken the variable out of the offer, the offer should say what saying
yes means: "Turn tracing on and run the crew now? The run's trace is sent to
CrewAI AMP", and afterwards "Tracing is on for this project — its runs are
traced to CrewAI AMP."
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* feat(tui): an Evaluate button beside View Traces and Deploy
A run finishes in the terminal UI and the next question is usually "was that
any good?" — which `crewai eval` answers, from the command line, about the run
that just happened. The button is there now, with `e` bound to it.
It behaves like Deploy rather than like View Traces: it leaves the UI and then
runs the command, because `crewai eval` talks — it prints the link to follow,
waits for the verdict and prints that — and none of it belongs inside a
full-screen app. It calls `eval_crew()` itself rather than reimplementing any
of it, so the button and the command can never say different things; a clean
SystemExit is the command deciding, and a failure is reported without failing
the run that already succeeded.
Declarative flows get it too — same button, same tail.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* feat(telemetry): count `crewai eval`, not only the button that runs it
The TUI's buttons each record a `cli_usage:` feature span, so the new Evaluate
button was counted — but the command it runs was not, and most people will type
`crewai eval` rather than press a button. Every evaluation now records
`cli_usage:eval`; the button keeps its own `cli_usage:evaluate`, so the
difference between the two is how many were started from the TUI.
It counts the command and nothing about the run: no id, no project, no verdict.
And it is wrapped like every other telemetry call here — a missing tracer or a
refused span never stops an evaluation.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* feat(telemetry): say which run was evaluated, and by whom
`project_id` already rides every span. What only this command knows is which
run is being graded, whether the caller was logged in, and — when they are —
which organization they are logged in to. A feature span can carry attributes
now, and `cli_usage:eval` carries those three.
What it does not carry is anything about the work: no inputs, no output, no
verdict, no criteria. The execution id is the join key; what was found lives in
AMP, where the evaluation itself is.
An anonymous caller reports `authenticated=false` and an empty organization, and
is not asked for one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* Address the review: a terminal, a thread, and the right run
Three findings worth acting on, one already fixed, and ruff.
**The security review is right.** `prompt_user_for_trace_viewing` fails closed
where no one can answer, so an anonymous run in CI has never uploaded. Treating
the switch as consent everywhere would have changed that: a project `.env`
copied into CI carries the variable, not the person. `tracing_asked_for()` now
answers only where a prompt could have been shown — the double-ask this PR is
for happens in a terminal, and nothing else moves.
**The ContextVar never crosses the worker.** `set_tracing_enabled` runs in the
crew's validator, on the thread the crew was BUILT on, and a flow never sets it
at all — so `tracing=True` without the env var still opened the modal. The TUI
reads the declaration off the object as well: the same yes, wherever written.
**The button now names the run it watched.** `_chain_eval()` called `crewai eval`
with no id, which looks up the last recorded run — a PREVIOUS run when this one
was not traced, and an offer to run the crew again when there is no record at
all. The TUI remembers what was recorded before the run, hands the new id to the
command, and says "this run was not traced" rather than grading someone else's.
**Already fixed**: telemetry is recorded after `get_or_create_project_id()`, not
before — that moved when the span gained its attributes.
492 telemetry tests, 187 CLI tests. Ruff format and check clean.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* Close all three doors the prompt closes, not just the terminal
The security review named three places `prompt_user_for_trace_viewing` fails
closed — no terminal, a suite under test, and a host that suppressed tracing
messages — and the first pass only honoured the terminal. A run under test, or
inside a host that asked for silence, could still have uploaded on the strength
of a variable.
The three conditions are one function now, `_prompt_can_be_shown()`, used by the
prompt itself and by `tracing_asked_for`, so the rule and the prompt it answers
cannot drift apart. Where the question could not have been put to anybody, there
is nothing to answer and nothing leaves the machine.
The terminal UI decides for itself, and says why: it silences crewAI's console
to keep it out of its own layout, which is a reason a PROMPT cannot be shown,
not a reason to ask a second time — it has a screen and somebody in front of it.
494 telemetry tests, 88 TUI tests. Ruff clean.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* Tests: a real session for the consent rule, and a laid-out screen for a click
The rule that a declared CREWAI_TRACING_ENABLED needs no modal only holds
where a prompt could have been shown, and a test suite is one of the places
it could not — so the three cases about a real session say a person is here.
The consent click waited for the screen, which is not the same as waiting for
its buttons to have a region; it now waits for the layout it clicks on.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* A run started for an evaluation closes itself, even in a child process
A project's crew and flow are run through `uv run …`, so the app lives in
another process and the ContextVar saying "an evaluation is waiting" never
reaches it: the screen stayed open and `crewai eval` waited behind it, which
is the trap the handshake was meant to remove. The environment is the one
thing that crosses into that process, so the command sets a variable there and
clears it after; the id travels back the way it already did, through the
project's last-run record. A run that FAILED leaves too — the command behind
the screen has its own way of saying there is nothing to grade, and it cannot
say it while somebody has to quit a screen first.
And the spans a run exports reach AMP a moment after the run ends, sometimes
later than that, so a command that just watched the run would ask for a trace
still in flight and be told, correctly and uselessly, that there is nothing
there. A run we know is fresh now waits up to two minutes and says it is
waiting; an id somebody typed fails at once, as it always has.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* The evaluation happens in the app, on the screen the reader is already on
Closing the app the moment the run ended and leaving somebody in a bare
terminal, waiting some seconds for a link to appear and a browser to open, is
worse than the screen it replaced. So the evaluation runs where the run does:
`Evaluating this run` under the run's own line, the report link beside it, the
browser opened from there, and `Goal gate PASSED · goal 5/5 · …` when it comes
back. Nothing closes; the reader leaves when they are done reading, and the
link is printed once more on the way out so it outlives the screen.
The Evaluate button does the same thing now instead of quitting the app and
chaining into the command, so there is one implementation of one experience —
`_chain_eval` and `_want_eval` are gone. Pressing it again opens the report it
already made rather than paying for a second evaluation.
Underneath, `crewai eval`'s two steps became callable as values: they raise
`EvaluationStoppedError` with the sentence to show instead of printing and
exiting, and `evaluate_run()` is the app's one door — the url as soon as AMP
gives it, every unfinished answer while it waits. The command keeps the
terminal wording it had; when it ran the crew itself, it now has nothing left
to say and steps aside for the app it opened.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* The button says which report it opens
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* The button works while it waits: it spins, and it stays pressable
A dimmed label that never moves reads as a screen that has stopped. The
Evaluate button now carries the same spinner the run's own line does, the
clock goes back up to eight frames a second while the evaluation runs, and the
button is no longer disabled — the report page is where the progress is, so
pressing it while it waits is the useful thing to do.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* View Traces opens the traces
The button never had a link. crewAI grants one for every exported run and
prints it after a run — but never under a TUI, where its console is kept out
of the layout — so the app had nothing to open and said so, which read as a
feature that had stopped working.
The link is now recorded beside the rest of what crewAI writes when a run's
spans reach AMP, and the button reads it off that record, matched to THIS
app's execution rather than to whatever ran last in the project. When there is
no link the notice says which of the two reasons it is: a run nobody traced,
or a traced run AMP granted no viewer for — only one of those is the reader's
to fix.
Verified against a real anonymous run: pressing `t` opens
app.crewai.com/crewai_plus/ephemeral_traces/<id> and shows it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* The review's five: a trust boundary, a fallback, a retry, an exit and a count
**A project's `.env` cannot lend itself the login.** `evaluate_run()` built its
AMP client after the project's `.env` had been loaded with override, so a
`CREWAI_PLUS_URL` a project put there landed in the set of origins trusted with
the saved `crewai login` bearer. The command reads that variable BEFORE the
`.env` — a shell export is this machine speaking — but inside the app there is
no such "before", so the variable is left out of the set entirely. The request
still follows it; the credential does not.
**A run no app evaluated is still evaluated.** The command returned as soon as
a trace existed, assuming the app had graded it — but a conversational session
never ends the way a crew does and a flow with human feedback takes the
terminal instead, so `crewai eval` could exit 0 having graded nothing. The app
now leaves a marker naming the run it started evaluating, and the command
grades anything that marker does not name.
**An evaluation that stopped can be tried again.** The button re-enabled itself
after a failure but the action still treated any existing evaluation as
finished, so a 502 could only be escaped by re-running the crew.
**A login that exits is shown on screen.** `saved_login()` fails by exiting,
which is not an `Exception` and so slipped past the worker's hands: the button
could sit on "Evaluating…" forever with the reason gone with the exit.
**An evaluation the app runs is counted.** `cli_usage:eval` is the count of
evaluations that begin, and the ones that begin in the app were missing from
it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* A stop is a sentence, and the app's own word settles what it graded
**"1" is not a reason.** A credential that exists but cannot be read stopped
the evaluation by printing its sentence and exiting — under the app the print
lands beneath the layout and the exit carries only a code, so the screen said
`1`. `saved_login()` now raises the stop like everything else that stops an
evaluation, the command catches it where it builds the client and prints it as
before, and the app shows it. The test that claimed to cover this raised a
`SystemExit` carrying a sentence, which is not what `_fail` raises; it now
breaks the credential store and reads what reaches the screen. The worker
still catches an exit, and says that the reason did not survive rather than
echoing its code.
**The record is the project's last finish, not this run.** Another run
finishing in the same project replaces it while the app is open, so "is the
marker about the id in the record" could send the command off to evaluate a
different execution. The marker is now the app's own word about the run it
watched — the id it evaluated, or null when it looked and found nothing to
grade — and the command trusts it over the record, matching on time rather
than on an id that may already have been overwritten. No marker still means no
app got here, which is the conversational session and the flow that takes the
terminal, and the command grades those itself.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* A project can say what good means, and `crewai eval` carries it
An evaluation grades a run against the crew's own `expected_output`s — which
is the prompt that produced the output, so it catches a crew that ignored its
own contract and nothing else. What discriminates lives in criteria somebody
writes, and there was nowhere to write them.
Now there is. After the first evaluation the criteria it used are written to
`eval.jsonc` in the project — once, never over a file that exists, because the
one thing worse than no criteria is criteria that vanish whenever somebody
evaluates — and the line that says so points at what to do with it. From then
on the file travels with every evaluation, as it was written, comments and
all: what a criterion means is the grader's to read, not this command's.
Both doors do it, since most evaluations now happen inside the run app: the
app writes the file and names it in the line that outlives its screen, and
`crewai eval` in a terminal says it there.
The file is bounded at the same 64KB AMP holds the field to, so one that would
be refused is never uploaded and the run is graded anyway.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
* The review's asks: anonymous eval stats, a handshake a project cannot forge
Usage stats are anonymous, so the `crewai eval` count says whether the caller
was logged in and nothing else — no execution id, no organization (the product
decision on this PR). Which runs were evaluated, and by whom, is AMP's record.
A feature span now keeps only the keys its own feature lists, in both
telemetry classes; everything else — the dimensions every span carries
included — is dropped.
The run app's "an evaluation is waiting" handshake was an environment variable
any project's `.env` could set, which would start an evaluation with the
machine's login on a plain `crewai run`. Its value is now a random token that
counts only while the command's file of that name exists in the user's crewAI
data directory; a `.env` cannot create a file.
And three smaller ones: when no app evaluated the run, the command grades the
project's record only if it was written after this run began; the "Wrote
eval.jsonc" line after the app closes now appears (the key it read was never
set); an oversized eval.jsonc is said once, through the caller's note, rather
than printed under the TUI on every retry.
uv.lock is back to main's: the change was re-resolution drift, nothing this PR
needs.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
* A report link is a web address, and an anonymous request names no organization
Two from the security review of 52fa5f3ae.
The report url comes from whichever AMP answered — which a project's `.env`
may choose — and is printed as a link and opened without a click. It is now
checked where it arrives, before anything shows or opens it: HTTPS, or plain
HTTP to this machine, no credentials in it, no control characters. Anything
else costs the link and nothing more. The host is not pinned: the report is
served by the evaluation service, not by AMP.
A request sent without the login — nobody logged in, or an AMP this machine is
not logged in to — no longer carries the saved organization id. AMP reads it
only beside a credential, so it said nothing to AMP and told an AMP the login
does not go to which organization was asking. Dropped, as anonymous trace
grants already drop it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
* fix(deps): pin instructor <1.16 to keep OpenAI Mode.TOOLS registered
instructor 1.16.0 removed the OpenAI Mode.TOOLS handler in favor of
Mode.RESPONSES_TOOLS. `InternalInstructor` calls `instructor.from_provider`,
which still defaults to Mode.TOOLS internally, so any native-OpenAI kickoff
that hits a structured-output path (guardrails, output_pydantic, LLM
guardrails) fails immediately with `ModeError: Mode.TOOLS is not registered
for provider Provider.OPENAI`.
Cap the range while we teach `_create_instructor_client` to negotiate a
supported mode against `mode_registry`.
Refs COR-861 (Docusign, deployment 134108).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix: avoid redundant cache saves in tests workflow
Co-authored-by: vinibrsl <[email protected]>
* docs: explain instructor compatibility pin
Co-authored-by: vinibrsl <[email protected]>
---------
Co-authored-by: Daniel Minella <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Nothing about the command changes. It is still `crewai eval`, still listed in
`--help`, still takes `--run`, still prints the same lines. Only the module
moved, from `crewai_cli/eval_crew.py` to `crewai_cli/experimental/eval_crew.py`,
with its tests alongside.
The package says what the command's location could not: its output and its
options may change between releases without the deprecation cycle a settled
command gets. That is honest about where it stands — most of what it depends
on lives outside this repository, in an AMP endpoint and the evaluator behind
it, and both are new enough that the report's shape is still moving. The
command is already built to absorb some of that, printing whatever the
evaluation graded rather than a list of its own, but the promise it makes to a
user should match.
`crewai/experimental/` has the same role on the library side, so this is the
convention the repository already has rather than a new one. The CLI had no
experimental package before; it does now, and `crewai eval` is its first
tenant.
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Lorenze Jay