Files
João MouraandClaude Opus 5.5 1b9bbccbed feat(cli): crewai eval --models, and llm_overlay swaps models as well as roles (#7811)
* feat(llm): llm_overlay maps models too, not only agent roles

A key `model:<provider/model>` maps every LLM built from that model string
inside the block, and `model:*` every LLM built from any model string. That
reaches what a role cannot: a flow step's own `LLM(...).call()`. The key is
read in `LLM.__new__` (and so `create_llm` and an agent's `llm="..."`), and
on an agent's declared llm built outside the block; a role key still wins for
that agent. The caller's settings follow the mapped model by the rule a
declared llm's do. `MODEL_KEY_PREFIX` is public so another package can
feature-detect the form.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* feat(telemetry): allow-list cli_usage:eval_models attributes

The span `crewai eval --models` records carries whether the caller was
logged in, the models compared (provider/model names) and how many, in both
telemetry classes. Nothing about the run, its output or the organization.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* feat(cli): crewai eval --models compares models on the deployment

`crewai eval --models "a,b"` (and `--deployment UUID` when AMP cannot tell
the deployment from the project id) asks AMP for a models evaluation: the
project id, the models from one comma-separated list (stripped, repeats
dropped, provider/model required, 1-5) and the project's eval.jsonc. It
needs the login, prints and opens the report link, says where the
comparison is while it runs, then prints one row per model (goal, tasks,
agents, tools, cost, time; the deployed one marked, the best per column
starred) and the top three suggestions. Exit 0 when it finished, 1 when it
failed or could not start. Everything from the wire is printed as Text.
The comparison is counted as `cli_usage:eval_models` once AMP accepts it.

Mode 1 now sends the project id with `create_evaluation` too, so every
evaluation is filed under the project it is about.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* feat(cli): crewai deploy create and push send the project id

Every create (git or ZIP) and every push (redeploy by name or uuid, or a
ZIP update) carries `[tool.crewai].project_id` when the project has one, so
AMP knows which project a deployment runs and `crewai eval --models` can find
it. The id is read, never minted: a deploy does not rewrite pyproject.toml.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(llm): a model key maps an LLM whose declared SDK is not installed

Nothing is built on the declared model, so a missing SDK for it is no
reason to fail the mapped build; its credentials cannot be matched to the
new provider, so none are carried.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* feat(cli): crewai eval --models leaves its criteria behind as eval.jsonc

When the comparison's answer carries `eval_config` and the project has no
eval.jsonc, it is written through `write_eval_config` and the same one line
Mode 1 prints says so. An existing file is never overwritten.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(llm): a model key compares providers by provider, not by class

`_same_route_provider` now follows `_same_provider`'s rule case for case: a
LiteLLM-routed `LLM(...)` is on the same provider as a native route when its
provider names that native class, so its key and endpoint follow a model
key to another model of the same provider. Another provider still gets none.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* chore(llm): say why mark_mapped may skip an object, and one import form in its tests

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(cli): usage stats name a compared model only when it is a public one

`cli_usage:eval_models` carried the `--models` strings as typed, and those
can identify a customer: a fine-tune id, an Azure deployment name, a
self-hosted model or host. A model is now sent by name only when it is an
exact entry of crewAI's model catalog (`LLM_CONTEXT_WINDOW_SIZES`), else as
`<provider>/other`, and a provider crewAI does not route as `other/other`.
An `ft:` id is never sent.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(cli): a refused comparison's test no longer keeps a None return

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(llms): re-importing OpenAI's completion module restores its package attribute

`test_openai_completion_module_is_imported` deletes the module from
sys.modules and lets `LLM(...)` import it again, which binds the new module
on `crewai.llms.providers.openai` as well. monkeypatch restored only the
sys.modules entry, so every later test on that worker saw two module
objects for one name: `patch("crewai.llms.providers.openai.completion.X")`
patched the attribute's module while `from ... import X` read sys.modules'.
On Python 3.10 that made tests/llms/azure/test_azure_responses.py fail
whenever it shared a worker with test_openai.py and test_crew_loader.py,
which the new tests' even split now did. The attribute is restored too.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(llm): model keys strip only a model's own prefix; a role builds from the declaration

Two review findings on the model-for-model overlay:

- `overlay_model_for_model` took the text after the first slash as another
  name for the model, so an aggregator's route (`openrouter/openai/gpt-4o`)
  matched a `model:openai/gpt-4o` or `model:gpt-4o` key meant for the
  native model. A prefix is now stripped only when it is a native provider's
  own and what remains has no slash; an instance on an aggregator is compared
  as `<provider>/<its model id>`.
- With a role key and a model key both active, the model key mapped the
  declared llm first and the role's model was built like that mapped
  instance, losing the caller's key and endpoint. The overlay now remembers
  what the caller declared for every llm it built, and a role key is built
  from that declaration.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(llm): model keys read a provider's other names; a LiteLLM choice survives a missing SDK

- The router's provider aliases are now one table, `crewai.llm.PROVIDER_ALIASES`,
  and `llm_overlay` compares model names through it: `google/x` and
  `gemini/x` are one model whether the llm is built in the block or was
  built before it (a prebuilt instance records `gemini`). An exact key still
  wins, and an aggregator's route is still compared whole.
- When the declared model's SDK is missing, a caller's explicit
  `is_litellm=True` is kept for the mapped model instead of dropping to a
  native route.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(deploy): a git create sends the project id once, at the top level

AMP reads project_id at the top level on every deploy route, so the git
create no longer repeats it inside `deploy`. The redeploy docstring records
why its optional body is safe on an AMP that predates project ids.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-30 10:53:42 -07:00

497 lines
18 KiB
Python

import os
import unittest
from unittest.mock import ANY, MagicMock, patch
from crewai_cli.plus_api import PlusAPI
class TestPlusAPI(unittest.TestCase):
def setUp(self):
self.api_key = "test_api_key"
self.api = PlusAPI(self.api_key)
self.org_uuid = "test-org-uuid"
def test_init(self):
self.assertEqual(self.api.api_key, self.api_key)
self.assertEqual(self.api.headers["Authorization"], f"Bearer {self.api_key}")
self.assertEqual(self.api.headers["Content-Type"], "application/json")
self.assertIn("CrewAI-CLI/", self.api.headers["User-Agent"])
self.assertTrue(self.api.headers["X-Crewai-Version"])
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_create_evaluation(self, mock_make_request):
mock_response = MagicMock()
mock_make_request.return_value = mock_response
response = self.api.create_evaluation("6f31fe1a-20bd-4bfe-a011-25d6b9341f62")
mock_make_request.assert_called_once_with(
"POST",
"/crewai_plus/api/v1/tracing/evaluations",
json={"execution_id": "6f31fe1a-20bd-4bfe-a011-25d6b9341f62"},
timeout=120.0, # AMP reads the run's spans inside this request
)
self.assertEqual(response, mock_response)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_create_evaluation_files_it_under_the_project(self, mock_make_request):
self.api.create_evaluation("run-1", eval_config="{}", project_id="proj-1")
mock_make_request.assert_called_once_with(
"POST",
"/crewai_plus/api/v1/tracing/evaluations",
json={"execution_id": "run-1", "eval_config": "{}", "project_id": "proj-1"},
timeout=120.0,
)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_create_models_evaluation(self, mock_make_request):
mock_response = MagicMock()
mock_make_request.return_value = mock_response
response = self.api.create_models_evaluation(
["openai/gpt-4o-mini", "anthropic/claude-haiku-4-5"],
project_id="proj-1",
eval_config="// ours\n{}",
deployment_id="dep-1",
)
mock_make_request.assert_called_once_with(
"POST",
"/crewai_plus/api/v1/tracing/evaluations",
json={
"kind": "models",
"project_id": "proj-1",
"models": ["openai/gpt-4o-mini", "anthropic/claude-haiku-4-5"],
"eval_config": "// ours\n{}",
"deployment_id": "dep-1",
},
timeout=120.0,
)
self.assertEqual(response, mock_response)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_create_models_evaluation_sends_only_what_it_was_given(
self, mock_make_request
):
self.api.create_models_evaluation(["openai/gpt-4o-mini"], project_id="proj-1")
mock_make_request.assert_called_once_with(
"POST",
"/crewai_plus/api/v1/tracing/evaluations",
json={
"kind": "models",
"project_id": "proj-1",
"models": ["openai/gpt-4o-mini"],
},
timeout=120.0,
)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_get_evaluation(self, mock_make_request):
mock_response = MagicMock()
mock_make_request.return_value = mock_response
response = self.api.get_evaluation("ev-1")
mock_make_request.assert_called_once_with(
"GET", "/crewai_plus/api/v1/tracing/evaluations/ev-1", timeout=30.0
)
self.assertEqual(response, mock_response)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_login_to_tool_repository(self, mock_make_request):
mock_response = MagicMock()
mock_make_request.return_value = mock_response
response = self.api.login_to_tool_repository()
mock_make_request.assert_called_once_with(
"POST", "/crewai_plus/api/v1/tools/login", json={}
)
self.assertEqual(response, mock_response)
def assert_request_with_org_id(
self, mock_client_instance, method: str, endpoint: str, **kwargs
):
mock_client_instance.request.assert_called_once_with(
method,
f"{os.getenv('CREWAI_PLUS_URL')}{endpoint}",
headers={
"Authorization": ANY,
"Content-Type": ANY,
"User-Agent": ANY,
"X-Crewai-Version": ANY,
"X-Crewai-Organization-Id": self.org_uuid,
},
**kwargs,
)
@patch("crewai_core.plus_api.Settings")
@patch("crewai_core.plus_api.httpx.Client")
def test_login_to_tool_repository_with_org_uuid(
self, mock_client_class, mock_settings_class
):
mock_settings = MagicMock()
mock_settings.org_uuid = self.org_uuid
mock_settings.enterprise_base_url = os.getenv('CREWAI_PLUS_URL')
mock_settings_class.return_value = mock_settings
self.api = PlusAPI(self.api_key)
mock_client_instance = MagicMock()
mock_response = MagicMock()
mock_client_instance.request.return_value = mock_response
mock_client_class.return_value.__enter__.return_value = mock_client_instance
response = self.api.login_to_tool_repository()
self.assert_request_with_org_id(
mock_client_instance, "POST", "/crewai_plus/api/v1/tools/login", json={}
)
self.assertEqual(response, mock_response)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_get_tool(self, mock_make_request):
mock_response = MagicMock()
mock_make_request.return_value = mock_response
response = self.api.get_tool("test_tool_handle")
mock_make_request.assert_called_once_with(
"GET", "/crewai_plus/api/v1/tools/test_tool_handle"
)
self.assertEqual(response, mock_response)
@patch("crewai_core.plus_api.Settings")
@patch("crewai_core.plus_api.httpx.Client")
def test_get_tool_with_org_uuid(self, mock_client_class, mock_settings_class):
mock_settings = MagicMock()
mock_settings.org_uuid = self.org_uuid
mock_settings.enterprise_base_url = os.getenv('CREWAI_PLUS_URL')
mock_settings_class.return_value = mock_settings
self.api = PlusAPI(self.api_key)
mock_client_instance = MagicMock()
mock_response = MagicMock()
mock_client_instance.request.return_value = mock_response
mock_client_class.return_value.__enter__.return_value = mock_client_instance
response = self.api.get_tool("test_tool_handle")
self.assert_request_with_org_id(
mock_client_instance, "GET", "/crewai_plus/api/v1/tools/test_tool_handle"
)
self.assertEqual(response, mock_response)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_publish_tool(self, mock_make_request):
mock_response = MagicMock()
mock_make_request.return_value = mock_response
handle = "test_tool_handle"
public = True
version = "1.0.0"
description = "Test tool description"
encoded_file = "encoded_test_file"
response = self.api.publish_tool(
handle, public, version, description, encoded_file
)
params = {
"handle": handle,
"public": public,
"version": version,
"file": encoded_file,
"description": description,
"available_exports": None,
"tools_metadata": None,
}
mock_make_request.assert_called_once_with(
"POST", "/crewai_plus/api/v1/tools", json=params
)
self.assertEqual(response, mock_response)
@patch("crewai_core.plus_api.Settings")
@patch("crewai_core.plus_api.httpx.Client")
def test_publish_tool_with_org_uuid(self, mock_client_class, mock_settings_class):
mock_settings = MagicMock()
mock_settings.org_uuid = self.org_uuid
mock_settings.enterprise_base_url = os.getenv('CREWAI_PLUS_URL')
mock_settings_class.return_value = mock_settings
self.api = PlusAPI(self.api_key)
mock_client_instance = MagicMock()
mock_response = MagicMock()
mock_client_instance.request.return_value = mock_response
mock_client_class.return_value.__enter__.return_value = mock_client_instance
handle = "test_tool_handle"
public = True
version = "1.0.0"
description = "Test tool description"
encoded_file = "encoded_test_file"
response = self.api.publish_tool(
handle, public, version, description, encoded_file
)
expected_params = {
"handle": handle,
"public": public,
"version": version,
"file": encoded_file,
"description": description,
"available_exports": None,
"tools_metadata": None,
}
self.assert_request_with_org_id(
mock_client_instance, "POST", "/crewai_plus/api/v1/tools", json=expected_params
)
self.assertEqual(response, mock_response)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_publish_tool_without_description(self, mock_make_request):
mock_response = MagicMock()
mock_make_request.return_value = mock_response
handle = "test_tool_handle"
public = False
version = "2.0.0"
description = None
encoded_file = "encoded_test_file"
response = self.api.publish_tool(
handle, public, version, description, encoded_file
)
params = {
"handle": handle,
"public": public,
"version": version,
"file": encoded_file,
"description": description,
"available_exports": None,
"tools_metadata": None,
}
mock_make_request.assert_called_once_with(
"POST", "/crewai_plus/api/v1/tools", json=params
)
self.assertEqual(response, mock_response)
@patch("crewai_core.plus_api.httpx.Client")
def test_make_request(self, mock_client_class):
mock_client_instance = MagicMock()
mock_response = MagicMock()
mock_client_instance.request.return_value = mock_response
mock_client_class.return_value.__enter__.return_value = mock_client_instance
response = self.api._make_request("GET", "test_endpoint")
mock_client_class.assert_called_once_with(trust_env=False, verify=True)
mock_client_instance.request.assert_called_once_with(
"GET", f"{self.api.base_url}/test_endpoint", headers=self.api.headers
)
self.assertEqual(response, mock_response)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_deploy_by_name(self, mock_make_request):
self.api.deploy_by_name("test_project")
mock_make_request.assert_called_once_with(
"POST", "/crewai_plus/api/v1/crews/by-name/test_project/deploy"
)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_deploy_by_uuid(self, mock_make_request):
self.api.deploy_by_uuid("test_uuid")
mock_make_request.assert_called_once_with(
"POST", "/crewai_plus/api/v1/crews/test_uuid/deploy"
)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_deploy_sends_the_project_id_when_there_is_one(self, mock_make_request):
self.api.deploy_by_name("test_project", project_id="proj-1")
mock_make_request.assert_called_with(
"POST",
"/crewai_plus/api/v1/crews/by-name/test_project/deploy",
json={"project_id": "proj-1"},
)
self.api.deploy_by_uuid("test_uuid", project_id="proj-1")
mock_make_request.assert_called_with(
"POST",
"/crewai_plus/api/v1/crews/test_uuid/deploy",
json={"project_id": "proj-1"},
)
@patch("crewai_cli.plus_api.PlusAPI._make_multipart_request")
def test_zip_deploys_send_the_project_id_when_there_is_one(
self, mock_make_multipart_request
):
self.api.create_crew_from_zip("/tmp/test.zip", name="c", project_id="proj-1")
mock_make_multipart_request.assert_called_with(
"POST",
"/crewai_plus/api/v1/crews/zip",
zip_file_path="/tmp/test.zip",
data={"name": "c", "project_id": "proj-1"},
timeout=300,
)
self.api.update_crew_from_zip("u", "/tmp/test.zip", project_id="proj-1")
mock_make_multipart_request.assert_called_with(
"POST",
"/crewai_plus/api/v1/crews/u/zip_update",
zip_file_path="/tmp/test.zip",
data={"project_id": "proj-1"},
timeout=300,
)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_crew_status_by_name(self, mock_make_request):
self.api.crew_status_by_name("test_project")
mock_make_request.assert_called_once_with(
"GET", "/crewai_plus/api/v1/crews/by-name/test_project/status"
)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_crew_status_by_uuid(self, mock_make_request):
self.api.crew_status_by_uuid("test_uuid")
mock_make_request.assert_called_once_with(
"GET", "/crewai_plus/api/v1/crews/test_uuid/status"
)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_crew_by_name(self, mock_make_request):
self.api.crew_by_name("test_project")
mock_make_request.assert_called_once_with(
"GET", "/crewai_plus/api/v1/crews/by-name/test_project/logs/deployment"
)
self.api.crew_by_name("test_project", "custom_log")
mock_make_request.assert_called_with(
"GET", "/crewai_plus/api/v1/crews/by-name/test_project/logs/custom_log"
)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_crew_by_uuid(self, mock_make_request):
self.api.crew_by_uuid("test_uuid")
mock_make_request.assert_called_once_with(
"GET", "/crewai_plus/api/v1/crews/test_uuid/logs/deployment"
)
self.api.crew_by_uuid("test_uuid", "custom_log")
mock_make_request.assert_called_with(
"GET", "/crewai_plus/api/v1/crews/test_uuid/logs/custom_log"
)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_delete_crew_by_name(self, mock_make_request):
self.api.delete_crew_by_name("test_project")
mock_make_request.assert_called_once_with(
"DELETE", "/crewai_plus/api/v1/crews/by-name/test_project"
)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_delete_crew_by_uuid(self, mock_make_request):
self.api.delete_crew_by_uuid("test_uuid")
mock_make_request.assert_called_once_with(
"DELETE", "/crewai_plus/api/v1/crews/test_uuid"
)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_list_crews(self, mock_make_request):
self.api.list_crews()
mock_make_request.assert_called_once_with("GET", "/crewai_plus/api/v1/crews")
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_create_crew(self, mock_make_request):
payload = {"name": "test_crew"}
self.api.create_crew(payload)
mock_make_request.assert_called_once_with(
"POST", "/crewai_plus/api/v1/crews", json=payload
)
@patch("crewai_cli.plus_api.PlusAPI._make_multipart_request")
def test_create_crew_from_zip(self, mock_make_multipart_request):
self.api.create_crew_from_zip(
"/tmp/test.zip",
name="test_crew",
env={"ENV_VAR": "value"},
)
mock_make_multipart_request.assert_called_once_with(
"POST",
"/crewai_plus/api/v1/crews/zip",
zip_file_path="/tmp/test.zip",
data={"name": "test_crew", "env[ENV_VAR]": "value"},
timeout=300,
)
@patch("crewai_cli.plus_api.PlusAPI._make_multipart_request")
def test_update_crew_from_zip(self, mock_make_multipart_request):
self.api.update_crew_from_zip(
"test_uuid",
"/tmp/test.zip",
env={"ENV_VAR": "value"},
)
mock_make_multipart_request.assert_called_once_with(
"POST",
"/crewai_plus/api/v1/crews/test_uuid/zip_update",
zip_file_path="/tmp/test.zip",
data={"env[ENV_VAR]": "value"},
timeout=300,
)
@patch("crewai_core.plus_api.Settings")
@patch.dict(os.environ, {"CREWAI_PLUS_URL": ""})
def test_custom_base_url(self, mock_settings_class):
mock_settings = MagicMock()
mock_settings.enterprise_base_url = "https://custom-url.com/api"
mock_settings_class.return_value = mock_settings
custom_api = PlusAPI("test_key")
self.assertEqual(
custom_api.base_url,
"https://custom-url.com/api",
)
@patch.dict(os.environ, {"CREWAI_PLUS_URL": "https://custom-url-from-env.com"})
def test_custom_base_url_from_env(self):
custom_api = PlusAPI("test_key")
self.assertEqual(
custom_api.base_url,
"https://custom-url-from-env.com",
)
@patch("crewai_core.plus_api.PlusAPI._make_request")
def test_get_agent(mock_make_request):
api = PlusAPI("test_api_key")
mock_response = MagicMock()
mock_make_request.return_value = mock_response
response = api.get_agent("test_agent_handle")
mock_make_request.assert_called_once_with(
"GET", "/crewai_plus/api/v1/agents/test_agent_handle"
)
assert response == mock_response
@patch("crewai_core.plus_api.PlusAPI._make_request")
@patch("crewai_core.plus_api.Settings")
def test_get_agent_with_org_uuid(mock_settings_class, mock_make_request):
org_uuid = "test-org-uuid"
mock_settings = MagicMock()
mock_settings.org_uuid = org_uuid
mock_settings.enterprise_base_url = os.getenv("CREWAI_PLUS_URL")
mock_settings_class.return_value = mock_settings
api = PlusAPI("test_api_key")
mock_response = MagicMock()
mock_make_request.return_value = mock_response
response = api.get_agent("test_agent_handle")
mock_make_request.assert_called_once_with(
"GET", "/crewai_plus/api/v1/agents/test_agent_handle"
)
assert "X-Crewai-Organization-Id" in api.headers
assert api.headers["X-Crewai-Organization-Id"] == org_uuid
assert response == mock_response