mirror of
https://github.com/crytic/echidna.git
synced 2026-10-12 02:12:32 +00:00
test: drive the MCP server the way its clients do (#1608)
The MCP server is the one part of Echidna that another program talks to, and the Haskell suite cannot say whether that conversation works: it runs in process, while what matters here is what goes over the wire and what the tools answer about a campaign that is actually fuzzing. So this suite starts one. conftest.py launches a real echidna against a fixture contract and waits for the server to accept an MCP initialize rather than merely for the socket to open, then hands the same campaign to every module. Four layers ride on it: test_mcp.py covers the nine tools -- what each one reports about a live campaign, and what it says when it cannot answer. status while the campaign fuzzes, coverage marked up line by line, an LCOV file that is really on disk, an injected sequence that the workers keep fuzzing through, a replayed sequence reported call by call including the revert reason, a sampled function whose counts show up in status a few seconds later. Anything that steers the campaign is undone afterwards, so no test depends on the order it ran in. test_mcp_conformance.py is the regression guard for "real clients can still connect". A notification gets 202 with no body; a GET asking for a server stream gets 405 rather than the JSON blob a strict client tries to read as SSE; initialize negotiates a version and names the server. These are exactly the two behaviours mcp-server 0.2.0.1 fixed, and the reason that version is the floor. test_mcp_codex.py replays Codex's rmcp handshake step by step, and test_mcp_claude.py drives the server with the reference `mcp` SDK through a real session. Both are transport-level and need no API key, so both run in CI; a live-model run is the LangGraph example instead. That example is here too. It reads status, and when the campaign has gone a minute without finding coverage it hands Claude the ABI and the coverage report and injects the sequences that come back -- a demonstration that those tools are enough to close the loop, not a tuned strategy. Some notes on what differs from the version this is adapted from: The tool tests are written against the server as it is rather than the prototype: status answers JSON, show_coverage takes a contract, and a tool that cannot do what it was asked answers with an MCP error result. They assert on all three, where the originals asserted that some substring appeared somewhere in the text. They also go through the shared wire helpers after a handshake, rather than a second raw client that posted tools/call with no handshake and no headers -- asserting leniency next to a suite that asserts strictness said two different things about what the server promises. The `mcp` SDK's 2.0 renamed its transport helper, dropped the third stream it used to yield, and moved its result models to snake_case. Rather than straddle both majors, requirements-test.txt follows the current one and the test skips itself on an older SDK. The other pins are bounded by major version only, since a compatibility suite should be running against what clients actually ship. The fixture contracts lost their spec-kit headers, two properties that could not fail (one returned a literal `true`), and the `test` prefix on their entry points, which means something else in Foundry mode. What is left is a contract that gives the tools something to report on: calls that succeed, calls that revert with a reason, and properties that hold so the campaign keeps running for as long as the session needs it. The workflow now triggers on master and on pull requests against it, like every other workflow here, and only for paths that can change what the server answers -- it builds the whole closure and then fuzzes, which is not something to do on every push to every branch. Its Cachix token is unavailable to pull requests from forks, so those runs read the public cache and push nothing; that is noted where it will be read. tests/mcp/requirements.txt is gone: nothing referenced it, and it contradicted both its sibling and the example's README on what to install. So is pytest.ini, which was only read when pytest was given the directory as an argument -- the one async test carries an explicit marker instead, which works either way. Co-authored-by: Dani Tradito <[email protected]> Co-authored-by: gustavo-grieco <[email protected]>
This commit is contained in:
co-authored by
Dani Tradito
gustavo-grieco
parent
e6baf93926
commit
5403657dbf
@@ -0,0 +1,99 @@
|
||||
name: MCP integration tests
|
||||
|
||||
# Out-of-process MCP testing: spins up a real Echidna campaign with the MCP
|
||||
# server and drives it the way real agent clients do. Besides the tool tests it
|
||||
# runs the client-compatibility suite:
|
||||
# * test_mcp_conformance.py -- wire-protocol regression guard (202/405/handshake)
|
||||
# * test_mcp_codex.py -- replays Codex's strict rmcp handshake
|
||||
# * test_mcp_claude.py -- drives the server with the official `mcp` SDK
|
||||
# Both client checks are transport-level and need no API keys, so they run on
|
||||
# every push/PR that touches the code they cover. (A live-model smoke using
|
||||
# examples/mcp_agent.py needs ANTHROPIC_API_KEY / Codex auth and is
|
||||
# intentionally not part of CI.)
|
||||
|
||||
# Building the closure and running a campaign is expensive, so this is limited
|
||||
# to changes that can affect what the server answers. (GitHub Actions does not
|
||||
# support YAML anchors, hence the repetition.)
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
paths:
|
||||
- '.github/workflows/mcp-tests.yml'
|
||||
- 'tests/mcp/**'
|
||||
- 'lib/**'
|
||||
- 'src/**'
|
||||
- 'package.yaml'
|
||||
- 'flake.nix'
|
||||
- 'flake.lock'
|
||||
pull_request:
|
||||
branches:
|
||||
- master
|
||||
paths:
|
||||
- '.github/workflows/mcp-tests.yml'
|
||||
- 'tests/mcp/**'
|
||||
- 'lib/**'
|
||||
- 'src/**'
|
||||
- 'package.yaml'
|
||||
- 'flake.nix'
|
||||
- 'flake.lock'
|
||||
|
||||
# Least-privilege GITHUB_TOKEN: this workflow only checks out the repo and runs
|
||||
# tests, so it needs no write scopes.
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# A new push to the same ref cancels an in-flight run: this one builds the whole
|
||||
# closure and then fuzzes, so it is worth not piling up.
|
||||
concurrency:
|
||||
group: mcp-tests-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
mcp-client-compat:
|
||||
name: MCP client compatibility (Claude + Codex)
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Install Nix
|
||||
uses: DeterminateSystems/nix-installer-action@ef8a148080ab6020fd15196c2084a2eea5ff2d25 # v22
|
||||
|
||||
# Pull prebuilt echidna/hevm/deps from the project's Cachix so the build
|
||||
# takes minutes instead of compiling the whole closure from source.
|
||||
# CACHIX_AUTH_TOKEN is not exposed to pull requests from forks, so those
|
||||
# runs read the public cache and push nothing back: a fork PR that misses
|
||||
# is slow rather than red.
|
||||
- name: Configure Cachix
|
||||
uses: cachix/cachix-action@5f2d7c5294214f71b873db4b969586b980625e71 # v17
|
||||
with:
|
||||
name: trailofbits
|
||||
authToken: ${{ secrets.CACHIX_AUTH_TOKEN }}
|
||||
|
||||
- name: Build echidna
|
||||
run: nix build .#echidna --out-link result
|
||||
|
||||
- name: Add echidna to PATH
|
||||
run: echo "$GITHUB_WORKSPACE/result/bin" >> "$GITHUB_PATH"
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- name: Install test dependencies
|
||||
run: pip install -r tests/mcp/requirements-test.txt
|
||||
|
||||
# Echidna shells out to crytic-compile + solc to compile the test contract.
|
||||
- name: Install crytic-compile and solc
|
||||
run: |
|
||||
pip install crytic-compile solc-select
|
||||
solc-select install 0.8.25
|
||||
solc-select use 0.8.25
|
||||
|
||||
- name: Run MCP test suite (tools + conformance + Claude + Codex)
|
||||
run: pytest tests/mcp -v
|
||||
@@ -318,6 +318,25 @@ $ nix develop # alternatively nix-shell
|
||||
[nix-shell]$ cabal new-repl
|
||||
```
|
||||
|
||||
### Running the test suites
|
||||
|
||||
`cabal run tests` is the Haskell suite, and covers everything Echidna does in
|
||||
process.
|
||||
|
||||
The MCP server is tested out of process instead, by starting a real campaign and
|
||||
driving it the way an agent client would. That suite is Python, and needs an
|
||||
`echidna` on `$PATH`:
|
||||
|
||||
```sh
|
||||
$ pip install -r tests/mcp/requirements-test.txt
|
||||
$ pytest tests/mcp -v
|
||||
```
|
||||
|
||||
Point it at a campaign you started yourself with `ECHIDNA_MCP_URL`, at a
|
||||
different port with `ECHIDNA_MCP_PORT`, or at a specific binary with
|
||||
`ECHIDNA_BIN`. An example of an agent driving a campaign through the same
|
||||
interface is in [examples/](examples/README.md).
|
||||
|
||||
## Public use of Echidna
|
||||
|
||||
### Property testing suites
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
# Echidna MCP agent example
|
||||
|
||||
An agent that connects to a live Echidna campaign through its [MCP](https://modelcontextprotocol.io/)
|
||||
server, watches what the fuzzer is reaching, and aims it at what it is not.
|
||||
|
||||
## Requirements
|
||||
|
||||
```
|
||||
pip install langchain-anthropic langgraph httpx
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
**1. Start Echidna with the MCP server:**
|
||||
|
||||
```bash
|
||||
echidna MyContract.sol --server 8080 --format text
|
||||
```
|
||||
|
||||
`--format text` is required — the interactive TUI otherwise owns the terminal.
|
||||
|
||||
**2. Run the agent:**
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY=your_key_here
|
||||
python examples/mcp_agent.py
|
||||
```
|
||||
|
||||
It polls until you stop it with Ctrl-C. Point it at another campaign with
|
||||
`ECHIDNA_MCP_URL=http://127.0.0.1:9000/mcp`.
|
||||
|
||||
## What it does
|
||||
|
||||
Every 30 seconds it calls `status`. When the campaign has gone a minute without
|
||||
finding new coverage, it reads `target` and `show_coverage`, asks Claude for
|
||||
call sequences that would reach lines the campaign has not, and injects them
|
||||
with `inject_fuzz_transactions` — clearing the previous round first, so the
|
||||
fuzzer is not left splitting its budget across every ordering ever suggested.
|
||||
|
||||
## The tools the server exposes
|
||||
|
||||
Four report on the campaign:
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `status` | Corpus size, iterations, coverage, failing tests, optimization values, how long since the last coverage and which functions found it, and whatever `sample` is recording |
|
||||
| `target` | The contract under test and the functions it exposes |
|
||||
| `show_coverage` | One contract's source, line by line, marked with what the campaign reached |
|
||||
| `dump_lcov` | Write the coverage so far to an LCOV file |
|
||||
|
||||
Five steer it:
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `inject_fuzz_transactions` | Spend part of the fuzzer's budget on a specific ordering of calls |
|
||||
| `clear_fuzz_priorities` | Forget every injected ordering and return to the corpus |
|
||||
| `execute_sequence` | Run a concrete sequence and report what each call did, without disturbing the campaign |
|
||||
| `sample` | Record what one function does as the campaign calls it; results appear in `status` |
|
||||
| `reload_corpus` | Pick up whatever was written to the corpus directory since the campaign started |
|
||||
|
||||
`inject_fuzz_transactions` and `execute_sequence` take a sequence written the way
|
||||
the calls would be in Solidity, separated by `;`. `inject_fuzz_transactions`
|
||||
additionally accepts `?` for an argument the fuzzer should choose:
|
||||
|
||||
```
|
||||
approve(0x10, ?); transferFrom(?, ?, 100)
|
||||
```
|
||||
|
||||
## Testing the server
|
||||
|
||||
`tests/mcp/` drives a real campaign the way these clients do — tool semantics,
|
||||
wire-protocol conformance, and the official MCP SDK. See the
|
||||
[README](../README.md#running-the-test-suites) for how to run it.
|
||||
@@ -0,0 +1,190 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
LangGraph agent for Echidna's MCP server.
|
||||
|
||||
Connects to a running campaign, watches what the fuzzer is reaching, and when
|
||||
coverage stalls asks Claude for call sequences to aim it at. A demonstration of
|
||||
what the server is for rather than a tuned strategy: the interesting part is
|
||||
that `status` and `show_coverage` are enough for a model to decide what to try
|
||||
next, and `inject_fuzz_transactions` is enough to act on it.
|
||||
|
||||
Requirements:
|
||||
pip install langchain-anthropic langgraph httpx
|
||||
|
||||
Usage:
|
||||
# Start Echidna with the MCP server. --format text is required: the
|
||||
# interactive TUI otherwise owns the terminal.
|
||||
echidna MyContract.sol --server 8080 --format text
|
||||
|
||||
# Run the agent:
|
||||
ANTHROPIC_API_KEY=... python examples/mcp_agent.py
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from typing import TypedDict
|
||||
|
||||
import httpx
|
||||
from langchain_anthropic import ChatAnthropic
|
||||
from langgraph.graph import END, StateGraph
|
||||
|
||||
MCP_URL = os.environ.get("ECHIDNA_MCP_URL", "http://127.0.0.1:8080/mcp")
|
||||
PROTOCOL_VERSION = "2025-06-18"
|
||||
|
||||
# How long the campaign may go without finding coverage before the agent steps
|
||||
# in, and how often it looks.
|
||||
STALL_SECONDS = 60
|
||||
INTERVAL_SECONDS = 30
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MCP client
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _rpc(method: str, params: dict | None = None, request_id: int | None = 1):
|
||||
body = {"jsonrpc": "2.0", "method": method}
|
||||
if request_id is not None:
|
||||
body["id"] = request_id
|
||||
if params is not None:
|
||||
body["params"] = params
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
"Accept": "application/json, text/event-stream",
|
||||
}
|
||||
if method != "initialize":
|
||||
headers["MCP-Protocol-Version"] = PROTOCOL_VERSION
|
||||
response = httpx.post(MCP_URL, json=body, headers=headers, timeout=60)
|
||||
response.raise_for_status()
|
||||
return response.json() if request_id is not None else None
|
||||
|
||||
|
||||
def connect() -> str:
|
||||
"""Run the MCP handshake and return the server's name."""
|
||||
result = _rpc("initialize", {
|
||||
"protocolVersion": PROTOCOL_VERSION,
|
||||
"capabilities": {},
|
||||
"clientInfo": {"name": "echidna-mcp-agent", "version": "0"},
|
||||
}, request_id=0)["result"]
|
||||
_rpc("notifications/initialized", request_id=None)
|
||||
return result["serverInfo"]["name"]
|
||||
|
||||
|
||||
def call_tool(name: str, arguments: dict | None = None) -> str:
|
||||
"""Call a tool and return its report, raising if the tool could not answer."""
|
||||
result = _rpc("tools/call", {"name": name, "arguments": arguments or {}})["result"]
|
||||
text = "".join(part.get("text", "") for part in result.get("content", []))
|
||||
if result.get("isError"):
|
||||
raise RuntimeError(f"{name}: {text}")
|
||||
return text
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Graph
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class State(TypedDict):
|
||||
contract: str
|
||||
iterations: int
|
||||
coverage: int
|
||||
stalled_for: int
|
||||
|
||||
|
||||
def observe(state: State) -> State:
|
||||
"""Read how the campaign is going."""
|
||||
status = json.loads(call_tool("status"))
|
||||
stalled_for = status["time_since_last_coverage_sec"] or 0
|
||||
print(f" coverage={status['coverage_points']} "
|
||||
f"iterations={status['iterations']} "
|
||||
f"last coverage {stalled_for}s ago "
|
||||
f"({', '.join(status['recent_covered_functions'][:3]) or 'nothing yet'})")
|
||||
return {
|
||||
**state,
|
||||
"iterations": status["iterations"],
|
||||
"coverage": status["coverage_points"],
|
||||
"stalled_for": stalled_for,
|
||||
}
|
||||
|
||||
|
||||
def inject(state: State) -> State:
|
||||
"""Ask Claude which orderings to try, and point the fuzzer at them."""
|
||||
# temperature is not accepted on current models; steer with the prompt.
|
||||
llm = ChatAnthropic(model="claude-opus-4-8", max_tokens=1024)
|
||||
|
||||
abi = call_tool("target")
|
||||
coverage = call_tool("show_coverage", {"contract": state["contract"]})
|
||||
|
||||
answer = llm.invoke(
|
||||
"You are helping an Echidna fuzzing campaign reach code it has not "
|
||||
"reached. It has found no new coverage for a while.\n\n"
|
||||
f"The contract under test exposes:\n{abi}\n\n"
|
||||
f"Its coverage so far — lines marked * were executed, r reverted, "
|
||||
f"o ran out of gas, e errored, blank was never reached:\n"
|
||||
f"{coverage[:8000]}\n\n"
|
||||
"Reply with up to 3 call sequences that would reach unmarked lines, one "
|
||||
"per line and nothing else. Separate the calls in a sequence with ';', "
|
||||
"and write '?' for any argument the fuzzer should choose. For example:\n"
|
||||
"approve(0x10, ?); transferFrom(?, ?, 100)\n"
|
||||
).content
|
||||
|
||||
# Anything with a call in it; the model's prose, if any, has no parentheses.
|
||||
sequences = [line.strip() for line in answer.splitlines()
|
||||
if re.search(r"\w+\s*\(", line)]
|
||||
|
||||
# Drop what was injected last time first, so the fuzzer is not left
|
||||
# splitting its budget across every ordering the agent has ever suggested.
|
||||
call_tool("clear_fuzz_priorities")
|
||||
for sequence in sequences[:3]:
|
||||
try:
|
||||
print(f" injecting: {sequence}")
|
||||
print(f" {call_tool('inject_fuzz_transactions', {'transactions': sequence})}")
|
||||
except RuntimeError as complaint:
|
||||
print(f" rejected: {complaint}")
|
||||
|
||||
return state
|
||||
|
||||
|
||||
def route(state: State) -> str:
|
||||
return "inject" if state["stalled_for"] >= STALL_SECONDS else END
|
||||
|
||||
|
||||
def build_graph():
|
||||
graph = StateGraph(State)
|
||||
graph.add_node("observe", observe)
|
||||
graph.add_node("inject", inject)
|
||||
graph.set_entry_point("observe")
|
||||
graph.add_conditional_edges("observe", route, {"inject": "inject", END: END})
|
||||
graph.add_edge("inject", END)
|
||||
return graph.compile()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
if not os.getenv("ANTHROPIC_API_KEY"):
|
||||
print("Set ANTHROPIC_API_KEY before running.")
|
||||
return
|
||||
|
||||
try:
|
||||
print(f"Connected to {connect()} at {MCP_URL}.")
|
||||
except Exception as unreachable:
|
||||
print(f"Cannot reach the MCP server: {unreachable}")
|
||||
print("Start Echidna with: echidna MyContract.sol --server 8080 --format text")
|
||||
return
|
||||
|
||||
# "Contract: path/to/File.sol:Name" — the name is what show_coverage wants.
|
||||
contract = call_tool("target").splitlines()[0].rsplit(":", 1)[1].strip()
|
||||
print(f"Watching {contract}. Stepping in after {STALL_SECONDS}s without coverage.")
|
||||
|
||||
graph = build_graph()
|
||||
state: State = {"contract": contract, "iterations": 0, "coverage": 0, "stalled_for": 0}
|
||||
|
||||
step = 0
|
||||
while True:
|
||||
step += 1
|
||||
print(f"\n--- step {step} ---")
|
||||
state = graph.invoke(state)
|
||||
time.sleep(INTERVAL_SECONDS)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,100 @@
|
||||
"""
|
||||
Minimal, dependency-light MCP-over-HTTP client helpers shared by every module
|
||||
in this suite.
|
||||
|
||||
These intentionally do NOT use a high-level MCP SDK: they speak the wire
|
||||
protocol directly so the tests can assert on exact status codes, headers and
|
||||
bodies — the things real clients (Codex's rmcp, Anthropic's MCP client) are
|
||||
strict about and that a lenient `httpx.post(...).json()` would silently hide.
|
||||
The SDK does get exercised, as a client rather than as a helper, in
|
||||
test_mcp_claude.py.
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
import httpx
|
||||
|
||||
PROTOCOL_VERSION = "2025-06-18"
|
||||
SUPPORTED_VERSIONS = ("2025-06-18", "2025-03-26", "2024-11-05")
|
||||
|
||||
|
||||
class ToolError(Exception):
|
||||
"""A tool answered with an MCP error result instead of a report."""
|
||||
|
||||
|
||||
def rpc(url, method, params=None, id=1, protocol_version=PROTOCOL_VERSION,
|
||||
accept="application/json, text/event-stream", timeout=30):
|
||||
"""POST a single JSON-RPC message; return the raw httpx.Response (never parsed)."""
|
||||
body = {"jsonrpc": "2.0", "method": method}
|
||||
if id is not None:
|
||||
body["id"] = id
|
||||
if params is not None:
|
||||
body["params"] = params
|
||||
headers = {"Content-Type": "application/json", "Accept": accept}
|
||||
# Per the spec the MCP-Protocol-Version header is sent on requests *after*
|
||||
# initialization, not on `initialize` itself.
|
||||
if protocol_version is not None and method != "initialize":
|
||||
headers["MCP-Protocol-Version"] = protocol_version
|
||||
return httpx.post(url, json=body, headers=headers, timeout=timeout)
|
||||
|
||||
|
||||
def http_get(url, accept="text/event-stream", timeout=10):
|
||||
"""GET the MCP endpoint (clients use this to open the server->client SSE stream)."""
|
||||
return httpx.get(url, headers={"Accept": accept}, timeout=timeout)
|
||||
|
||||
|
||||
def handshake(url, protocol_version=PROTOCOL_VERSION):
|
||||
"""Run the client half of the MCP handshake; return the InitializeResult dict.
|
||||
|
||||
initialize -> (read result) -> notifications/initialized
|
||||
"""
|
||||
resp = rpc(url, "initialize", {
|
||||
"protocolVersion": protocol_version,
|
||||
"capabilities": {},
|
||||
"clientInfo": {"name": "echidna-mcp-tests", "version": "0"},
|
||||
}, id=0, protocol_version=None)
|
||||
resp.raise_for_status()
|
||||
result = resp.json()["result"]
|
||||
negotiated = result.get("protocolVersion", protocol_version)
|
||||
rpc(url, "notifications/initialized", id=None, protocol_version=negotiated)
|
||||
return result
|
||||
|
||||
|
||||
def call_tool(url, name, arguments=None, protocol_version=PROTOCOL_VERSION,
|
||||
timeout=60):
|
||||
"""Call a tool and return its report.
|
||||
|
||||
A tool that cannot do what it was asked answers with an MCP error result
|
||||
rather than with a report, so that becomes a ToolError here: a test that
|
||||
did not expect one fails at the call rather than on some later assertion
|
||||
about the text. Tests that do expect one use `pytest.raises(ToolError)`.
|
||||
"""
|
||||
resp = rpc(url, "tools/call", {"name": name, "arguments": arguments or {}},
|
||||
id=2, protocol_version=protocol_version, timeout=timeout)
|
||||
resp.raise_for_status()
|
||||
body = resp.json()
|
||||
if "error" in body:
|
||||
raise ToolError(f"{name}: JSON-RPC error {body['error']}")
|
||||
result = body["result"]
|
||||
text = "".join(part.get("text", "") for part in result.get("content", []))
|
||||
if result.get("isError"):
|
||||
raise ToolError(f"{name}: {text}")
|
||||
return text
|
||||
|
||||
|
||||
def eventually(read, timeout=60, interval=0.5):
|
||||
"""Poll `read` until it answers with something truthy, and return that.
|
||||
|
||||
Nothing a client asks for takes effect at once: a worker only looks at the
|
||||
bus between sequences, and what it does then only shows up in `status` on
|
||||
the next call. Returns None if the timeout runs out, so the caller can say
|
||||
what it was waiting for.
|
||||
"""
|
||||
deadline = time.time() + timeout
|
||||
while True:
|
||||
answer = read()
|
||||
if answer:
|
||||
return answer
|
||||
if time.time() >= deadline:
|
||||
return None
|
||||
time.sleep(interval)
|
||||
@@ -0,0 +1,110 @@
|
||||
"""
|
||||
Shared pytest fixtures for the out-of-process MCP test suite.
|
||||
|
||||
A single session-scoped Echidna campaign (with the MCP server enabled) is
|
||||
started once and reused by every test module: the tool tests (test_mcp.py), the
|
||||
wire-protocol conformance tests (test_mcp_conformance.py), and the
|
||||
client-simulation tests (test_mcp_claude.py, test_mcp_codex.py).
|
||||
|
||||
The Echidna launch/wait logic was extracted from the original test_mcp.py
|
||||
contributed by Dani Tradito (@datradito) in crytic/echidna#1509.
|
||||
"""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from _mcp_client import PROTOCOL_VERSION, handshake, rpc
|
||||
|
||||
MCP_PORT = int(os.environ.get("ECHIDNA_MCP_PORT", "8080"))
|
||||
MCP_URL = f"http://127.0.0.1:{MCP_PORT}/mcp"
|
||||
ECHIDNA_BIN = os.environ.get("ECHIDNA_BIN", "echidna")
|
||||
CONTRACT = os.path.join(os.path.dirname(__file__), "contracts", "EchidnaMCPTest.sol")
|
||||
|
||||
|
||||
def _server_ready() -> bool:
|
||||
"""True once the MCP server answers a real `initialize` request."""
|
||||
try:
|
||||
resp = rpc(MCP_URL, "initialize", {
|
||||
"protocolVersion": PROTOCOL_VERSION,
|
||||
"capabilities": {},
|
||||
"clientInfo": {"name": "readiness-probe", "version": "0"},
|
||||
}, id=0, protocol_version=None, timeout=2)
|
||||
return resp.status_code == 200 and "result" in resp.json()
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def echidna_server(tmp_path_factory):
|
||||
"""Start an Echidna campaign with the MCP server for the whole test session.
|
||||
|
||||
If ECHIDNA_MCP_URL is set, use that already-running server instead of
|
||||
starting one (handy for pointing the suite at a live campaign or, in CI, a
|
||||
deliberately broken server to prove the conformance tests have teeth). The
|
||||
campaign is then someone else's, so tests that look at what it wrote to
|
||||
disk skip themselves.
|
||||
"""
|
||||
external = os.environ.get("ECHIDNA_MCP_URL")
|
||||
if external:
|
||||
yield {"url": external, "local": False}
|
||||
return
|
||||
|
||||
corpus_dir = tmp_path_factory.mktemp("corpus")
|
||||
|
||||
env = os.environ.copy()
|
||||
env["PATH"] = os.path.expanduser("~/.local/bin") + os.pathsep + env.get("PATH", "")
|
||||
# Echidna shells out to solc/crytic-compile; an active VIRTUAL_ENV breaks that.
|
||||
env.pop("VIRTUAL_ENV", None)
|
||||
|
||||
cmd = [
|
||||
ECHIDNA_BIN, CONTRACT,
|
||||
"--contract", "EchidnaMCPTest",
|
||||
"--server", str(MCP_PORT),
|
||||
# Required: the interactive TUI otherwise owns the terminal.
|
||||
"--format", "text",
|
||||
# The campaign has to outlive the session; it is stopped from here.
|
||||
"--test-limit", "1000000000",
|
||||
"--corpus-dir", str(corpus_dir),
|
||||
]
|
||||
proc = subprocess.Popen(
|
||||
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, env=env
|
||||
)
|
||||
|
||||
# Wait (compiling the contract can take a while) for the server to accept a
|
||||
# real MCP initialize, not merely an open socket.
|
||||
deadline = time.time() + 120
|
||||
while time.time() < deadline:
|
||||
if proc.poll() is not None:
|
||||
out, err = proc.communicate()
|
||||
raise RuntimeError(f"Echidna exited early.\nstdout:\n{out}\nstderr:\n{err}")
|
||||
if _server_ready():
|
||||
break
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
proc.terminate()
|
||||
out, err = proc.communicate(timeout=5)
|
||||
raise RuntimeError(f"MCP server did not become ready.\nstdout:\n{out}\nstderr:\n{err}")
|
||||
|
||||
yield {"url": MCP_URL, "local": True, "corpus_dir": str(corpus_dir)}
|
||||
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.communicate(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait()
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def mcp_url(echidna_server):
|
||||
"""The endpoint, with the handshake already done.
|
||||
|
||||
Every test but the conformance ones is about what the campaign answers, not
|
||||
about how a session is established, and a real client has initialized
|
||||
before it asks anything.
|
||||
"""
|
||||
handshake(echidna_server["url"])
|
||||
return echidna_server["url"]
|
||||
@@ -0,0 +1,48 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
pragma solidity ^0.8.0;
|
||||
|
||||
import "./SimpleToken.sol";
|
||||
|
||||
/// The contract the MCP test campaign runs against.
|
||||
///
|
||||
/// It is here to give the tools something to report on rather than to find a
|
||||
/// bug: a handful of functions to inject, sample and replay, one of which
|
||||
/// reverts or succeeds depending on its arguments, and properties that hold so
|
||||
/// the campaign keeps running for as long as the test session needs it.
|
||||
contract EchidnaMCPTest {
|
||||
SimpleToken public token;
|
||||
uint256 constant INITIAL_SUPPLY = 1000000 * 10**18;
|
||||
|
||||
constructor() {
|
||||
token = new SimpleToken(INITIAL_SUPPLY);
|
||||
}
|
||||
|
||||
/// Succeeds for an amount this contract still holds, reverts otherwise --
|
||||
/// which is most of the time, the amount being a random uint256.
|
||||
function transferTokens(address to, uint256 amount) public {
|
||||
token.transfer(to, amount);
|
||||
}
|
||||
|
||||
function approveSpender(address spender, uint256 amount) public {
|
||||
token.approve(spender, amount);
|
||||
}
|
||||
|
||||
/// Succeeds for any recipient but the zero address.
|
||||
function mintTokens(address to, uint256 amount) public {
|
||||
token.mint(to, amount);
|
||||
}
|
||||
|
||||
function tokenAddress() public view returns (address) {
|
||||
return address(token);
|
||||
}
|
||||
|
||||
/// Nothing burns, so the supply only ever grows.
|
||||
function echidna_supply_never_shrinks() public view returns (bool) {
|
||||
return token.totalSupply() >= INITIAL_SUPPLY;
|
||||
}
|
||||
|
||||
/// No account can hold more than was ever issued.
|
||||
function echidna_balance_within_supply() public view returns (bool) {
|
||||
return token.balanceOf(address(this)) <= token.totalSupply();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
pragma solidity ^0.8.0;
|
||||
|
||||
/// A minimal ERC20-ish token for the MCP test campaign to fuzz.
|
||||
///
|
||||
/// The `require`s are the point: they give the campaign calls that revert with
|
||||
/// a reason, which is what `sample` records and `execute_sequence` reports on.
|
||||
contract SimpleToken {
|
||||
string public name = "Test Token";
|
||||
string public symbol = "TEST";
|
||||
uint8 public decimals = 18;
|
||||
uint256 public totalSupply;
|
||||
|
||||
mapping(address => uint256) public balances;
|
||||
mapping(address => mapping(address => uint256)) public allowances;
|
||||
|
||||
event Transfer(address indexed from, address indexed to, uint256 value);
|
||||
event Approval(address indexed owner, address indexed spender, uint256 value);
|
||||
|
||||
constructor(uint256 initialSupply) {
|
||||
totalSupply = initialSupply;
|
||||
balances[msg.sender] = initialSupply;
|
||||
emit Transfer(address(0), msg.sender, initialSupply);
|
||||
}
|
||||
|
||||
function transfer(address to, uint256 amount) public returns (bool) {
|
||||
require(to != address(0), "Invalid recipient");
|
||||
require(balances[msg.sender] >= amount, "Insufficient balance");
|
||||
|
||||
balances[msg.sender] -= amount;
|
||||
balances[to] += amount;
|
||||
|
||||
emit Transfer(msg.sender, to, amount);
|
||||
return true;
|
||||
}
|
||||
|
||||
function approve(address spender, uint256 amount) public returns (bool) {
|
||||
require(spender != address(0), "Invalid spender");
|
||||
|
||||
allowances[msg.sender][spender] = amount;
|
||||
|
||||
emit Approval(msg.sender, spender, amount);
|
||||
return true;
|
||||
}
|
||||
|
||||
function transferFrom(address from, address to, uint256 amount) public returns (bool) {
|
||||
require(to != address(0), "Invalid recipient");
|
||||
require(balances[from] >= amount, "Insufficient balance");
|
||||
require(allowances[from][msg.sender] >= amount, "Insufficient allowance");
|
||||
|
||||
balances[from] -= amount;
|
||||
balances[to] += amount;
|
||||
allowances[from][msg.sender] -= amount;
|
||||
|
||||
emit Transfer(from, to, amount);
|
||||
return true;
|
||||
}
|
||||
|
||||
/// Unguarded on purpose: the campaign needs a call that always succeeds.
|
||||
function mint(address to, uint256 amount) public {
|
||||
require(to != address(0), "Invalid recipient");
|
||||
|
||||
totalSupply += amount;
|
||||
balances[to] += amount;
|
||||
|
||||
emit Transfer(address(0), to, amount);
|
||||
}
|
||||
|
||||
function balanceOf(address account) public view returns (uint256) {
|
||||
return balances[account];
|
||||
}
|
||||
|
||||
function allowance(address owner, address spender) public view returns (uint256) {
|
||||
return allowances[owner][spender];
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
# Dependencies for the out-of-process MCP test suite (tests/mcp/).
|
||||
#
|
||||
# Bounded by major version only: the suite is a compatibility check, so it
|
||||
# should be running against what clients actually ship rather than against
|
||||
# whatever was current when it was written.
|
||||
|
||||
httpx>=0.27,<1
|
||||
pytest>=8.0,<10
|
||||
pytest-asyncio>=0.24,<2
|
||||
|
||||
# Official Model Context Protocol SDK -- used as the reference "Claude-family"
|
||||
# client in test_mcp_claude.py (transport only; no API key required).
|
||||
mcp>=2,<3
|
||||
@@ -0,0 +1,245 @@
|
||||
"""
|
||||
Tool tests for the Echidna MCP server: what the nine tools answer when they are
|
||||
driven against a live campaign, and what they say when they cannot answer.
|
||||
|
||||
The transport is somebody else's problem here (test_mcp_conformance.py), as is
|
||||
the promise that `execute_sequence` leaves the campaign alone, which the
|
||||
Haskell suite pins down from the inside. What is left is the interface an agent
|
||||
sees.
|
||||
|
||||
Originally contributed by Dani Tradito (@datradito) in crytic/echidna#1509;
|
||||
adapted here to the session-scoped campaign in conftest.py.
|
||||
|
||||
Run with:
|
||||
pytest tests/mcp/test_mcp.py
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
|
||||
import pytest
|
||||
|
||||
from _mcp_client import ToolError, call_tool, eventually, rpc
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def fixture_campaign(echidna_server, mcp_url):
|
||||
"""These tests are about the campaign conftest.py starts, and they leave it
|
||||
as they found it.
|
||||
|
||||
Injection and sampling are campaign-wide and outlive the test that turned
|
||||
them on, so undoing them here is what keeps the tests independent of the
|
||||
order they run in.
|
||||
"""
|
||||
if not echidna_server["local"]:
|
||||
pytest.skip("ECHIDNA_MCP_URL points at a campaign this module knows nothing about")
|
||||
yield
|
||||
call_tool(mcp_url, "clear_fuzz_priorities")
|
||||
call_tool(mcp_url, "sample", {"function": "off"})
|
||||
|
||||
|
||||
def status(url) -> dict:
|
||||
return json.loads(call_tool(url, "status"))
|
||||
|
||||
|
||||
def test_every_tool_is_advertised(mcp_url):
|
||||
"""The tools an agent can reach, which examples/README.md documents."""
|
||||
listed = rpc(mcp_url, "tools/list", id=3).json()["result"]["tools"]
|
||||
|
||||
assert {tool["name"] for tool in listed} == {
|
||||
"status", "target", "show_coverage", "dump_lcov", "reload_corpus",
|
||||
"inject_fuzz_transactions", "clear_fuzz_priorities", "execute_sequence",
|
||||
"sample",
|
||||
}
|
||||
for tool in listed:
|
||||
assert tool["description"], f"{tool['name']} is undescribed"
|
||||
|
||||
|
||||
def test_status_reports_campaign_metrics(mcp_url):
|
||||
report = status(mcp_url)
|
||||
|
||||
# The two echidna_ properties of the fixture contract, both of which hold.
|
||||
assert report["tests_total"] == 2
|
||||
assert report["tests_failed"] == 0
|
||||
assert report["optimization_values"] == []
|
||||
|
||||
assert eventually(lambda: status(mcp_url)["iterations"] > 0), \
|
||||
"the campaign never reported an iteration"
|
||||
assert eventually(lambda: status(mcp_url)["coverage_points"] > 0), \
|
||||
"the campaign never reported any coverage"
|
||||
assert eventually(lambda: status(mcp_url)["recent_covered_functions"]), \
|
||||
"no function was credited with finding coverage"
|
||||
|
||||
|
||||
def test_target_lists_the_abi(mcp_url):
|
||||
report = call_tool(mcp_url, "target")
|
||||
|
||||
assert "EchidnaMCPTest" in report
|
||||
for signature in ("transferTokens(address,uint256)",
|
||||
"approveSpender(address,uint256)",
|
||||
"mintTokens(address,uint256)"):
|
||||
assert signature in report, f"{signature} missing from:\n{report}"
|
||||
|
||||
|
||||
def test_show_coverage_marks_up_the_source(mcp_url):
|
||||
report = call_tool(mcp_url, "show_coverage", {"contract": "EchidnaMCPTest"})
|
||||
|
||||
assert "EchidnaMCPTest.sol" in report
|
||||
assert "contract EchidnaMCPTest {" in report, "the source is not in the report"
|
||||
# " 12 | *r | contract EchidnaMCPTest {": line number, markers, source.
|
||||
assert re.search(r"^\s*\d+ \| *\*", report, re.M), \
|
||||
f"nothing is marked as executed:\n{report[:2000]}"
|
||||
|
||||
|
||||
def test_show_coverage_needs_a_contract_that_exists(mcp_url):
|
||||
with pytest.raises(ToolError, match="required"):
|
||||
call_tool(mcp_url, "show_coverage")
|
||||
|
||||
with pytest.raises(ToolError, match="No contract by that name"):
|
||||
call_tool(mcp_url, "show_coverage", {"contract": "NotAContract"})
|
||||
|
||||
|
||||
def test_dump_lcov_writes_a_file(echidna_server, mcp_url):
|
||||
report = call_tool(mcp_url, "dump_lcov")
|
||||
|
||||
written = re.fullmatch(r"Wrote LCOV coverage to (.+)\.", report.strip())
|
||||
assert written, f"unexpected dump_lcov report: {report}"
|
||||
path = written.group(1)
|
||||
|
||||
assert os.path.dirname(path) == echidna_server["corpus_dir"], \
|
||||
f"LCOV went somewhere other than the corpus directory: {path}"
|
||||
with open(path) as f:
|
||||
assert "SF:" in f.read(), "no source-file record in the LCOV output"
|
||||
|
||||
|
||||
def test_reload_corpus_reads_the_corpus_directory(mcp_url):
|
||||
assert eventually(lambda: status(mcp_url)["corpus_size"] > 0), \
|
||||
"the campaign never wrote a corpus entry to reload"
|
||||
|
||||
report = call_tool(mcp_url, "reload_corpus")
|
||||
|
||||
# Every sequence on disk is one this campaign put there, so it has them all
|
||||
# already; what matters is that it read the directory and said what it found.
|
||||
assert re.match(r"Added \d+ of \d+ transaction sequences from ", report), \
|
||||
f"unexpected reload_corpus report: {report}"
|
||||
|
||||
|
||||
def test_inject_fuzz_transactions_keeps_the_campaign_running(mcp_url):
|
||||
before = status(mcp_url)["iterations"]
|
||||
|
||||
report = call_tool(mcp_url, "inject_fuzz_transactions", {
|
||||
"transactions": "mintTokens(0x1111111111111111111111111111111111111111, 100);"
|
||||
"transferTokens(?, ?)"
|
||||
})
|
||||
assert re.fullmatch(r"Fuzzing that sequence on [1-9]\d* workers\.", report), \
|
||||
f"unexpected injection report: {report}"
|
||||
|
||||
# A worker that choked on the injected sequence would stop reporting here.
|
||||
assert eventually(lambda: status(mcp_url)["iterations"] > before), \
|
||||
"the campaign stopped fuzzing after an injection"
|
||||
|
||||
|
||||
def test_inject_fuzz_transactions_checks_the_sequence(mcp_url):
|
||||
with pytest.raises(ToolError, match="No function 'nosuchfunction'"):
|
||||
call_tool(mcp_url, "inject_fuzz_transactions",
|
||||
{"transactions": "nosuchfunction(1)"})
|
||||
|
||||
with pytest.raises(ToolError, match="takes 2 arguments, not 1"):
|
||||
call_tool(mcp_url, "inject_fuzz_transactions",
|
||||
{"transactions": "transferTokens(?)"})
|
||||
|
||||
with pytest.raises(ToolError, match="Could not parse"):
|
||||
call_tool(mcp_url, "inject_fuzz_transactions",
|
||||
{"transactions": "not a call sequence"})
|
||||
|
||||
|
||||
def test_clear_fuzz_priorities_returns_the_workers_to_the_corpus(mcp_url):
|
||||
call_tool(mcp_url, "inject_fuzz_transactions",
|
||||
{"transactions": "mintTokens(?, ?)"})
|
||||
|
||||
report = call_tool(mcp_url, "clear_fuzz_priorities")
|
||||
assert re.fullmatch(r"Cleared injected sequences on [1-9]\d* workers\.", report), \
|
||||
f"unexpected clear report: {report}"
|
||||
|
||||
assert eventually(lambda: status(mcp_url)["iterations"] > 0), \
|
||||
"the campaign stopped fuzzing after clearing its priorities"
|
||||
|
||||
|
||||
def test_execute_sequence_reports_every_call(mcp_url):
|
||||
report = json.loads(call_tool(mcp_url, "execute_sequence", {
|
||||
"transactions": "mintTokens(0x10, 1000); transferTokens(0x20, 500)"
|
||||
}))
|
||||
|
||||
assert report["status"] == "completed"
|
||||
assert report["failed_tx_index"] is None
|
||||
assert report["transaction_count"] == 2
|
||||
|
||||
minted, transferred = report["transactions"]
|
||||
assert minted["call"].endswith("mintTokens(0x10,1000)")
|
||||
assert transferred["call"].endswith("transferTokens(0x20,500)")
|
||||
for tx in (minted, transferred):
|
||||
assert tx["status"] == "completed", f"{tx['call']} did not complete: {tx}"
|
||||
assert tx["gas_used"] > 0
|
||||
assert any("Transfer" in log for log in tx["logs"]), \
|
||||
f"{tx['call']} emitted no Transfer: {tx['logs']}"
|
||||
|
||||
|
||||
def test_execute_sequence_reports_a_revert(mcp_url):
|
||||
# More than the whole supply, so SimpleToken.transfer's require fails.
|
||||
report = json.loads(call_tool(mcp_url, "execute_sequence", {
|
||||
"transactions": "transferTokens(0x20, 999999999999999999999999999999)"
|
||||
}))
|
||||
|
||||
assert report["status"] == "reverted"
|
||||
assert report["failed_tx_index"] == 1
|
||||
reverted, = report["transactions"]
|
||||
assert reverted["status"] == "reverted"
|
||||
assert any("Insufficient balance" in log for log in reverted["logs"]), \
|
||||
f"the revert reason is not in the report: {reverted['logs']}"
|
||||
|
||||
|
||||
def test_execute_sequence_can_return_a_trace(mcp_url):
|
||||
report = json.loads(call_tool(mcp_url, "execute_sequence", {
|
||||
"transactions": "mintTokens(0x10, 1)", "trace": True
|
||||
}))
|
||||
|
||||
assert "SimpleToken::mint" in report["trace"], \
|
||||
f"the inner call is not in the trace: {report.get('trace')}"
|
||||
|
||||
|
||||
def test_execute_sequence_needs_concrete_arguments(mcp_url):
|
||||
with pytest.raises(ToolError, match="has to be concrete"):
|
||||
call_tool(mcp_url, "execute_sequence",
|
||||
{"transactions": "transferTokens(?, 1)"})
|
||||
|
||||
|
||||
def test_sample_records_what_a_function_does(mcp_url):
|
||||
report = call_tool(mcp_url, "sample", {"function": "transferTokens"})
|
||||
assert re.fullmatch(
|
||||
r"Sampling transferTokens\(address,uint256\) on [1-9]\d* workers\.", report
|
||||
), f"unexpected sample report: {report}"
|
||||
|
||||
sampled = eventually(lambda: status(mcp_url)["samples"])
|
||||
assert sampled, "the campaign never reported a sample"
|
||||
stats, = sampled
|
||||
assert stats["function"] == "transferTokens(address,uint256)"
|
||||
assert stats["calls"] > 0
|
||||
assert 0 <= stats["reverts"] <= stats["calls"]
|
||||
# A random uint256 is more than the supply, so most calls revert with a
|
||||
# reason, and the tail of them is what an agent reads.
|
||||
def recent_reverts():
|
||||
sampled = status(mcp_url)["samples"]
|
||||
return sampled and sampled[0]["recent_reverts"]
|
||||
|
||||
assert eventually(recent_reverts), \
|
||||
"a function that reverts reported no revert summaries"
|
||||
|
||||
assert call_tool(mcp_url, "sample", {"function": "off"}).startswith("Stopped sampling")
|
||||
assert eventually(lambda: status(mcp_url)["samples"] == []), \
|
||||
"the campaign kept sampling after being told to stop"
|
||||
|
||||
|
||||
def test_sample_needs_a_function_that_exists(mcp_url):
|
||||
with pytest.raises(ToolError, match="No function 'nosuchfunction'"):
|
||||
call_tool(mcp_url, "sample", {"function": "nosuchfunction"})
|
||||
@@ -0,0 +1,60 @@
|
||||
"""
|
||||
Claude (Anthropic MCP client) compatibility test.
|
||||
|
||||
Anthropic's clients speak MCP via the reference `mcp` SDK. This test drives the
|
||||
server with that SDK's Streamable-HTTP client through a full, real session
|
||||
(initialize -> list_tools -> call_tool). If the server's transport is
|
||||
non-conformant, the SDK -- like Claude -- fails to establish the session, so
|
||||
this is a faithful "would Claude break?" check.
|
||||
|
||||
No API key is required: it exercises the MCP transport, not the model. (An
|
||||
end-to-end run with a live model is the LangGraph demo in examples/mcp_agent.py,
|
||||
which needs ANTHROPIC_API_KEY and is not part of CI.)
|
||||
|
||||
Requires the `mcp` SDK and pytest-asyncio (see requirements-test.txt); the test
|
||||
skips itself if neither is installed.
|
||||
|
||||
Run with:
|
||||
pip install -r tests/mcp/requirements-test.txt
|
||||
pytest tests/mcp/test_mcp_claude.py
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
transport = pytest.importorskip(
|
||||
"mcp.client.streamable_http",
|
||||
reason="install the `mcp` SDK to run the Claude client test",
|
||||
)
|
||||
|
||||
# The SDK's 2.0 renamed the transport helper, stopped yielding a third stream
|
||||
# alongside the two, and moved its result models to snake_case. Rather than
|
||||
# straddle both spellings, follow the current major and say so.
|
||||
if not hasattr(transport, "streamable_http_client"):
|
||||
pytest.skip(
|
||||
"needs the `mcp` SDK 2.x; see tests/mcp/requirements-test.txt",
|
||||
allow_module_level=True,
|
||||
)
|
||||
|
||||
from mcp import ClientSession # noqa: E402
|
||||
from mcp.client.streamable_http import streamable_http_client # noqa: E402
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_claude_mcp_sdk_session(echidna_server):
|
||||
"""A real MCP SDK client completes a session and calls a tool."""
|
||||
async with streamable_http_client(echidna_server["url"]) as (read, write):
|
||||
async with ClientSession(read, write) as session:
|
||||
init = await session.initialize()
|
||||
assert init.server_info is not None, "initialize returned no serverInfo"
|
||||
|
||||
tools = await session.list_tools()
|
||||
names = {t.name for t in tools.tools}
|
||||
assert "status" in names, (
|
||||
f"tools not visible to the MCP SDK client: {sorted(names)}"
|
||||
)
|
||||
|
||||
result = await session.call_tool("status", {})
|
||||
assert not result.is_error, f"status tool reported an error: {result}"
|
||||
assert result.content, "status returned no content to the MCP SDK client"
|
||||
text = getattr(result.content[0], "text", "")
|
||||
assert text, f"empty status content: {result.content}"
|
||||
@@ -0,0 +1,67 @@
|
||||
"""
|
||||
Codex (rmcp) client-compatibility test.
|
||||
|
||||
Codex connects to MCP servers with the Rust `rmcp` StreamableHttpClientTransport,
|
||||
which is strict about the wire protocol. We can't run rmcp from Python, so this
|
||||
module *replays Codex's exact observed handshake* against the live server and
|
||||
asserts the server satisfies rmcp's expectations at each step. A regression that
|
||||
would make Codex drop the server fails here -- without needing Codex installed
|
||||
or authenticated.
|
||||
|
||||
The sequence and failure modes below were reverse-engineered from a real
|
||||
`codex exec` run against an Echidna MCP server.
|
||||
|
||||
For an end-to-end smoke with the real binary (needs Codex auth, so it is NOT in
|
||||
CI): point Codex's `~/.codex/config.toml` at the running campaign and run
|
||||
`codex mcp get echidna`.
|
||||
|
||||
Run with:
|
||||
pytest tests/mcp/test_mcp_codex.py
|
||||
"""
|
||||
|
||||
from _mcp_client import PROTOCOL_VERSION, SUPPORTED_VERSIONS, rpc, http_get
|
||||
|
||||
|
||||
def test_codex_rmcp_handshake_replay(echidna_server):
|
||||
url = echidna_server["url"]
|
||||
|
||||
# 1. initialize -- rmcp sends this first (Accept: application/json, text/event-stream).
|
||||
init = rpc(url, "initialize", {
|
||||
"protocolVersion": PROTOCOL_VERSION,
|
||||
"capabilities": {"roots": {"listChanged": True}},
|
||||
"clientInfo": {"name": "codex", "version": "0"},
|
||||
}, id=0, protocol_version=None)
|
||||
assert init.status_code == 200, f"initialize: HTTP {init.status_code}"
|
||||
init_result = init.json()["result"]
|
||||
assert init_result.get("protocolVersion") in SUPPORTED_VERSIONS, (
|
||||
f"protocol negotiation: {init_result.get('protocolVersion')}"
|
||||
)
|
||||
negotiated = init_result["protocolVersion"]
|
||||
|
||||
# 2. notifications/initialized -- rmcp DIES here if the server answers `200 {}`
|
||||
# ("data did not match any variant of untagged enum JsonRpcMessage").
|
||||
note = rpc(url, "notifications/initialized", id=None, protocol_version=negotiated)
|
||||
assert note.status_code == 202 and note.content == b"", (
|
||||
"Codex's rmcp client requires 202 + empty body for notifications; "
|
||||
f"got {note.status_code} {note.content!r}"
|
||||
)
|
||||
|
||||
# 3. GET the endpoint to open the server->client SSE stream. rmcp tolerates
|
||||
# 405 ("no server stream"); a 200 application/json discovery blob breaks it.
|
||||
stream = http_get(url, accept="text/event-stream")
|
||||
ctype = stream.headers.get("content-type", "")
|
||||
assert stream.status_code == 405 or "text/event-stream" in ctype, (
|
||||
f"GET stream must be 405 or SSE; got {stream.status_code} ({ctype})"
|
||||
)
|
||||
|
||||
# 4. tools/list -- only reached if the handshake survived.
|
||||
tools = rpc(url, "tools/list", id=1, protocol_version=negotiated)
|
||||
assert tools.status_code == 200, f"tools/list: HTTP {tools.status_code}"
|
||||
names = {t["name"] for t in tools.json()["result"]["tools"]}
|
||||
assert "status" in names, f"tools not exposed to Codex: {sorted(names)}"
|
||||
|
||||
# 5. tools/call -- the agent's first real action.
|
||||
call = rpc(url, "tools/call", {"name": "status", "arguments": {}}, id=2,
|
||||
protocol_version=negotiated)
|
||||
assert call.status_code == 200, f"tools/call: HTTP {call.status_code}"
|
||||
assert call.json()["result"]["content"][0]["text"], "status returned no content"
|
||||
@@ -0,0 +1,97 @@
|
||||
"""
|
||||
Wire-protocol conformance tests for the Echidna MCP server.
|
||||
|
||||
This is the regression guard for "real MCP clients (Claude, Codex) can still
|
||||
connect." It asserts exact transport behaviour that a lenient
|
||||
`httpx.post(...).json()` hides but strict clients depend on -- in particular
|
||||
the behaviours that broke Codex's rmcp client and were fixed in
|
||||
haskell-mcp-server:
|
||||
|
||||
* a notification (no `id`) -> 202 Accepted with NO body
|
||||
* GET on the endpoint (an SSE-stream request) -> 405, never 200/JSON
|
||||
* initialize -> a negotiated protocolVersion + serverInfo + capabilities
|
||||
|
||||
Run with:
|
||||
pytest tests/mcp/test_mcp_conformance.py
|
||||
"""
|
||||
|
||||
from _mcp_client import PROTOCOL_VERSION, SUPPORTED_VERSIONS, rpc, http_get, handshake
|
||||
|
||||
|
||||
def test_initialize_negotiates_protocol_version(echidna_server):
|
||||
url = echidna_server["url"]
|
||||
resp = rpc(url, "initialize", {
|
||||
"protocolVersion": PROTOCOL_VERSION,
|
||||
"capabilities": {},
|
||||
"clientInfo": {"name": "conformance", "version": "0"},
|
||||
}, id=0, protocol_version=None)
|
||||
assert resp.status_code == 200, f"initialize must return 200, got {resp.status_code}"
|
||||
result = resp.json()["result"]
|
||||
assert result.get("protocolVersion") in SUPPORTED_VERSIONS, (
|
||||
f"unexpected negotiated protocolVersion: {result.get('protocolVersion')}"
|
||||
)
|
||||
assert "serverInfo" in result, "initialize result missing serverInfo"
|
||||
assert "capabilities" in result, "initialize result missing capabilities"
|
||||
|
||||
|
||||
def test_notification_returns_202_no_body(echidna_server):
|
||||
"""The Codex/rmcp regression guard.
|
||||
|
||||
A notification (no `id`) MUST get HTTP 202 with an empty body. Returning
|
||||
`200 {}` makes rmcp fail with
|
||||
'data did not match any variant of untagged enum JsonRpcMessage' and drop
|
||||
the whole server -- the exact bug this suite exists to prevent.
|
||||
"""
|
||||
url = echidna_server["url"]
|
||||
resp = rpc(url, "notifications/initialized", id=None)
|
||||
assert resp.status_code == 202, (
|
||||
f"notifications/initialized must return 202, got {resp.status_code} "
|
||||
f"with body {resp.content!r}"
|
||||
)
|
||||
assert resp.content == b"", (
|
||||
f"a 202 response must have an empty body, got {resp.content!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_get_is_sse_or_405_never_plain_json(echidna_server):
|
||||
"""A GET asking for an SSE stream must be 405 (no server stream) or a real
|
||||
text/event-stream -- never 200 application/json, which strict clients try
|
||||
to read as SSE and choke on."""
|
||||
url = echidna_server["url"]
|
||||
resp = http_get(url, accept="text/event-stream")
|
||||
ctype = resp.headers.get("content-type", "")
|
||||
assert resp.status_code == 405 or "text/event-stream" in ctype, (
|
||||
f"GET must be 405 or an SSE stream, got {resp.status_code} ({ctype})"
|
||||
)
|
||||
|
||||
|
||||
def test_tools_list_after_handshake(echidna_server):
|
||||
url = echidna_server["url"]
|
||||
handshake(url)
|
||||
resp = rpc(url, "tools/list", id=3)
|
||||
assert resp.status_code == 200
|
||||
tools = resp.json()["result"]["tools"]
|
||||
names = {t["name"] for t in tools}
|
||||
assert {"status", "inject_fuzz_transactions", "clear_fuzz_priorities"} <= names, (
|
||||
f"missing expected tools, got {sorted(names)}"
|
||||
)
|
||||
for t in tools:
|
||||
assert "inputSchema" in t, f"tool {t.get('name')} missing inputSchema"
|
||||
|
||||
|
||||
def test_unknown_method_returns_jsonrpc_error(echidna_server):
|
||||
url = echidna_server["url"]
|
||||
resp = rpc(url, "this/method/does/not/exist", id=4)
|
||||
body = resp.json()
|
||||
assert "error" in body, f"unknown method should yield a JSON-RPC error, got {body}"
|
||||
|
||||
|
||||
def test_tool_call_roundtrip(echidna_server):
|
||||
"""A full handshake + tools/call returns content (what an agent actually does)."""
|
||||
url = echidna_server["url"]
|
||||
handshake(url)
|
||||
resp = rpc(url, "tools/call", {"name": "status", "arguments": {}}, id=5)
|
||||
assert resp.status_code == 200
|
||||
result = resp.json()["result"]
|
||||
content = result.get("content", [])
|
||||
assert content and content[0].get("text"), f"status returned no content: {result}"
|
||||
Reference in New Issue
Block a user