test: drive the MCP server the way its clients do (#1608)

The MCP server is the one part of Echidna that another program talks to,
and the Haskell suite cannot say whether that conversation works: it runs
in process, while what matters here is what goes over the wire and what
the tools answer about a campaign that is actually fuzzing.

So this suite starts one. conftest.py launches a real echidna against a
fixture contract and waits for the server to accept an MCP initialize
rather than merely for the socket to open, then hands the same campaign
to every module. Four layers ride on it:

test_mcp.py covers the nine tools -- what each one reports about a live
campaign, and what it says when it cannot answer. status while the
campaign fuzzes, coverage marked up line by line, an LCOV file that is
really on disk, an injected sequence that the workers keep fuzzing
through, a replayed sequence reported call by call including the revert
reason, a sampled function whose counts show up in status a few seconds
later. Anything that steers the campaign is undone afterwards, so no test
depends on the order it ran in.

test_mcp_conformance.py is the regression guard for "real clients can
still connect". A notification gets 202 with no body; a GET asking for a
server stream gets 405 rather than the JSON blob a strict client tries to
read as SSE; initialize negotiates a version and names the server. These
are exactly the two behaviours mcp-server 0.2.0.1 fixed, and the reason
that version is the floor.

test_mcp_codex.py replays Codex's rmcp handshake step by step, and
test_mcp_claude.py drives the server with the reference `mcp` SDK through
a real session. Both are transport-level and need no API key, so both run
in CI; a live-model run is the LangGraph example instead.

That example is here too. It reads status, and when the campaign has gone
a minute without finding coverage it hands Claude the ABI and the coverage
report and injects the sequences that come back -- a demonstration that
those tools are enough to close the loop, not a tuned strategy.

Some notes on what differs from the version this is adapted from:

The tool tests are written against the server as it is rather than the
prototype: status answers JSON, show_coverage takes a contract, and a tool
that cannot do what it was asked answers with an MCP error result. They
assert on all three, where the originals asserted that some substring
appeared somewhere in the text. They also go through the shared wire
helpers after a handshake, rather than a second raw client that posted
tools/call with no handshake and no headers -- asserting leniency next to
a suite that asserts strictness said two different things about what the
server promises.

The `mcp` SDK's 2.0 renamed its transport helper, dropped the third stream
it used to yield, and moved its result models to snake_case. Rather than
straddle both majors, requirements-test.txt follows the current one and
the test skips itself on an older SDK. The other pins are bounded by major
version only, since a compatibility suite should be running against what
clients actually ship.

The fixture contracts lost their spec-kit headers, two properties that
could not fail (one returned a literal `true`), and the `test` prefix on
their entry points, which means something else in Foundry mode. What is
left is a contract that gives the tools something to report on: calls that
succeed, calls that revert with a reason, and properties that hold so the
campaign keeps running for as long as the session needs it.

The workflow now triggers on master and on pull requests against it, like
every other workflow here, and only for paths that can change what the
server answers -- it builds the whole closure and then fuzzes, which is
not something to do on every push to every branch. Its Cachix token is
unavailable to pull requests from forks, so those runs read the public
cache and push nothing; that is noted where it will be read.

tests/mcp/requirements.txt is gone: nothing referenced it, and it
contradicted both its sibling and the example's README on what to install.
So is pytest.ini, which was only read when pytest was given the directory
as an argument -- the one async test carries an explicit marker instead,
which works either way.

Co-authored-by: Dani Tradito <[email protected]>
Co-authored-by: gustavo-grieco <[email protected]>
This commit is contained in:
Emilio López
2026-09-29 17:10:51 -03:00
committed by GitHub
co-authored by Dani Tradito gustavo-grieco
parent e6baf93926
commit 5403657dbf
13 changed files with 1197 additions and 0 deletions
+99
View File
@@ -0,0 +1,99 @@
name: MCP integration tests
# Out-of-process MCP testing: spins up a real Echidna campaign with the MCP
# server and drives it the way real agent clients do. Besides the tool tests it
# runs the client-compatibility suite:
# * test_mcp_conformance.py -- wire-protocol regression guard (202/405/handshake)
# * test_mcp_codex.py -- replays Codex's strict rmcp handshake
# * test_mcp_claude.py -- drives the server with the official `mcp` SDK
# Both client checks are transport-level and need no API keys, so they run on
# every push/PR that touches the code they cover. (A live-model smoke using
# examples/mcp_agent.py needs ANTHROPIC_API_KEY / Codex auth and is
# intentionally not part of CI.)
# Building the closure and running a campaign is expensive, so this is limited
# to changes that can affect what the server answers. (GitHub Actions does not
# support YAML anchors, hence the repetition.)
on:
push:
branches:
- master
paths:
- '.github/workflows/mcp-tests.yml'
- 'tests/mcp/**'
- 'lib/**'
- 'src/**'
- 'package.yaml'
- 'flake.nix'
- 'flake.lock'
pull_request:
branches:
- master
paths:
- '.github/workflows/mcp-tests.yml'
- 'tests/mcp/**'
- 'lib/**'
- 'src/**'
- 'package.yaml'
- 'flake.nix'
- 'flake.lock'
# Least-privilege GITHUB_TOKEN: this workflow only checks out the repo and runs
# tests, so it needs no write scopes.
permissions:
contents: read
# A new push to the same ref cancels an in-flight run: this one builds the whole
# closure and then fuzzes, so it is worth not piling up.
concurrency:
group: mcp-tests-${{ github.ref }}
cancel-in-progress: true
jobs:
mcp-client-compat:
name: MCP client compatibility (Claude + Codex)
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Install Nix
uses: DeterminateSystems/nix-installer-action@ef8a148080ab6020fd15196c2084a2eea5ff2d25 # v22
# Pull prebuilt echidna/hevm/deps from the project's Cachix so the build
# takes minutes instead of compiling the whole closure from source.
# CACHIX_AUTH_TOKEN is not exposed to pull requests from forks, so those
# runs read the public cache and push nothing back: a fork PR that misses
# is slow rather than red.
- name: Configure Cachix
uses: cachix/cachix-action@5f2d7c5294214f71b873db4b969586b980625e71 # v17
with:
name: trailofbits
authToken: ${{ secrets.CACHIX_AUTH_TOKEN }}
- name: Build echidna
run: nix build .#echidna --out-link result
- name: Add echidna to PATH
run: echo "$GITHUB_WORKSPACE/result/bin" >> "$GITHUB_PATH"
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.11"
- name: Install test dependencies
run: pip install -r tests/mcp/requirements-test.txt
# Echidna shells out to crytic-compile + solc to compile the test contract.
- name: Install crytic-compile and solc
run: |
pip install crytic-compile solc-select
solc-select install 0.8.25
solc-select use 0.8.25
- name: Run MCP test suite (tools + conformance + Claude + Codex)
run: pytest tests/mcp -v
+19
View File
@@ -318,6 +318,25 @@ $ nix develop # alternatively nix-shell
[nix-shell]$ cabal new-repl
```
### Running the test suites
`cabal run tests` is the Haskell suite, and covers everything Echidna does in
process.
The MCP server is tested out of process instead, by starting a real campaign and
driving it the way an agent client would. That suite is Python, and needs an
`echidna` on `$PATH`:
```sh
$ pip install -r tests/mcp/requirements-test.txt
$ pytest tests/mcp -v
```
Point it at a campaign you started yourself with `ECHIDNA_MCP_URL`, at a
different port with `ECHIDNA_MCP_PORT`, or at a specific binary with
`ECHIDNA_BIN`. An example of an agent driving a campaign through the same
interface is in [examples/](examples/README.md).
## Public use of Echidna
### Property testing suites
+73
View File
@@ -0,0 +1,73 @@
# Echidna MCP agent example
An agent that connects to a live Echidna campaign through its [MCP](https://modelcontextprotocol.io/)
server, watches what the fuzzer is reaching, and aims it at what it is not.
## Requirements
```
pip install langchain-anthropic langgraph httpx
```
## Usage
**1. Start Echidna with the MCP server:**
```bash
echidna MyContract.sol --server 8080 --format text
```
`--format text` is required — the interactive TUI otherwise owns the terminal.
**2. Run the agent:**
```bash
export ANTHROPIC_API_KEY=your_key_here
python examples/mcp_agent.py
```
It polls until you stop it with Ctrl-C. Point it at another campaign with
`ECHIDNA_MCP_URL=http://127.0.0.1:9000/mcp`.
## What it does
Every 30 seconds it calls `status`. When the campaign has gone a minute without
finding new coverage, it reads `target` and `show_coverage`, asks Claude for
call sequences that would reach lines the campaign has not, and injects them
with `inject_fuzz_transactions` — clearing the previous round first, so the
fuzzer is not left splitting its budget across every ordering ever suggested.
## The tools the server exposes
Four report on the campaign:
| Tool | Description |
|------|-------------|
| `status` | Corpus size, iterations, coverage, failing tests, optimization values, how long since the last coverage and which functions found it, and whatever `sample` is recording |
| `target` | The contract under test and the functions it exposes |
| `show_coverage` | One contract's source, line by line, marked with what the campaign reached |
| `dump_lcov` | Write the coverage so far to an LCOV file |
Five steer it:
| Tool | Description |
|------|-------------|
| `inject_fuzz_transactions` | Spend part of the fuzzer's budget on a specific ordering of calls |
| `clear_fuzz_priorities` | Forget every injected ordering and return to the corpus |
| `execute_sequence` | Run a concrete sequence and report what each call did, without disturbing the campaign |
| `sample` | Record what one function does as the campaign calls it; results appear in `status` |
| `reload_corpus` | Pick up whatever was written to the corpus directory since the campaign started |
`inject_fuzz_transactions` and `execute_sequence` take a sequence written the way
the calls would be in Solidity, separated by `;`. `inject_fuzz_transactions`
additionally accepts `?` for an argument the fuzzer should choose:
```
approve(0x10, ?); transferFrom(?, ?, 100)
```
## Testing the server
`tests/mcp/` drives a real campaign the way these clients do — tool semantics,
wire-protocol conformance, and the official MCP SDK. See the
[README](../README.md#running-the-test-suites) for how to run it.
+190
View File
@@ -0,0 +1,190 @@
#!/usr/bin/env python3
"""
LangGraph agent for Echidna's MCP server.
Connects to a running campaign, watches what the fuzzer is reaching, and when
coverage stalls asks Claude for call sequences to aim it at. A demonstration of
what the server is for rather than a tuned strategy: the interesting part is
that `status` and `show_coverage` are enough for a model to decide what to try
next, and `inject_fuzz_transactions` is enough to act on it.
Requirements:
pip install langchain-anthropic langgraph httpx
Usage:
# Start Echidna with the MCP server. --format text is required: the
# interactive TUI otherwise owns the terminal.
echidna MyContract.sol --server 8080 --format text
# Run the agent:
ANTHROPIC_API_KEY=... python examples/mcp_agent.py
"""
import json
import os
import re
import time
from typing import TypedDict
import httpx
from langchain_anthropic import ChatAnthropic
from langgraph.graph import END, StateGraph
MCP_URL = os.environ.get("ECHIDNA_MCP_URL", "http://127.0.0.1:8080/mcp")
PROTOCOL_VERSION = "2025-06-18"
# How long the campaign may go without finding coverage before the agent steps
# in, and how often it looks.
STALL_SECONDS = 60
INTERVAL_SECONDS = 30
# ---------------------------------------------------------------------------
# MCP client
# ---------------------------------------------------------------------------
def _rpc(method: str, params: dict | None = None, request_id: int | None = 1):
body = {"jsonrpc": "2.0", "method": method}
if request_id is not None:
body["id"] = request_id
if params is not None:
body["params"] = params
headers = {
"Content-Type": "application/json",
"Accept": "application/json, text/event-stream",
}
if method != "initialize":
headers["MCP-Protocol-Version"] = PROTOCOL_VERSION
response = httpx.post(MCP_URL, json=body, headers=headers, timeout=60)
response.raise_for_status()
return response.json() if request_id is not None else None
def connect() -> str:
"""Run the MCP handshake and return the server's name."""
result = _rpc("initialize", {
"protocolVersion": PROTOCOL_VERSION,
"capabilities": {},
"clientInfo": {"name": "echidna-mcp-agent", "version": "0"},
}, request_id=0)["result"]
_rpc("notifications/initialized", request_id=None)
return result["serverInfo"]["name"]
def call_tool(name: str, arguments: dict | None = None) -> str:
"""Call a tool and return its report, raising if the tool could not answer."""
result = _rpc("tools/call", {"name": name, "arguments": arguments or {}})["result"]
text = "".join(part.get("text", "") for part in result.get("content", []))
if result.get("isError"):
raise RuntimeError(f"{name}: {text}")
return text
# ---------------------------------------------------------------------------
# Graph
# ---------------------------------------------------------------------------
class State(TypedDict):
contract: str
iterations: int
coverage: int
stalled_for: int
def observe(state: State) -> State:
"""Read how the campaign is going."""
status = json.loads(call_tool("status"))
stalled_for = status["time_since_last_coverage_sec"] or 0
print(f" coverage={status['coverage_points']} "
f"iterations={status['iterations']} "
f"last coverage {stalled_for}s ago "
f"({', '.join(status['recent_covered_functions'][:3]) or 'nothing yet'})")
return {
**state,
"iterations": status["iterations"],
"coverage": status["coverage_points"],
"stalled_for": stalled_for,
}
def inject(state: State) -> State:
"""Ask Claude which orderings to try, and point the fuzzer at them."""
# temperature is not accepted on current models; steer with the prompt.
llm = ChatAnthropic(model="claude-opus-4-8", max_tokens=1024)
abi = call_tool("target")
coverage = call_tool("show_coverage", {"contract": state["contract"]})
answer = llm.invoke(
"You are helping an Echidna fuzzing campaign reach code it has not "
"reached. It has found no new coverage for a while.\n\n"
f"The contract under test exposes:\n{abi}\n\n"
f"Its coverage so far — lines marked * were executed, r reverted, "
f"o ran out of gas, e errored, blank was never reached:\n"
f"{coverage[:8000]}\n\n"
"Reply with up to 3 call sequences that would reach unmarked lines, one "
"per line and nothing else. Separate the calls in a sequence with ';', "
"and write '?' for any argument the fuzzer should choose. For example:\n"
"approve(0x10, ?); transferFrom(?, ?, 100)\n"
).content
# Anything with a call in it; the model's prose, if any, has no parentheses.
sequences = [line.strip() for line in answer.splitlines()
if re.search(r"\w+\s*\(", line)]
# Drop what was injected last time first, so the fuzzer is not left
# splitting its budget across every ordering the agent has ever suggested.
call_tool("clear_fuzz_priorities")
for sequence in sequences[:3]:
try:
print(f" injecting: {sequence}")
print(f" {call_tool('inject_fuzz_transactions', {'transactions': sequence})}")
except RuntimeError as complaint:
print(f" rejected: {complaint}")
return state
def route(state: State) -> str:
return "inject" if state["stalled_for"] >= STALL_SECONDS else END
def build_graph():
graph = StateGraph(State)
graph.add_node("observe", observe)
graph.add_node("inject", inject)
graph.set_entry_point("observe")
graph.add_conditional_edges("observe", route, {"inject": "inject", END: END})
graph.add_edge("inject", END)
return graph.compile()
def main() -> None:
if not os.getenv("ANTHROPIC_API_KEY"):
print("Set ANTHROPIC_API_KEY before running.")
return
try:
print(f"Connected to {connect()} at {MCP_URL}.")
except Exception as unreachable:
print(f"Cannot reach the MCP server: {unreachable}")
print("Start Echidna with: echidna MyContract.sol --server 8080 --format text")
return
# "Contract: path/to/File.sol:Name" — the name is what show_coverage wants.
contract = call_tool("target").splitlines()[0].rsplit(":", 1)[1].strip()
print(f"Watching {contract}. Stepping in after {STALL_SECONDS}s without coverage.")
graph = build_graph()
state: State = {"contract": contract, "iterations": 0, "coverage": 0, "stalled_for": 0}
step = 0
while True:
step += 1
print(f"\n--- step {step} ---")
state = graph.invoke(state)
time.sleep(INTERVAL_SECONDS)
if __name__ == "__main__":
main()
+100
View File
@@ -0,0 +1,100 @@
"""
Minimal, dependency-light MCP-over-HTTP client helpers shared by every module
in this suite.
These intentionally do NOT use a high-level MCP SDK: they speak the wire
protocol directly so the tests can assert on exact status codes, headers and
bodies — the things real clients (Codex's rmcp, Anthropic's MCP client) are
strict about and that a lenient `httpx.post(...).json()` would silently hide.
The SDK does get exercised, as a client rather than as a helper, in
test_mcp_claude.py.
"""
import time
import httpx
PROTOCOL_VERSION = "2025-06-18"
SUPPORTED_VERSIONS = ("2025-06-18", "2025-03-26", "2024-11-05")
class ToolError(Exception):
"""A tool answered with an MCP error result instead of a report."""
def rpc(url, method, params=None, id=1, protocol_version=PROTOCOL_VERSION,
accept="application/json, text/event-stream", timeout=30):
"""POST a single JSON-RPC message; return the raw httpx.Response (never parsed)."""
body = {"jsonrpc": "2.0", "method": method}
if id is not None:
body["id"] = id
if params is not None:
body["params"] = params
headers = {"Content-Type": "application/json", "Accept": accept}
# Per the spec the MCP-Protocol-Version header is sent on requests *after*
# initialization, not on `initialize` itself.
if protocol_version is not None and method != "initialize":
headers["MCP-Protocol-Version"] = protocol_version
return httpx.post(url, json=body, headers=headers, timeout=timeout)
def http_get(url, accept="text/event-stream", timeout=10):
"""GET the MCP endpoint (clients use this to open the server->client SSE stream)."""
return httpx.get(url, headers={"Accept": accept}, timeout=timeout)
def handshake(url, protocol_version=PROTOCOL_VERSION):
"""Run the client half of the MCP handshake; return the InitializeResult dict.
initialize -> (read result) -> notifications/initialized
"""
resp = rpc(url, "initialize", {
"protocolVersion": protocol_version,
"capabilities": {},
"clientInfo": {"name": "echidna-mcp-tests", "version": "0"},
}, id=0, protocol_version=None)
resp.raise_for_status()
result = resp.json()["result"]
negotiated = result.get("protocolVersion", protocol_version)
rpc(url, "notifications/initialized", id=None, protocol_version=negotiated)
return result
def call_tool(url, name, arguments=None, protocol_version=PROTOCOL_VERSION,
timeout=60):
"""Call a tool and return its report.
A tool that cannot do what it was asked answers with an MCP error result
rather than with a report, so that becomes a ToolError here: a test that
did not expect one fails at the call rather than on some later assertion
about the text. Tests that do expect one use `pytest.raises(ToolError)`.
"""
resp = rpc(url, "tools/call", {"name": name, "arguments": arguments or {}},
id=2, protocol_version=protocol_version, timeout=timeout)
resp.raise_for_status()
body = resp.json()
if "error" in body:
raise ToolError(f"{name}: JSON-RPC error {body['error']}")
result = body["result"]
text = "".join(part.get("text", "") for part in result.get("content", []))
if result.get("isError"):
raise ToolError(f"{name}: {text}")
return text
def eventually(read, timeout=60, interval=0.5):
"""Poll `read` until it answers with something truthy, and return that.
Nothing a client asks for takes effect at once: a worker only looks at the
bus between sequences, and what it does then only shows up in `status` on
the next call. Returns None if the timeout runs out, so the caller can say
what it was waiting for.
"""
deadline = time.time() + timeout
while True:
answer = read()
if answer:
return answer
if time.time() >= deadline:
return None
time.sleep(interval)
+110
View File
@@ -0,0 +1,110 @@
"""
Shared pytest fixtures for the out-of-process MCP test suite.
A single session-scoped Echidna campaign (with the MCP server enabled) is
started once and reused by every test module: the tool tests (test_mcp.py), the
wire-protocol conformance tests (test_mcp_conformance.py), and the
client-simulation tests (test_mcp_claude.py, test_mcp_codex.py).
The Echidna launch/wait logic was extracted from the original test_mcp.py
contributed by Dani Tradito (@datradito) in crytic/echidna#1509.
"""
import os
import subprocess
import time
import pytest
from _mcp_client import PROTOCOL_VERSION, handshake, rpc
MCP_PORT = int(os.environ.get("ECHIDNA_MCP_PORT", "8080"))
MCP_URL = f"http://127.0.0.1:{MCP_PORT}/mcp"
ECHIDNA_BIN = os.environ.get("ECHIDNA_BIN", "echidna")
CONTRACT = os.path.join(os.path.dirname(__file__), "contracts", "EchidnaMCPTest.sol")
def _server_ready() -> bool:
"""True once the MCP server answers a real `initialize` request."""
try:
resp = rpc(MCP_URL, "initialize", {
"protocolVersion": PROTOCOL_VERSION,
"capabilities": {},
"clientInfo": {"name": "readiness-probe", "version": "0"},
}, id=0, protocol_version=None, timeout=2)
return resp.status_code == 200 and "result" in resp.json()
except Exception:
return False
@pytest.fixture(scope="session")
def echidna_server(tmp_path_factory):
"""Start an Echidna campaign with the MCP server for the whole test session.
If ECHIDNA_MCP_URL is set, use that already-running server instead of
starting one (handy for pointing the suite at a live campaign or, in CI, a
deliberately broken server to prove the conformance tests have teeth). The
campaign is then someone else's, so tests that look at what it wrote to
disk skip themselves.
"""
external = os.environ.get("ECHIDNA_MCP_URL")
if external:
yield {"url": external, "local": False}
return
corpus_dir = tmp_path_factory.mktemp("corpus")
env = os.environ.copy()
env["PATH"] = os.path.expanduser("~/.local/bin") + os.pathsep + env.get("PATH", "")
# Echidna shells out to solc/crytic-compile; an active VIRTUAL_ENV breaks that.
env.pop("VIRTUAL_ENV", None)
cmd = [
ECHIDNA_BIN, CONTRACT,
"--contract", "EchidnaMCPTest",
"--server", str(MCP_PORT),
# Required: the interactive TUI otherwise owns the terminal.
"--format", "text",
# The campaign has to outlive the session; it is stopped from here.
"--test-limit", "1000000000",
"--corpus-dir", str(corpus_dir),
]
proc = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, env=env
)
# Wait (compiling the contract can take a while) for the server to accept a
# real MCP initialize, not merely an open socket.
deadline = time.time() + 120
while time.time() < deadline:
if proc.poll() is not None:
out, err = proc.communicate()
raise RuntimeError(f"Echidna exited early.\nstdout:\n{out}\nstderr:\n{err}")
if _server_ready():
break
time.sleep(0.5)
else:
proc.terminate()
out, err = proc.communicate(timeout=5)
raise RuntimeError(f"MCP server did not become ready.\nstdout:\n{out}\nstderr:\n{err}")
yield {"url": MCP_URL, "local": True, "corpus_dir": str(corpus_dir)}
proc.terminate()
try:
proc.communicate(timeout=5)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait()
@pytest.fixture(scope="session")
def mcp_url(echidna_server):
"""The endpoint, with the handshake already done.
Every test but the conformance ones is about what the campaign answers, not
about how a session is established, and a real client has initialized
before it asks anything.
"""
handshake(echidna_server["url"])
return echidna_server["url"]
+48
View File
@@ -0,0 +1,48 @@
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
import "./SimpleToken.sol";
/// The contract the MCP test campaign runs against.
///
/// It is here to give the tools something to report on rather than to find a
/// bug: a handful of functions to inject, sample and replay, one of which
/// reverts or succeeds depending on its arguments, and properties that hold so
/// the campaign keeps running for as long as the test session needs it.
contract EchidnaMCPTest {
SimpleToken public token;
uint256 constant INITIAL_SUPPLY = 1000000 * 10**18;
constructor() {
token = new SimpleToken(INITIAL_SUPPLY);
}
/// Succeeds for an amount this contract still holds, reverts otherwise --
/// which is most of the time, the amount being a random uint256.
function transferTokens(address to, uint256 amount) public {
token.transfer(to, amount);
}
function approveSpender(address spender, uint256 amount) public {
token.approve(spender, amount);
}
/// Succeeds for any recipient but the zero address.
function mintTokens(address to, uint256 amount) public {
token.mint(to, amount);
}
function tokenAddress() public view returns (address) {
return address(token);
}
/// Nothing burns, so the supply only ever grows.
function echidna_supply_never_shrinks() public view returns (bool) {
return token.totalSupply() >= INITIAL_SUPPLY;
}
/// No account can hold more than was ever issued.
function echidna_balance_within_supply() public view returns (bool) {
return token.balanceOf(address(this)) <= token.totalSupply();
}
}
+76
View File
@@ -0,0 +1,76 @@
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
/// A minimal ERC20-ish token for the MCP test campaign to fuzz.
///
/// The `require`s are the point: they give the campaign calls that revert with
/// a reason, which is what `sample` records and `execute_sequence` reports on.
contract SimpleToken {
string public name = "Test Token";
string public symbol = "TEST";
uint8 public decimals = 18;
uint256 public totalSupply;
mapping(address => uint256) public balances;
mapping(address => mapping(address => uint256)) public allowances;
event Transfer(address indexed from, address indexed to, uint256 value);
event Approval(address indexed owner, address indexed spender, uint256 value);
constructor(uint256 initialSupply) {
totalSupply = initialSupply;
balances[msg.sender] = initialSupply;
emit Transfer(address(0), msg.sender, initialSupply);
}
function transfer(address to, uint256 amount) public returns (bool) {
require(to != address(0), "Invalid recipient");
require(balances[msg.sender] >= amount, "Insufficient balance");
balances[msg.sender] -= amount;
balances[to] += amount;
emit Transfer(msg.sender, to, amount);
return true;
}
function approve(address spender, uint256 amount) public returns (bool) {
require(spender != address(0), "Invalid spender");
allowances[msg.sender][spender] = amount;
emit Approval(msg.sender, spender, amount);
return true;
}
function transferFrom(address from, address to, uint256 amount) public returns (bool) {
require(to != address(0), "Invalid recipient");
require(balances[from] >= amount, "Insufficient balance");
require(allowances[from][msg.sender] >= amount, "Insufficient allowance");
balances[from] -= amount;
balances[to] += amount;
allowances[from][msg.sender] -= amount;
emit Transfer(from, to, amount);
return true;
}
/// Unguarded on purpose: the campaign needs a call that always succeeds.
function mint(address to, uint256 amount) public {
require(to != address(0), "Invalid recipient");
totalSupply += amount;
balances[to] += amount;
emit Transfer(address(0), to, amount);
}
function balanceOf(address account) public view returns (uint256) {
return balances[account];
}
function allowance(address owner, address spender) public view returns (uint256) {
return allowances[owner][spender];
}
}
+13
View File
@@ -0,0 +1,13 @@
# Dependencies for the out-of-process MCP test suite (tests/mcp/).
#
# Bounded by major version only: the suite is a compatibility check, so it
# should be running against what clients actually ship rather than against
# whatever was current when it was written.
httpx>=0.27,<1
pytest>=8.0,<10
pytest-asyncio>=0.24,<2
# Official Model Context Protocol SDK -- used as the reference "Claude-family"
# client in test_mcp_claude.py (transport only; no API key required).
mcp>=2,<3
+245
View File
@@ -0,0 +1,245 @@
"""
Tool tests for the Echidna MCP server: what the nine tools answer when they are
driven against a live campaign, and what they say when they cannot answer.
The transport is somebody else's problem here (test_mcp_conformance.py), as is
the promise that `execute_sequence` leaves the campaign alone, which the
Haskell suite pins down from the inside. What is left is the interface an agent
sees.
Originally contributed by Dani Tradito (@datradito) in crytic/echidna#1509;
adapted here to the session-scoped campaign in conftest.py.
Run with:
pytest tests/mcp/test_mcp.py
"""
import json
import os
import re
import pytest
from _mcp_client import ToolError, call_tool, eventually, rpc
@pytest.fixture(autouse=True)
def fixture_campaign(echidna_server, mcp_url):
"""These tests are about the campaign conftest.py starts, and they leave it
as they found it.
Injection and sampling are campaign-wide and outlive the test that turned
them on, so undoing them here is what keeps the tests independent of the
order they run in.
"""
if not echidna_server["local"]:
pytest.skip("ECHIDNA_MCP_URL points at a campaign this module knows nothing about")
yield
call_tool(mcp_url, "clear_fuzz_priorities")
call_tool(mcp_url, "sample", {"function": "off"})
def status(url) -> dict:
return json.loads(call_tool(url, "status"))
def test_every_tool_is_advertised(mcp_url):
"""The tools an agent can reach, which examples/README.md documents."""
listed = rpc(mcp_url, "tools/list", id=3).json()["result"]["tools"]
assert {tool["name"] for tool in listed} == {
"status", "target", "show_coverage", "dump_lcov", "reload_corpus",
"inject_fuzz_transactions", "clear_fuzz_priorities", "execute_sequence",
"sample",
}
for tool in listed:
assert tool["description"], f"{tool['name']} is undescribed"
def test_status_reports_campaign_metrics(mcp_url):
report = status(mcp_url)
# The two echidna_ properties of the fixture contract, both of which hold.
assert report["tests_total"] == 2
assert report["tests_failed"] == 0
assert report["optimization_values"] == []
assert eventually(lambda: status(mcp_url)["iterations"] > 0), \
"the campaign never reported an iteration"
assert eventually(lambda: status(mcp_url)["coverage_points"] > 0), \
"the campaign never reported any coverage"
assert eventually(lambda: status(mcp_url)["recent_covered_functions"]), \
"no function was credited with finding coverage"
def test_target_lists_the_abi(mcp_url):
report = call_tool(mcp_url, "target")
assert "EchidnaMCPTest" in report
for signature in ("transferTokens(address,uint256)",
"approveSpender(address,uint256)",
"mintTokens(address,uint256)"):
assert signature in report, f"{signature} missing from:\n{report}"
def test_show_coverage_marks_up_the_source(mcp_url):
report = call_tool(mcp_url, "show_coverage", {"contract": "EchidnaMCPTest"})
assert "EchidnaMCPTest.sol" in report
assert "contract EchidnaMCPTest {" in report, "the source is not in the report"
# " 12 | *r | contract EchidnaMCPTest {": line number, markers, source.
assert re.search(r"^\s*\d+ \| *\*", report, re.M), \
f"nothing is marked as executed:\n{report[:2000]}"
def test_show_coverage_needs_a_contract_that_exists(mcp_url):
with pytest.raises(ToolError, match="required"):
call_tool(mcp_url, "show_coverage")
with pytest.raises(ToolError, match="No contract by that name"):
call_tool(mcp_url, "show_coverage", {"contract": "NotAContract"})
def test_dump_lcov_writes_a_file(echidna_server, mcp_url):
report = call_tool(mcp_url, "dump_lcov")
written = re.fullmatch(r"Wrote LCOV coverage to (.+)\.", report.strip())
assert written, f"unexpected dump_lcov report: {report}"
path = written.group(1)
assert os.path.dirname(path) == echidna_server["corpus_dir"], \
f"LCOV went somewhere other than the corpus directory: {path}"
with open(path) as f:
assert "SF:" in f.read(), "no source-file record in the LCOV output"
def test_reload_corpus_reads_the_corpus_directory(mcp_url):
assert eventually(lambda: status(mcp_url)["corpus_size"] > 0), \
"the campaign never wrote a corpus entry to reload"
report = call_tool(mcp_url, "reload_corpus")
# Every sequence on disk is one this campaign put there, so it has them all
# already; what matters is that it read the directory and said what it found.
assert re.match(r"Added \d+ of \d+ transaction sequences from ", report), \
f"unexpected reload_corpus report: {report}"
def test_inject_fuzz_transactions_keeps_the_campaign_running(mcp_url):
before = status(mcp_url)["iterations"]
report = call_tool(mcp_url, "inject_fuzz_transactions", {
"transactions": "mintTokens(0x1111111111111111111111111111111111111111, 100);"
"transferTokens(?, ?)"
})
assert re.fullmatch(r"Fuzzing that sequence on [1-9]\d* workers\.", report), \
f"unexpected injection report: {report}"
# A worker that choked on the injected sequence would stop reporting here.
assert eventually(lambda: status(mcp_url)["iterations"] > before), \
"the campaign stopped fuzzing after an injection"
def test_inject_fuzz_transactions_checks_the_sequence(mcp_url):
with pytest.raises(ToolError, match="No function 'nosuchfunction'"):
call_tool(mcp_url, "inject_fuzz_transactions",
{"transactions": "nosuchfunction(1)"})
with pytest.raises(ToolError, match="takes 2 arguments, not 1"):
call_tool(mcp_url, "inject_fuzz_transactions",
{"transactions": "transferTokens(?)"})
with pytest.raises(ToolError, match="Could not parse"):
call_tool(mcp_url, "inject_fuzz_transactions",
{"transactions": "not a call sequence"})
def test_clear_fuzz_priorities_returns_the_workers_to_the_corpus(mcp_url):
call_tool(mcp_url, "inject_fuzz_transactions",
{"transactions": "mintTokens(?, ?)"})
report = call_tool(mcp_url, "clear_fuzz_priorities")
assert re.fullmatch(r"Cleared injected sequences on [1-9]\d* workers\.", report), \
f"unexpected clear report: {report}"
assert eventually(lambda: status(mcp_url)["iterations"] > 0), \
"the campaign stopped fuzzing after clearing its priorities"
def test_execute_sequence_reports_every_call(mcp_url):
report = json.loads(call_tool(mcp_url, "execute_sequence", {
"transactions": "mintTokens(0x10, 1000); transferTokens(0x20, 500)"
}))
assert report["status"] == "completed"
assert report["failed_tx_index"] is None
assert report["transaction_count"] == 2
minted, transferred = report["transactions"]
assert minted["call"].endswith("mintTokens(0x10,1000)")
assert transferred["call"].endswith("transferTokens(0x20,500)")
for tx in (minted, transferred):
assert tx["status"] == "completed", f"{tx['call']} did not complete: {tx}"
assert tx["gas_used"] > 0
assert any("Transfer" in log for log in tx["logs"]), \
f"{tx['call']} emitted no Transfer: {tx['logs']}"
def test_execute_sequence_reports_a_revert(mcp_url):
# More than the whole supply, so SimpleToken.transfer's require fails.
report = json.loads(call_tool(mcp_url, "execute_sequence", {
"transactions": "transferTokens(0x20, 999999999999999999999999999999)"
}))
assert report["status"] == "reverted"
assert report["failed_tx_index"] == 1
reverted, = report["transactions"]
assert reverted["status"] == "reverted"
assert any("Insufficient balance" in log for log in reverted["logs"]), \
f"the revert reason is not in the report: {reverted['logs']}"
def test_execute_sequence_can_return_a_trace(mcp_url):
report = json.loads(call_tool(mcp_url, "execute_sequence", {
"transactions": "mintTokens(0x10, 1)", "trace": True
}))
assert "SimpleToken::mint" in report["trace"], \
f"the inner call is not in the trace: {report.get('trace')}"
def test_execute_sequence_needs_concrete_arguments(mcp_url):
with pytest.raises(ToolError, match="has to be concrete"):
call_tool(mcp_url, "execute_sequence",
{"transactions": "transferTokens(?, 1)"})
def test_sample_records_what_a_function_does(mcp_url):
report = call_tool(mcp_url, "sample", {"function": "transferTokens"})
assert re.fullmatch(
r"Sampling transferTokens\(address,uint256\) on [1-9]\d* workers\.", report
), f"unexpected sample report: {report}"
sampled = eventually(lambda: status(mcp_url)["samples"])
assert sampled, "the campaign never reported a sample"
stats, = sampled
assert stats["function"] == "transferTokens(address,uint256)"
assert stats["calls"] > 0
assert 0 <= stats["reverts"] <= stats["calls"]
# A random uint256 is more than the supply, so most calls revert with a
# reason, and the tail of them is what an agent reads.
def recent_reverts():
sampled = status(mcp_url)["samples"]
return sampled and sampled[0]["recent_reverts"]
assert eventually(recent_reverts), \
"a function that reverts reported no revert summaries"
assert call_tool(mcp_url, "sample", {"function": "off"}).startswith("Stopped sampling")
assert eventually(lambda: status(mcp_url)["samples"] == []), \
"the campaign kept sampling after being told to stop"
def test_sample_needs_a_function_that_exists(mcp_url):
with pytest.raises(ToolError, match="No function 'nosuchfunction'"):
call_tool(mcp_url, "sample", {"function": "nosuchfunction"})
+60
View File
@@ -0,0 +1,60 @@
"""
Claude (Anthropic MCP client) compatibility test.
Anthropic's clients speak MCP via the reference `mcp` SDK. This test drives the
server with that SDK's Streamable-HTTP client through a full, real session
(initialize -> list_tools -> call_tool). If the server's transport is
non-conformant, the SDK -- like Claude -- fails to establish the session, so
this is a faithful "would Claude break?" check.
No API key is required: it exercises the MCP transport, not the model. (An
end-to-end run with a live model is the LangGraph demo in examples/mcp_agent.py,
which needs ANTHROPIC_API_KEY and is not part of CI.)
Requires the `mcp` SDK and pytest-asyncio (see requirements-test.txt); the test
skips itself if neither is installed.
Run with:
pip install -r tests/mcp/requirements-test.txt
pytest tests/mcp/test_mcp_claude.py
"""
import pytest
transport = pytest.importorskip(
"mcp.client.streamable_http",
reason="install the `mcp` SDK to run the Claude client test",
)
# The SDK's 2.0 renamed the transport helper, stopped yielding a third stream
# alongside the two, and moved its result models to snake_case. Rather than
# straddle both spellings, follow the current major and say so.
if not hasattr(transport, "streamable_http_client"):
pytest.skip(
"needs the `mcp` SDK 2.x; see tests/mcp/requirements-test.txt",
allow_module_level=True,
)
from mcp import ClientSession # noqa: E402
from mcp.client.streamable_http import streamable_http_client # noqa: E402
@pytest.mark.asyncio
async def test_claude_mcp_sdk_session(echidna_server):
"""A real MCP SDK client completes a session and calls a tool."""
async with streamable_http_client(echidna_server["url"]) as (read, write):
async with ClientSession(read, write) as session:
init = await session.initialize()
assert init.server_info is not None, "initialize returned no serverInfo"
tools = await session.list_tools()
names = {t.name for t in tools.tools}
assert "status" in names, (
f"tools not visible to the MCP SDK client: {sorted(names)}"
)
result = await session.call_tool("status", {})
assert not result.is_error, f"status tool reported an error: {result}"
assert result.content, "status returned no content to the MCP SDK client"
text = getattr(result.content[0], "text", "")
assert text, f"empty status content: {result.content}"
+67
View File
@@ -0,0 +1,67 @@
"""
Codex (rmcp) client-compatibility test.
Codex connects to MCP servers with the Rust `rmcp` StreamableHttpClientTransport,
which is strict about the wire protocol. We can't run rmcp from Python, so this
module *replays Codex's exact observed handshake* against the live server and
asserts the server satisfies rmcp's expectations at each step. A regression that
would make Codex drop the server fails here -- without needing Codex installed
or authenticated.
The sequence and failure modes below were reverse-engineered from a real
`codex exec` run against an Echidna MCP server.
For an end-to-end smoke with the real binary (needs Codex auth, so it is NOT in
CI): point Codex's `~/.codex/config.toml` at the running campaign and run
`codex mcp get echidna`.
Run with:
pytest tests/mcp/test_mcp_codex.py
"""
from _mcp_client import PROTOCOL_VERSION, SUPPORTED_VERSIONS, rpc, http_get
def test_codex_rmcp_handshake_replay(echidna_server):
url = echidna_server["url"]
# 1. initialize -- rmcp sends this first (Accept: application/json, text/event-stream).
init = rpc(url, "initialize", {
"protocolVersion": PROTOCOL_VERSION,
"capabilities": {"roots": {"listChanged": True}},
"clientInfo": {"name": "codex", "version": "0"},
}, id=0, protocol_version=None)
assert init.status_code == 200, f"initialize: HTTP {init.status_code}"
init_result = init.json()["result"]
assert init_result.get("protocolVersion") in SUPPORTED_VERSIONS, (
f"protocol negotiation: {init_result.get('protocolVersion')}"
)
negotiated = init_result["protocolVersion"]
# 2. notifications/initialized -- rmcp DIES here if the server answers `200 {}`
# ("data did not match any variant of untagged enum JsonRpcMessage").
note = rpc(url, "notifications/initialized", id=None, protocol_version=negotiated)
assert note.status_code == 202 and note.content == b"", (
"Codex's rmcp client requires 202 + empty body for notifications; "
f"got {note.status_code} {note.content!r}"
)
# 3. GET the endpoint to open the server->client SSE stream. rmcp tolerates
# 405 ("no server stream"); a 200 application/json discovery blob breaks it.
stream = http_get(url, accept="text/event-stream")
ctype = stream.headers.get("content-type", "")
assert stream.status_code == 405 or "text/event-stream" in ctype, (
f"GET stream must be 405 or SSE; got {stream.status_code} ({ctype})"
)
# 4. tools/list -- only reached if the handshake survived.
tools = rpc(url, "tools/list", id=1, protocol_version=negotiated)
assert tools.status_code == 200, f"tools/list: HTTP {tools.status_code}"
names = {t["name"] for t in tools.json()["result"]["tools"]}
assert "status" in names, f"tools not exposed to Codex: {sorted(names)}"
# 5. tools/call -- the agent's first real action.
call = rpc(url, "tools/call", {"name": "status", "arguments": {}}, id=2,
protocol_version=negotiated)
assert call.status_code == 200, f"tools/call: HTTP {call.status_code}"
assert call.json()["result"]["content"][0]["text"], "status returned no content"
+97
View File
@@ -0,0 +1,97 @@
"""
Wire-protocol conformance tests for the Echidna MCP server.
This is the regression guard for "real MCP clients (Claude, Codex) can still
connect." It asserts exact transport behaviour that a lenient
`httpx.post(...).json()` hides but strict clients depend on -- in particular
the behaviours that broke Codex's rmcp client and were fixed in
haskell-mcp-server:
* a notification (no `id`) -> 202 Accepted with NO body
* GET on the endpoint (an SSE-stream request) -> 405, never 200/JSON
* initialize -> a negotiated protocolVersion + serverInfo + capabilities
Run with:
pytest tests/mcp/test_mcp_conformance.py
"""
from _mcp_client import PROTOCOL_VERSION, SUPPORTED_VERSIONS, rpc, http_get, handshake
def test_initialize_negotiates_protocol_version(echidna_server):
url = echidna_server["url"]
resp = rpc(url, "initialize", {
"protocolVersion": PROTOCOL_VERSION,
"capabilities": {},
"clientInfo": {"name": "conformance", "version": "0"},
}, id=0, protocol_version=None)
assert resp.status_code == 200, f"initialize must return 200, got {resp.status_code}"
result = resp.json()["result"]
assert result.get("protocolVersion") in SUPPORTED_VERSIONS, (
f"unexpected negotiated protocolVersion: {result.get('protocolVersion')}"
)
assert "serverInfo" in result, "initialize result missing serverInfo"
assert "capabilities" in result, "initialize result missing capabilities"
def test_notification_returns_202_no_body(echidna_server):
"""The Codex/rmcp regression guard.
A notification (no `id`) MUST get HTTP 202 with an empty body. Returning
`200 {}` makes rmcp fail with
'data did not match any variant of untagged enum JsonRpcMessage' and drop
the whole server -- the exact bug this suite exists to prevent.
"""
url = echidna_server["url"]
resp = rpc(url, "notifications/initialized", id=None)
assert resp.status_code == 202, (
f"notifications/initialized must return 202, got {resp.status_code} "
f"with body {resp.content!r}"
)
assert resp.content == b"", (
f"a 202 response must have an empty body, got {resp.content!r}"
)
def test_get_is_sse_or_405_never_plain_json(echidna_server):
"""A GET asking for an SSE stream must be 405 (no server stream) or a real
text/event-stream -- never 200 application/json, which strict clients try
to read as SSE and choke on."""
url = echidna_server["url"]
resp = http_get(url, accept="text/event-stream")
ctype = resp.headers.get("content-type", "")
assert resp.status_code == 405 or "text/event-stream" in ctype, (
f"GET must be 405 or an SSE stream, got {resp.status_code} ({ctype})"
)
def test_tools_list_after_handshake(echidna_server):
url = echidna_server["url"]
handshake(url)
resp = rpc(url, "tools/list", id=3)
assert resp.status_code == 200
tools = resp.json()["result"]["tools"]
names = {t["name"] for t in tools}
assert {"status", "inject_fuzz_transactions", "clear_fuzz_priorities"} <= names, (
f"missing expected tools, got {sorted(names)}"
)
for t in tools:
assert "inputSchema" in t, f"tool {t.get('name')} missing inputSchema"
def test_unknown_method_returns_jsonrpc_error(echidna_server):
url = echidna_server["url"]
resp = rpc(url, "this/method/does/not/exist", id=4)
body = resp.json()
assert "error" in body, f"unknown method should yield a JSON-RPC error, got {body}"
def test_tool_call_roundtrip(echidna_server):
"""A full handshake + tools/call returns content (what an agent actually does)."""
url = echidna_server["url"]
handshake(url)
resp = rpc(url, "tools/call", {"name": "status", "arguments": {}}, id=5)
assert resp.status_code == 200
result = resp.json()["result"]
content = result.get("content", [])
assert content and content[0].get("text"), f"status returned no content: {result}"