mirror of
https://github.com/Heretek-AI/awesome-harness.git
synced 2026-11-07 10:46:11 -10:00
Awesome Open Source AI Agent Harnesses
A dynamically curated list of open source AI agent testbeds, evaluation harnesses, sandboxed runtimes, and benchmarking frameworks, automatically updated and grouped by GitHub topics.
What is an AI Agent Harness?
An AI Agent Harness is the runtime environment, benchmarking framework, evaluation testbed, or execution sandbox that provides:
- Task Environments & Datasets: Realistic scenarios (codebases, browser interfaces, terminal shells, APIs).
- Execution Runtimes: Isolated sandboxes (Docker, WebContainers, microVMs) to safely run agent actions.
- Observation & Tooling Interfaces: Standardized protocols (e.g., Model Context Protocol / MCP, tool-calling APIs).
- Scoring & Verification: Deterministic or LLM-judged metrics to validate agent completion and test against regressions.
This repository dynamically indexes, categorizes, and tracks open source harnesses across the GitHub ecosystem using GitHub Topics.
Contents
- Categories
- 💻 Software Engineering & Coding Harnesses (52)
- 📊 Evaluation, Leaderboards & General Benchmarks (96)
- 🌐 Web, Browser & OS Agent Environments (46)
- 🐝 Multi-Agent & Swarm Harnesses (24)
- 🛡️ Safety, Security & Red-Teaming Harnesses (3)
- 📦 Execution Sandboxes, Runtimes & Tooling (56)
- 🚀 Emerging & General Agent Harnesses (105)
- Topic Cloud
- How It Works
- Contributing
Categories
💻 Software Engineering & Coding Harnesses
Harnesses and testbeds for evaluating and executing autonomous coding agents against real-world repositories, issues, and pull requests.
| Repository | Stars | License | Language | Description | Key Topics |
|---|---|---|---|---|---|
| openinterpreter | ★ 68.8k |
Apache-2.0 |
Rust |
A coding agent for open models like Kimi K3 and GLM 5.3 | acp coding-agent deepseek kimi |
| jcode | ★ 21.4k |
MIT |
Rust |
High performance coding agent harness written in Rust, built for heavy multi-agent dev workflows | ai-agent ai-coding-agent claude cli |
| PI-Desktop | ★ 7.6k |
LGPL-3.0 |
TypeScript |
Local-first AI coding agent desktop: Electron + Rust host core + pi Agent Harness + user-installable plugins | ai-agent coding-agent desktop-app electron |
| SWE-bench | ★ 6.1k |
MIT |
Python |
SWE-bench: Can Language Models Resolve Real-world Github Issues? | benchmark language-model software-engineering |
| harnessrouter | ★ 3.9k |
Apache-2.0 |
Python |
HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, a... | agent-api agent-framework agent-harness ai-agents |
| openlit | ★ 2.9k |
Apache-2.0 |
TypeScript |
OpenLIT is the open-source agent harness engineering platform: trace, evaluate, guard, and improve everything around the model in your AI agents, on OTEL. | agent-harness agent-harness-engineering ai-engineering ai-observability |
| Tianshu-harness | ★ 1.2k |
Apache-2.0 |
TypeScript |
天枢harness是一个的终端编程智能体，它要回答的核心问题是：模型何以稳定地交付——目标不漂移、完成有证据、验证有闭环，不说"应该修好了"。为此它在模型与真实世界之间建立一层认知执行环境（CVM），把目标、状态、证据、资源、权限与终止条件从对话历史中外部化，由运行时持续管理。同时对deepseek v4极度优化。 | agent-harness ai-agent cli coding-agent |
| cuga-agent | ★ 1.1k |
Apache-2.0 |
Python |
CUGA is an open-source generalist agent harness for the enterprise, supporting complex task execution on web and APIs, OpenAPI/MCP integrations, composable a... | agentic-ai ai-agents coding-agent computer-use |
| dscode | ★ 999 |
MIT |
JavaScript |
A DeepSeek coding agent harness: persistent shell, Ultra subagents, auto approval, Chrome MCP and session telemetry | agentic ai-agent cli code-review |
| Repo2RLEnv | ★ 763 |
Apache-2.0 |
Python |
Turn any repository into verifiable RL environments for coding agents - Harbor tasks you can train on, evaluate and share on the Hugging Face Hub | coding-agents datasets harbor huggingface |
| CodeAF | ★ 616 |
Apache-2.0 |
Go |
Open-Source Software factory for Open Models | agent-fleet agent-harness agentfield agentic-coding |
| mu | ★ 571 |
MIT |
TypeScript |
mu (μ): most of what a coding agent decides isn't code. mu hands 40 kinds of yes/no calls to a small model: fewer wasted tokens, fewer approvals on autopilot... | agent-harness ai-agent ai-coding cli |
| uncut | ★ 473 |
MIT |
Go |
The lossless, headless coding agent harness. One 7 MB static binary, four tools, plain stdin/stdout, no silent cuts. | agent-harness cli coding-agent go |
| harness-remote | ★ 433 |
Apache-2.0 |
JavaScript |
Native-session control plane for Codex, Claude Code, OpenCode, OMP and PI. Run, resume and hand off coding sessions across your machines. | acp agent-harness agent-orchestration ai-agents |
| zot | ★ 350 |
MIT |
Go |
Yet another coding agent harness, lightweight and written in go. | agent-framework agent-harness ai-agent cli |
| no_human | ★ 326 |
MIT |
Python |
From ticket to reviewed pull request. Free and open-source, on your machine. | agent-harness agent-orchestration agentic-ai ai-agent |
| pi-bluebook | ★ 313 |
MIT |
Markdown |
Pi Coding Agent 中文学习蓝皮书：从安装与第一个可验收任务开始，逐步掌握 Session、Context、Skill、Extension、Subagent 与长期 Agent 工作流。 | agent-harness agent-skills ai-agent chinese |
| SE-Agent | ★ 290 |
MIT |
Python |
SE-Agent is a self-evolution framework for LLM Code agents. It enables trajectory-level evolution to exchange information across reasoning paths via Revision... | claude-code code-agent code-fix mcts |
| learn-agent | ★ 282 |
MIT |
JavaScript |
学习Agent开发的笔记，尝试让Agent从可用到可靠 | agent-harness agent-loop ai-agent aider |
| minimal | ★ 235 |
Apache-2.0 |
Rust |
Run your agent in a Minimal box. Ship and run software with isolation on your own computer. | agent-isolation agent-sandbox ai-agents ai-security |
| eido | ★ 215 |
GPL-3.0 |
Rust |
AI coding agent workspace for macOS, built on Zed and pi. Native code editing and review, browser use, computer use, and global agent access controls. | ai-assistant browser-use code-editor code-review |
| RepoLaunch | ★ 212 |
MIT |
Python |
[NeurIPS 2026 Main] Automate the build, execution and test of software repositories across programming languages and operating systems. | agentic-rl build-tool coding-agent swe-bench |
| crux | ★ 186 |
MIT |
TypeScript |
Taking the pi coding agent to Claude Code-level performance: 0.539 → 0.773 pass@1 on Terminal-Bench 2.1 with the same self-hosted 27B model, by re-engineerin... | agent-evaluation ai-agent coding-agent context-engineering |
| coder_eval | ★ 156 |
Apache-2.0 |
Python |
Playwright for coding agents. Test that your skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, A/B experiments, CI... | agent-evaluation agent-skills agent-testing anthropic |
| DM-Code-Agent | ★ 153 |
MIT |
Python |
Local-first, auditable Python code agent. Ships its own 30-task hidden-test benchmark plus SWE-bench Verified scored by the official harness -- every number ... | agent-evaluation agent-skills code-agent llm-agent |
| Flowness | ★ 107 |
Apache-2.0 |
Python |
A work-centered runtime for agentic software engineering. Work persists; agents, context, and graphs assemble around it. | agent-evaluation agent-governance agent-orchestration agent-runtime |
| godmode | ★ 99 |
MIT |
Python |
Production-grade Agent Skills for AI coding agents—composable workflows for planning, TDD, debugging, review, UI/UX, releases, incidents, and evals. | agent-evaluation agent-skills ai-agents ai-coding |
| vieval | ★ 71 |
MIT |
TypeScript |
🧪💥 Evaluation framework based on Vitest, the testing framework you familiar with, for agents, models, and more. | agentic-evals evals evalscope evaluation |
| retro-harness | ★ 68 |
MIT |
Python |
RHO: Evolving Agents in the Dark — Retrospective Harness Optimization via Self-Preference. Improving LLM agents from unlabeled past trajectories (arXiv:2606.... | agent-optimization llm-agents prompt-optimization research |
| benchmark-harnesses | ★ 67 |
Apache-2.0 |
Python |
Strands-based agents and harnesses for agentic benchmarks. | agentic agentic-ai benchmarks genai |
| agentic-loop-engineering-course | ★ 61 |
MIT |
Jupyter Notebook |
An 18 notebook course that isolates and measures each component of agentic loop engineering on real, industry standard software datasets. | agent-orchestration agentic-ai ai-agents code-generation |
| clawker | ★ 58 |
MIT |
Go |
AI coding agent sandbox platform | agent-sandbox agentic-ai agentic-coding ai-agent |
| Xenon | ★ 56 |
MIT |
Python |
Verifiable AI coding agent harness / 可验证的 AI 编码代理：evidence hash-chain · hollow-answer gate · manual verification checklist · 6 engines + MCP · multi-model · ... | agent-framework agent-harness ai-coding-assistant cli |
| kotlin-swe-bench | ★ 55 |
Apache-2.0 |
Python |
The official Kotlin benchmark by JetBrains | agent-evaluation ai-agents kotlin swe-bench |
| insights | ★ 52 |
MIT |
TypeScript |
We track and analyze the activity and performance of autonomous code agents in the wild | swe-agent swe-bench |
| why-was-fable-banned | ★ 47 |
MIT |
Python |
Fable-style spec + evidence gate for Claude Code + Codex. Makes Opus/Codex work under Fable-like discipline: blocks every edit until a deterministic spec pas... | agent-harness ai-agent-guardrails ai-coding claude-code |
| agro | ★ 43 |
Apache-2.0 |
TypeScript |
🏗️ AGRO — Agent Governance Runtime Orchestrator. A portable home for autonomous coding agents. | agent-sandbox ai-agents ai-coding claude |
| NanoHarness | ★ 40 |
MIT |
Python |
Compact AI agent runtime control plane with governed tools, HITL approval, resumable execution, trace-driven evaluation, and SWE-bench-shaped evidence. | agent-evaluation agent-harness coding-agent llm-agent |
| AgentPatchCheck | ★ 38 |
Apache-2.0 |
TypeScript |
Coding Agent Runtime & Evaluation Harness for controlled repository-level software repair | agent-runtime benchmarking coding-agent evaluation-harness |
| simple-long-horizon-agent | ★ 34 |
Apache-2.0 |
Python |
Simple Long Horizon Agent - A simple yet effective AI agent for learning, experimentation, and long horizon work. | agent-harness harness harness-engineering long-horizon |
| mini-ork | ★ 32 |
Apache-2.0 |
Python |
Prove an AI-written fix actually fixes the bug. mini-ork certify reproduces the bug against your repo's own code, attacks the patch with generated invariants... | agent-framework agentic-ai ai-agents ai-code-review |
| repoagentbench | ★ 26 |
MIT |
Python |
SWE-bench for your codebase — mine your merged PRs into local, contamination-free coding-agent benchmarks. Adapters: claude-code, aider (Opus 4.7 / GPT-5.5 /... | agent-evals ai-agents aider benchmark |
| squeez | ★ 24 |
Apache-2.0 |
Python |
Squeeze verbose LLM agent tool output down to only the relevant lines | coding-agent context-compression lora pytorch |
| coding-agent-kernel | ★ 23 |
Apache-2.0 |
Python |
Independent Python coding-agent kernel with observable runs, tools, sessions, permissions, DeepSeek, and SWE-bench. | agent-kernel coding-agent deepseek llm-agents |
| deepseek-v4.1-flash-a100-turbo | ★ 20 |
Apache-2.0 |
Python |
DeepSeek-V4.1-Flash inference on NVIDIA Ampere with vLLM, SM80 patches, reproducible throughput and coding-agent benchmarks. Tested on 8x A800 80GB. | a100 a800 deepseek deepseek-v4 |
| ai-workflow-benchmark | ★ 14 |
MIT |
Python |
Benchmark harness measuring AI coding tool+workflow performance, not just model capability. 100 tasks, sigmoid scoring, 12 capability dimensions, gap analysis. | ai-agents ai-coding benchmark claude-code |
| Parsimony | ★ 13 |
MIT |
Python |
Measures how much code AI coding agents add, from patches benchmarks already publish | ai-agents benchmark code-complexity code-quality |
| Agent-Sandbox | ★ 12 |
Apache-2.0 |
TypeScript |
Fast, Multi-Cloud Sandbox Engine for AI Agents | agent-sandbox agentic-rl e2b e2b-compatible |
| mcpbr | ★ 11 |
MIT |
Python |
Benchmark your MCP server. | benchmarking llm-evaluation mcp ml-evaluation |
| agents | ★ 11 |
Apache-2.0 |
Python |
Minimal coding agents, easy to play with (opencode TUI) and easy to benchmark (ACP on BenchFlow) — built around mini-swe-agent | acp ai-agents coding-agent llm-evaluation |
| harness-bench | ★ 10 |
MIT |
TypeScript |
Benchmark leaderboard for AI coding harnesses by model. | ai-agents benchmark coding-agent leaderboard |
| reason-agent | ★ 10 |
MIT |
Multi |
Reason Agent, the autonomous software engineer: how to use it, releases and benchmark results. | ai-software-engineer autonomous-agents benchmarks coding-agent |
📊 Evaluation, Leaderboards & General Benchmarks
Harnesses designed to evaluate LLMs and agent capabilities across diverse benchmarks, tasks, and standardized leaderboards.
| Repository | Stars | License | Language | Description | Key Topics |
|---|---|---|---|---|---|
| iFixAi | ★ 42.9k |
Apache-2.0 |
Python |
Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is ... | agent-evaluation ai-alignment ai-evaluation ai-governance |
| giskard-oss | ★ 5.9k |
Apache-2.0 |
Python |
🐢 Open-Source Evaluation & Testing library for LLM Agents | agent-evaluation ai-red-team ai-security ai-testing |
| coze-loop | ★ 5.8k |
Apache-2.0 |
Go |
Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities fro... | agent-evaluation agent-observability agentops coze |
| trulens | ★ 3.6k |
MIT |
Python |
Evaluation and Tracking for LLM Experiments and AI Agents | agent-evaluation agentops ai-agents ai-monitoring |
| future-agi | ★ 2.2k |
Apache-2.0 |
Python |
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway ·... | agent-evaluation ai-agents ai-evals ai-gateway |
| any-agent | ★ 1.3k |
Apache-2.0 |
Python |
A single interface to use and evaluate different agent frameworks | a2a agent-evaluation mcp |
| Tracely-ai | ★ 1.2k |
MIT |
Python |
Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-h... | agent-evaluation ai-agents ai-testing ci-cd |
| agy-staff | ★ 810 |
MIT |
JavaScript |
Hire Google's Antigravity CLI (agy) as a fast Gemini staffer for Claude Code and OpenAI Codex. | agent-evaluation agentic-ai ai-agents antigravity |
| TheAgentCompany | ★ 808 |
MIT |
Python |
An agent benchmark with tasks in a simulated software company. | ai-benchmark ai-research benchmark |
| pandaprobe | ★ 793 |
Apache-2.0 |
Python |
open source agent engineering platform: traces, evals, and metrics to debug and improve your AI agents. Integrates with LangGraph, CrewAI, Claude Agent SDK, ... | agent-engineering agent-evaluation agent-observability agentic-ai |
| SkillEvaluator | ★ 622 |
Apache-2.0 |
Python |
Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent e... | agent-evaluation agent-plugins agent-security agent-skills |
| Harness-RSI | ★ 433 |
CC-BY-4.0 |
TeX |
Harness self-evolution: update targets, theory, evaluation, and reported evidence. Bilingual reading list and project website. | agent-evaluation agent-harness agent-memory agent-skills |
| agentenv-framework | ★ 402 |
Apache-2.0 |
Python |
Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments ... | a2a agent-evaluation agentic-ai ai-agents |
| AdaRubrics | ★ 368 |
Apache-2.0 |
Python |
AdaRubric: Adaptive Dynamic Rubric Evaluator for Agent Trajectories | agent-evaluation llm-evaluation reward-model rlhf |
| benchmark-radar | ★ 313 |
MIT |
Python |
Track 42,331+ AI benchmark, eval, dataset, and data-quality records from 40 public sources, with linked evidence and daily updates. | agent-bench agent-benchmark agent-benchmarking agent-benchmarks |
| mirobody-eval | ★ 277 |
MIT |
Python |
An evaluation harness for medical/health AI agents — reproduce and cover multiple benchmarks under one scoring discipline | agent-evals benchmark-reproducibility evaluation-framework evaluation-harness |
| agent-leaderboard | ★ 227 |
MIT |
Jupyter Notebook |
Ranking LLMs on agentic tasks | agent-evaluation ai-agents ai-benchmark ai-evaluation |
| AgentMeasure | ★ 218 |
MIT |
Python |
Inspect Codex / Claude logs, verify metering samples and re-count supported billing exports. Free local tools; bounded $10 diagnosis and $20 QA. | agent-commerce agent-economy agent-evaluation agent-observability |
| ai-agents-tutorial | ★ 206 |
Apache-2.0 |
Multi |
Learn AI Agents step by step, from scratch - from function calling to agent loops to multi-agent systems, orchestration, and evaluation. | agent-evaluation agent-loop agent-orchestration ai-agent |
| Awesome-AI4AI | ★ 190 |
MIT |
Python |
Updated weekly — until humanity achieves AI4AI. A survey-grounded catalog of 432 papers on long-horizon agents, benchmarks, harness design, and recursive sel... | agent-benchmarks agent-harness agi ai-agents |
| LLM-Agent-Benchmark-List | ★ 173 |
Apache-2.0 |
Multi |
A banchmark list for evaluation of large language models. | benchmark survey |
| model-serving-minefield | ★ 167 |
MIT |
Python |
Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool parsers, reasoning fields, quant kernel paths, CUDA... | benchmarking chat-template cuda debugging |
| mirobody-env | ★ 155 |
MIT |
Python |
HAEnv — synthetic longitudinal patients with code-derived gold, for evaluating health agents over time | agent-evals benchmark clinical-decision-support evaluation-harness |
| eval-view | ★ 140 |
Apache-2.0 |
Python |
Regression testing for AI agents. Snapshot behavior,diff tool calls,catch regressions in CI. Works with LangGraph, CrewAI, OpenAI, Anthropic. | agent-benchmark agent-evaluation agentic-ai ai-agents |
| every_eval_ever | ★ 138 |
MIT |
Python |
Every Eval Ever is a shared schema and crowdsourced eval database. It defines a standardized metadata format for storing AI evaluation results — from leaderb... | agent-evaluation ai-evaluation evaluations infra |
| nemo-helix | ★ 134 |
Apache-2.0 |
Python |
NeMo Helix: Make the agents you ship faster, more accurate, and safer. | agent-evaluation ai-agents guardrails kubernetes |
| AgentKernelArena | ★ 124 |
Apache-2.0 |
Python |
AgentKernelArena is a platform for evaluating and training AI agents for GPU kernel optimization, combining realistic tasks, correctness validation, and perf... | agent-evaluation gpu-kernels llamas |
| claimpilot-harness | ★ 117 |
MIT |
Python |
Crash-test insurance claim AI agents before production. | agent-evaluation ai-agents insurance llm-evals |
| fde-gym | ★ 112 |
Apache-2.0 |
TypeScript |
The online judge for the LLM era. Every case is a real customer delivery: you work it with a coding agent, and your system is judged once it is live. | agent-evaluation ai-agents benchmark coding-agent |
| ResolveLab | ★ 106 |
MIT |
Python |
Fault-injected tool environments with side-effect grading for evaluating and post-training LLM agents | agent-evaluation ai-agents dagger fastapi |
| awesome-llm-agent-papers | ★ 105 |
MIT |
Python |
LLM agents: curated papers on planning, memory, tool use, multi-agent systems, evaluation, and safety. | agent-benchmark agent-survey ai-agents ai-safety |
| jev-curate | ★ 104 |
MIT |
Rust |
High-throughput synthetic and pretraining dataset sifter for TypeSafe Jev. Rust streaming core, Parquet and JSONL I/O, typed Choice/Score/Noul judgments, spe... | arrow cli data-cleaning data-engineering |
| dsh-harness-evolution | ★ 102 |
MIT |
HTML |
Evolved DSH plugin harnesses for five agent benchmarks, with versioned artifacts, evolution curves and Life-Harness comparisons | agent-benchmark dsh harness leaderboard |
| forsy-trace-skill | ★ 96 |
MIT |
Python |
Open skill for capturing AI agent work as structured traces. | agent-evaluation agent-traces agent-workflows ai-agents |
| heurigym | ★ 96 |
Apache-2.0 |
Python |
Agentic Benchmark for LLM-Crafted Heuristics in Combinatorial Optimization (ICLR'26) | benchmark chatgpt optimization |
| checkup | ★ 95 |
AGPL-3.0 |
Shell |
AgentVitals Checkup (/checkup) — an AI agent skill that gives your agent a professional health checkup: dual-axis Stability + Welfare scoring, a personality-... | agent-benchmark agent-health agent-skill agentvitals |
| reforge-ai-evals | ★ 95 |
MIT |
HTML |
Open-source AI evals toolkit: skills that guide AI coding agents to write eval rubrics, eval datasets (User Input Grid), trace codes, code evaluators and LLM... | agent-evaluation agent-skills ai-evals ai-product-management |
| AgentCheck | ★ 73 |
Apache-2.0 |
Python |
Behavioral testing for AI agents. | agent-evaluation ai-agents ai-safety developer-tools |
| human-value-allocation | ★ 71 |
MIT |
Python |
Evidence-linked analysis of human contributions in human–AI conversations. Research preview. | agent-evaluation human-ai-collaboration human-in-the-loop llm-evaluation |
| on-panda | ★ 65 |
MIT |
JavaScript |
🐼 Steer your LLMs and agents at the token level: A web tool for token visualization & control, model inspection, installation-free harness, and LLM & agent t... | agent-evaluation agent-trajectory agent-trajectory-annotation agent-trajectory-annotation-tool |
| HarnessBench | ★ 64 |
Apache-2.0 |
Python |
Benchmark for comparing agent harnesses on everyday online tasks — fixes the base model, varies the harness. Sister project of ClawBench, same scoring pipeline. | agent-evaluation agent-harness agentic-ai ai-agents |
| ai-agents-reality-check | ★ 61 |
Apache-2.0 |
Python |
Benchmarking the gap between AI agent hype and architecture. Three agent archetypes, 73-point performance spread, stress testing, network resilience, and ens... | agent-architecture agent-benchmark agent-evaluation agent-performance |
| dokimos | ★ 61 |
MIT |
Java |
LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client. | agent-evaluation agentic-ai embabel evaluation |
| cloudbox | ★ 55 |
MIT |
TypeScript |
Durable Cloudflare computers for agents: run repos, steer live workspaces, verify results, and return receipts. | agent-computer agent-evaluation ai-agents cloudflare |
| isnad | ★ 53 |
Apache-2.0 |
Python |
Grade every agent, scraper and model in a claim's chain — provenance, trust scoring and audit evidence for LLM pipelines | agent-evaluation agent-observability ai-agents ai-governance |
| benchjack | ★ 53 |
Apache-2.0 |
Python |
AI agent benchmark hackability scanner — find evaluation vulnerabilities before they undermine your results | ai-agents ai-security benchmark evaluation |
| ConsciousDB | ★ 53 |
Apache-2.0 |
Python |
A database-agnostic Agentic GraphRAG framework using swappable System One models (local Laya / cloud Jev). A plug-and-play intelligence layer featuring a com... | astar-algorithm eval-harness fine-tuning graphdatabase |
| laya-jev-GraphRAG | ★ 51 |
Apache-2.0 |
Python |
A database-agnostic Agentic GraphRAG framework using swappable System One models (local Laya / cloud Jev). A plug-and-play intelligence layer featuring a com... | astar-algorithm eval-harness fine-tuning graphdatabase |
| tianji-ai-agent | ★ 47 |
MIT |
Java |
Production-oriented multi-agent course advisor with deterministic action governance, offline regression evaluation, SSE contracts and observable tool execution. | agent-evaluation java multi-agent rag |
| ai-agent-eval-harness | ★ 45 |
Apache-2.0 |
Python |
The open-source MultiAgentOps evaluation and verification harness for any industry business workflow. | agent-assurance agent-observability agent-testing agent-verification |
| AgentSWE | ★ 45 |
Apache-2.0 |
Python |
Can coding agents build the agent you actually want? A benchmark of 25 agent software engineering tasks across creation, editing and optimization. | agent-evaluation benchmark coding-agents llm-agents |
| A2E | ★ 44 |
MIT |
Python |
No description provided. | agent-audit agent-evaluation agent-observability agentic-ai |
| RubricLoop | ★ 43 |
MIT |
Python |
Rubric-driven evaluation for AI agents. Executable criteria, code and model judges, and inspectable trace reports. | agent-evaluation data-quality human-in-the-loop llm-evaluation |
| agentune | ★ 40 |
Apache-2.0 |
Python |
Tune your AI Agent to best meet its KPI with a cyclic process of analyze, improve and simulate | agent-evaluation agent-optimization agent-simulator ai-agents |
| DataSpace | ★ 36 |
MIT |
Python |
Benchmarking data agents for verifiable analytics over heterogeneous workspaces. | agent-evaluation ai-agents benchmark data-agents |
| money-agent | ★ 36 |
MIT |
Python |
Overnight experiment: can an AI agent make money, and can you trust what it reports? The agent produces claims; a separate verifier it cannot invoke produces... | agent-evaluation agentic-ai ai-agent ai-agents |
| self-bench | ★ 34 |
MIT |
TypeScript |
Benchmark coding agents and models on your own repository with private Harbor-formatted evals built from its merged pull requests. | agent-evaluation ai-agents benchmark coding-agents |
| done-is-a-claim | ★ 33 |
Apache-2.0 |
Python |
Your coding agent says done. Show the receipt. Evidence-first workflows, practical skills, and plugins for Claude Code and Codex. | agent-evaluation agent-reliability agent-skills agents-md |
| done-is-a-claim | ★ 33 |
Apache-2.0 |
Python |
Your coding agent says done. Show the receipt. Evidence-first workflows, practical skills, and plugins for Claude Code and Codex. | agent-evaluation agent-reliability agent-skills agents-md |
| TwinRouterBench | ★ 31 |
Apache-2.0 |
Python |
Per-step LLM routing benchmark with 970 static labels, live SWE-bench evaluation, an open data pipeline, and a public leaderboard. | agentic-ai benchmark cost-optimization leaderboard |
| bg3-agent-harness | ★ 31 |
MIT |
Python |
An AI agent plays Baldur's Gate 3 on Linux through a Lua mod and an MCP server, seeing only what a player sees. Includes a model benchmark and a playthrough ... | agent-evaluation agent-harness ai-benchmark baldurs-gate-3 |
| hermes-skilleval | ★ 29 |
MIT |
Python |
Verification-gated skill routing and self-improvement harness for Hermes-style agent skills | agent-evaluation benchmark ci llm-agents |
| self-care | ★ 28 |
MIT |
JavaScript |
Agent trace analysis and context remediation plugin for Claude Code. Detects quality issues in your AI agent traces — goal drift, hallucinations, missed acti... | agent-evaluation ai-agents claude-code observability |
| agentshield-benchmark | ★ 28 |
Apache-2.0 |
TypeScript |
Open benchmark for AI agent security tools — prompt injection, data exfiltration, tool abuse, provenance | agent-security ai-security benchmark guardrails |
| jeju | ★ 27 |
MIT |
Go |
Declarative, local-first runtime for bounded AI agents — define an agent in one manifest, run it headless, and audit every effect. | agent-evaluation agent-harness ai-agents evolution |
| Agentic_Evals | ★ 27 |
MIT |
Jupyter Notebook |
Practical evaluation techniques, benchmarks, and examples for production AI agents: RAG, tool use, planning, memory, safety, reliability, and cost. | agent-evaluation agentic-ai ai-agents benchmark |
| foundry-cicd | ★ 25 |
MIT |
Multi |
Enterprise-ready CI/CD reference for Microsoft Foundry AI agents, with parallel GitHub Actions and Azure DevOps pipelines, evaluation-driven quality gates, a... | agent-evaluation ai-agents azure azure-devops |
| eval-genius | ★ 24 |
Apache-2.0 |
Python |
Teach your agent to work with evals: WHEN you actually need an eval or benchmark, HOW to build one that holds up, and how to read what it tells you. Determin... | agent-skill ai-agents benchmark-framework benchmarks |
| langeval | ★ 24 |
Apache-2.0 |
TypeScript |
Evaluation Infrastructure for AI Agents | agent-evaluation ai-evals ai-evaluation |
| calibrate | ★ 23 |
CC-BY-SA-4.0 |
JavaScript |
Core engine behind Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations | agent-evaluation agent-evaluation-tools ai-agent-evaluation ai-agents |
| oh-my-knowledge | ★ 23 |
MIT |
TypeScript |
OMK — Evidence-backed evaluation and observability for prompts, RAG, skills, agents, and workflows. Native Codex, Claude Code, and DeepSeek Harness support. | agent-evaluation benchmark bootstrap-ci claude |
| ctf-agent-benchmark | ★ 23 |
GPL-3.0 |
Python |
Benchmarking platform for evaluating AI agents on CTF-style tasks and tool-use workflows. | ai-agent benchmark ctf evaluation |
| basically-agent-evals | ★ 22 |
MIT |
TypeScript |
Your Evals Are Lying To You: a hands-on TypeScript curriculum for LLM evals, from a five-line assertion to a live diagnostic console. By Tejas Kumar. | ai-agents eval-harness evals llm-evaluation |
| amber | ★ 22 |
Apache-2.0 |
Python |
AMBER (sealed-in-amber historical replay evaluation): real-event replay, physical sealing, pre-registered scoring — method spec; cases private, results public | agent-evaluation ai-agents benchmark benchmarking |
| swe-serve | ★ 22 |
Apache-2.0 |
Python |
SWE-Serve: an agentic benchmark of 53 production inference-engineering tasks derived from merged SGLang pull requests, run with Harbor. | benchmark evaluation llm-agents llm-inference |
| browseruse-agent-bench | ★ 21 |
Apache-2.0 |
Python |
Real-world browser-agent benchmark: 210 tasks across 107 websites, multi-agent/multi-browser evaluation, reproducible leaderboard and result submissions. | agent-evaluation ai-agents benchmark browser-agent |
| EC2-Bench | ★ 21 |
MIT |
Python |
EC2-Bench (EC²-Bench): benchmark for edge-cloud collaboration in long-horizon LLM agents — task performance, resource usage & privacy exposure. arXiv 2610.17816 | agent-benchmark agent-evaluation ai-agents benchmark |
| guard-eval-harness | ★ 20 |
MIT |
Python |
One command to benchmark AI guardrails and coding agents across safety, security, jailbreak, prompt-injection, and secure-code tasks. | ai-safety benchmark cli eval-harness |
| ATLAS | ★ 20 |
Apache-2.0 |
Python |
No description provided. | agent-evaluation agent-observability claude-code codex |
| agentrial | ★ 20 |
MIT |
Python |
Statistical evaluation framework for AI agents | agent-evaluation ai-agents ai-testing ci-cd |
| tau-trait | ★ 19 |
MIT |
Python |
TraitBasis applied to TauBench | agent-benchmark rl-envs rl-training |
| BAGEN | ★ 19 |
MIT |
Python |
BAGEN (Budget-Aware Agent): benchmark and training code for budget-aware LLM agents that estimate token, time, and money cost mid-task. | agent-benchmark budget-aware-agent budget-aware-agents budget-awareness |
| ratchet | ★ 18 |
MIT |
TypeScript |
BYOA Agent orchestration that ensures alignment end-to-end – with built-in eval system. | agent-orchestration agent-workflow ai-sdlc bdd |
| tiny_qa_benchmark_pp | ★ 18 |
Apache-2.0 |
Python |
Tiny QA Benchmark++ a micro-benchmark suite (52-item gold + on-demand multilingual synthetic packs), generator CLI, and CI-ready eval harness for ultra-fast ... | benchmark dataset evaluation hacktoberfest |
| BABY | ★ 17 |
MIT |
TypeScript |
BABY - Build Agent Benchmarks for Yourself | agent-benchmark agent-workflows |
| baba-is-harbor | ★ 15 |
Apache-2.0 |
Python |
Baba Is Harbor - AI agent benchmark on Baba is You | ai-agents baba-is-you benchmark game |
| tycoon-learning-environment | ★ 15 |
MIT |
Python |
A JAX transport-economy learning environment for route planning, cargo flow, financing, and replayable agent benchmarks. | benchmark economy jax logistics |
| nasde-toolkit | ★ 14 |
MIT |
Python |
CLI for benchmarks & evals of AI coding agents — on tasks you already understand, using your Claude / Codex / Gemini individual subscriptions or API keys. | agent-benchmark agent-evaluation ai-coding-agents ai-evaluation |
| memory_and_context_eval_harness | ★ 14 |
MIT |
Python |
Open evaluation harness for AI agent memory systems. Runs LongMemEval and LoCoMo against Synap, Mem0, Zep and Supermemory with pluggable provider adapters. S... | agent-memory ai-agents benchmark evaluation |
| SRE-bench | ★ 14 |
MIT |
Shell |
An SRE agent benchmark inspired by SWE-bench, focused on real-world Site Reliability Engineering tasks: incident response, infra changes, observability triag... | benchmark devops language-model platform-engineering |
| dojo.md | ★ 11 |
MIT |
TypeScript |
University for AI agents. 92 courses, 4400+ scenarios, any model via OpenRouter. Auto-training loops generate per-model SKILL.md documents. Works with Claude... | agent-benchmark agent-skills ai-agents ai-testing |
| joblander | ★ 11 |
Apache-2.0 |
Python |
Multi-agent engine for running a job search — guardrails, human approval gates, and an eval harness. Built while laid off; open-sourced after the offer. | agent-evaluation ai-agents fastapi human-in-the-loop |
| aobench | ★ 10 |
Apache-2.0 |
Python |
Role-aware, permission-enforced benchmark for LLM agents that operate HPC systems — SLURM, telemetry, RBAC, docs, facility. A policy violation hard-fails the... | agent-benchmark agent-evaluation ai-agents ai-safety |
| halu-core | ★ 10 |
MIT |
Python |
Scores whether an autonomous AI agent actually did the work or just hallucinated its report, by checking every claim against recorded API calls. The core eng... | agent-benchmark agent-evaluation ai-agents ai-evaluation |
| titration | ★ 10 |
Apache-2.0 |
TypeScript |
Let your coding agent fix a prompt until it actually works, and prove it did. A self-hosted MCP server: cross-vendor judges, frozen baselines and memory. | ai-agents eval-harness evals llm-as-a-judge |
| HEAR | ★ 10 |
MIT |
Python |
Run 23 speech language models on your own audio task through one interface: vLLM, transformers, and API backends behind a single JSONL contract. Ships HEAR, ... | audio audio-language-model benchmark emnlp2026 |
🌐 Web, Browser & OS Agent Environments
Sandbox environments, browser harnesses, and multi-modal testbeds for web navigation, DOM interaction, and operating system automation.
| Repository | Stars | License | Language | Description | Key Topics |
|---|---|---|---|---|---|
| browser-use | ★ 119.8k |
MIT |
Python |
Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API. | ai-agents ai-tools browser-agent browser-automation |
| UI-TARS-desktop | ★ 39.5k |
Apache-2.0 |
TypeScript |
The Open-Source Multimodal AI Agent Stack: Connecting Cutting-Edge AI Models and Agent Infra | agent-tars browser-use computer-use cowork |
| OpenCLI | ★ 30.8k |
Apache-2.0 |
JavaScript |
Make Any Website into CLI & Use your logged-in browser by AI agent. | ai-agent ai-agents ai-tools browser-automation |
| browser-harness | ★ 18.8k |
MIT |
Python |
Let Claude Code, Codex or Cursor use your real Chrome, logged in as you. One prompt to set up. Open source, MIT. | ai-agent browser-agent browser-automation browser-use |
| midscene | ★ 15.5k |
MIT |
TypeScript |
See interfaces like a human. E2E testing in natural language. | browser-use computer-use e2e-testing gpt-operator |
| nanobrowser | ★ 14.2k |
Apache-2.0 |
TypeScript |
Open-source AI agent for web automation. Use your real, logged-in browser to automate daily tasks. | ai-agents ai-tools automation browser |
| magentic-ui | ★ 10.1k |
MIT |
Python |
MagenticLite is an experimental agent that works across the browser and local file system | ai-ux autogen browser-use computer-use-agent |
| BrowserSkill | ★ 9.9k |
MIT |
TypeScript |
Let AI agents use your real, logged-in browser without interrupting your work. CLI + extension for browser automation across any shell-capable AI agent. | browser-use dsh-plugin |
| fara | ★ 6.2k |
MIT |
Python |
Fara1.5 – A family of frontier computer use agent models | browser-use computer-use computer-use-agent cua |
| mobile-use | ★ 3.4k |
Apache-2.0 |
Python |
AI agents can now use real Android and iOS apps, just like a human. | browser-use langchain langgraph langgraph-python |
| TuriX-CUA | ★ 3.2k |
MIT |
Python |
This is the official website for TuriX Computer-use-Agent | ai-agents browser-use computer-automation computer-use |
| webcmd | ★ 2.8k |
Apache-2.0 |
TypeScript |
Self-learning agent browser | ai-agents browser-automation browser-use cli |
| OpenAdapt | ★ 1.9k |
MIT |
Python |
For EHRs, payer portals, and any app your team clicks through. A person or an agent shows the task; OpenAdapt turns it into a local program that repeats the ... | agent-harness ai-agents browser-automation citrix |
| AIPex | ★ 1.3k |
MIT |
TypeScript |
AIPex: AI browser automation assistant, no migration and privacy first. Alternative to Manus Browser Operator、 Claude Chrome and Agent Browser | accessbility-testing agent-browser agent-skills browser-automation |
| browserable | ★ 1.2k |
MIT |
JavaScript |
Open source and self-hostable browser automation library for AI agents | ai-agents ai-tools browser-automation browser-use |
| ClawBench | ★ 1.2k |
Apache-2.0 |
Python |
Open-source benchmark for browser AI agents on daily tasks. | agent-evaluation agentic-ai ai-agent-benchmark ai-agents |
| celesto | ★ 1.1k |
Apache-2.0 |
Python |
Secure and persistent computer for AI agents | agent-runtime ai-sandbox browser-agent browser-use |
| AutoCua | ★ 1.1k |
MIT |
Python |
Automate with computer-use agent (ComputerUse / CUA) for macOS, Windows, Linux, Browser, iOS & Android. It sees the screen, clicks, types, browses and writes... | browser-use browser-use-agent codex computer-agent |
| unbrowse | ★ 842 |
MIT |
TypeScript |
Unbrowse - Reverse engineer any website so that your agents can access its APIs directly | agentskills browser-use claude-code-plugin clawdbot |
| desktop | ★ 710 |
MIT |
TypeScript |
Browser Use Desktop App | agent-browser browser browser-agent browser-agents |
| terminal | ★ 659 |
MIT |
Rust |
Terminal UI to get stuff done in the browser | browser-automation browser-use cdp terminal |
| web-agent-protocol | ★ 507 |
MIT |
Python |
🌐Web Agent Protocol (WAP) - Record and replay user interactions in the browser with MCP support | ai-agents ai-tools browser-automation browser-use |
| bux | ★ 460 |
MIT |
Python |
Browser Use Box: a 24/7 Claude Code agent for Playwright-style browser automation with Browser Use Cloud, Telegram, and a real browser on any box you own. | ai-agent ai-automation automation browser-agent |
| open-browser-use | ★ 410 |
MIT |
JavaScript |
🔮 Platform-neutral Browser Use for AI agents: real Chrome automation with a CLI + SDKs, no lock-in, dead simple. | ai-agent ai-agents browser-automation browser-use |
| opencow | ★ 395 |
Apache-2.0 |
TypeScript |
One task, one agent, delivered. The open-source platform for task-driven autonomous AI agents.OpenCow assigns an autonomous AI agent to every task — features... | ai-agent ai-first autonomous-agent browser-use |
| RLVR-World | ★ 283 |
MIT |
Python |
Official repository for "RLVR-World: Training World Models with Reinforcement Learning" (NeurIPS 2025), https://arxiv.org/abs/2505.13934 | grpo real2sim reinforcement-learning-with-verifiable-rewards rlvr |
| browsentic | ★ 267 |
Apache-2.0 |
TypeScript |
An agentic browser harness: an AI side panel & orb in your own logged-in Chrome, Edge, Arc, Brave or even Firefox run by the agent/subscription you already u... | agentic-browser agentic-browsing agy-cli ai-agent |
| socai | ★ 252 |
Apache-2.0 |
Rust |
Agent that actually understands social platforms. Fast. Precise. Deep. | browser-agent browser-use claude codex |
| Meterless | ★ 249 |
Apache-2.0 |
TypeScript |
Local-first context layer for AI agents. Open-source memory, reasoning, world state, intent, and swarm engines. | agent-memory agentic-ai ai-agents browser-use |
| clearcote-browser | ★ 238 |
BSD-3-Clause |
Python |
Open-source stealth Chromium with engine-level fingerprint spoofing - de-Googled, drop-in Playwright, fully buildable and verifiable from source. | ai-agents anti-detect-browser anti-fingerprinting antidetect |
| spidercreator | ★ 227 |
AGPL-3.0 |
Python |
Automated web scraping spider generation using Browser Use and LLMs. Streamline the creation of Playwright-based spiders with minimal manual coding. Ideal fo... | automation browser-use crawling low-code |
| os-ai-computer-use | ★ 182 |
Apache-2.0 |
Python |
AI controls your OS. OS AI Computer Use, OS and API agnostic. For now on OpenAI and Anthropic API. Desktop app ready. | ai-agents-framework ai-os anthropic browser-use |
| BrowserPilot | ★ 180 |
MIT |
TypeScript |
Open‑source alternative to Perplexity Comet, director.ai and firecrawl combined | ai-agent ai-agents browser browser-agent |
| clawbrowser | ★ 136 |
MIT |
C++ |
Agent-native browser with fingerprint control and proxy support. | ai-agents ai-tools browser-automation browser-use |
| jev-use | ★ 128 |
MIT |
Swift |
Say it, and your Mac does it. A computer-use harness on Jev that reads the screen through Accessibility. Fast, no vision model | accessibility ai-agents browser-use computer-use |
| bubus | ★ 126 |
MIT |
Python |
📢 Production-ready python event bus library with support for async and sync handlers, forwarding betwen busses w/ parent event tracking + loop prevention, F... | async asyncio browser-use concurrency |
| dsh-tabbit | ★ 121 |
MIT |
TypeScript |
Tabbit Browser plugins for Deepseek Harness | awesome-dsh-plugin browser-automation browser-use deepseek-harness |
| SwiftAutoGUI | ★ 94 |
MIT |
Swift |
A Swift library for macOS automation — mouse, keyboard, screenshots, image recognition, and AI-powered agents. | ai-agents automation browser-use browser-use-agent |
| octop-browser | ★ 68 |
MIT |
Python |
Agent-first lightweight browser automation — direct CDP, no Playwright, more accurate and more reliable. | agent-skills browser-use |
| mcp-browser-use | ★ 62 |
MIT |
Python |
FastAPI MCP server that exposes browser-use automation through a testable Model Context Protocol boundary. | agentic-ai browser-automation browser-use fastapi |
| browser-use-vision | ★ 57 |
MIT |
Python |
Vision-grounding plugin for browser-use agents with SoM, Florence-2, Vision–DOM alignment, adaptive visual context, and objective evaluation. | ai-agent browser-agent browser-use florence-2 |
| agentitest | ★ 48 |
Apache-2.0 |
Python |
Write browser tests in natural language, built on pytest and allure. | allure-report browser-use gemini playwright |
| jev-for-chrome | ★ 46 |
MIT |
TypeScript |
Jev for Chrome: drives the tab you are looking at with TypeSafe Jev, a sub-second decision model. Community port of browser-use/jev-ultrafast, not affiliated... | ai-agent browser-agent browser-automation browser-use |
| cappuccino | ★ 45 |
Apache-2.0 |
Python |
Cappuccino is an GUI Agent based on desktop screen. It is a Manus-like AI Agent that can be deployed locally. | browser-use computeruse deepseek omniparser |
| browser-use-rs | ★ 41 |
MIT |
Rust |
A Rust library for browser automation via Chrome DevTools Protocol with built-in AI integration through Model Context Protocol (MCP) | browser-use mcp |
| Operator-Use | ★ 41 |
MIT |
Python |
🐧Operator-Use: AI that can do stuffs on your computer | ai-agents browser-use computer-use openclaw |
🐝 Multi-Agent & Swarm Harnesses
Harnesses and testbeds designed for multi-agent collaboration, swarms, role-playing, and distributed agent execution.
| Repository | Stars | License | Language | Description | Key Topics |
|---|---|---|---|---|---|
| ruflo | ★ 80.6k |
MIT |
TypeScript |
🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive... | agentic-ai agentic-framework agentic-workflow ai-agents |
| anything-llm | ★ 67.8k |
MIT |
JavaScript |
Stop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experience | agent-computer agent-harness agent-orchestration agentic-ai |
| CowAgent | ★ 47.8k |
MIT |
Python |
Open-source personal AI assistant & Agent Harness. Plans tasks, runs tools and skills, self-evolves with memory and knowledge. Multi-agent, multi-model, mult... | ai-agent ai-agents chatgpt-on-wechat claude |
| openhuman | ★ 42.5k |
GPL-3.0 |
Rust |
The fastest, cheapest, most efficient open-source agent harness. Run more than 500 agents on a $10 VPS. | agent-orchestration ai-agents ai-assistant desktop |
| munder-difflin | ★ 9.3k |
MIT |
JavaScript |
an open-source alternative to the dots, bots and muses of the world, run an office of claude code/codex like agents on your laptop, sandboxes or anywhere, us... | agent-harness agent-orchestration ai-agents autonomous-agents |
| openrig | ★ 8.6k |
Apache-2.0 |
TypeScript |
Build your own network of agents from Claude Code, Codex and Pi: persistent teams with roles, shared context and owned work. | agent-harness agent-orchestration agent-skills ai-coding |
| Orkas | ★ 7.7k |
MIT |
JavaScript |
Orkas is an open-source (MIT), local-first AI team for your desktop: a Commander turns your goal into to-dos and coordinates 16 built-in specialist agents pl... | agent-collaboration agent-harness agent-orchestration agent-team |
| loopx | ★ 6.5k |
Apache-2.0 |
Python |
Turn long-horizon goals into verifiable results. A local-first semantic control plane and research workbench for long-running agent teams, across sessions an... | agent-control-plane agent-harness agentic-software ai-agents |
| agentops | ★ 5.9k |
MIT |
Python |
Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI ... | agentops agents-sdk anthropic autogen |
| devspace | ★ 5.4k |
MIT |
TypeScript |
Minimal Coding Agent Harness on MCP for ChatGPT, Claude, Hermes, Grok Bot, OpenClaw | agent-orchestration chatgpt claude claude-code |
| zeroshot | ★ 2.1k |
MIT |
Rust |
Open-source meta-harness for coding agents. Define your own workflows, reviewers and repair loops. zeroshot enforces the process you configure. | agent-harness agent-loops agent-orchestration agentic-workflow |
| starnet | ★ 1.7k |
MIT |
JavaScript |
A living pixel-art station where real AI agents do real work. Local-first desktop agent harness - bring your own key, watch your crew actually run. | agent-harness ai-agents byok desktop-app |
| agentconnect | ★ 1.5k |
Apache-2.0 |
TypeScript |
The open-source, multi-agent alternative to Claude Tag. @ any agent, wherever work happens, they work alongside your team, learning as they go. | agent-client-protocol agent-sandbox ai-agent claude-tag-alternative |
| openharness | ★ 1.4k |
MIT |
TypeScript |
The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and bu... | 3d-modeling agent-framework agent-harness ai-agents |
| commonly | ★ 1.4k |
Apache-2.0 |
TypeScript |
One pod for your team and the agents each of you runs: your own Claude Code, or a hosted agent in one click. Open source (Apache-2.0), self-hostable. | agent-framework agent-harness agent-protocol agentic |
| oh-my-agent | ★ 1.4k |
MIT |
TypeScript |
Agent skills and specialist teams for Claude Code, Codex, Cursor and other coding agents. Load one skill or install domain presets, project hooks and paralle... | agent-harness agent-orchestration agent-skills ai-agents |
| Sage | ★ 1.2k |
MIT |
Python |
An RSI Agent Harness for running, evaluating, and evolving agents—with tools, skills, MCP, stateful execution, and multi-agent workflows. | flutter manus mcp multi-agent |
| Chorus | ★ 1.2k |
AGPL-3.0 |
TypeScript |
The Agent Harness for AI-Human Collaboration, inspired by the AI-DLC (AI-Driven Development Lifecycle) | agent-harness ai-agents ai-dlc claude-code |
| mateclaw | ★ 1.2k |
Apache-2.0 |
Java |
🤖 MateClaw — Your second brain with Multi-Agent Orchestration, MCP Protocol, Skills & Memory, Dream, and Multi-Channel Support. Built on Spring AI Alibaba. | agent-harness agentscope-java ai-agent dingtalk-robot |
| pi-harness | ★ 1.0k |
AGPL-3.0 |
TypeScript |
The operational superset of Pi Coding Agent — everything Pi, plus observability, governance, recovery, evaluation and multi-agent orchestration. Pi Coding Ag... | agent-harness agent-observability agent-orchestration ai-agent |
| zenith | ★ 389 |
Apache-2.0 |
Python |
Zenith: a continuous-improvement harness for long-running agent tasks. Turns Claude Code, Codex, or Hermes into a multi-agent mission orchestrator via MCP/ACP. | agent-client-protocol agent-harness ai-agents claude-code |
| my-virtual-office | ★ 356 |
AGPL-3.0 |
Python |
A self-hosted 2D AI workspace for AI Agents | agent-character agent-harness agent-management agent-office |
| water | ★ 338 |
Apache-2.0 |
Python |
The production-ready agent harness framework for Python | agent-harness framework harness multi-agent |
| multi-agents | ★ 158 |
MIT |
Python |
Multi-agent teams for real work: run teams of coding agents (Claude Code, Codex, any model) in isolated git worktrees, and measure whether the team beats a s... | agent-orchestration ai-agents benchmark claude-code |
🛡️ Safety, Security & Red-Teaming Harnesses
Evaluation harnesses and testbeds for agent jailbreaking, prompt injection, security audits, and behavioral safety alignment.
| Repository | Stars | License | Language | Description | Key Topics |
|---|---|---|---|---|---|
| OpenART | ★ 268 |
AGPL-3.0 |
Python |
OpenART is an open-source framework designed to evaluate the safety and robustness of autonomous AI agents in dynamic, long-horizon, and stateful environment... | agent-environment agent-evaluation agent-redteam agent-safety |
| proofagent-harness | ★ 35 |
Apache-2.0 |
Python |
Open-source test harness for AI agents. Stress-test production agents with adversarial multi-turn scenarios in CI | adversarial-testing agent-evaluation agent-testing ai-agents |
| opencode-seatbelt | ★ 11 |
MIT |
TypeScript |
A real kernel sandbox for OpenCode on macOS 🛡️ | agent-sandbox ai-safety macos opencode |
📦 Execution Sandboxes, Runtimes & Tooling
Secure execution sandboxes, containerized runtimes, local code execution environments, and Model Context Protocol (MCP) harnesses.
| Repository | Stars | License | Language | Description | Key Topics |
|---|---|---|---|---|---|
| ECC | ★ 295.1k |
MIT |
JavaScript |
The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Curs... | ai-agents anthropic claude claude-code |
| QwenPaw | ★ 37.2k |
Apache-2.0 |
TypeScript |
Your Personal AI Assistant; easy to install, deploy on your own machine or on the cloud; supports multiple chat apps with easily extensible capabilities. | agent-harness agentscope ai-agent ai-agents |
| harness-sdk | ★ 9.0k |
Apache-2.0 |
Python |
Build an agent harness and control it end-to-end. Open-source SDK for production AI agents in Python & TypeScript - any model, any cloud. | agent-framework agentic agentic-ai ai-agents |
| smolvm | ★ 6.9k |
Apache-2.0 |
Rust |
An embeddable, portable, branchable virtual machine to safely run Agents locally. | agent-sandbox ai-agents code-execution containers |
| nono | ★ 4.8k |
Apache-2.0 |
Rust |
agent runtime security - zero trust, zero setup, zero latency agent sandboxing | agent-runtime-security agent-sandbox agent-security ai-agent-sandbox |
| mirage | ★ 3.8k |
Apache-2.0 |
TypeScript |
The World's First Virtual Terminal for AI Agents | agent-sandbox agent-tools ai-agents bash |
| onecli | ★ 3.6k |
Apache-2.0 |
TypeScript |
Open-source sandboxed agent harness for teams. Giving every employee a secured personal agent. | ai-agents cli mcp nodejs |
| agents-best-practices | ★ 2.4k |
MIT |
Multi |
Provider-neutral Agent Skill for Codex, Claude Code, and agentic harness design. | agent-skill agent-skills agentic-workflows ai-agents |
| gentle-shell | ★ 1.5k |
MIT |
TypeScript |
Gentle Shell is a Pi-native workspace for AI-assisted development, bringing parallel coding agents, live tasks, Git diffs, persistent memory, and permission ... | agent-harness agent-skills ai-agents ai-coding |
| lime | ★ 1.5k |
GPL-3.0 |
TypeScript |
Full-stack AI agent for coding, files, terminals, tools, research, content, multimodal work, and multi-agent workflows. | agent-collaboration agent-harness agentic-ai claw |
| best-of-Agent-Harnesses | ★ 1.4k |
CC-BY-SA-4.0 |
Python |
🏆 Ranked list of 187 AI agent harnesses, plus templates, playbooks, MCP, and learning resources. Rescored weekly. | agent-framework agent-harness agentic-ai ai-agent |
| thClaws | ★ 1.3k |
Apache-2.0 |
Rust |
Open-source AI agent harness in native Rust — GUI, CLI, headless, and webapp from one binary. Multi-provider, MCP, skills, plugins, agent teams. | agent-harness agent-teams ai-agent anthropic |
| flock | ★ 1.1k |
Apache-2.0 |
Rust |
A desktop multi-agent harness built with Rust, Tauri, and React, powered by langgraph-rust. | chatbot deekseek harness langchain |
| clawk | ★ 1.0k |
Apache-2.0 |
Go |
Give coding agents a disposable Linux VM, not your laptop | agent-sandbox ai-agent-sandbox ai-agents claude |
| nuwax | ★ 899 |
Apache-2.0 |
TypeScript |
Nuwax Agent OS - An enterprise-grade AI Agent Development and Operation Platform - Providing a complete solution for agent creation and distribution, knowled... | a2a agent-os agent-sandbox agentic |
| Agenvoy | ★ 893 |
AGPL-3.0 |
Go |
A local AI agent harness in Go that runs 24/7 with auto-built tools, MCP, and Ollama | agentic-ai ai-agent ai-agents ai-assistant |
| browser-use-mcp-server | ★ 849 |
MIT |
Python |
Browse the web, directly from Cursor etc. | browser browser-use cursor mcp |
| BossConsole | ★ 815 |
Apache-2.0 |
Kotlin |
Open-source, multi-platform harness for AI agents - a native, multi-threaded operator's console (JVM, not Electron) to run Claude Code, Codex, Gemini or Open... | agent-harness ai-agents browser claude-code |
| metaharness | ★ 737 |
MIT |
TypeScript |
🛠️ The meta-harness for AI agents — scaffold your own focused, branded agent harness with its own npx CLI, MCP server, memory, learning loop, and witness-sig... | agent-harness agent-harness-generator agent-scaffolding agentic-ai |
| sandbase-harness | ★ 640 |
Apache-2.0 |
TypeScript |
Local-first, self-hosted AI agent runtime and MCP bridge with sandboxed sessions, memory, credentials, audit/replay, and a local Console. | agent-harness agent-observability agent-plugin agent-runtime |
| lemma-platform | ★ 637 |
AGPL-3.0 |
Python |
Multiplayer harness that shapes around your work. Agents, apps and workflows your team shares, used from Slack, Telegram, WhatsApp, Teams, Email, Web or your... | agent-harness agentic-ai ai-agents claude-code |
| openclaw.net | ★ 532 |
MIT |
C# |
Self-hosted Enterprise and Personal AI + agent runtime in .NET (NativeAOT-friendly) | agent-harness agent-runtime agentqi ai-agent |
| mantishack | ★ 505 |
Apache-2.0 |
Rust |
Mantis Hack | agent-harness ai-agents autonomous-agents bug-bounty |
| accelerated-container-image | ★ 480 |
Apache-2.0 |
Go |
A production-ready remote/on-demand container/sandbox/vm image format (overlaybd) and snapshotter | agent-sandbox container-images containerd containerd-snapshotter |
| Sponsio | ★ 454 |
Apache-2.0 |
Python |
Deterministic safety solutions for probabilistic AI agents | agent-guardrails agent-harness agent-runtime agent-safety |
| Hypha | ★ 451 |
Apache-2.0 |
TypeScript |
Harness-oriented agent system framework for production-grade LLM agent applications | agent-framework agent-harness agent-memory agent-runtime |
| overlaybd | ★ 432 |
Apache-2.0 |
C++ |
Overlaybd: a block based remote image format for containers/vms/sandboxes. The storage backend of containerd/accelerated-container-image. | agent-sandbox blockdevice container-images iscsi-target |
| open-managed-agents | ★ 346 |
Apache-2.0 |
TypeScript |
OpenMA handles sessions, execution and recovery, so you can build and run agents with any model, any harness, any sandbox, on any cloud. An open-source, self... | agent-client-protocol agent-platform agent-sandbox ai-agents |
| Containarium | ★ 314 |
Apache-2.0 |
Go |
Open-source agent runtime — SSH-native isolation, eBPF egress policy, Kubernetes + LXC backends, GPU passthrough, MCP-native CLI | agent-native agent-runtime agent-sandbox agentic-ai |
| deltabox-lite | ★ 300 |
Apache-2.0 |
Rust |
Lightweight, agent-native sandboxing with diff, commit and abort | agent-sandbox dsh-plugin harness pi-plugin |
| devbox | ★ 270 |
MIT |
TypeScript |
Open source autonomous engineering harness in your cloud. One sandbox per task, no real secrets inside. | agent-harness agent-sandbox agent-security agentic-coding |
| agent-sandbox | ★ 228 |
Apache-2.0 |
Go |
Agent-Sandbox is an easy-to-use, enterprise-grade sandbox platform for AI Agents — letting them securely run untrusted LLM-generated code, Browser use, Compu... | agent-sandbox ai-infra ai-sandbox browser-use |
| agent-sandbox | ★ 212 |
MIT |
Python |
Secure local dev environment for collaboration with AI coding agents | agent-harness agent-sandbox coding-agents |
| sample-autonomous-cloud-coding-agents | ★ 168 |
MIT-0 |
TypeScript |
Autonomous background coding agents on AWS. Turn tasks into pull requests via isolated runtimes, with built-in orchestration, observability, and governance. | agent-sandbox agentcore autonomous-agents aws |
| mosoo | ★ 166 |
Apache-2.0 |
TypeScript |
Open-source managed runtime for OpenAI Codex, Claude Agent SDK, OpenCode, and Pi Agent. Run coding agents through one Agents API with isolated cloud sandboxe... | agent-api agent-runtime agent-sandbox agents-api |
| ephemeral-sandbox | ★ 124 |
MIT |
Rust |
Open-source agent sandbox infrastructure for parallel coding agents: isolated workspaces, MCP/CLI control, observability, and atomic publication. | agent-infrastructure agent-sandbox agent-swarm ai-agents |
| intentic | ★ 88 |
MIT |
TypeScript |
An open-source workspace for coding agents, running on your own machine. More work. Less AI waste. Same subscriptions. | agent-infrastructure agent-orchestrator agent-sandbox agentic-coding |
| sandbox-operator | ★ 81 |
AGPL-3.0 |
Go |
Aggregated Kubernetes apiserver and e2b-compatible data plane for warm-pooled agent sandboxes on microVMs. Sandboxes are served from per-node inventory with ... | agent-sandbox ai-agents cocoon crd |
| axern | ★ 65 |
Apache-2.0 |
Go |
Secure, reproducible sandboxes for AI agent evaluation, training, and data synthesis. | agent-sandbox agentic-infrastructure ai-agents cloud-native |
| hivebox | ★ 61 |
BSD-3-Clause |
Python |
Lightweight Linux sandbox for AI agents. Kernel-native isolation (namespaces, cgroups, seccomp, Landlock) with REST API, MCP bridge, and web dashboard. Singl... | agent-sandbox ai-agents ai-infrastructure cli |
| TheGrandQuiz | ★ 56 |
MIT |
Python |
Assessment-driven, local-first learning agent built on an observable Agent Runtime and eval harness—grounded ingestion, trace replay, HITL assessment, durabl... | adaptive-learning agent-harness agent-observability agent-runtime |
| greyproxy | ★ 47 |
MIT |
Go |
SOCKS5 and DNS proxy for greywall | agent-firewall agent-sandbox firewall proxy |
| mvm | ★ 46 |
Apache-2.0 |
Rust |
Manage secure microVMs — one command from zero to secure sessions, locally and remotely. | agent-sandbox agent-security ai-agent-sandbox ai-agent-security |
| boxlite-mcp | ★ 43 |
Apache-2.0 |
Python |
Powered by BoxLite - embeddable sandbox with hardware-level isolation and no daemon. The SQLite of sandbox, coming soon as open source. | ai-agents browser-use code-interpreter computer-use |
| agent-box | ★ 40 |
MIT |
Rust |
Sandboxed containers for AI coding agents with disposable Git/Jujutsu workspaces | agent-sandbox claude-code codex codex-cli |
| sprites-js | ★ 36 |
MIT |
TypeScript |
JavaScript and TypeScript SDK for Fly.io Sprites: computers for agents. Node-style APIs for creating Sprites, running remote commands, moving files, and mana... | agent-sandbox ai-agents development-environments fly-io |
| agent-sandbox | ★ 34 |
ISC |
TypeScript |
Give any AI agent its own disposable Linux machine. Firecracker microVM sandboxing with millisecond boot times, full network access, and native MCP support. | agent-sandbox ai-agents code-execution code-interpreter |
| sprites-go | ★ 32 |
MIT |
Go |
Go SDK for Fly.io Sprites: computers for agents. An exec.Cmd-style API for running commands on persistent, hardware-isolated Linux machines. | agent-sandbox ai-agents development-environments fly-io |
| agent-run | ★ 28 |
GPL-3.0 |
Rust |
Run a coding agent in a sandboxed environment | agent-harness agent-sandbox |
| console | ★ 26 |
AGPL-3.0 |
TypeScript |
Open-source, self-hosted AI gateway and control plane: OpenAI-compatible LLM routing, RAG, an MCP hub, guardrails and agent tracing for multi-tenant teams. | agent-sandbox agent-tracing agentic-ai ai-gateway |
| microsandbox-reef | ★ 25 |
MIT |
Rust |
Run OpenClaw, Hermes and your own agents in microsandbox microVMs. One reviewable TOML role sets image, allowed domains and host-bound secrets. No daemon. | agent-sandbox ai-agents cli egress-filtering |
| sprites-ex | ★ 25 |
MIT |
Elixir |
Elixir SDK for Fly.io Sprites: computers for agents. Manage Sprites and run remote commands from Elixir, with APIs that match the language's conventions. | agent-sandbox ai-agents code-execution development-environments |
| sprites-py | ★ 22 |
MIT |
Python |
Python SDK for Fly.io Sprites: computers for agents. Sprite management, remote command execution, filesystems, checkpoints, services, and network policy from... | agent-sandbox ai-agents development-environments fly-io |
| pvisor | ★ 22 |
Apache-2.0 |
Rust |
Run a coding agent in a staged workspace, review its file changes, and apply only what you keep. | agent-sandbox ai-agents claude-code cli |
| YoloFS | ★ 21 |
GPL-2.0 |
Rust |
Don't Let AI Agents YOLO Your Files: Information and Control in Agent-Native Filesystems | agent-harness agent-sandbox filesystem |
| sample-aws-self-hosted-sandbox | ★ 11 |
MIT-0 |
Python |
a production-grade AI Agent sandbox platform built on AWS, microVM architecture — with lower cost, full data sovereignty, and native Kubernetes integration. | agent-sandbox eks firecracker sandbox |
🚀 Emerging & General Agent Harnesses
Discovered open source agent harnesses and experimental frameworks with emerging topic tags.
| Repository | Stars | License | Language | Description | Key Topics |
|---|---|---|---|---|---|
| ragflow | ★ 93.3k |
Apache-2.0 |
Go |
RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engine that grew into context infrastructure. | agent-harness agent-native-database agentic-ai agentic-retrieval |
| OpenHands | ★ 92.5k |
MIT |
TypeScript |
🙌 OpenHands: AI-Driven Development | chatgpt claude-ai cli developer-tools |
| learn-claude-code | ★ 79.7k |
MIT |
Python |
Bash is all you need - A nano claude code–like 「agent harness」, built from 0 to 1 | agent-development ai-agent claude claude-code |
| aider | ★ 49.9k |
Apache-2.0 |
Python |
aider is AI pair programming in your terminal | anthropic chatgpt claude-3 cli |
| deepagents | ★ 30.9k |
MIT |
Python |
The batteries-included agent harness. | deepagents harness harness-engineering langchain |
| grok-build | ★ 27.7k |
Apache-2.0 |
Rust |
SpaceXAI's coding agent harness and TUI. Fullscreen, mouse interactive, extensible. | |
| SWE-agent | ★ 20.7k |
MIT |
Python |
SWE-agent takes a GitHub issue and tries to automatically fix it, using your LM of choice. It can also be employed for offensive cybersecurity or competitive... | agent-based-model cybersecurity developer-tools lms |
| evals | ★ 19.7k |
OTHER |
Python |
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks. | |
| DeepCode | ★ 16.8k |
MIT |
Python |
"DeepCode: Open Agentic Coding (Agent Harness & Loop Engineering & Multi-Agent Orchestration)" | agentic-coding harness-engineering llm-agent |
| OpenHarness | ★ 16.2k |
MIT |
Python |
"OpenHarness: Open Agent Harness with a Built-in Personal Agent--Ohmo!" | |
| qm | ★ 15.6k |
MIT |
TypeScript |
Multiplayer agent harness for work. | assistant harness qm |
| Trellis | ★ 15.1k |
AGPL-3.0 |
TypeScript |
The best agent harness. | agentic-coding ai-workflow claudecode codex |
| lm-evaluation-harness | ★ 14.4k |
MIT |
Python |
A framework for few-shot evaluation of language models. | evaluation-framework language-model transformer |
| hive | ★ 11.1k |
Apache-2.0 |
Python |
Multi-Agent Harness for Production AI | agent-framework agent-skills anthropic automation |
| awesome-prompts | ★ 9.2k |
GPL-3.0 |
Python |
Curated prompts, frameworks and papers for LLMs and agents, with an engineering bias. Prompt engineering, context engineering, agent harnesses, red teaming. | awesome awesome-list chatgpt context-engineering |
| ZCode | ★ 8.4k |
Apache-2.0 |
TypeScript |
Z.ai's coding agent harness. Powerful, intelligent, extensible. | |
| trueforge | ★ 6.4k |
MIT |
TypeScript |
The open-source agent harness - the runtime layer that turns an LLM into a working agent. | agentic-ai harness harness-engineering |
| learnharness | ★ 4.8k |
AGPL-3.0 |
TypeScript |
Agent harness for learning difficult things. | |
| knowhere | ★ 3.9k |
Apache-2.0 |
Python |
Knowhere extracts, parses, and outputs structured chunks ready for AI Agents and RAG. | agent-harness ai-agents claude-code codex |
| lazycodex | ★ 3.9k |
MIT |
TypeScript |
The one and only agent harness for complex codebases. Project memory, planning, execution, and verified completion inside Codex. | ai-agents claude claude-code cli |
| AgentBench | ★ 3.8k |
Apache-2.0 |
Python |
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24) | chatgpt gpt-4 llm-agent |
| SoL-Pi | ★ 3.6k |
MIT |
TypeScript |
SoL-Pi: Scaling Auto-Research Loops for Efficient Agent Harnesses | |
| llm-as-a-verifier | ★ 3.3k |
MIT |
Python |
LLM-as-a-Verifier is a general-purpose framework that provides fine-grained feedback for any agent without requiring additional training. It achieves SOTA pe... | |
| Qwen-MM-Plugins | ★ 3.2k |
Apache-2.0 |
Python |
Make any agent harness multimodal-native. | |
| unreal-agent | ★ 2.2k |
MIT |
Go |
Async-first agent harness | |
| waku-agent | ★ 2.1k |
MIT |
Python |
Waku Waku! Waku Agent is a local-first AI agent harness you actually own, including loop, memory, eval, all in code built to stay legible as it grows. | agent-framework ai-agent ai-assistant claude |
| dataclaw | ★ 2.1k |
MIT |
Python |
Agent harness to publish your agent chat history as Huggingface datasets. | |
| ante | ★ 2.0k |
Apache-2.0 |
Rust |
Ghost in your shell. Ante is a self-contained agent harness with a highly optimized core. It works like Claude Code or Codex, with none of their dependencies... | agentic-ai ai-agents artifical-life rust |
| loushang | ★ 1.9k |
Apache-2.0 |
Python |
AI-native agent harness for coding workflows by python: multi-model LLM orchestration, stateful sessions, tool governance, traceable delivery, and provider... | agent-harness agentic chatgpt claude |
| Introduction-to-Quantitative-Finance | ★ 1.9k |
MIT |
Python |
AI+金融（量化）：1.多因子股票量化框架开源教程 2.学界和业界的经典资料收录 3.AI + 金融的相关工作，包括LLM, Agent, benchmark(evaluation), etc. | ai4fin finance investment llm4fin |
| webarena | ★ 1.6k |
Apache-2.0 |
Python |
Code repo for "WebArena: A Realistic Web Environment for Building Autonomous Agents" | nlp |
| deepagentsjs | ★ 1.6k |
MIT |
TypeScript |
The batteries included agent harness. | deepagents langchain langgraph |
| exo | ★ 1.6k |
MIT |
Rust |
Exo is an agent + harness architecture that is fully recursive, able to safely edit all aspects of itself at runtime to get better at your tasks. | harness self-evolution |
| agnes-harness | ★ 1.6k |
Apache-2.0 |
TypeScript |
Agnes Harness: A pluggable agent harness for Forward Deployed Engineering (FDE). Built for trust. Made for the real world. | |
| Autonomous-Agents | ★ 1.4k |
MIT |
Multi |
Autonomous Agents (LLMs) research papers. Updated Daily. | agent-context agent-harness agentic agentic-ai |
| mini-coding-agent | ★ 1.2k |
Apache-2.0 |
Python |
Minimal and readable coding agent harness implementation in Python to explain the core components of coding agents. | llms |
| whip | ★ 1.1k |
Apache-2.0 |
Go |
A fast coding-agent harness in Go. Tool-use loop, bubbletea TUI, provider-routable models with live catalog discovery, MCP support, background subagents. One... | |
| bigcode-evaluation-harness | ★ 1.1k |
Apache-2.0 |
Python |
A framework for the evaluation of autoregressive code generation language models. | |
| Mind2Web | ★ 1.0k |
MIT |
Jupyter Notebook |
[NeurIPS'23 Spotlight] "Mind2Web: Towards a Generalist Agent for the Web" -- the first LLM-based web agent and benchmark for generalist web agents | |
| agentic-harness-engineering | ★ 959 |
MIT |
Python |
Official AHE code — Agentic Harness Engineering: observability-driven automatic evolution of coding-agent harnesses (concurrent w/ meta-harness). NexAU-AHE r... | |
| VulnHunter | ★ 798 |
Apache-2.0 |
Python |
Agentic AI security scanner that hunts exploitable vulnerabilities like an adversary, proves them with executable PoCs, and fixes them test-first. A maintain... | agent-harness agentic-ai appsec exploit-verification |
| cheetahclaws | ★ 789 |
Apache-2.0 |
Python |
CheetahClaws: A Fast and Easy-to-Use Agent Harness Infrastructure for Long-Horizon, Multi-Model, and Tool-Using AI Systems | agentic-ai claude claude-code memory |
| RSI-Harness | ★ 726 |
MIT |
TypeScript |
RSIH — versionable, shareable agent harness: Pi coding agent + Genome config layer | |
| Awesome-Code-as-Agent-Harness-Papers | ★ 722 |
MIT |
Multi |
A curated list of papers and resources based on the survey "Code as Agent Harness" | |
| baoyan-agent-benchmark | ★ 699 |
MIT |
Python |
Sprix AI at 屿智同行 × 保研岛 — an open-source benchmark for AI agents in Baoyan advising and application planning. | |
| vla-evaluation-harness | ★ 667 |
Apache-2.0 |
Python |
One framework to evaluate any VLA model on any robot simulation benchmark. | |
| waggle | ★ 646 |
Apache-2.0 |
Rust |
Attributed, resolvable artifact references for agent handoffs — a ~30-byte token instead of pasted context. MCP-native; the reference layer for the agent-har... | ai-agents harness-engineering llm-tools loop-engineering |
| inferoa | ★ 567 |
Apache-2.0 |
TypeScript |
Inference-native Tokenmaxxing Agent Harness for Loop Engineering | agent-harness agentic-ai harness-engineering inference |
| Awesome-DeepSeek-Harness-Plugins | ★ 566 |
MIT |
Multi |
Curated DeepSeek Harness (DSH) plugins, extensions, tools, skills, clients, runtimes, integrations, and verified references — English and Chinese. | agent-harness ai-agents awesome-list bilingual |
| learn-workbuddy | ★ 504 |
MIT |
Python |
从 0 复刻 WorkBuddy-style 桌面 AI 助手 Harness：24 章 Python 教程，覆盖 Agent Loop、工具调用、记忆系统、Sidecar、沙盒审计、DeepSeek/OpenAI 评测轨迹 | agent-harness ai-agent anthropic context-engineering |
| genomi | ★ 489 |
Apache-2.0 |
Python |
Local-first, open-source Claude Science alternative, before Claude Science is a thing. Turn your AI agent into personal DNA expert. | agent-harness genome-analysis whole-genome-sequencing |
| ViNote | ★ 480 |
MIT |
Python |
ViNote（视记AI）｜DeepSeek Harness SDK 驱动的开源视频知识 Agent。以工具编排、SQLite 会话恢复和流式交互，串联 YouTube / Bilibili 检索、下载、转写与笔记生成，支持视频问答、知识卡片和思维导图。FastAPI + React，支持 OpenAI 兼容接口模型。 | agent-harness ai-agent bilibili deepseek-harness |
| dryforge | ★ 435 |
Apache-2.0 |
Python |
Bounded-autonomy plugin harness for agents. Intent to implementation: ready, then go. | agent-harness agent-plugins agent-skills agentic-coding |
| ScienceBuddy | ★ 408 |
MIT |
Python |
ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents | agent-harness aiforscience autoresearch llm-agents |
| gear | ★ 391 |
MIT |
TypeScript |
Scikit-learn for Recursive Self-improvement (RSI). A modular framework for recursive harness evolution. | agent-harness ai-agents continual-learning harness-optimization |
| AgentClinic | ★ 374 |
MIT |
Python |
Agent benchmark for medical diagnosis | |
| Awesome-Agent-Harness | ★ 373 |
CC-BY-4.0 |
Multi |
Official repository for Agent Harness for Large Language Model Agents: A Survey — 168 references, 34 evidence-coded systems, and 30 harness benchmarks. | agent-framework agent-harness awesome-list harness |
| lacp | ★ 305 |
MIT |
Shell |
Local-first policy, evidence, memory, and recovery controls for Claude, Codex, Hermes, and other CLI coding agents. | agent-harness ai-agents claude-code codex |
| rsi_eval | ★ 305 |
Apache-2.0 |
Python |
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks. | ai-agents benchmarking llm-evaluation recursive-self-improvement |
| math-evaluation-harness | ★ 280 |
MIT |
Python |
A simple toolkit for benchmarking LLMs on mathematical reasoning tasks. 🧮✨ | |
| claw-bench | ★ 180 |
Apache-2.0 |
Python |
The Definitive AI Agent Benchmark | |
| shellbench | ★ 141 |
MIT |
Python |
The agent benchmark that scores the full stack — harness, config, and model — not just the LLM. Trace-based scoring, reliability metrics, configuration diagn... | |
| AgentRE-Bench | ★ 138 |
MIT |
Python |
AgentRE-Bench is an agentic benchmark that evaluates state-of-the-art models on long-horizon reverse engineering tasks, measuring their ability to analyze bi... | |
| RODE | ★ 89 |
Apache-2.0 |
Python |
Codes accompanying the paper "RODE: Learning Roles to Decompose Multi-Agent Tasks (ICLR 2021, https://arxiv.org/abs/2010.01523). RODE is a scalable role-base... | |
| lemans | ★ 86 |
MIT |
Ruby |
A Ruby harness for running agent benchmarks | evals rails ruby |
| ko-lm-evaluation-harness | ★ 81 |
MIT |
Python |
Forked repo from https://github.com/EleutherAI/lm-evaluation-harness/commit/1f66adc | |
| skill-eval-harness | ★ 79 |
MIT |
Python |
Agent Skill evaluation harness for paired variants, trace artifacts, and runner adapters | agent-skills evals skills |
| Apex | ★ 78 |
MIT |
Python |
Agents, and RL environment, for optimizing GPU kernels on AMD ROCm using LLM agents. Benchmarks LLM serving workloads end-to-end, profiles bottleneck kernels... | gpu-kernels llamas optimization-pipeline |
| Agent-Benchmarks | ★ 62 |
Apache-2.0 |
Java |
skywalking agent performance test | |
| QwenClawBench | ★ 61 |
MIT |
Python |
General Agent Benchmark for OpenClaw, made by Qwen Team, Alibaba Group. | |
| agent-eval-harness | ★ 48 |
Apache-2.0 |
Python |
No description provided. | |
| agent-eval-harness | ★ 45 |
Apache-2.0 |
Python |
No description provided. | |
| eval-harness | ★ 43 |
MIT |
Python |
My own personal evaluation harness | |
| GeoBenchX | ★ 35 |
MIT |
Jupyter Notebook |
LLM-agents benchmark set of geospatial tasks requiring multi-step tool use; and LLM-as-Judge based evaluation framework. | |
| geo-eval-harness | ★ 35 |
MIT |
Python |
Benchmark harness/toolkit for evaluating Geospatial Foundation Models (GFM). Independently developed with long-term updates and maintenance. | |
| GentPool | ★ 31 |
MIT |
Python |
Gentopia Agent Zoo and Agent Benchmark | |
| ai-agent-benchmark | ★ 29 |
MIT |
Multi |
AI coding agents comparison - 80+ agents, SWE-Bench leaderboard, pricing. Devin, Cursor, Claude Code, Copilot, and more. December 2025. | |
| NexusBench | ★ 28 |
Apache-2.0 |
Python |
Nexusflow function call, tool use, and agent benchmarks. | |
| RAG-evaluation-harnesses | ★ 27 |
MIT |
Python |
An evaluation suite for Retrieval-Augmented Generation (RAG). | evaluation lm-evaluation rag retrieval-augmented-generation |
| hermes-best-models | ★ 27 |
MIT |
Python |
LLM evals for Hermes agents. Benchmark to find the best models for Hermes | |
| write-like-me | ★ 22 |
MIT |
Python |
Measures how you write and builds a voice profile, then enforces it with a deterministic checker: em dashes, filler, hedges, and AI tells are caught by code,... | claude-code claude-skills nlp style-checker |
| agent-eval-harness | ★ 20 |
MIT |
Python |
An open-source evaluation framework specifically for agentic systems — not just LLM outputs, but full agent behavior. | |
| FORTE | ★ 20 |
MIT |
Python |
FORTE (Full-cycle Office Real-world Task Evaluation) is a general agent benchmark for evaluating AI agents on daily office productivity across 15 corporate p... | |
| ACESEvals | ★ 20 |
MIT |
Python |
Security Agent Benchmarking and Evaluation Research | agentic-ai benchmark-framework cyber-security evaluation-framework |
| tamil-lm | ★ 20 |
Apache-2.0 |
Python |
Tamil continued pretraining and instruction tuning of Qwen3.5-2B-Base: serving stack, eval harness, training recipe and knowledge base build scripts (Timegra... | |
| agent-runtime-patterns | ★ 19 |
MIT |
Python |
Runtime architecture patterns for production AI/ LLM Agents. Runnable LangGraph + Google ADK code, a composed 90-day contract-renewal example, IBM Telco data... | |
| amazon-log-agent-benchmark-tool | ★ 19 |
MIT-0 |
Go |
Simple log generation and process resource monitor used to benchmark log agents | |
| judgejudy | ★ 18 |
MIT |
Go |
Eval Harness | |
| Neural_Forge | ★ 18 |
MIT |
Python |
NeuralForge is an end-to-end LLM fine-tuning & serving platform. It leverages LoRA (PyTorch/Hugging Face) for domain adaptation, cutting error rates vs. base... | ci-cd containerization docker finetuning |
| cloud-attack-range | ★ 18 |
Apache-2.0 |
Python |
Self-hosted vulnerable cloud lab for authorized security training and AI-agent benchmarking | |
| Multi-Agent-Benchmark-Tool | ★ 17 |
MIT |
Python |
Interested in running a conspiracy of agents (technical term) on your Local AI Infra? Who isn't! | ai-agents hermes-agent llm-agent llm-inference |
| query-agent-benchmarking | ★ 17 |
BSD-3-Clause |
Jupyter Notebook |
Tools for various benchmarking scenarios of Weaviate's Query Agent. | |
| K-BrowseComp | ★ 16 |
MIT |
Python |
Repository for "K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts" | |
| PHMForge-A-Scenario-Driven-Agentic-Benchmark-for-Industrial-Asset-Lifecycle-Maintenance | ★ 16 |
MIT |
Jupyter Notebook |
Agentic approach to predicting RUL, cost while adhering to safety standards. | |
| LongVidSearch | ★ 15 |
MIT |
Python |
An Agentic Benchmark for Multi-hop Evidence Retrieval Planning in Long Videos | |
| eval-harness | ★ 14 |
MIT |
Shell |
Typed evaluation for OpenCode skills and AI agents: capability, regression and product checks; provider-neutral evidence grading, provenance, and CI reports. | agent-testing ai-agents anthropic bash |
| Awesome-General-Agents-Benchmark | ★ 14 |
MIT |
Multi |
Awesome list of general agent benchmarks | |
| fusion-engine | ★ 12 |
MIT |
Python |
Self-hosted Fusion API and OpenRouter Fusion alternative for building reliable multi-model LLM ensembles — fan out one prompt to N models, then synthesize on... | ai-agents benchmarking fastapi fusion-api |
| encoder_eval | ★ 11 |
Apache-2.0 |
Python |
Reproducible encoder downstream fine-tune eval harness | |
| skillroll | ★ 11 |
MIT |
Python |
The simple eval harness for your skills | |
| q-evaluation-harness | ★ 10 |
MIT |
Python |
An open-source framework by KX for evaluating Large Language Models on Q/kdb+ code generation tasks. | |
| lm-evaluation-harness | ★ 10 |
MIT |
Python |
A framework for few-shot evaluation of autoregressive language models. | |
| lm-eval-harness | ★ 10 |
MIT |
Python |
No description provided. | |
| lm-evaluation-harness | ★ 10 |
MIT |
Python |
A framework for few-shot evaluation of autoregressive language models. | |
| agent-anvil | ★ 10 |
MIT |
Python |
Agent Anvil is a CI-first eval harness for tool-using AI agents. It runs scenario suites, captures traces, grades agent behavior, clusters failures, and sugg... |
Topic Cloud
Discovered topics across indexed open source harnesses:
#ai-agents (142)
#llm (101)
#agent-evaluation (71)
#agent-harness (70)
#mcp (65)
#claude-code (62)
#python (55)
#codex (49)
#browser-use (48)
#llm-evaluation (46)
#agent (45)
#agentic-ai (42)
#agent-sandbox (41)
#benchmark (41)
#ai (40)
#ai-agent (37)
#developer-tools (34)
#coding-agent (33)
#swe-bench (31)
#multi-agent (30)
#sandbox (29)
#cli (28)
#claude (27)
#coding-agents (27)
#llm-agents (27)
#harness (26)
#agents (24)
#harness-engineering (23)
#typescript (23)
#browser-automation (22)
#agent-skills (21)
#computer-use (20)
#openai (20)
#self-hosted (20)
#evaluation (18)
#model-context-protocol (18)
#agent-orchestration (16)
#autonomous-agents (16)
#rag (16)
#evals (15)
#opencode (15)
#agent-benchmark (14)
#anthropic (14)
#browser-agent (14)
#local-first (14)
#openclaw (14)
#rust (14)
#skills (14)
#deepseek (13)
#llmops (13)
#multi-agent-systems (13)
#open-source (13)
#agent-framework (12)
#playwright (12)
#evaluation-framework (11)
#mcp-server (11)
#agent-observability (10)
#agent-runtime (10)
#ai-coding (10)
#ai-safety (10)
#jev (10)
#llm-agent (10)
#pi-coding-agent (10)
#ai-evaluation (9)
#benchmarking (9)
#context-engineering (9)
#docker (9)
#dsh-plugin (9)
#kubernetes (9)
#loop-engineering (9)
#ai-tools (8)
#chatgpt (8)
#claude-skills (8)
#deepseek-harness (8)
#desktop-app (8)
#eval-harness (8)
#guardrails (8)
#human-in-the-loop (8)
#langchain (8)
#langgraph (8)
#prompt-engineering (8)
#prompt-injection (8)
#security (8)
#terminal (8)
#testing (8)
#tool-use (8)
#web-automation (8)
#agent-security (7)
#agentic (7)
#agentic-coding (7)
#ai-security (7)
#awesome-list (7)
#code-review (7)
#computer-use-agent (7)
#cursor (7)
#fastapi (7)
#gemini (7)
#observability (7)
#reinforcement-learning (7)
#tool-calling (7)
#ai-assistant (6)
#automation (6)
#cdp (6)
#chrome-extension (6)
#codex-cli (6)
#evaluation-harness (6)
#hallucination-detection (6)
#large-language-models (6)
#llm-observability (6)
#llm-security (6)
#llm-tools (6)
#machine-learning (6)
#macos (6)
#microvm (6)
#opentelemetry (6)
#pi (6)
#recursive-self-improvement (6)
#sdk (6)
#tui (6)
#workflow (6)
#ai-sandbox (5)
#browser (5)
#claude-code-plugin (5)
#code-execution (5)
#long-horizon-agents (5)
#openai-codex (5)
#orchestration (5)
#remote-execution (5)
#synthetic-data (5)
#terminal-bench (5)
#tutorial (5)
#verification (5)
#vllm (5)
#web-agent (5)
#agent-memory (4)
#agent-testing (4)
#agentic-workflow (4)
#agentops (4)
#ai-benchmark (4)
#ai-evals (4)
#ai-testing (4)
#benchmarks (4)
#ci-cd (4)
#code-sandbox (4)
#cua (4)
#desktop-automation (4)
#development-environments (4)
#e2b (4)
#electron (4)
#fly-io (4)
#gemini-cli (4)
#genai (4)
#go (4)
#hermes-agent (4)
#leaderboard (4)
#llm-as-a-judge (4)
#llm-eval (4)
#llm-testing (4)
#llms (4)
#memory (4)
#nodejs (4)
#openai-compatible (4)
#openrouter (4)
#pi-agent (4)
#pytest (4)
#rag-evaluation (4)
#react (4)
#red-teaming (4)
#rsi (4)
#self-evolving-agents (4)
#software-engineering (4)
#software-factory (4)
#sprites (4)
#sqlite (4)
#survey (4)
#tauri (4)
#tracing (4)
#typesafe-ai (4)
#a2a (3)
#acp (3)
#agent-client-protocol (3)
#agent-evals (3)
#agent-skill (3)
#ai-agent-sandbox (3)
#ai-agents-framework (3)
#ai-code-review (3)
#ai-infrastructure (3)
#ai-observability (3)
#antigravity (3)
#autogen (3)
#awesome (3)
#benchmark-framework (3)
#browser-use-agent (3)
#chatbot (3)
#cloud-browser (3)
#computer-use-agents (3)
#crewai (3)
#datasets (3)
#dsh (3)
#eval (3)
#fine-tuning (3)
#generative-ai (3)
#git-worktree (3)
#github-copilot (3)
#golang (3)
#gui (3)
#gui-agent (3)
#hacktoberfest (3)
#harbor (3)
#hermes (3)
#javascript (3)
#language-model (3)
#llm-benchmark (3)
#llm-inference (3)
#local-ai (3)
#mlops (3)
#multi-tenant (3)
#multimodal (3)
#muse (3)
#ollama (3)
#paper-list (3)
#parallel-agents (3)
#personal-assistant (3)
#qa (3)
#qwen (3)
#regression-testing (3)
#responsible-ai (3)
#retrieval-augmented-generation (3)
#sandboxing (3)
#sdlc (3)
#spec-driven-development (3)
#spring-ai (3)
#vector-database (3)
#virtualization (3)
#vm (3)
#web-agents (3)
#web-scraping (3)
#workflow-automation (3)
#agent-api (2)
#agent-benchmarks (2)
#agent-browser (2)
#agent-collaboration (2)
#agent-computer (2)
#agent-infrastructure (2)
#agent-loop (2)
#agent-optimization (2)
#agent-plugins (2)
#agent-reliability (2)
#agent-safety (2)
#agent-workflow (2)
#agent-workflows (2)
#agentic-framework (2)
#agentic-rl (2)
#agents-md (2)
#ai-agent-security (2)
#ai-automation (2)
#ai-coding-agent (2)
#ai-engineering (2)
#ai-gateway (2)
#ai-governance (2)
#ai-research (2)
#ai-verification (2)
#ai-workflow (2)
#aider (2)
#artificial-intelligence (2)
#astar-algorithm (2)
#background-agents (2)
#bash (2)
#browser-use-box (2)
#browser-use-cloud (2)
#byok (2)
#chrome-agent (2)
#chrome-devtools-protocol (2)
#claude-agent-sdk (2)
#claude-code-skills (2)
#claude-md (2)
#clickhouse (2)
#cloudflare (2)
#code-agent (2)
#code-generation (2)
#code-interpreter (2)
#code-quality (2)
#codex-plugin (2)
#coding-assistant (2)
#computeruse (2)
#container-images (2)
#containers (2)
#context-management (2)
#cost-estimation (2)
#cybersecurity (2)
#dataset (2)
#deep-learning (2)
#deep-research (2)
#deepagents (2)
#desktop-agent (2)
#devops (2)
#devtools (2)
#durable-objects (2)
#e2b-compatible (2)
#eu-ai-act (2)
#evaluation-metrics (2)
#evaluations (2)
#event-sourcing (2)
#firecracker (2)
#generalist-agent (2)
#github-actions (2)
#gpt-4 (2)
#gpu-kernels (2)
#graphdatabase (2)
#graprag (2)
#grpo (2)
#gui-operator (2)
#health (2)
#inference (2)
#jailbreak (2)
#java (2)
#jev-model (2)
#kimi (2)
#knowledge-base (2)
#kotlin (2)
#laya (2)
#llamas (2)
#llm-as-judge (2)
#llm-evals (2)
#llm-orchestration (2)
#llm-routing (2)
#llm-safety (2)
#llm-tracing (2)
#llm-wiki (2)
#lora (2)
#manus (2)
#mcp-security (2)
#meta-harness (2)
#model-evaluation (2)
#monitoring (2)
#multi-agent-system (2)
#neo4j (2)
#nix (2)
#nlp (2)
#open-standard (2)
#openai-agents (2)
#openai-agents-sdk (2)
#openclaw-skills (2)
#opencode-plugin (2)
#optimization (2)
#persistent-browser (2)
#personal-ai (2)
#personal-ai-assistant (2)
#planning (2)
#plugin (2)
#plugins (2)
#policy-engine (2)
#post-training (2)
#privacy (2)
#prompt-optimization (2)
#provenance (2)
#puppeteer (2)
#python-sdk (2)
#pytorch (2)
#rbac (2)
#react-agent (2)
#reasoning-models (2)
#record-replay (2)
#red-team (2)
#remote-image (2)
#replay (2)
#reproducibility (2)
#research (2)
#rlvr (2)
#rpa (2)
#scraping (2)
#self-improving (2)
#simulation (2)
#skill-evaluation (2)
#spring-boot (2)
#ssh (2)
#stealth-browser (2)
#strands-agents (2)
#super-agent (2)
#supply-chain-security (2)
#swe-agent (2)
#swift (2)
#telegram-agent (2)
#telegram-bot (2)
#tmux (2)
#trustworthy-ai (2)
#typesafe (2)
#vibe-coding (2)
#virtual-machine (2)
#vision (2)
#vps-agent (2)
#vulnerability-scanner (2)
#world-model (2)
How It Works
This repository is maintained autonomously by a GitHub Actions workflow:
- Discovery: Queries the GitHub Search API on a scheduled cron and on demand using targeted topic queries (
topic:agent-harness,topic:eval-harness,topic:swe-bench, etc.). - Validation: Filters candidates strictly for open-source licenses (OSI-approved SPDX keys), non-archived status, and minimum star thresholds.
- Topic Grouping: Clusters repositories into functional harness categories based on their defined GitHub topics and indexes all raw tags.
- Autonomous Publishing: Generates
README.mdand machine-readabledata/harnesses.json, committing updates automatically if changes are detected.
Contributing
Want to add your AI agent harness or testbed to this list?
Method 1: Tag Your GitHub Repository (Automatic)
Add relevant topics to your GitHub repository (under the repo's "About" section):
- Core topics:
agent-harness,eval-harness,swe-bench,browser-use,agent-benchmark,agent-sandbox. - Ensure your repository has an OSI-approved open source license (
MIT,Apache-2.0,BSD,GPL, etc.). The automated workflow will pick up your repository on its next run!
Method 2: Submit a Pull Request
Add your repository to seed_repos in config/settings.yaml:
seed_repos:
- "your-org/your-agent-harness"
Or propose a new topic category in config/topics.yaml.
License Requirements
All indexed repositories must be licensed under an OSI-approved open source license.