anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-23 02:36:57 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

[PR #10699] feat: add SWE-bench evaluation tool #13526

New Issue
Open
opened 2026-02-16 18:18:22 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 18:18:22 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/anomalyco/opencode/pull/10699

State: open
Merged: No

Fixes #10690

Summary

  • Add SWE-bench dataset integration (Lite/Verified/Full)
  • Automated patch generation for bug fixing tasks
  • Output predictions.jsonl in SWE-bench standard format
  • Support concurrent processing with configurable parallelism
  • Checkpoint support for resumable runs
  • Interactive and quick start modes

What does this PR do?

This PR adds a SWE-bench evaluation tool that allows users to evaluate opencode's performance on real-world software engineering tasks.

Features:

  • Load SWE-bench datasets directly from HuggingFace
  • Clone target repositories and checkout to specific commits
  • Use opencode to generate patches for bug fixes
  • Output results in SWE-bench standard format (predictions.jsonl)
  • Support resumable runs with checkpoints

Usage:

# Interactive mode
bun swebench/run.ts

# Quick mode (smoke test with 2 instances)
bun swebench/run.ts -q

# Full control
bun swebench/index.ts --model anthropic/claude-sonnet-4-20250514 --limit 10

How did you verify your code works?

  • Ran smoke test with 2 instances
  • Verified output format matches SWE-bench standard
  • Tested checkpoint resume functionality
**Original Pull Request:** https://github.com/anomalyco/opencode/pull/10699 **State:** open **Merged:** No --- Fixes #10690 ## Summary - Add SWE-bench dataset integration (Lite/Verified/Full) - Automated patch generation for bug fixing tasks - Output predictions.jsonl in SWE-bench standard format - Support concurrent processing with configurable parallelism - Checkpoint support for resumable runs - Interactive and quick start modes ### What does this PR do? This PR adds a SWE-bench evaluation tool that allows users to evaluate opencode's performance on real-world software engineering tasks. **Features:** - Load SWE-bench datasets directly from HuggingFace - Clone target repositories and checkout to specific commits - Use opencode to generate patches for bug fixes - Output results in SWE-bench standard format (`predictions.jsonl`) - Support resumable runs with checkpoints **Usage:** ```bash # Interactive mode bun swebench/run.ts # Quick mode (smoke test with 2 instances) bun swebench/run.ts -q # Full control bun swebench/index.ts --model anthropic/claude-sonnet-4-20250514 --limit 10 ``` ### How did you verify your code works? - Ran smoke test with 2 instances - Verified output format matches SWE-bench standard - Tested checkpoint resume functionality
yindo added the pull-request label 2026-02-16 18:18:22 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#13526
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?