anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 10:15:31 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

[FEATURE]:Add SWE-bench Evaluation Support #7641

New Issue
Open
opened 2026-02-16 18:07:48 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 18:07:48 -05:00
Owner
Copy Link
Copy Source

Originally created by @JeffC0628 on GitHub (Jan 26, 2026).

Originally assigned to: @thdxr on GitHub.

Feature hasn't been suggested before.

  • I have verified this feature I'm about to request hasn't been suggested before.

Describe the enhancement you want to request

Description

SWE-bench is a critical benchmark for evaluating code generation models on real-world software engineering tasks, including bug fixing, feature implementation, and code refactoring. Currently, opencode does not support SWE-bench evaluation, which limits our ability to:

  • Directly compare opencode's performance with other state-of-the-art code generation models (e.g., Cluade Code, Cline).
  • Quantify how well opencode handles practical, complex programming scenarios that developers face daily.

Request

I would like to request adding SWE-bench evaluation support to opencode, which should include:

  1. Integration with the official SWE-bench dataset.
  2. Automated execution of SWE-bench tasks (e.g., applying model-generated patches to codebases and running test suites).

Benefits

  • Improved credibility: SWE-bench is widely recognized in the research and developer community, so adding this evaluation will make opencode more comparable to industry-standard models.
  • Better user insights: Developers and researchers will be able to see how opencode performs on real engineering tasks, not just synthetic benchmarks.
  • Community appeal: This feature will attract users who prioritize SWE-bench results when choosing a code generation tool.
Originally created by @JeffC0628 on GitHub (Jan 26, 2026). Originally assigned to: @thdxr on GitHub. ### Feature hasn't been suggested before. - [x] I have verified this feature I'm about to request hasn't been suggested before. ### Describe the enhancement you want to request #### Description SWE-bench is a critical benchmark for evaluating code generation models on real-world software engineering tasks, including bug fixing, feature implementation, and code refactoring. Currently, opencode does not support SWE-bench evaluation, which limits our ability to: - Directly compare opencode's performance with other state-of-the-art code generation models (e.g., Cluade Code, Cline). - Quantify how well opencode handles practical, complex programming scenarios that developers face daily. #### Request I would like to request adding SWE-bench evaluation support to opencode, which should include: 1. Integration with the official SWE-bench dataset. 2. Automated execution of SWE-bench tasks (e.g., applying model-generated patches to codebases and running test suites). #### Benefits - **Improved credibility**: SWE-bench is widely recognized in the research and developer community, so adding this evaluation will make opencode more comparable to industry-standard models. - **Better user insights**: Developers and researchers will be able to see how opencode performs on real engineering tasks, not just synthetic benchmarks. - **Community appeal**: This feature will attract users who prioritize SWE-bench results when choosing a code generation tool.
yindo added the discussion label 2026-02-16 18:07:48 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label discussion
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#7641
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?