anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-23 02:36:57 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

Feat: Framework/Module for proper A/B testing of prompts within opencode #1237

New Issue
Open
opened 2026-02-16 17:30:07 -05:00 by yindo · 3 comments
yindo commented 2026-02-16 17:30:07 -05:00
Owner
Copy Link
Copy Source

Originally created by @spoons-and-mirrors on GitHub (Aug 11, 2025).

Originally assigned to: @thdxr on GitHub.

The title says it all, I have nothing on this just yet, but it would sure be a great addition.
Feel free to share your view on how this should be implemented, I'm stumped at the moment.

Originally created by @spoons-and-mirrors on GitHub (Aug 11, 2025). Originally assigned to: @thdxr on GitHub. The title says it all, I have nothing on this just yet, but it would sure be a great addition. Feel free to share your view on how this should be implemented, I'm stumped at the moment.
yindo commented 2026-02-16 17:30:08 -05:00
Author
Owner
Copy Link
Copy Source

@thdxr commented on GitHub (Aug 11, 2025):

not sure what this means!

@thdxr commented on GitHub (Aug 11, 2025): not sure what this means!
yindo commented 2026-02-16 17:30:08 -05:00
Author
Owner
Copy Link
Copy Source

@spoons-and-mirrors commented on GitHub (Aug 11, 2025):

tbh, I don't either...
I think that in order to benchmark prompt changes made to parts of OC or to agents made by users, having a proper way to A/B test those against a set of pre-made tasks and context or something a user could 'save' and re-use for testing purposes would be incredibly valuable.

A testing framework of sort, so everyone's prompts, including OC internal prompting can be fine tuned using an automated, scientific methodology. Less winging it, better handling of run to run variance and such...

That being said, idk what shape this 'thing' should be

@spoons-and-mirrors commented on GitHub (Aug 11, 2025): tbh, I don't either... I think that in order to benchmark prompt changes made to parts of OC or to agents made by users, having a proper way to A/B test those against a set of pre-made tasks and context or something a user could 'save' and re-use for testing purposes would be incredibly valuable. A testing framework of sort, so everyone's prompts, including OC internal prompting can be fine tuned using an automated, scientific methodology. Less winging it, better handling of run to run variance and such... That being said, idk what shape this 'thing' should be
yindo commented 2026-02-16 17:30:08 -05:00
Author
Owner
Copy Link
Copy Source

@LegendEvent commented on GitHub (Nov 4, 2025):

I think this is a great idea. A benchmark to actually visualize how the prompts affect speed/code processing. It would also be good because it would allow you to benchmark different agent orchestrations.

I was thinking of predefined prompts that ALWAYS force the same output, and then you could output a score based on time, etc. Internally, you could address this as /benchmark.

@LegendEvent commented on GitHub (Nov 4, 2025): I think this is a great idea. A benchmark to actually visualize how the prompts affect speed/code processing. It would also be good because it would allow you to benchmark different agent orchestrations. I was thinking of predefined prompts that ALWAYS force the same output, and then you could output a score based on time, etc. Internally, you could address this as /benchmark.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#1237
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?