[PR #564] [MERGED] feat: add translation A/B testing framework #578

New Issue

2026-02-16T06:15:57-05:00

yindo commented

2026-02-16 06:15:57 -05:00

📋 Pull Request Information

Original PR: https://github.com/langgenius/dify-docs/pull/564
Author: @alterxyz
Created: 11/28/2025
Status: ✅ Merged
Merged: 11/28/2025
Merged by: @crazywoola

Base: main ← Head: tools/translation-ab-testing

📝 Commits (1)

5fc0482 feat: add translation A/B testing framework

📊 Changes

8 files changed (+677 additions, -0 deletions)

View changed files

📝 CLAUDE.md (+18 -0)
➕ tools/translate-test-dify/.gitignore (+25 -0)
➕ tools/translate-test-dify/README.md (+92 -0)
➕ tools/translate-test-dify/compare.py (+169 -0)
➕ tools/translate-test-dify/example-model-comparison.md (+27 -0)
➕ tools/translate-test-dify/requirements.txt (+10 -0)
➕ tools/translate-test-dify/run_test.py (+284 -0)
➕ tools/translate-test-dify/setup.sh (+52 -0)

📄 Description

Summary

Add tools/translate-test-dify/ - A/B testing framework for comparing translation quality between different models or prompt variations
Update CLAUDE.md with usage instructions

Features

run_test.py: Test runner using Dify API with streaming mode
compare.py: Generate similarity reports and side-by-side comparisons
Support for inline test content or referencing existing docs
Example spec included (example-model-comparison.md)

Use Case

Evaluate translation improvements before deploying changes:

Compare Sonnet vs Opus output quality
Test prompt modifications
Verify punctuation handling

Usage

cd tools/translate-test-dify
./setup.sh
source venv/bin/activate
python run_test.py <spec.md>
python compare.py results/<folder>/

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/langgenius/dify-docs/pull/564 **Author:** [@alterxyz](https://github.com/alterxyz) **Created:** 11/28/2025 **Status:** ✅ Merged **Merged:** 11/28/2025 **Merged by:** [@crazywoola](https://github.com/crazywoola) **Base:** `main` ← **Head:** `tools/translation-ab-testing` --- ### 📝 Commits (1) - [`5fc0482`](https://github.com/langgenius/dify-docs/commit/5fc04821e8686f680c9ba3e28b3456dfb910030e) feat: add translation A/B testing framework ### 📊 Changes **8 files changed** (+677 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `CLAUDE.md` (+18 -0) ➕ `tools/translate-test-dify/.gitignore` (+25 -0) ➕ `tools/translate-test-dify/README.md` (+92 -0) ➕ `tools/translate-test-dify/compare.py` (+169 -0) ➕ `tools/translate-test-dify/example-model-comparison.md` (+27 -0) ➕ `tools/translate-test-dify/requirements.txt` (+10 -0) ➕ `tools/translate-test-dify/run_test.py` (+284 -0) ➕ `tools/translate-test-dify/setup.sh` (+52 -0) </details> ### 📄 Description ## Summary - Add `tools/translate-test-dify/` - A/B testing framework for comparing translation quality between different models or prompt variations - Update `CLAUDE.md` with usage instructions ## Features - **run_test.py**: Test runner using Dify API with streaming mode - **compare.py**: Generate similarity reports and side-by-side comparisons - Support for inline test content or referencing existing docs - Example spec included (`example-model-comparison.md`) ## Use Case Evaluate translation improvements before deploying changes: - Compare Sonnet vs Opus output quality - Test prompt modifications - Verify punctuation handling ## Usage ```bash cd tools/translate-test-dify ./setup.sh source venv/bin/activate python run_test.py <spec.md> python compare.py results/<folder>/ ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

yindo added the pull-request label 2026-02-16 06:15:57 -05:00

yindo closed this issue

2026-02-16 06:15:57 -05:00

yindo referenced this issue

2026-02-16 06:16:02 -05:00

[PR #578] [MERGED] add code owner #587

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langgenius/dify-docs#578