Proposal: report-only CI for app/api/** chain outputs (PromptProof) #39

New Issue

2026-02-16T00:15:10-05:00

yindo commented

2026-02-16 00:15:10 -05:00

Originally created by @geminimir on GitHub (Aug 14, 2025).

I can add a tiny report-only CI step that replays one recorded chain output under app/api/** (no live model calls) and uploads a single HTML report on PRs.

Scope / paths
Triggered only on app/api/** and the three added files below (keeps CI focused).

Files to add

.github/workflows/promptproof.yml

name: PromptProof
on:
  pull_request:
    paths:
      - "app/api/**"
      - ".github/workflows/promptproof.yml"
      - "promptproof.yaml"
      - "fixtures/promptproof/**"
jobs:
  proof:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: geminimir/promptproof-action@v0
        with:
          config: promptproof.yaml
          runs: 3
          seed: 1337
          max-run-cost: 0.50
          report-artifact: promptproof-report
          mode: report-only

promptproof.yaml

mode: fail
format: html
fixtures:
  - path: fixtures/promptproof/template_chat.json
checks:
  - id: chain_output_schema
    type: schema
    json_schema:
      type: object
      properties:
        output:
          type: object
          properties:
            text: { type: string, minLength: 1 }
            tokens: { type: number, minimum: 0 }
          required: [text]
      required: [output]
budgets:
  max_run_cost: 0.50
stability:
  runs: 3
  seed: 1337

fixtures/promptproof/template_chat.json

{
  "record_id": "lc-template-chat",
  "input": { "query": "Brief greeting" },
  "output": { "text": "Hello there — sample deterministic text.", "tokens": 12 }
}

What maintainers get

A single HTML report artifact per PR (schema/regex/cost summary).
Zero live calls; easy to delete if unwanted.

References
Sample report: https://geminimir.github.io/promptproof-action/reports/before.html

If this sounds okay, I’ll open a 3-file PR and can tweak the checks/paths to your preference.

Marketplace: https://github.com/marketplace/actions/promptproof-eval
Demo project: https://github.com/geminimir/promptproof-demo-project

Originally created by @geminimir on GitHub (Aug 14, 2025). I can add a tiny **report-only** CI step that replays one recorded chain output under **`app/api/**`** (no live model calls) and uploads a single **HTML report** on PRs. **Scope / paths** Triggered only on `app/api/**` and the three added files below (keeps CI focused). **Files to add** 1. `.github/workflows/promptproof.yml` ```yaml name: PromptProof on: pull_request: paths: - "app/api/**" - ".github/workflows/promptproof.yml" - "promptproof.yaml" - "fixtures/promptproof/**" jobs: proof: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: geminimir/promptproof-action@v0 with: config: promptproof.yaml runs: 3 seed: 1337 max-run-cost: 0.50 report-artifact: promptproof-report mode: report-only ``` 2. `promptproof.yaml` ```yaml mode: fail format: html fixtures: - path: fixtures/promptproof/template_chat.json checks: - id: chain_output_schema type: schema json_schema: type: object properties: output: type: object properties: text: { type: string, minLength: 1 } tokens: { type: number, minimum: 0 } required: [text] required: [output] budgets: max_run_cost: 0.50 stability: runs: 3 seed: 1337 ``` 3. `fixtures/promptproof/template_chat.json` ```json { "record_id": "lc-template-chat", "input": { "query": "Brief greeting" }, "output": { "text": "Hello there — sample deterministic text.", "tokens": 12 } } ``` **What maintainers get** * A single **HTML report** artifact per PR (schema/regex/cost summary). * Zero live calls; easy to delete if unwanted. **References** Sample report: https://geminimir.github.io/promptproof-action/reports/before.html If this sounds okay, I’ll open a **3-file PR** and can tweak the checks/paths to your preference. Marketplace: https://github.com/marketplace/actions/promptproof-eval Demo project: https://github.com/geminimir/promptproof-demo-project

yindo referenced this issue

2026-02-16 00:15:14 -05:00

[PR #39] [MERGED] Update tool responses readme title #53

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langchain-ai/langchain-nextjs-template#39