mirror of
https://github.com/Niko1221/Strata.git
synced 2026-11-07 06:15:58 -10:00
hanchengme
243fdb213c
In the single-request speculative paths the accepted rows of a verify window were committed first and the output loop cut to max_new or EOS afterwards, so a request's last window could park a snapshot with tokens the client never received. conversation_prefix() needs the whole snapshot to be a prefix of the next prompt, so the snapshot was refused and the older checkpoint was used (about 6% of a 12K-token prompt read again). serve_stop_bound() caps the accepted rows at the rows the request can still emit and ends at the first EOS, before pl_commit_async. Merged prompt windows keep committing every row. HIP only (STRATA_USE_HIP), where it was measured. Patch and policy test (2,736 cases against the old loop) by hanchengme, from the issue's attachment. Two changes on our side: it is on by default, STRATA_SERVE_STOP_BOUND=0 turns it off; the test is registered in CMake. Measured on 4 x R9700, Qwen3.8-Flash-Next IQ3_S, 2 replicas x 2 cards, --batch 8 --batch-groups 2, four 11-15K-token conversations with 768-token answers, conversations opened one at a time: 4 of 16 continuations fell back to the older checkpoint without it, 0 of 16 with it (reuse 98.36% against 99.74%). Opened together as batch rows both builds miss the same continuations (slot overwritten before it is parked); not changed here.
32 lines
1.6 KiB
C++
32 lines
1.6 KiB
C++
// (asserts stay on in a Release build: the test is its own check)
|
|
#undef NDEBUG
|
|
#include "strata/core/serve_stop_bound.hpp"
|
|
#include <cassert>
|
|
#include <iostream>
|
|
#include <vector>
|
|
|
|
int main() {
|
|
int cases = 0;
|
|
// Independent reference: the existing emission loop, including its budget and EOS rules.
|
|
for (int accepted = 1; accepted <= 16; ++accepted)
|
|
for (int remaining = 1; remaining <= 18; ++remaining)
|
|
for (int eos = -1; eos < accepted; ++eos) {
|
|
std::vector<int> outputs(accepted);
|
|
for (int i = 0; i < accepted; ++i) outputs[i] = i == eos ? -1 : 100 + i;
|
|
std::vector<int> reference;
|
|
for (int i = 0; i < accepted && static_cast<int>(reference.size()) < remaining; ++i) {
|
|
reference.push_back(outputs[i]);
|
|
if (outputs[i] == -1) break;
|
|
}
|
|
const int kept = strata::core::serve_stop_bound(accepted, remaining, outputs.data(),
|
|
[](int t) { return t == -1; });
|
|
assert(kept > 0 && kept <= accepted && kept <= remaining);
|
|
assert(std::vector<int>(outputs.begin(), outputs.begin() + kept) == reference);
|
|
// Unlike the old full-window commit, no unexposed input suffix enters the cached history.
|
|
assert(kept == static_cast<int>(reference.size()));
|
|
++cases;
|
|
}
|
|
assert(strata::core::serve_stop_bound(4, 0, static_cast<const int*>(nullptr), [](int) { return false; }) == 0);
|
|
std::cout << "SUMMARY stop_bound cases=" << cases << " passed\n";
|
|
}
|