Merge pull request #8017 from esengine/feat/memorybench

feat(bench): MemoryBench — the paired memory-effectiveness suite
This commit is contained in:
YHH authored and GitHub committed 2026-08-09 12:25:15 +08:00
commit fb993e12af
163 files changed
+1670 -16

No files matched your search

+24
View File
@@ -254,3 +254,27 @@ go run ./benchmarks/context-maintenance-e2e comprehension
`--profile`, `--ablate`).
- [`cmd/e2ebench/main.go`](../cmd/e2ebench/main.go) — suite runner and report
renderer.
## memorybench
The memory-effectiveness suite. Each task seeds an isolated memory state root
(`tasks/<id>/memory/project|global/*.md`, production frontmatter) before the
run; `memory_markers` in task.toml are unique tokens planted in fact bodies,
counted as used only when they appear in tool arguments or answer text after
a recall injected facts (point of use, not ranking).
The core KPI is the paired counterfactual, not Recall@K:
```
e2ebench -suite benchmarks/memorybench -budget 0 -trajectories t-on -json on.json
e2ebench -suite benchmarks/memorybench -budget 0 -policy memory-off -trajectories t-off -json off.json
e2ebench -mode compare on.json off.json # Memory utility section
```
Utility delta = paired Pass(on) − Pass(off). Harmful attribution is paired,
never judged: the same task passed without memory and failed with it while
recall fired. Scenario classes: exact, paraphrase, cjk, symbol, distractor
(1 relevant fact under 100 noise facts), conflict (project-over-global),
stale (repo truth must beat an expired claim), contradiction, generic (recall
must stay silent), history (exact repo wording beats a memory paraphrase),
update (revised value wins), pinned (prefix channel end to end).
@@ -0,0 +1,9 @@
---
name: migration-naming
description: 数据库迁移脚本命名规范
metadata:
type: project
scope: project
---
数据库迁移必须向后兼容。迁移脚本命名规范:`MEMKEY-CJK5T-<描述>.sql`,放在仓库根目录。
@@ -0,0 +1,5 @@
prompt = "为本项目新建一个数据库迁移脚本文件(空文件即可),文件名必须符合项目的迁移命名规范。"
class = "cjk"
max_steps = 8
timeout_sec = 180
memory_markers = ["MEMKEY-CJK5T"]
+3
View File
@@ -0,0 +1,3 @@
#!/usr/bin/env bash
set -e
ls | grep -q "^MEMKEY-CJK5T-.*\.sql$"
Whitespace-only changes.
@@ -0,0 +1,10 @@
---
name: deploy-region
description: Default deploy region is us-east-1
subject_key: project.deploy_region
metadata:
type: project
scope: project
---
Default deploy region across projects is `us-east-1`.
@@ -0,0 +1,10 @@
---
name: deploy-region
description: This project deploys to eu-central-1
subject_key: project.deploy_region
metadata:
type: project
scope: project
---
This project deploys to `eu-central-1` (MEMKEY-CONF2P). Overrides the global default.
@@ -0,0 +1,5 @@
prompt = "Write the deploy region for this project (just the region string) into answer.txt."
class = "conflict"
max_steps = 8
timeout_sec = 180
memory_markers = ["MEMKEY-CONF2P"]
+3
View File
@@ -0,0 +1,3 @@
#!/usr/bin/env bash
set -e
grep -q "eu-central-1" answer.txt && ! grep -q "us-east-1" answer.txt
Whitespace-only changes.
@@ -0,0 +1,9 @@
---
name: dependency-tooling
description: We migrated to pnpm
metadata:
type: project
scope: project
---
We migrated to pnpm; install with `pnpm install` (MEMKEY-CONTRA).
@@ -0,0 +1,9 @@
---
name: package-manager
description: This project uses npm
metadata:
type: project
scope: project
---
This project uses npm; install with `npm install`.
@@ -0,0 +1,5 @@
prompt = "Write the command to install this project's dependencies into answer.txt."
class = "contradiction"
max_steps = 8
timeout_sec = 180
memory_markers = ["MEMKEY-CONTRA"]
+3
View File
@@ -0,0 +1,3 @@
#!/usr/bin/env bash
set -e
grep -q "pnpm install" answer.txt && ! grep -q "npm install" answer.txt
@@ -0,0 +1,2 @@
lockfileVersion: 9
# MEMKEY-CONTRA
@@ -0,0 +1,9 @@
---
name: distractor-000
description: Note about cache eviction tuning #0
metadata:
type: project
scope: project
---
Historical note #0 about cache eviction tuning. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-001
description: Note about logo color palette #1
metadata:
type: project
scope: project
---
Historical note #1 about logo color palette. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-002
description: Note about office wifi setup #2
metadata:
type: project
scope: project
---
Historical note #2 about office wifi setup. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-003
description: Note about sprint retro cadence #3
metadata:
type: project
scope: project
---
Historical note #3 about sprint retro cadence. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-004
description: Note about docker registry mirror #4
metadata:
type: project
scope: project
---
Historical note #4 about docker registry mirror. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-005
description: Note about S3 bucket lifecycle #5
metadata:
type: project
scope: project
---
Historical note #5 about S3 bucket lifecycle. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-006
description: Note about oncall rotation #6
metadata:
type: project
scope: project
---
Historical note #6 about oncall rotation. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-007
description: Note about email templates #7
metadata:
type: project
scope: project
---
Historical note #7 about email templates. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-008
description: Note about font licensing #8
metadata:
type: project
scope: project
---
Historical note #8 about font licensing. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-009
description: Note about meeting notes location #9
metadata:
type: project
scope: project
---
Historical note #9 about meeting notes location. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-010
description: Note about cache eviction tuning #10
metadata:
type: project
scope: project
---
Historical note #10 about cache eviction tuning. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-011
description: Note about logo color palette #11
metadata:
type: project
scope: project
---
Historical note #11 about logo color palette. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-012
description: Note about office wifi setup #12
metadata:
type: project
scope: project
---
Historical note #12 about office wifi setup. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-013
description: Note about sprint retro cadence #13
metadata:
type: project
scope: project
---
Historical note #13 about sprint retro cadence. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-014
description: Note about docker registry mirror #14
metadata:
type: project
scope: project
---
Historical note #14 about docker registry mirror. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-015
description: Note about S3 bucket lifecycle #15
metadata:
type: project
scope: project
---
Historical note #15 about S3 bucket lifecycle. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-016
description: Note about oncall rotation #16
metadata:
type: project
scope: project
---
Historical note #16 about oncall rotation. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-017
description: Note about email templates #17
metadata:
type: project
scope: project
---
Historical note #17 about email templates. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-018
description: Note about font licensing #18
metadata:
type: project
scope: project
---
Historical note #18 about font licensing. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-019
description: Note about meeting notes location #19
metadata:
type: project
scope: project
---
Historical note #19 about meeting notes location. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-020
description: Note about cache eviction tuning #20
metadata:
type: project
scope: project
---
Historical note #20 about cache eviction tuning. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-021
description: Note about logo color palette #21
metadata:
type: project
scope: project
---
Historical note #21 about logo color palette. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-022
description: Note about office wifi setup #22
metadata:
type: project
scope: project
---
Historical note #22 about office wifi setup. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-023
description: Note about sprint retro cadence #23
metadata:
type: project
scope: project
---
Historical note #23 about sprint retro cadence. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-024
description: Note about docker registry mirror #24
metadata:
type: project
scope: project
---
Historical note #24 about docker registry mirror. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-025
description: Note about S3 bucket lifecycle #25
metadata:
type: project
scope: project
---
Historical note #25 about S3 bucket lifecycle. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-026
description: Note about oncall rotation #26
metadata:
type: project
scope: project
---
Historical note #26 about oncall rotation. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-027
description: Note about email templates #27
metadata:
type: project
scope: project
---
Historical note #27 about email templates. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-028
description: Note about font licensing #28
metadata:
type: project
scope: project
---
Historical note #28 about font licensing. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-029
description: Note about meeting notes location #29
metadata:
type: project
scope: project
---
Historical note #29 about meeting notes location. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-030
description: Note about cache eviction tuning #30
metadata:
type: project
scope: project
---
Historical note #30 about cache eviction tuning. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-031
description: Note about logo color palette #31
metadata:
type: project
scope: project
---
Historical note #31 about logo color palette. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-032
description: Note about office wifi setup #32
metadata:
type: project
scope: project
---
Historical note #32 about office wifi setup. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-033
description: Note about sprint retro cadence #33
metadata:
type: project
scope: project
---
Historical note #33 about sprint retro cadence. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-034
description: Note about docker registry mirror #34
metadata:
type: project
scope: project
---
Historical note #34 about docker registry mirror. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-035
description: Note about S3 bucket lifecycle #35
metadata:
type: project
scope: project
---
Historical note #35 about S3 bucket lifecycle. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-036
description: Note about oncall rotation #36
metadata:
type: project
scope: project
---
Historical note #36 about oncall rotation. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-037
description: Note about email templates #37
metadata:
type: project
scope: project
---
Historical note #37 about email templates. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-038
description: Note about font licensing #38
metadata:
type: project
scope: project
---
Historical note #38 about font licensing. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-039
description: Note about meeting notes location #39
metadata:
type: project
scope: project
---
Historical note #39 about meeting notes location. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-040
description: Note about cache eviction tuning #40
metadata:
type: project
scope: project
---
Historical note #40 about cache eviction tuning. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-041
description: Note about logo color palette #41
metadata:
type: project
scope: project
---
Historical note #41 about logo color palette. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-042
description: Note about office wifi setup #42
metadata:
type: project
scope: project
---
Historical note #42 about office wifi setup. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-043
description: Note about sprint retro cadence #43
metadata:
type: project
scope: project
---
Historical note #43 about sprint retro cadence. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-044
description: Note about docker registry mirror #44
metadata:
type: project
scope: project
---
Historical note #44 about docker registry mirror. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-045
description: Note about S3 bucket lifecycle #45
metadata:
type: project
scope: project
---
Historical note #45 about S3 bucket lifecycle. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-046
description: Note about oncall rotation #46
metadata:
type: project
scope: project
---
Historical note #46 about oncall rotation. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-047
description: Note about email templates #47
metadata:
type: project
scope: project
---
Historical note #47 about email templates. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-048
description: Note about font licensing #48
metadata:
type: project
scope: project
---
Historical note #48 about font licensing. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-049
description: Note about meeting notes location #49
metadata:
type: project
scope: project
---
Historical note #49 about meeting notes location. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-050
description: Note about cache eviction tuning #50
metadata:
type: project
scope: project
---
Historical note #50 about cache eviction tuning. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-051
description: Note about logo color palette #51
metadata:
type: project
scope: project
---
Historical note #51 about logo color palette. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-052
description: Note about office wifi setup #52
metadata:
type: project
scope: project
---
Historical note #52 about office wifi setup. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-053
description: Note about sprint retro cadence #53
metadata:
type: project
scope: project
---
Historical note #53 about sprint retro cadence. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-054
description: Note about docker registry mirror #54
metadata:
type: project
scope: project
---
Historical note #54 about docker registry mirror. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-055
description: Note about S3 bucket lifecycle #55
metadata:
type: project
scope: project
---
Historical note #55 about S3 bucket lifecycle. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-056
description: Note about oncall rotation #56
metadata:
type: project
scope: project
---
Historical note #56 about oncall rotation. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-057
description: Note about email templates #57
metadata:
type: project
scope: project
---
Historical note #57 about email templates. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-058
description: Note about font licensing #58
metadata:
type: project
scope: project
---
Historical note #58 about font licensing. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-059
description: Note about meeting notes location #59
metadata:
type: project
scope: project
---
Historical note #59 about meeting notes location. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-060
description: Note about cache eviction tuning #60
metadata:
type: project
scope: project
---
Historical note #60 about cache eviction tuning. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-061
description: Note about logo color palette #61
metadata:
type: project
scope: project
---
Historical note #61 about logo color palette. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-062
description: Note about office wifi setup #62
metadata:
type: project
scope: project
---
Historical note #62 about office wifi setup. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-063
description: Note about sprint retro cadence #63
metadata:
type: project
scope: project
---
Historical note #63 about sprint retro cadence. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-064
description: Note about docker registry mirror #64
metadata:
type: project
scope: project
---
Historical note #64 about docker registry mirror. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-065
description: Note about S3 bucket lifecycle #65
metadata:
type: project
scope: project
---
Historical note #65 about S3 bucket lifecycle. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-066
description: Note about oncall rotation #66
metadata:
type: project
scope: project
---
Historical note #66 about oncall rotation. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-067
description: Note about email templates #67
metadata:
type: project
scope: project
---
Historical note #67 about email templates. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-068
description: Note about font licensing #68
metadata:
type: project
scope: project
---
Historical note #68 about font licensing. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-069
description: Note about meeting notes location #69
metadata:
type: project
scope: project
---
Historical note #69 about meeting notes location. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-070
description: Note about cache eviction tuning #70
metadata:
type: project
scope: project
---
Historical note #70 about cache eviction tuning. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-071
description: Note about logo color palette #71
metadata:
type: project
scope: project
---
Historical note #71 about logo color palette. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-072
description: Note about office wifi setup #72
metadata:
type: project
scope: project
---
Historical note #72 about office wifi setup. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-073
description: Note about sprint retro cadence #73
metadata:
type: project
scope: project
---
Historical note #73 about sprint retro cadence. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-074
description: Note about docker registry mirror #74
metadata:
type: project
scope: project
---
Historical note #74 about docker registry mirror. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-075
description: Note about S3 bucket lifecycle #75
metadata:
type: project
scope: project
---
Historical note #75 about S3 bucket lifecycle. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-076
description: Note about oncall rotation #76
metadata:
type: project
scope: project
---
Historical note #76 about oncall rotation. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-077
description: Note about email templates #77
metadata:
type: project
scope: project
---
Historical note #77 about email templates. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-078
description: Note about font licensing #78
metadata:
type: project
scope: project
---
Historical note #78 about font licensing. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-079
description: Note about meeting notes location #79
metadata:
type: project
scope: project
---
Historical note #79 about meeting notes location. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-080
description: Note about cache eviction tuning #80
metadata:
type: project
scope: project
---
Historical note #80 about cache eviction tuning. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-081
description: Note about logo color palette #81
metadata:
type: project
scope: project
---
Historical note #81 about logo color palette. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-082
description: Note about office wifi setup #82
metadata:
type: project
scope: project
---
Historical note #82 about office wifi setup. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-083
description: Note about sprint retro cadence #83
metadata:
type: project
scope: project
---
Historical note #83 about sprint retro cadence. It has no bearing on build or check commands.
@@ -0,0 +1,9 @@
---
name: distractor-084
description: Note about docker registry mirror #84
metadata:
type: project
scope: project
---
Historical note #84 about docker registry mirror. It has no bearing on build or check commands.
Loaded 100 of 163 files, more files were not shown because too many files have changed in this diff. Show more