feder-cr/Jobs_Applier_AI_Agent
1
0 Fork 0
mirror of https://github.com/feder-cr/Jobs_Applier_AI_Agent.git synced 2026-11-07 10:36:05 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity

docs: the wiki is written in docs/wiki and published on every push to main by publish-wiki, with invisible_playwright_mcp's converter; its first articles are the studies behind a Dot's memory, its token sizing, its compaction and its benchmarks

Browse Source
This commit is contained in:
feder-cr committed 2026-10-09 19:19:59 +02:00
1 parent ad4fbf3306
commit 3820581c6a
8 files changed
+633
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/publish-wiki.yml
+82
View File
@@ -0,0 +1,82 @@
# Publishes docs/wiki/ to the GitHub wiki of this repository on every push to main that changes it.
#
# The wiki is its own git repository (<repo>.wiki.git): flat pages, no front matter, a _Sidebar.md. The render is
# invisible_playwright_mcp's own (scripts/build_wiki.py and scripts/sync_article_pixels.py, MIT, fetched at a fixed
# commit rather than copied): it strips the front matter, rewrites [x](page.md) links to the wiki's form, names
# index.md Home, builds the sidebar from parent/nav_order, and gives every page its view pixel (a release asset of
# the `article-views` release, created on first use). The wiki checkout is emptied of pages before the copy, so a
# page deleted from docs/wiki/ leaves the wiki too.
name: publish-wiki
on:
push:
branches: [main]
paths:
- "docs/wiki/**"
- ".github/workflows/publish-wiki.yml"
workflow_dispatch: {}
permissions:
contents: write
concurrency:
group: publish-wiki
cancel-in-progress: true
env:
# invisible_playwright_mcp's main on 2026-10-09: the converter this wiki is rendered with.
CONVERTER_REPO: feder-cr/invisible_playwright_mcp
CONVERTER_REF: b72e8b1937771149ed06b2ba73ccb9cb32a687f8
jobs:
publish-wiki:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Fetch the converter
run: |
set -euo pipefail
mkdir -p wiki_tools
for script in build_wiki.py sync_article_pixels.py; do
curl -fsSL -o "wiki_tools/$script" \
"https://raw.githubusercontent.com/${CONVERTER_REPO}/${CONVERTER_REF}/scripts/$script"
done
- name: Render docs/wiki/ into wiki pages
run: python wiki_tools/build_wiki.py docs/wiki wiki_build --pixels "${{ github.repository }}"
# Upload before the push: a page published without its asset counts nothing until the next run.
- name: Give every new page its view pixel
env:
GITHUB_TOKEN: ${{ github.token }}
run: python wiki_tools/sync_article_pixels.py --repo "${{ github.repository }}" wiki_build
- name: Check every page carries its own pixel
run: |
python wiki_tools/sync_article_pixels.py --selftest
python wiki_tools/sync_article_pixels.py --repo "${{ github.repository }}" --check wiki_build
- name: Publish to the wiki
env:
WIKI_PUSH_TOKEN: ${{ secrets.WIKI_TOKEN != '' && secrets.WIKI_TOKEN || github.token }}
run: |
set -euo pipefail
repo="${{ github.repository }}"
git clone "https://x-access-token:${WIKI_PUSH_TOKEN}@github.com/${repo}.wiki.git" wiki
find wiki -maxdepth 1 -name '*.md' -delete
cp wiki_build/*.md wiki/
cd wiki
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add -A
if git diff --cached --quiet; then
echo "wiki already up to date"
else
git commit -m "docs: sync wiki from docs/wiki (${{ github.ref_name }})"
git push origin HEAD:master
fi
docs/wiki/agent-memory-systems-compared.md
+133
View File
@@ -0,0 +1,133 @@
---
title: "Agent memory systems, measured on the same questions"
description: "Mastra Observational Memory, LangMem, Hindsight, an agentic dream pass and a single request, run on the same LongMemEval questions on real agents, with accuracy and cost per pass. Plus what changes with years of history."
parent: "Studies"
nav_order: 2
---
# Agent memory systems, measured on the same questions
A Dot keeps a profile of the person, `MEMORY.md`, in every prompt (see
[How a Dot remembers](how-a-dot-remembers.md)). Something has to write it from
the conversations. Many open-source projects do this; most publish their own
scores on their own setups, and the setups differ. So the candidates were run
here on the same questions, the same agent and the same model.
## The setup
- **Questions:** the 30 LongMemEval questions whose answer depends on what the
person likes ("can you suggest a hotel for my trip?"), each with about 48
dated chats, 115,000 tokens, as its history. These are the ones a profile is
for; the other kinds were checked separately on 116 mixed questions.
- **Agent:** a new Dot per question, with the history as its conversation
files. Each method writes `MEMORY.md` from that history; then the question
goes to the Dot as a task. The Dot can also grep the files, so every method
is measured on top of the same baseline.
- **Model:** z-ai/glm-5.3-flash for everything, through OpenRouter; GPT-4o
grades with LongMemEval's own prompt.
- **Runs:** two per method where the budget allowed. Two identical runs with no
profile scored 19 and 20 out of 30, and others moved by up to three
questions, so a single run of 30 cannot tell methods apart; a difference of
two questions out of 60 is noise.
- **Faithfulness:** each project's own prompts, copied verbatim (18 of 18
checked against the sources), its own call pattern, and its own benchmark
settings where it publishes them. Every adaptation is marked in the adapters,
which are kept in commit `de3d9a1b` of the repository.
## Results
| Method | Correct | % | Cost of a pass over 115K tokens | Calls | Needs |
|---|---|---|---|---|---|
| No profile (the Dot greps the files) | 39/60 | 65% | 0 | 0 | nothing |
| **One request (ours)** | **50/60** | **83%** | **~$0.015** | **1** | nothing |
| Hindsight (MIT) | 25/30 | 83% | ~$0.25 | ~255 | Postgres + pgvector, ~1.9 GB RAM |
| Mastra Observational Memory (Apache-2.0) | 48/60 | 80% | ~$0.05 | ~6 | (TypeScript) |
| LangMem profile manager (MIT) | 48/60 | 80% | ~$0.02 | 1 | LangChain |
| Agentic pass with file tools (nanobot's Dream, MIT) | 44/59 | 75% | ~$0.08 | ~20 steps | nothing |
| Memobase (Apache-2.0) | stopped | | ~$0.25-0.40 | ~200 | Postgres + Redis |
| Mem0 (Apache-2.0) | not run | | ~$0.40 | ~250 | a vector store |
| Letta sleep-time agents (Apache-2.0) | not run | | ~$1.10-1.60 | ~120, growing | a server + Postgres |
The costs of the last three are estimated from their own call patterns,
measured against a stand-in model: running them through would have cost more
than the rest of the study together, for designs the measured ones already
covered (Letta's is an agentic pass like Dream's, which was measured).
## What the numbers say
**The profile matters more than the method.** Every method that writes one
moves the score from 65% to about 80%. Between the one request, Hindsight,
Mastra and LangMem the differences are two questions out of 60 or less:
inside the noise.
**At equal quality, cost decides, and it varies seventeenfold.** The single
request reads the conversations once and answers with the whole new profile.
Mastra reads them in 30,000-token chunks and rewrites its notes when they grow.
Hindsight extracts facts from every ~1,800 characters, one call each. The
agentic pass opens files one at a time with tools, and every step sends the
whole growing context again: 115,000 tokens of history become about 500,000
tokens paid. Letta's sleep-time agent does the same while keeping its own
transcript, which is why its estimate is the highest.
**Simpler is also more robust.** The single request has no tools to call and
no state of its own; the agentic pass failed in 13 of 30 runs before a token
sizing bug was fixed (see
[Token counts are not portable between models](token-counts-across-models.md)),
and Memobase's run ended when the credit did.
**On the 116 mixed questions the profile helped where expected and hurt
nowhere:** updates from 78% to 94% (the profile keeps the newest version of a
fact, with its date), preferences from 57% to 71%, the other kinds within one
question.
## What was chosen
The single request, as a background pass: once the Dot has been quiet for
five minutes, the conversation files that changed go to the summary model with
the current `MEMORY.md`, and the answer is the new `MEMORY.md`. From Mastra it
takes two ideas rather than code: reading in pieces when the new conversations
do not fit one request, and saying that a changed fact replaces the old one.
From LangMem, waiting for a quiet spell rather than running after every
message.
## With years of history (reasoning, not measured)
A Dot used every day for a year holds on the order of ten million tokens of
conversation. What changes:
- **The files and grep hold.** Searching 40 MB of text is instant, and nothing
is lost, because the files are the words as said. What degrades is
precision: a common word matches hundreds of times. The first remedy costs no
model call: a ranked keyword index (BM25) inside the Dot's computer.
- **One profile stops being enough.** A year of a working life does not fit in
25,000 characters. The pass then needs two levels: `MEMORY.md` as the core,
and notes by subject (clients, health, home) that it writes when the core is
full, each named by a line in the core. That is the part of Mastra's design
(its Reflector) worth taking later.
- **The cost of the pass stays flat**, because it reads only what is new:
around a cent a day, a few dollars a year. A monthly rebuild of the profile
from the files, about $2 at that size, would stop small errors from
compounding over hundreds of rewrites.
- **Hindsight gains, and so does its bill.** Retrieving facts for each question
from millions of tokens is its strength, and the gap to a bounded profile
would grow for specific details. Its cost grows with the history (about $22 a
year of extraction at this rate, plus a database per Dot), and its facts are
extractions, lossy where the files are verbatim. For specific details the
files already do that job.
- **The agentic designs and per-message extraction get worse**: the first
reread more every pass, the second pay on every message.
- **Contradictions are the real risk** ("I live in Milan", then months later "I
moved to Rome"). A rewrite with dated facts handles them; systems that only
accumulate facts keep both.
## Sources
- LongMemEval: [paper](https://arxiv.org/abs/2410.10813), [code](https://github.com/xiaowu0162/LongMemEval)
- Mastra Observational Memory: [research](https://mastra.ai/research/observational-memory), [code](https://github.com/mastra-ai/mastra)
- LangMem: [code](https://github.com/langchain-ai/langmem)
- Hindsight: [paper](https://arxiv.org/abs/2512.12818), [code](https://github.com/vectorize-io/hindsight), [benchmark harness](https://github.com/vectorize-io/agent-memory-benchmark)
- Memobase: [code](https://github.com/memodb-io/memobase)
- Mem0: [paper](https://arxiv.org/abs/2504.19413), [code](https://github.com/mem0ai/mem0)
- Letta: [code](https://github.com/letta-ai/letta), [sleep-time compute](https://arxiv.org/abs/2504.13171)
- nanobot (Dream): [code](https://github.com/HKUDS/nanobot)
- Is Grep All You Need?: [arXiv 2605.15184](https://arxiv.org/html/2605.15184)
docs/wiki/benchmarking-agents-on-real-vms.md
+92
View File
@@ -0,0 +1,92 @@
---
title: "Benchmarking an agent on real VMs"
description: "How Dots are tested as a person meets them: Harbor tasks in a new VM each, long builds of two to three hours, eight Dots on one host, and the defects these runs found."
parent: "Studies"
nav_order: 5
---
# Benchmarking an agent on real VMs
A Dot is an agent with its own computer: a QEMU virtual machine on the host,
reached through an API. Unit tests and a stand-in model check the engine's
logic; they cannot tell whether a Dot gets real work done. These runs do, on
real VMs and a real model, and every one of them found something.
## The harness
[Harbor](https://github.com/harbor-framework/harbor) (Apache-2.0), the harness
of Terminal-Bench, runs a task in an environment and grades what is left. Here
the environment is a Dot: a new one per trial, created through the product's
API, with the task's Dockerfile replayed inside it as the Dot's own user (there
is no root: `apt-get install` becomes the Dot's `sudo dot-install`), and the
task's paths moved under its home. The agent is the Dot itself, given the
instruction as a task. A task counts only once its reference solution scores 1
and doing nothing scores 0 **in a Dot**: a task that needs root fails its
oracle, which is how such a task is found. Code: `tests/bench/`.
## Short tasks
25 tasks written for this, in shell, files and documents, coding, data
analysis, the computer's settings, questions with one answer, two long
pipelines, and three about what a Dot is for (its own automation must run
twice; clean up leftovers and nothing else; say a request is impossible). The
data is generated with a fixed seed and the expected answers are computed by
the authoring script, never by an agent.
- Oracle 25/25, doing nothing 0/25.
- A Dot on z-ai/glm-5.3-flash: **73 of 75 trials** (each task three times).
Both misses were the same task, a Bash script to port to Python.
## Long tasks: two to three hours
Five tasks where the Dot builds or researches something whole, each graded by
checking the result itself, never by asking a model:
| Task | What the Dot does | What the grader checks |
|---|---|---|
| chess-perft | a chess move generator, every rule included | published perft counts of five positions to depth 4 and 5 (up to 4.9 million), and four positions it has not seen |
| lisp-interpreter | an interpreter for a Scheme subset | 30 programs it has not seen: closures, tail calls a million deep, recursion 5000 deep, errors |
| kv-store | a key-value server over HTTP | the API, TTLs, 1000 writes from 20 clients at once, every acknowledged write after `kill -9` |
| language-history | research on 18 programming languages, on the web | years and creators, influences, a cited 1500-word report from five sites |
| fraud-investigation | an investigation of 292,467 generated card transactions | exactly the planted cases of four kinds of fraud, a script that finds them again |
These found the token limits described in
[Compacting a thread at the model's own window](compacting-at-the-models-window.md):
with a 4,096-token answer cap, a Dot writing a whole program in one call looped
until it failed. After the fix all five passed, the server and the interpreter
two out of two, in runs of 18 to 75 minutes.
## Memory: LongMemEval
The same harness runs [LongMemEval](https://github.com/xiaowu0162/LongMemEval)
on Dots: a history of 48 chats becomes a Dot's past conversations, the question
is a task, GPT-4o grades outside the Dot. The results and what they changed are
in [How a Dot remembers](how-a-dot-remembers.md) and
[Agent memory systems, measured on the same questions](agent-memory-systems-compared.md).
This run found two defects of its own: a token estimate OpenRouter refuses
(see [Token counts are not portable between models](token-counts-across-models.md)),
and in the harness a call to a Dot that could wait forever, which once held
four questions for seven hours; every call now has a bound.
## Many Dots on one host
`tests/e2e/scale.ts` starts 1, 2, 4 and 8 Dots at once on one host; each must
come up, answer a message and do a task, within budgets that are ratios to one
Dot. All four steps passed. Measured: the time to READY at the 95th percentile
went from 21 s with one Dot to 45 s with eight; the API answered in 6 to 10 ms
at the 95th percentile throughout; each Dot's QEMU took about 700 MB.
## What the other tests found
- **Approvals**: every tool under allow, deny, ask-then-approve and
ask-then-reject, in the chat and in a task (258 cases), plus a race of an
approve and a reject sent at the same moment, 50 times on PostgreSQL: one
wins, the other is refused, the Dot hears one decision. It found that a task
cancelled while a call waited for approval left the approval pending forever,
so the Dot never went back to READY and never slept.
- **Upgrades**: the databases a Dot and a host of a released version left
behind open under the new build and keep working.
Every run here costs real money on a real model; the numbers above were bought
for a few dollars each, and the harness keeps each Dot's events, engine log and
memory with its result so a wrong answer can be read after the Dot is gone.
docs/wiki/compacting-at-the-models-window.md
+81
View File
@@ -0,0 +1,81 @@
---
title: "Compacting a thread at the model's own window"
description: "Why a Dot sets no token limit of its own, when it compacts a long thread, and why it clears old tool results before it summarizes anything: the rules taken from opencode, Codex, OpenHands and a SWE-bench study."
parent: "Studies"
nav_order: 4
---
# Compacting a thread at the model's own window
An agent working for hours fills its context. Something has to give, and
every choice costs something: a cap on the answer cuts work in half, a summary
loses detail, a truncation loses more. This page is how a Dot decides, and the
failures that decided it.
## What went wrong with fixed limits
Until October 2026 a Dot sent `max_tokens: 4096` on every request and kept each
thread under a configurable 32,000 tokens. On long benchmark tasks (writing a
Scheme interpreter, a key-value server: see
[Benchmarking an agent on real VMs](benchmarking-agents-on-real-vms.md)) this
failed in a specific way: the model wrote a whole program in one `write_file`
call, the answer was cut at 4,096 tokens in the middle of the call, the engine
asked it to "continue from its exact endpoint", and with nothing delivered to
continue from, the model made the same call again until the task failed. A
model with a one-million-token window was being summarized at 32,000.
So the limits went, all of them: a Dot has no token setting. Every request uses
its model's own limits, read once from OpenRouter's list of models (the window
and the longest answer of the provider OpenRouter routes to by default), and
asks for the longest answer the window leaves room for. Without `max_tokens`
each provider would apply a default of its own, often far lower. A response
cut before it delivered anything is now told that nothing ran and to work in
smaller steps, instead of to continue.
## When to compact
A thread is compacted only when the next request would not fit: at the window,
less the room kept for the answer (20,000 tokens, or the longest answer when
that is shorter) and a safety margin. That is opencode's rule. How the prompt is
measured turned out to matter as much as the rule: see
[Token counts are not portable between models](token-counts-across-models.md).
## How: clear first, summarize second
1. **Old tool results are cleared.** Walking back from the newest message, the
first 40,000 tokens of tool results stay; older ones become a short note that
the tool can be called again, if that frees at least 20,000 tokens. The calls,
the reasoning and every message stay. No model call is needed. On SWE-bench
this costs about half as much as summarizing for the same solve rate
(JetBrains, "The Complexity Trap", arXiv 2508.21433); the thresholds are
opencode's.
2. **Only if that is not enough, a summary.** It is written as a handoff for the
model that resumes the work (Codex's framing), under the headings of
OpenHands' summarizing prompt, plus what only a Dot has: approvals given and
files written. After it, the person's latest messages are kept as they wrote
them (Codex keeps 20,000 tokens; a Dot keeps at most a quarter of the budget,
because a model with a small window still needs room after them). The summary
may be written by a cheaper model (`models.summary`), within that model's own
window; a thread too long even for it loses its oldest messages first, never
a call without its result.
OpenRouter's own `context-compression` plugin was looked at and not used: it is
a middle-out truncation, which drops the middle of the thread without a trace.
## What it changed
On the five long tasks (two to three hours each), the key-value server and the
Scheme interpreter went from failing most runs to passing 2 out of 2; all five
passed in the final round, in runs of 18 to 75 minutes.
One side effect is worth knowing: OpenRouter holds the cost of `max_tokens` up
front, so asking for the longest answer makes a nearly empty balance show
sooner. When it refuses for credit, the Dot now shows OpenRouter's own message
(how much the balance still allows, where to add credit) after its own.
## Sources
- opencode: [code](https://github.com/sst/opencode) (compaction thresholds)
- Codex: [code](https://github.com/openai/codex) (handoff summary, recent user messages)
- OpenHands: [code](https://github.com/All-Hands-AI/OpenHands) (summarizing condenser)
- JetBrains, "The Complexity Trap": [arXiv 2508.21433](https://arxiv.org/abs/2508.21433)
docs/wiki/how-a-dot-remembers.md
+118
View File
@@ -0,0 +1,118 @@
---
title: "How a Dot remembers"
description: "Past conversations kept as files the agent greps, a profile of the person in every prompt, and a background pass that keeps it current. Measured on LongMemEval: 7.8% to 90.5%."
parent: "Studies"
nav_order: 1
---
# How a Dot remembers
A Dot talks to one person for weeks: in its chat, and in the tasks it is
given. Three things make it remember what it was told, and each one was
added because a measurement asked for it.
| Layer | What it holds | Who writes it | In the prompt? |
|---|---|---|---|
| Conversations | every word said, by day and by task | the engine, after every turn | no: the Dot searches it with grep |
| `MEMORY.md` | what the Dot should always know about the person | the Dot, and a pass in the background | always |
| Notes and skills | details by subject, and how to do things | the Dot | when it opens them |
## The measure: LongMemEval
[LongMemEval](https://github.com/xiaowu0162/LongMemEval) (ICLR 2025, MIT)
gives each question a history of about 48 dated chats, around 115,000
tokens, and then asks something only that history answers: a detail the
person mentioned, one that changed later, how many days passed between two
events, what they would like, or something they never said (the right answer
is then that it is not known). The answer is graded by GPT-4o with the
benchmark's own prompts.
On Dots it runs as a person would meet it: a new Dot per question, the
history written as its past conversations, the question sent as a task, the
grading done outside the Dot so the OpenRouter key never enters it. 116
questions, drawn with a fixed seed in the benchmark's proportions; model
z-ai/glm-5.3-flash. The harness is `tests/bench/longmemeval.py`.
| | Correct | % |
|---|---|---|
| No history at all (the control) | 9/116 | 7.8% |
| Conversations as files | 96/116 | 82.8% |
| + the fixes that run found | 102/116 | 87.9% |
| + `MEMORY.md` from the memory pass | 105/116 | **90.5%** |
The control only gets the four questions whose answer is "you never told
me", and five it can guess.
## Conversations as files
The transcripts live in the engine's database, which the model's commands
cannot read, and a long chat reaches the model only as a summary of its older
part. So after every turn the engine also writes what was said to the Dot's
own computer:
```
/home/dot/conversations/chat/2023-05-20.md one file a day of the chat
/home/dot/conversations/tasks/2023-05-21-t1.md one file a task
```
Each message sits under a heading with its time, and each call the Dot made
is one line (not what it returned). The prompt says to search there with grep
when something said before matters. This one change took the Dot from 7.8% to
82.8%. Plain files and grep were chosen over embeddings or a vector store
because that is what wins in the published measures: in "Is Grep All You
Need?" grep beat vector search in every harness and model tried, and on
LongMemEval-V2 the best result came from an agent searching files (74.9%)
rather than from extracted facts with retrieval (58.6%).
## What the first run found
Twenty answers were wrong, and reading each one showed where:
- **4 of the 5 preference misses searched nothing.** Asked "can you suggest
accessories for my phone?", the Dot gave generic advice: nothing in the
question says "remember". The prompt now says to look in past conversations
before advice or a recommendation.
- **One task was refused by OpenRouter** for asking an answer 2,779 tokens
too long for the window: the token estimate was off. See
[Token counts are not portable between models](token-counts-across-models.md).
- The system prompt carried the time to the minute, so it changed on every
turn and the provider's cache never held. It now carries the day.
Together: 82.8% to 87.9%.
## MEMORY.md, and the pass that keeps it
The remaining preference misses had a common cause grep cannot fix: the fact
that matters shares no word with the question. "I've been sneezing, could it
be my living room?" needs "my cat Luna sheds", and a search for "sneez|dust"
does not find the cat. What fixes that is a short profile of the person in
every prompt, so the Dot does not have to guess what to look for.
`MEMORY.md` is that profile: who the person is, the people and animals in
their life, what they have and use, what they like and dislike, their
routines and plans, each fact with the day it was said, and a line for every
other note. The Dot sees it whole in every prompt, up to 25,000 characters
(cut there, with a word to shorten it, as Claude Code does with its own).
A model often does not think to write it while it works, so the engine keeps
it current itself: once the Dot has been quiet for five minutes, it takes the
conversation files that changed since the last time and has the summary model
rewrite `MEMORY.md` from them, in one request with no tools. A changed fact
replaces the old one with its date; nothing a web page or a command said is
copied in as an instruction. If the conversations do not fit one request they
go in several, oldest first. A pass that is cut, empty, unpriced, or races
with the Dot editing the file writes nothing, and the next one takes the same
conversations again. Its cost goes to the person's usage like any other turn.
On the 30 preference questions it raised the right answers from 65% to 83%,
and on the 116 mixed ones updates went from 78% to 94% (the profile keeps the
newest version of a fact), with nothing else moving beyond the noise of a run.
How this was chosen among five methods is in
[Agent memory systems, measured on the same questions](agent-memory-systems-compared.md).
## What it costs
With glm-5.3-flash a pass over a whole 115,000-token history costs about
$0.015. In use a pass reads only what changed since the last one, so a normal
day of chat comes to about a cent. Answering a question that needs the
history cost the Dot $0.003 to $0.005 on average.
docs/wiki/index.md
+33
View File
@@ -0,0 +1,33 @@
---
title: "Home"
nav_order: 1
---
# invisible_dots Wiki
invisible_dots runs Dots: autonomous AI agents, each with its own Linux
computer, that work for a person through chat and tasks. This wiki is where
the work behind them is written up: what was measured, how, what came out,
and what was chosen because of it. Every study has its numbers and its
method, so a reader can check them or run them again.
The [README](https://github.com/feder-cr/invisible_dots#readme) is how to run
invisible_dots; [the architecture](https://github.com/feder-cr/invisible_dots/blob/main/docs/architecture.md)
is how it is built.
## [Studies](studies.md)
- [How a Dot remembers](how-a-dot-remembers.md): past conversations as files,
a profile in every prompt, and the pass that keeps it current. 7.8% to 90.5%
on LongMemEval.
- [Agent memory systems, measured on the same questions](agent-memory-systems-compared.md):
Mastra, LangMem, Hindsight, an agentic "dream" pass and a single request,
with their cost per pass.
- [Token counts are not portable between models](token-counts-across-models.md):
tiktoken against five models' own counts, and why a fixed safety margin got
a request refused.
- [Compacting a thread at the model's own window](compacting-at-the-models-window.md):
no token cap of our own, old tool results cleared before anything is
summarized.
- [Benchmarking an agent on real VMs](benchmarking-agents-on-real-vms.md):
Harbor tasks, two-to-three-hour builds, and eight Dots on one host.
docs/wiki/studies.md
+18
View File
@@ -0,0 +1,18 @@
---
title: "Studies"
has_children: true
nav_order: 2
---
# Studies
What was measured to decide how a Dot works. Each page says the question, the
method, the numbers, and what was chosen; the code and the harness that
produced the numbers are in the repository, under `tests/bench/` and
`invisible_engine_dots/`.
- [How a Dot remembers](how-a-dot-remembers.md)
- [Agent memory systems, measured on the same questions](agent-memory-systems-compared.md)
- [Token counts are not portable between models](token-counts-across-models.md)
- [Compacting a thread at the model's own window](compacting-at-the-models-window.md)
- [Benchmarking an agent on real VMs](benchmarking-agents-on-real-vms.md)
docs/wiki/token-counts-across-models.md
+76
View File
@@ -0,0 +1,76 @@
---
title: "Token counts are not portable between models"
description: "tiktoken against the counts of five models on OpenRouter, by kind of text: from 0.52 to 1.33 times. Why a fixed safety margin got a request refused, and how a Dot sizes its requests now."
parent: "Studies"
nav_order: 3
---
# Token counts are not portable between models
Before an agent sends a request it has to know how big the prompt is: to
decide when to compact a long thread, and how long an answer it may ask for
(the prompt and the answer together must fit the model's window). Only the
provider knows its model's tokenizer, so engines estimate, usually with
OpenAI's tiktoken, and leave a small margin. A Dot did this with a margin of
1,024 tokens, and a long task on a one-million-token window was refused.
## The refusal
A memory task on z-ai/glm-5.3-flash (window 1,048,576 tokens, longest answer
943,717) asked OpenRouter for the longest answer that fitted after a prompt
the engine had estimated at 103,346 tokens. OpenRouter answered:
> This endpoint's maximum context length is 1048576 tokens. However, you
> requested about 1049513 tokens (100506 of text input, 5290 of tool input,
> 943717 in the output).
Its count of the prompt was 105,796: the estimate was 2,450 tokens (2.4%)
short, more than twice the margin, and the request overflowed the window by
937 tokens.
## The measurement
Six kinds of text were sent to five models through OpenRouter, asking for a
one-word answer, and the prompt tokens each model reported were compared with
tiktoken's count of the same request (ratio counted / estimated):
| Model | English chat | Python | TypeScript | JSON | Italian | Chinese |
|---|---|---|---|---|---|---|
| anthropic/claude-sonnet-4.5 | 1.115 | **1.334** | 1.242 | 1.119 | 1.040 | 1.097 |
| google/gemini-2.5-flash | 1.039 | 1.275 | 1.172 | 1.051 | 0.880 | 0.548 |
| deepseek/deepseek-chat-v3.1 | 1.008 | 1.120 | 1.066 | 1.004 | 0.920 | 0.516 |
| qwen/qwen3-235b-a22b-2507 | 1.020 | 1.001 | 1.011 | 1.028 | 0.960 | 0.549 |
| openai/gpt-4o-mini | 0.990 | 1.005 | 1.007 | 0.989 | 0.920 | 0.774 |
(October 2026; texts of 8,000 to 28,000 tokens.) The spread runs from about
half to a third more, and it depends on the model **and** on the text: Claude
counts Python a third above tiktoken, the Chinese text is counted at half by
four of the five. A margin of fixed size cannot cover a relative error: on a
600,000-token prompt of code, a third is 200,000 tokens.
## Why the provider's own count is not enough either
The obvious fix is to use what the provider reports: every response says how
many prompt tokens it took, so the next request is that count plus an estimate
of what was added. It was built, and the refusal came back. OpenRouter checks
a request against the window with **a count of its own**, made before it routes
the request and reported only in the refusal; on glm-5.3-flash it ran above the
model's count plus a scaled estimate of the rest. The judge is a number the
engine never sees on a request that succeeds.
## What a Dot does now
The whole estimate is scaled: a prompt is taken for its tiktoken count times
1.35 (the largest ratio measured above, with room), or times the largest ratio
the model's own counts have shown when that is more (learned from every
response of more than 1,000 tokens). Being wrong upwards costs little: an
answer limit of 900,000 tokens instead of 943,717, and a thread compacted a
little earlier. Being wrong downwards fails the request. Each request now logs
its size, how it was measured and the answer limit it sent, so the next
surprise shows its numbers.
A related fix: the system prompt carried the time to the minute, so it changed
on every turn, and neither the provider's prompt cache nor any count of the
previous request could carry over. It carries the day now.
Code: `invisible_engine_dots/nanobot/providers/prompt_count.py`.