## Description of changes
Validate effective persisted HNSW settings before appending local logs.
This prevents repeated failing writes from growing the queue while
retaining existing records for recovery after configuration repair.
Cover legacy invalid settings and verify the log remains empty.
## Test plan
CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
## Description of changes
Inspect persistent indexes and SQLite watermarks without modifying
them. Report missing files, dimension mismatches, ID-map
inconsistencies, and pending legacy watermark migration.
Use the configured log namespace, decode legacy binary watermarks,
and return meaningful exit codes from both command entry points.
Test namespace selection, legacy decoding, and read-only behavior.
## Test plan
CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
## Description of changes
Move version discovery out of module import into
pytest_generate_tests so spawned writer children no longer
depend on a fresh PyPI request at startup.
Assert the writer process exits cleanly and close the
multiprocessing connections in a finally block so failures
surface instead of leaking resources.
## Test plan
CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
## Description of changes
Delete all collection segments transactionally, including database
teardown. Invalidate live writers before removing HNSW files and reject
loads from stale segment descriptors. Retry orphan cleanup at startup
and hourly, checking each segment under its existing lifecycle lock.
Reuse transactional log deletion already on main. Avoid database-wide
mutexes and age-based guesses about which directories can be removed.
## Test plan
CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
## Description of changes
Cached local HNSW indexes held four CRT streams each, exhausting
Windows default 512-stream process limit after roughly 128 resident
segments.
Open streams only for bounded disk operations (load and
checkpoint) instead of for the index lifetime:
- Gate loads and saves behind a process-wide 32-permit semaphore.
- Close streams via a Drop guard held under the index write lock,
so a failed save still closes before the permit is released.
- Drop the eager open_fd() at reader and writer creation; queries
and mutations run from memory.
Add a test that re-execs under ulimit -n 512 and creates 160
cached indexes, then reloads them to verify checkpoints replay.
## Test plan
CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
## Description of changes
Attempt log purging even when backfill fails, reading durable SQLite
watermarks without reopening a broken HNSW index. Require both metadata
and vector progress so configuration errors never discard vector data.
## Test plan
CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
## Description of changes
Serialize cache misses by index identity to prevent concurrent native
writers sharing persistent files. Retain weak identities so eviction
cannot create a second index while readers or writers still hold one.
Cache hits keep their existing fast path.
## Test plan
CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
---------
Co-authored-by: dbeglord <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
The none-driver setup in `medyagh/setup-minikube` runs `chmod 755
/etc/cni/net.d` without creating the directory. On fresh runner images
this fails before `minikube start`, blocking Tilt integration tests.
Create the directory before invoking the action. The operation is
idempotent and preserves any existing CNI configuration.
Observed failure:
https://github.com/chroma-core/chroma/actions/runs/37825601610/job/113513898831?pr=7765
Validation: YAML parsing, prerequisite ordering, Bash syntax, and `git
diff --check` passed locally. Linux startup validation will run in this
PR's CI.
## Description of changes
Phase 3 of garbage collection could delete whatever plan it found in
gc/GARBAGE, trusting the manifest publisher's cached view and a
successful phase 2 response. An overlapping GC run could substitute a
newer plan, and cursor progress or a stale manifest could make phase 3
believe references were gone when they were not. Either path risked
deleting fragments still reachable by readers.
Bind the installed plan and its ETag into the phase-1 token:
- GarbageCollectionState now carries the plan and ETag; empty() is
unbound and rejected by phase 3, so sequential and UUID fragments both
require a successful phase 1.
- Phase 3 reloads gc/GARBAGE and refuses to proceed unless the plan and
ETag match the retained generation, restarting from phase 1 otherwise.
- Add ManifestPublisher::garbage_is_applied to read durable metadata;
the
Spanner impl queries fragment_regions directly so intrinsic-cursor
progress cannot authorize deletion, and the S3 impl checks the oldest
timestamp and dropped snapshots.
Update delete_unused_logs to thread an Option token and error if phase 1
produced no plan. Document the stricter contract in the README and add
gc_generation integration tests covering deterministic GC interleavings.
## Test plan
CI
## Migration plan
Backwards compatible.
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
Expose the existing single-region database count at `GET
/api/v2/tenants/{tenant}/databases_count`, using database-list
authorization and admission control. This lets the dashboard show a
total without listing every database.
Includes the generated JavaScript client and Rust 1.99 compatibility
fixes for async-trait and the atomic update call.
Validation: tenant isolation and create/delete count test passes
locally. CI passes, including JavaScript client tests, Rust feature
checks, Lint, and integration tests. The randomized index stress test
passed on rerun.
Required by https://github.com/chroma-core/hosted-chroma/pull/8457.
Deploy this endpoint before the dashboard count change. The existing
count RPC excludes topology-prefixed databases.
Forward limit and offset to the single-region SysDB even when a
secondary SysDB is configured. Return full pages directly; use a count
lookup when needed to continue pagination into the secondary results.
Validation:
- Reproduced the oversized gRPC response with 50,001 databases on the
old code; the regression test passes with the fix.
- Tested pagination across both sources, empty results, zero limits, and
unbounded requests.
- All 28 non-Kubernetes SysDB tests, Clippy, and formatting checks pass.
## Description of changes
Let SQLite choose the first dimension and reject conflicting writers
within the same transaction. Return the winning dimension through the
frontend without introducing collection or database mutexes.
Test concurrent initialization at the sysdb and frontend boundaries.
## Test plan
CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
---------
Co-authored-by: dbeglord <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
## Description of changes
Check persisted native dimensions before loading and reject mismatched
queries. Validate complete write batches before mutating the ID map,
tracking only touched IDs instead of cloning the entire collection.
Preserve the legacy sequence migration already present on main.
## Test plan
CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
---------
Co-authored-by: dbeglord <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
## Description of changes
This PR lands the HNSW resource-bounds validation from #7762 on `main`.
#7762 was reviewed and approved, but it merged into its stacked base
branch (`rescrv/security-fixes1`) instead of `main`, so its change is
not on `main` yet. #7761, the branch below it, had already merged to
`main`.
The change is Robert's, unchanged apart from a rebase onto current
`main`:
- **Validate HNSW resource bounds:** creation, updates, schemas and
legacy configuration loading all check index bounds before an index is
built. Invalid persisted data stays recoverable, and its pending vector
log is kept.
- **Bound generated HNSW settings:** the Python property tests generate
HNSW settings within those bounds.
The rest of Robert's security-fixes stack (#7763 and up) is based on
this branch.
## Test plan
CI. Both commits passed review in #7762.
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Robert Escriva <[email protected]>
## Description of changes
Reject filters deeper than 64 levels before recursive parsing can
exhaust the stack. Cover the accepted boundary and rejection for both
metadata and document filters.
## Test plan
CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
Dirty-log GC now discovers RLS nodes from the existing RLS memberlist
instead of scanning persisted dirty-log manifests. Manifests can outlive
their pods after a scale-down, causing GC to try nonexistent nodes such
as `rust-log-service-16`.
GC defers when the memberlist is empty. Cleanup is limited to 10
concurrent members, and a member failure no longer cancels cleanup for
other members. Failures are logged per member with a summary of the run.
Retired nodes’ stored logs remain untouched.
Validation: seven dirty-log GC tests (including MinIO-backed tests), six
memberlist tests, formatting, and commit hooks passed.
Safety follow-up: a separate overlapping-run test reproduced a
pre-existing WAL GC race in which phase three can reload a newer garbage
generation before its phase two completes. This PR does not fix that
race; the passing tests above do not establish safety against
overlapping runs.
## Description of changes
Manual GC of a soft-deleted collection accepts the request but can leave
the collection and its versions behind. The initial lookup includes
soft-deleted collections, while `get_collection_to_gc` hides them. The
scheduler interprets that empty result as `NoSuchCollection` and selects
the log-only cleanup fallback.
Include soft-deleted collections in the GC lookup so they reach the
normal collection/version cleanup path. Truly absent collections retain
the existing fallback behavior. Grace-period checks remain in the normal
GC path.
Reproduced against a production collection: the manual request completed
log-only cleanup, while a subsequent read-only SysDB query still showed
the soft-deleted row with two versions. Read-only GetCollections calls
returned an empty response with `include_soft_deleted=false` and the
collection with `true`.
This fix is independent of the scheduling reservation in #7821.
## Test plan
- Added a local SysDB gRPC integration regression covering live lookup,
database soft deletion, ordinary lookup hiding the collection, GC lookup
finding it, and missing/hard-deleted targets returning
`NoSuchCollection`.
- Confirmed the regression fails with `NoSuchCollection` before the fix
and passes afterward.
- All seven `sysdb::tests` passed, including the regression against the
local Kubernetes SysDB.
- Formatting and diff checks passed.
## Migration plan
No schema, RPC, or configuration change. Deploy the updated GC binary to
use the fixed client lookup. Production has not been deployed or retried
with this fix.
## Observability plan
After deployment, verify a manual request for a soft-deleted collection
enters the full `Garbage collection job` path rather than `Garbage
collection job (hard delete log)`, then verify the collection is removed
after successful cleanup.
## Documentation Changes
Added an inline explanation of why GC lookups include soft-deleted
collections.
Agent-query budget debits price only input and output tokens. Since the
planner started caching its prompt prefix (CHR-767), a large share of
each run's planner input arrives as cache reads and writes, which the
debit leaves out. In the last 7 days of prod events that was 1.65M
cache-read and 665k cache-write tokens.
foundation-api now prices each model's full usage:
- The price-card `TokenRate` copy
(`rust/foundation-api/src/budget/rates.rs`) gains optional
`cache_read_micros_per_m_tokens` / `cache_write_micros_per_m_tokens`. A
missing or `null` rate prices those tokens at zero, so the currently
served 2026-09-17 card still parses and debits exactly as before.
- `price_agent_usage` takes each model's `InferenceUsage` instead of an
`(model, input, output)` tuple, and the agent route passes its per-model
totals through unchanged.
The new card that sets planner cache rates ($0.36/M read, $4.50/M write)
is chroma-core/hosted-chroma#8434. Deploy sync with that card before
this PR. Either order is safe; this one just has no effect until the
card carries cache rates.
**Tests:** `cargo +1.92.0 test -p foundation-api budget` (11 pass,
including new cases for cache pricing and for parsing a card without
cache rates); clippy clean with `-D warnings`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <[email protected]>
Add `POST /api/v2/tenants/{tenant}/databases/batch/get` with an `ids`
array of at most 1,000 UUIDs. It returns matching active databases using
one Go SysDB query: `WHERE tenant_id = ? AND id IN (...) AND is_deleted
= false`. Missing IDs are omitted; duplicate IDs produce one result.
Empty input returns an empty array.
The endpoint requires tenant-wide `list_databases` permission before
reading storage. Both HTTP and gRPC reject oversized batches. MCMR is
unsupported; its server only has the required unimplemented RPC stub.
SQLite supports the same lookup for single-node use.
Used by https://github.com/chroma-core/hosted-chroma/pull/8433. Deploy
Go SysDB, then the data-plane frontend, before deploying that caller.
Validation: PostgreSQL lookup test, gRPC batch-limit test, frontend HTTP
test, and Rust compilation checks. Tests cover tenant isolation, soft
deletion, duplicate/missing IDs, empty input, and oversized batches.
Database-creation quota checks currently fetch every database row for a
tenant just to count them. Add an internal `CountDatabases` RPC and Rust
client method that return a scalar instead.
Go SysDB executes `SELECT COUNT(*) FROM databases WHERE tenant_id = ...
AND is_deleted = false`. The frontend client calls only Go SysDB; MCMR
counting is intentionally out of scope. The Rust SysDB server has only
the unimplemented stub required by the shared generated service trait,
with no Spanner or backend changes. SQLite and the in-memory test
backend implement the shared client interface. No public HTTP route or
schema migration is needed.
The caller change is
https://github.com/chroma-core/hosted-chroma/pull/8432. Merge this first
and deploy the new RPC on Go SysDB before deploying that caller. No
enumeration fallback is introduced.
Validation:
- Real PostgreSQL DAO test passes: empty tenant, active rows, soft
deletion, tenant isolation, and count/list agreement.
- Go coordinator and gRPC packages compile.
- `cargo check -p chroma-sysdb -p rust-sysdb` passes.
- Formatting and commit hooks pass.
- Focused SQLite count test passes (empty tenant, populated tenant,
isolation, deletion).
Anyone who copies one of our Claude code samples today gets a `404
not_found_error`. The samples use `claude-sonnet-4-20250514`, which
Anthropic retired on 2026-06-15. This PR moves all six references to
`claude-sonnet-5`. They're in the Package Search MCP page (Python and
Go), the building-with-AI guide (Python and TypeScript), and the
intro-to-retrieval guide (Python and TypeScript).
Two samples needed more than a model-id swap:
- **Package Search MCP (`cloud/package-search/mcp.mdx`).** These now use
the current MCP connector beta, `mcp-client-2025-11-20`. It requires a
`tools: [{type: "mcp_toolset", mcp_server_name: "package-search"}]`
entry that references the server. The Go sample also sets the beta
through the `Betas` request field instead of a raw header, and drops the
`tool_configuration` block that the older beta used. I checked the Go
type names (`BetaMCPToolsetParam`, `OfMCPToolset`,
`AnthropicBetaMCPClient2025_11_20`, `ModelClaudeSonnet5`) against the
current `anthropic-sdk-go` source.
- **Name extractor (`guides/build/building-with-ai.mdx`).** Sonnet 5
uses adaptive thinking by default, so `content[0]` can be a thinking
block. The Python and TypeScript samples now take the first `text` block
instead. I raised `max_tokens` to 4096 in the samples that produce
longer output, to leave room for thinking.
Same fix for our own MCP smoke tests: chroma-core/hosted-chroma#8422.
**Validation:** docs-only change. I checked the snippets against the SDK
sources, but I haven't run them.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
Moving existing Foundations into a new product-record table requires a
stable list of Foundations. If customers can create another Foundation
while that list is being registered, the new Foundation may be missing
when the service begins requiring a product record for access.
This change adds a temporary pause on Foundation creation so operators
can complete that registration without stopping ordinary memory reads
and writes. When the pause is enabled, both initialization of the
default Foundation and creation of a named Foundation return HTTP 503
before modifying storage or product records. The shared creation
function enforces the pause for every authorized caller, including
internal callers.
The pause is disabled by default. Operators must deploy the setting to
every service instance and allow requests already running on older
instances to finish before taking the final inventory. Reading existing
product records and ordinary memory operations remain available.
### Production preparation
The production service currently runs an older version that does not use
the new product-record table. It needs the same pause before the
migration starts. A [backport for that deployed
version](https://github.com/chroma-core/chroma/compare/5c4f800858336752b88d3fa70434483b23db82fc...dff1d8aea6adfc5abb0e7d710f3fb41db093288d)
is prepared and tested.
Merge this mainline change before promoting the backport through the
normal release process. Build the compatible image, record its image
digest, and rehearse the pause in staging before requesting production
release approval. After the pause is active, register the existing
Foundations, deploy and verify the service that requires those records,
then reopen creation.
Neither environment has been deployed or paused by this work.
### Validation
The two pause/configuration tests and all 20 product-record lifecycle
tests pass; these groups overlap. Formatting and whitespace checks pass.
The production backport's full Foundation library suite passes with 223
tests and two ignored. Its tests verify refusal before storage changes
and successful page reads and trajectory writes while paused;
trajectories are stored records of agent runs.
The setting is `foundation.provisioning_paused` in the Foundation API
configuration, or `foundation.foundation.provisioning_paused` in the
hosted service configuration. The affected routes are `POST /api/init`
and `POST /api/tenants/{tenant}/foundations`. Build the production
backport with Chroma source `dff1d8aea6adfc5abb0e7d710f3fb41db093288d`
and hosted source `5a61059205a307b8f3afe634d546fe5bdf836273`.
A failed metadata read must fail its request without crashing the
database metadata service. In staging, the compactor has a one-second
metadata request timeout. A matching request runs for 1,001.65 ms and
fails at the same instant the server logs a canceled first-row scan. The
canceled read leaves the first segment unset; continuing after that
error dereferences a nil pointer and terminates the Go sysdb process.
Other requests then fail while the service restarts.
Return immediately when scanning a segment row fails, and check for
errors after row iteration. Both paths return no partial results. This
prevents the crash; it does not change request timeouts or resolve the
source of the cancellation.
Validation: four regression cases cover first-row and later-row scan and
iteration failures. The first-row scan case reproduces the panic before
the fix. The regression tests and existing PostgreSQL-backed segment
tests pass with race detection.
## Description of changes
`/api/agent` accepted `claude-opus-4-5-20251101`, but no Orb metric and
no Foundation price-card rate matches that id: such a run bills $0 in
Orb and debits the internal budget at the Sonnet planner rate while
costing 5x per token (CHR-768). No Opus event has ever been sent
(verified in Orb 2026-09-23), so this closes the hole before anyone hits
it.
- Improvements & Bug fixes
- Drop `Opus4_5` from `AnthropicModel`: the enum now carries only priced
snapshots, and the route's existing parse turns any other id into a 400.
- The route test that used Opus as the explicit non-default model now
passes an upcased Sonnet id instead; the config test asserts the Opus
wire id is rejected.
**Product question this PR surfaces but does not decide:** should Opus
be offered on `/api/agent` at all? If yes, it needs Orb metrics + prices
+ a price-card rate first — that rides the CHR-769 version cut, and the
variant comes back with it.
## Test plan
- [x] `cargo test -p chroma-agent --lib` (30 passed)
- [x] `cargo test -p foundation-api --lib agent` (40 passed)
- [x] `cargo clippy -p chroma-agent -p foundation-api --lib --tests`
clean; `cargo fmt` applied
## Migration plan
None. The only caller that can pass a model is the `foundation agent
--model` CLI flag, which defaults to unset (server picks Sonnet); its
help text cites the Opus id as an example and gets a follow-up docstring
fix in the foundation repo.
## Observability plan
Rejected ids surface as 400 `UnknownModel` from the route, same as any
unknown string today.
## Documentation Changes
None.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5 <[email protected]>
foundation-api half of the agent-query budget debit. After each
`/api/agent` response stream completes — right where
`submit_search_agent_usage_events` posts the meter events — the run's
aggregated per-model token usage is priced against the Foundation price
card and posted as one debit to the sync-frontend's `POST
/foundation/budget-debit` (chroma-core/hosted-chroma#8404). The post
runs on a detached `tokio::spawn` task with the querying caller's own
`x-chroma-token`, so a slow sync, a failed debit, or a missing budget
row never touches the user's answer or the request latency; every
failure is a `warn!`, mirroring the meter events' posture.
**Pricing source:** the card is fetched from `GET
/foundation/price-card` through an in-process cache honoring the
response's `Cache-Control: max-age` (the endpoint serves 5 minutes); on
a failed refresh the last known rates serve stale, and with nothing
cached the debit is skipped with a warn — if sync is unreachable for the
card, the debit POST to the same host would fail anyway, so a
config-fallback rate table would not buy anything. Rates are micro-USD
per million tokens with the card's own integer floor division; the
request's planner model prices at the `planner` rate and all other usage
in a run (the context-1 deep-research subagent) at `context_1`. Cache
read/write tokens stay unpriced, matching the Orb meters (CHR-767 owns
changing that mix).
`ref_id` is the request's trace id (a fresh UUID when tracing is off, so
untraced runs don't all collapse onto the zero trace id) — the counter's
global `ref_id` key makes a re-post of the same run a no-op.
**Config:** new `foundation.sync_frontend_url` (e.g.
`https://sync.trychroma.com`). Absent means debits are disabled and
local/OSS deployments never call out. The Tilt value landed with the
sync half; the staging/production `configuration.yaml` values follow in
chroma-infra once this deploys.
## Testing
- `cargo +1.92.0 check` / `clippy` / `fmt` clean on `foundation-api`;
full crate suite passes (233 tests).
- New `budget::tests`: pricing math (planner vs context-1 mapping, floor
rounding), httpmock coverage of the debit POST shape (`spend_class:
"query"`, `source: "agent_query"`, priced `amount_micros`), card caching
across debits (one GET, two POSTs), zero-amount skip, and an
unreachable-sync run that drops quietly.
Linear: CHR-773
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5 <[email protected]>
MCP clients can select a Foundation through `/mcp/tenants/{tenant}/foundations/{foundation}`. The existing `/mcp/foundation` endpoint continues to select the caller's default Foundation and advertises the existing OAuth resource.
## Request scope and authorization
The named mount validates the path, authorizes the caller against its tenant and database, and stores the scope in request extensions. Each tool uses that request's scope. In the integrated catalog implementation (#7713), both mounts check the registered Foundation and exact backing database UUID before reaching storage. The bare mount resolves the caller's default scope; the named mount retains the explicitly requested pair. Memory operations continue through the frontend with the caller's token.
The named endpoint is registered as a routed service so the MCP library sees the original URI and Host header. Authentication middleware applies to the MCP routes, preserving ordinary 404 responses for unrelated paths.
## Client behavior
The default endpoint includes its OAuth discovery challenge for missing or rejected credentials, while permission refusals return 403 without triggering another token refresh. The named endpoint never advertises the default endpoint's discovery document. Its 403 response carries an insufficient-scope challenge without a discovery URL; a missing or rejected token returns 401 with `WWW-Authenticate: Bearer realm="foundation"`, without a `resource_metadata` parameter. Discovery continues to advertise the default endpoint.
Tenant and Foundation allowlists reject characters that could change a downstream URL's structure. Page links are omitted for non-default Foundations because the web interface remains on the default. The existing catch-all ingress can route this path without a new ingress rule.
## Validation and rollout
The named-endpoint authentication challenge fix passes all 40 MCP tests, including missing and rejected credentials, unchanged 403 responses, and absence of default-resource metadata on named endpoints. Foundation API Clippy (`--lib --tests -- -D warnings`), formatting, and commit hooks pass.
The explicit-route update passes all 318 Foundation API tests (two live tests ignored), including the API and MCP routing and catalog-identity checks. The previous integration also passes 24 sysdb and six worker HTTP-generation tests; this route-only change leaves those implementations unchanged. Clippy passes for Foundation API and worker with all targets/features and `-D warnings -D clippy::large_futures -D clippy::all`. Formatting and commit hooks pass. These are local integrated-stack results; remote CI and deployed acceptance remain separate gates.
The integrated suite exercises actual tool requests through both mounts, including rejection when a database name resolves to a replacement UUID and refusal when the catalog is unavailable or unconfigured.
The API routing dependency (#7712) is merged. The open Chroma stack is catalog-backed lifecycle (#7713) → named MCP (#7714). The MCP branch directly follows catalog-backed CRUD and preserves its catalog guards. Every service that checks Foundation permissions must name its database before hosted-chroma #8382 begins issuing database-scoped Foundation grants. The product table, reviewed backfill, and hosted adapter must be ready before catalog-enforced consumers roll out. Explicit-write enforcement remains permissive until writers migrate.
The [living project tracker](https://app.notion.com/p/3e258a6d81918162b7dfdc1f3274815c) records the remaining CI and deployed acceptance gates.
A benchmark dataset file is built once and cached: a callback writes the contents to a temporary file, that file is renamed to the cached path, and every later run reads the path instead of rebuilding.
## The defect
The callback was handed a file handle and trusted to have finished with it when it returned. A handle does not finish its pending writes just because it goes out of scope. Tokio's own documentation says so directly: a file is not closed immediately while it has operations that have not completed, and you should flush it before dropping it.
So the rename could publish a file whose tail had not been written. The next reader gets a short file rather than an error, which surfaces later as a deserialization failure with no obvious connection to the cause.
It depends on a write still being in flight when the handle is dropped, so it appears under load and not on an idle machine.
## Why it is worth fixing now
It is the cause of a `Rust tests` failure seen on several open pull requests, in `test_frozen_query_subset`, which writes a cache file and immediately reads it back and reports `unexpected end of file`. That test passes locally on an idle machine, on the default branch and on the branches where continuous integration failed.
All eight call sites had the same omission, which is the shape of a problem that wants fixing in one place rather than eight.
## The change
The callback now returns the writer it was given, and the helper flushes it before the rename. Returning the writer is the point: it moves the obligation from every callback to the one function that owns the file's lifecycle, so a new caller cannot omit it. The eight call sites each return their writer instead of the unit value.
## On the tests
The two tests added cover the round trip, not the race. A test cannot force a pending write to still be in flight, so on an idle machine they pass with or without the flush, and I have verified that rather than assuming it. The justification for the flush is the file handle's documented contract, not a failing test. I would rather say that than imply coverage this does not have.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Description of changes
A Foundation has a product identity and an immutable association with its backing Chroma database. This PR provides catalog-backed creation, listing, description, and memory access. The product catalog lives in dashboard-api's application database and is supplied through an injected registry interface; Chroma's system database stores the backing database, collections, and attached functions.
Creation and listing use `/api/tenants/{tenant}/foundations`. Description uses `/api/tenants/{tenant}/foundations/{foundation}`. Creation checks database-creation and Foundation-initialization permissions, reserves both identities before provisioning, and marks the Foundation ready only after all required storage and attached functions are complete. Retries resume the same reservation and UUIDs. Default initialization can adopt a verified database created by a first-party client; named creation rejects an arbitrary existing database. Interrupted attachment finalization resumes without restarting ready inputs.
Listing reads authorized product records and applies pagination. Every memory request, including default API aliases and the default MCP endpoint, verifies the catalog record and exact backing database UUID before accessing storage. Cached collection handles cannot bypass this check. Record operations continue through the Chroma frontend with the caller's token. Missing catalog access is an error; it cannot become an empty listing or an inferred Foundation. The standalone entrypoint requires an injected registry for catalog-dependent operations.
The companion product table and hosted registry adapter are in [hosted-chroma #8408](https://github.com/chroma-core/hosted-chroma/pull/8408). This branch directly follows API routing #7712. Named MCP routing #7714 is its child and retains the same catalog identity checks.
## Test plan
This standalone CRUD branch passes all 290 Foundation API tests, with two live tests ignored. Strict Clippy passes for Foundation API and worker with all targets/features and the CI warning, large-future, and all-Clippy lint flags. Formatting and commit hooks pass.
The combined tree at `ac541188702af60829736272fca338c03b8f6f9b` is identical to the verified explicit-route tree: 318 Foundation tests pass, two live tests are ignored, and strict Clippy passes for Foundation API and worker with all targets/features plus `-D warnings -D clippy::large_futures -D clippy::all`. Coverage includes complete provisioning, repeat reservations, authorization and pagination, exact identity checks, catalog completion failure, and interruption at all seven attachment-finish boundaries. The earlier integration also passes 24 sysdb tests and six worker payload tests.
Remote CI and deployed two-end-user ingestion/generation acceptance remain separate gates.
## Migration plan
Deploy and populate the product catalog before rolling catalog-enforced consumers. Final inventory/backfill and consumer rollout require a controlled pause in initialization and named creation. Preserve the catalog during rollback.
Deploy every database-aware permission consumer before hosted-chroma #8382 begins issuing database-scoped Foundation grants. Keep explicit-write enforcement permissive until writers migrate. Public Foundation deletion and name reuse remain outside this change. Name-addressed storage requests cannot make the UUID check and following operation atomic against out-of-band deletion/recreation.
## Observability plan
Catalog and storage failures remain request errors. Provisioning retains progress logs. The control-plane maintenance commands report inventory, reviewed backfill, and reconciliation outcomes. Production reconciliation scheduling and live verification remain rollout work.
## Documentation Changes
The Foundation API README describes the registry contract and endpoints. The companion [rollout runbook](https://github.com/chroma-core/hosted-chroma/blob/dbld/foundation-catalog/docs/foundation-catalog.md) covers identity, credentials, migration, reconciliation, and rollback. The [living tracker](https://app.notion.com/p/3e258a6d81918162b7dfdc1f3274815c) records the three-PR Chroma stack, companion services, and remaining gates.
A tenant can contain multiple private memory resources, each backed by its own Chroma database. This PR lets API callers select a tenant and Foundation explicitly. Foundation product identity is supplied by the catalog in #7713.
## What changes
Foundation memory routes use the explicit tenant/Foundation hierarchy: `/api/tenants/{tenant}/foundations/{foundation}/...` alongside the existing paths. Shared scope resolution reads the tenant and database from the path, or resolves the caller's tenant and configured default for an unprefixed request. Authorization receives that exact pair. Prefixed requests avoid the identity lookup needed by the default alias.
Collection lookups and cache keys carry the database name. Cached models whose tenant or database disagrees with the request are rejected and fetched again. The generation worker includes the source database name in every outgoing record batch so the generator can write pages to the correct Foundation.
Path components use allowlists because they become parts of downstream URLs. Tenant names require alphanumeric ends and allow only ASCII letters, digits, dots, underscores, and hyphens. Foundation names also enforce the database naming rules. Encoded separators cannot survive into another decoding step.
## Deliberate choices
Initialization keeps its unprefixed registration; named creation and catalog provisioning are handled directly by #7713. A configuration flag can require explicit Foundation prefixes for writes and defaults to permissive. Page links remain available for the configured default Foundation; other Foundations omit links because the web interface cannot address them. Rate limits remain per tenant.
The hosted authorization implementation checks the caller's tenant and permissions. The standalone authorization implementation remains permissive; the integrated catalog consumer requires an explicitly injected registry and otherwise returns an unavailable error for catalog-dependent operations.
## Validation and rollout
The explicit-route update passes all 318 Foundation API tests (two live tests ignored), including the API and MCP routing and catalog-identity checks. The previous integration also passes 24 sysdb and six worker HTTP-generation tests; this route-only change leaves those implementations unchanged. Clippy passes for Foundation API and worker with all targets/features and `-D warnings -D clippy::large_futures -D clippy::all`. Formatting and commit hooks pass. These are local integrated-stack results; remote CI and deployed acceptance remain separate gates.
The Chroma stack is #7712 → #7713 → #7714, rebased on main `0dad2c71e`. The combined MCP head is `ac541188702af60829736272fca338c03b8f6f9b`; its tree is identical to the verified explicit-route implementation. This branch includes the base API route change; the lifecycle and MCP branches extend the same hierarchy. Every service that checks Foundation permissions must name its database before hosted-chroma #8382 begins issuing database-scoped Foundation grants. The product table, reviewed backfill, and hosted adapter must be ready before catalog-enforced consumers roll out. Explicit-write enforcement remains permissive until writers migrate.
The [living project tracker](https://app.notion.com/p/3e258a6d81918162b7dfdc1f3274815c) records dependencies and remaining rollout gates.
`ExponentialBackoff::next` backs off over the time since the backoff was
constructed. It scales that elapsed time by the ratio of throughput to
reserve capacity, then by a random fraction below one.
## The defect
The test asserted that the first three values were each under one
second. That holds only while construction and the calls stay within a
few milliseconds of each other, because the elapsed time is itself the
quantity being scaled. A machine that schedules the test slowly raises
the value the assertion is made about, so the test fails for a reason
unrelated to the code it covers.
It failed that way on the `Rust tests` job of an unrelated pull request,
with `assertion failed: exp_backoff.next() < Duration::from_secs(1)`.
## The change
Assert the bound the implementation actually guarantees: a returned
value never exceeds the scaled elapsed time. The bound is sampled after
the call so it stays an upper bound, since elapsed time only grows.
The second half of the test, which sleeps and then checks the mean of a
hundred samples, is untouched. Its elapsed time is dominated by the
sleep, so it does not have this problem.
## On whether the test still catches anything
I checked rather than assumed. Making `next` return more than its
recovery window makes this assertion fail:
```
backoff 3.163ms exceeded the scaled recovery window 1.66042ms
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 <[email protected]>
## Description of changes
Enable serde_json's float_roundtrip feature in the log crate so
metadata float values survive the SQLite log JSON round trip
exactly. The default parser drops a bit of precision, which
causes equality filters to miss records after log replay.
Add a regression test and a proptest regression case covering the
exact-float round trip.
## Test plan
CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
## Description of changes
- Add an optional MDAC token bucket to WQS `GetWork`.
- Charge one token for each work item returned while preserving existing
FIFO, failure filtering, and rendezvous-shard assignment.
- Define `limit` as the maximum distinct function IDs and add
`max_items` as the independent total-record cap.
- Add `excluded_fn_ids` to `GetWorkRequest`; fn-consumers populate it
from active functions and WQS skips those rows before charging tokens.
- Return relative gRPC pushback on empty 429 responses and an absolute
UTC `retry_at_unix_ms` deadline on partial successful responses, and log
delayed fn-consumer polls.
- Reject invalid limiter configuration cleanly during startup.
- Validate `max_concurrent_workers` at 55,188 so worst-case exclusion
IDs use at most half of WQS's explicit 4 MiB gRPC decode limit.
- Emit `work_queue_get_work_rate_limited_count` and structured throttle
logs.
This carries the exclusion contract from #7586 onto current `main` and
composes it with the limiter. Token-bucket pacing is used here because
this bounds new work handed to consumers; fn-consumer already owns the
separate in-flight concurrency limit.
## Test plan
- [x] `cargo fmt --check`
- [x] `cargo test -p worker work_queue::config::tests --lib`
- [x] `cargo test -p worker work_queue::work_queue_manager::tests --lib`
- [x] `cargo test -p worker work_queue::work_queue_server::tests --lib`
- [x] `cargo test -p worker fn_consumer::fn_consumer_manager::tests
--lib`
- [x] `cargo test -p worker fn_consumer::config::tests --lib`
- [x] `cargo test -p mdac`
- [x] `cargo clippy -p mdac -p worker --lib -- -D warnings`
- [x] `git diff --check`
The manager regression test uses a one-token bucket with an excluded
in-progress function at the head and ready work behind it, verifying the
ready function receives the token. Config tests verify the protobuf UUID
wire-size assumption and both sides of the concurrency bound. Retry
tests cover absolute-deadline conversion, stale deadlines, millisecond
rounding, and the token bucket timestamp horizon.
## Migration plan
The protobuf additions are backward-compatible. The limiter is optional
and disabled when configuration is absent. chroma-core/k8s#3217 enables
it with a 100-item burst and one token per 100 ms (10 items/second)
after this image is available. Removing the config block disables it.
Existing fn-consumer values are far below the new 55,188 maximum.
Configurations above that value now fail deserialization rather than
risking an oversized `GetWork` request.
The limiter is process-local, which is global with the current single
WQS replica. Additional WQS replicas would each receive their own
allowance. Exclusions are advisory and request-scoped, preserving
at-least-once delivery without durable leases.
## Observability plan
Monitor `work_queue_get_work_rate_limited_count`, `returned_items`,
`excluded_fn_count`, and the structured `rate_limited` field. Watch for
shard starvation or a growing queue during staging rollout.
## Documentation changes
No user-facing API changes.
## Summary
- disable Parquet statistics for the opaque wal3 body column
- retain statistics for offsets and timestamps
- add a large incompressible-record regression test covering fragment
size, round-trip decoding, and setsum verification
Parquet's page-statistics default copies a single large binary value
into min/max metadata multiple times. A focused probe reduced a roughly
2 MB one-record fragment from 10.24 MB to 2.05 MB. Existing fragments
remain readable because this changes writer properties only.
S3 request count is unchanged, while fragment storage and transfer bytes
drop substantially for large records.
## Tests
- cargo test --manifest-path rust/wal3/Cargo.toml --lib writer::tests
- cargo test --manifest-path rust/wal3/Cargo.toml --lib
interfaces::tests::checksum_parquet
- cargo clippy --manifest-path rust/wal3/Cargo.toml --lib -- -D warnings
- cargo fmt --manifest-path rust/wal3/Cargo.toml -- --check
Tilt-backed integration tests were not validated locally because the
required services were not active.
---------
Co-authored-by: Robert Escriva <[email protected]>
## Description of changes
Storage consumers currently pull in `chroma-tracing`, `fastrace`, and
`fastant` solely to measure operation latency. Move the stopwatch into a
shared `chroma-metrics` crate, whose only normal dependency is
OpenTelemetry, and have storage use it directly.
`chroma-tracing::util` re-exports `Stopwatch` and `StopWatchUnit`,
preserving existing caller paths and `elapsed_micros()`. There is one
implementation shared by storage and tracing.
Explicit `finish()` now records exactly one histogram sample instead of
recording again on drop. Implicit drop continues to record one sample.
## Test plan
- Stopwatch regression tests: explicit completion, implicit drop, and
elapsed-time reads without extra samples.
- Storage library tests pass with the 27 Tilt-dependent tests skipped.
- Documentation tests for metrics, storage, and tracing.
- Clippy for all three crates, all targets and features, with warnings
and repository lint checks denied.
- `cargo fmt --all -- --check` and `git diff --check`.
- Verified the normal storage dependency tree excludes `chroma-tracing`,
`fastrace`, and `fastant`.
## Migration plan
No caller migration is needed: the existing tracing stopwatch API is
re-exported from the shared crate.
## Observability plan
Existing histogram names, attributes, and units are retained. Explicit
completion contributes one sample rather than two for both storage and
tracing callers.
## Documentation Changes
The shared crate and its public stopwatch API have Rust documentation.
## Description
Expose append contention as a structured `AppendOutcome` instead of
requiring callers to interpret `LogContention*` errors as commit status.
The additive `append_with_options_outcome` API distinguishes:
- committed appends
- durable publication contention
- retryable contention
- indeterminate durability
Non-contention failures remain in the existing error path, and the
legacy append API remains unchanged. This change applies directly to
`main` and does not depend on #7644 or #7645.
## Testing
- `cargo test -p wal3 append_result_preserves_contention_semantics
--lib`
- Casita RustFS racing-writer and racing-initializer integration tests
## Description of changes
The agent loop re-sends the whole conversation on every planner
iteration, and nothing in the request path ever asked for caching.
`parse_anthropic_usage` reads `cache_read_input_tokens` off the
response, so the `search_agent_usage` meter events have been honestly
reporting zeros — every iteration re-paid full input price for context
the previous iteration had already sent.
This sets
[`cache_control`](https://platform.claude.com/docs/en/build-with-claude/prompt-caching#automatic-caching)
at the **request root**. Anthropic places the breakpoint on the last
cacheable block and advances it as the conversation grows, which is
exactly the shape of an agent loop.
I first wrote this as two hand-placed block-level breakpoints — one
closing the static prefix, one rolling with the conversation. The
request-level marker is strictly better:
- no bookkeeping as the trajectory grows, so nothing to get wrong when
the message shape changes
- leaves all four block-level breakpoint slots free rather than spending
two
- handles blocks that cannot carry a marker on their own (it walks back
past `thinking` rather than erroring), so the special-casing goes away
- the static-prefix breakpoint was worthless anyway: system prompt plus
three tool schemas is ~530 tokens, under the 1024-token minimum
cacheable prefix
No beta header or `anthropic-version` bump needed; supported on all
active models. The one exclusion is the legacy Amazon Bedrock
integration, which returns 400 for a top-level `cache_control` — not a
path we use, since this calls `api.anthropic.com` directly.
### Sizing
From 195 production questions (`foundation_agent.infer` spans, 30 days),
planner input cost lands at **0.507** of today's — a ~2x saving, not the
10x the ticket estimated. 10x is the asymptote as the final context
shrinks relative to total input, and these conversations grow too fast
for that. Median question is 5 iterations and 150k input tokens.
The 5-minute TTL was chosen on measurement, not default: the 1-hour TTL
reaches only 0.773 because writes cost 2.0x instead of 1.25x, and of
2,062 `foundation_agent.act` spans over 30 days exactly one exceeded
five minutes. Twice the write price to cover a 0.3% tail.
`request_diagnostics` reports the request-level marker, so a run that
silently stopped asking for caching shows up in the error log.
**Billing:** the four Orb metrics for these tokens exist and are
correct, but are priced on no plan. Until CHR-769 lands, cached tokens
bill customers nothing — this lowers bills before it lowers them
deliberately. CHR-769 is sequenced after this change on purpose: with
the cache fields at zero today, a metric whose filter is wrong is
indistinguishable from one that is right, and this change is what makes
them verifiable.
Closes CHR-767.
## Test plan
`cargo test -p chroma-agent --lib` — 30 pass, including new coverage
that the marker is set once at the root and that no content block or
tool carries one. `cargo clippy -p chroma-agent --all-targets` and
`cargo fmt --check` clean. `cargo check -p foundation-api` builds.
Not verified against the live API — the acceptance check is a production
run showing `cache_read_tokens` non-zero in an emitted
`search_agent_usage` event.
## Migration plan
N/A
## Observability plan
Already covered: `foundation_agent.infer` spans record
`gen_ai.usage.cache_read_tokens` and `cache_write_tokens`, so the
realised ratio can be compared against the predicted 0.507 without new
instrumentation.
## Documentation Changes
N/A
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 <[email protected]>
## Description of changes
`rust/agent/src/inference/anthropic.rs` was 757 lines and about to pass
900 with the prompt-caching change stacked on top of this
([#7744](https://github.com/chroma-core/chroma/pull/7744)), so it gets
split by request lifecycle first.
| file | contents | lines |
| --- | --- | --- |
| `config.rs` | model snapshots, `anthropic-beta` flags, request knobs |
172 |
| `request.rs` | builds the Messages API body | 65 |
| `response.rs` | parses the response into an `Action` plus token counts
| 192 |
| `diagnostics.rs` | renders a log-safe copy of a failed request | 201 |
| `mod.rs` | the client struct, its constructors, and the infer loop |
202 |
**This is a pure move.** The only content changes are six items widening
to `pub(super)` because they now cross a module boundary
(`request_body`, `header_value`, `request_diagnostics`, both `parse_*`
fns, and the shared `weather_toolset` test helper), plus one signature
rustfmt reflowed. Verified by diffing every non-import, non-comment line
against the original — the six signature changes and the new module
scaffolding are the entire delta.
The six names the crate re-exports are untouched, so `foundation-api`
needs no change; `cargo check -p foundation-api` confirms it.
`mod.rs` and `diagnostics.rs` land 1–2 lines over the 200-line guideline
because their tests sit in-file. Moving tests into a `tests/` directory
— the pattern `foundation-api` already uses in `routes/agent/tests/` —
would bring both under, but that's a larger reshuffle and didn't seem
worth bundling here.
## Test plan
`cargo test -p chroma-agent --lib` — 28 pass, the same 28 as before the
split, with each test moved next to the code it exercises. `cargo clippy
-p chroma-agent --all-targets` and `cargo fmt --check` clean. `cargo
check -p foundation-api` builds.
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 <[email protected]>
## Summary
- forward `limit` and `offset` to the Go SysDB when no MCMR client is
configured
- return the already-paginated Go SysDB response without client-side
slicing
- add stable `created_at, id` ordering and a matching Postgres list
index
- preserve the existing MCMR merge behavior
## Why
The Rust SysDB client currently requests every database from the Go
SysDB and paginates in memory. That makes a bounded `ListDatabases` call
transfer all tenant database rows. The Postgres query also lacks an
index matching its tenant/deletion filters and ordering.
## Validation
- `cargo test -p chroma-sysdb list_databases_`
- `cargo check -p chroma-sysdb`
- `go test ./pkg/sysdb/metastore/db/dao -run ^'$'` (compile-only)
- `atlas migrate validate --dir file://migrations`
The focused database-backed Go test was added but could not run locally
because Docker is unavailable.
## Summary
- extend the existing GetDatabase request with an optional database ID
selector
- implement the ID selector only in the Go SysDB, backed by the database
primary key
- expose a GET database-by-ID endpoint in the Rust frontend
- retain tenant isolation by returning not found for cross-tenant IDs
- leave the MCMR rust-sysdb implementation unchanged
## Why
Cold database-scoped API-key verification currently resolves an ID by
listing every database in the tenant. This adds the constant-time
primitive needed to remove that tenant-wide scan for single-region
databases.
## Validation
- cargo check -p chroma-sysdb -p chroma-frontend -p rust-sysdb
- cargo test -p chroma-sysdb test_get_database_by_id_is_tenant_scoped
- go test ./pkg/sysdb/coordinator -run
TestCatalog_GetDatabaseByIDIsTenantScoped
- go test ./pkg/sysdb/grpc ./pkg/sysdb/coordinator -run ^$
The Docker-backed Go integration suites were not run because Docker is
unavailable in the local environment.
## Description of changes
Replace the mutable minio/minio:latest tag from Docker Hub with a
pinned quay.io digest for MinIO RELEASE.2025-09-07T16-13-09Z. Pulling
from quay.io avoids Docker Hub rate limits, and the digest pin makes
test runs reproducible instead of tracking a moving latest tag.
## Test plan
Fixes CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
## Description of changes
Wire the token bucket rate limiter into the local Tilt stack and the
distributed-chroma Helm chart so it runs alongside the other services.
- Add an mdac_service target to the Dockerfile that ships the
token_bucket_service binary and exposes port 8000.
- Add build/CI plumbing: docker-bake target, Tiltfile custom/docker
build, resource, port-forward (8002:8000), and group membership.
- Add a Helm template (ConfigMap, Deployment, Service) gated on
mdacService.enabled; run a single Recreate replica since token
buckets are process-local, and disable service links to avoid
MDAC_ env collisions.
- Enable mdacService in values.dev.yaml; default it off in values.yaml.
- Log the resolved config path, file contents, and MDAC_ env overrides
at startup to aid debugging deployed instances.
- Document the Tilt workflow in the mdac-service README.
## Test plan
CI
## Migration plan
N/A
## Observability plan
Improved: now logs config on start
## Documentation Changes
N/A
Co-authored-by: AI
## Description of changes
The put-back-and-drain endpoint now accepts either a single update
object (unchanged) or an array of updates via an untagged UpdatesRequest
enum. Array requests return HTTP 200 with a per-item result carrying
admitted and a numeric status (200, 429, or 404) in request order.
The whole body is validated before any bucket is mutated, and updates
run in order and continue after failures rather than aborting as a
group. Repeated names observe earlier updates within the array. The
existing 1 KiB body limit applies to the entire array, and an empty
array returns [].
Also refresh config/modal-main.yaml with per-provider
foundation-research
limits, document the array form in the README, and extend the HTTP tests
to cover single objects, validation failures, mixed outcomes, and the
empty array.
## Test plan
N/A
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
## Description of changes
- Rename `rust/modal.py` so it no longer shadows the installed `modal`
package.
- Update the deployment definition for Modal 1.2.6 (`context_dir`,
`@app.cls`, and `@modal.web_server`).
- Add Python to the Dockerfile-derived runtime image so Modal can launch
it.
- Keep a single warm container because token-bucket state is
process-local.
- Add a GitHub Actions workflow that deploys relevant changes on `main`
to Modal's `main` environment, with a manual dispatch fallback.
- Add a checked-in Modal main configuration containing `bucket-0`,
`bucket-1`, and `bucket-2`, each with capacity 100 and a 100 ms refill
interval.
- Copy the Figment config into `/config.yaml` in the runtime image.
- Remove the Docker `ENTRYPOINT` that prevented Modal's Python runner
from initializing; the deployment class starts the Rust service.
The workflow expects `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` to be
provisioned through Chroma's managed secrets process. No
repository-level deploy credentials are added by this PR. The existing
`MODAL_KEY` and `MODAL_SECRET` runtime infrastructure path remains
unchanged.
## Test plan
- Imported `rust/deploy_mdac.py` using Modal 1.2.6.
- Started `token_bucket_service` with `config/modal-main.yaml` and
verified health plus admitted requests for all three buckets; an unknown
bucket returned HTTP 404.
- `cargo test -p mdac-service` (7 tests passed).
- Passed repository pre-commit checks, including YAML validation, Black,
Flake8, and mypy.
## Migration plan
N/A
## Observability plan
The Actions run and Modal image-build logs record deployment status. The
Modal config enables stdout tracing.
## Documentation Changes
Documented local staging deployment, the remote Cargo/config image
build, and automatic deployment after relevant changes merge to `main`.
## Description of changes
Introduce mdac::TokenBucket, a thread-safe drainable GCRA rate
limiter. It starts with a full burst, refills lazily against a
monotonic clock with nanosecond precision, and supports atomic
put-back-and-drain so callers can refund unused work. Operations
use a lock-free compare-and-swap loop and never block on refills.
Add the mdac-service crate: an Axum harness exposing TokenBucket
over HTTP. It provides a healthcheck and a put-back-and-drain
endpoint over a startup-configured, immutable registry of named
buckets, reusing the frontend service pattern for shared state,
JSON extraction, body limits, TCP serving, and graceful shutdown.
Configuration loads from optional YAML overridden by MDAC_ env
vars, with a runnable token_bucket_service binary.
Add chroma_tracing::init_server_otel_tracing so servers share one
OTEL/stdout initialization path, and wire mdac-service telemetry
through the frontend OTLP exporter and HTTP middleware.
## Test plan
CI
## Migration plan
N/A
## Observability plan
Will add
## Documentation Changes
N/A
Co-authored-by: AI
The fn-consumer reads a window of log records between committed compaction boundaries. This restores it to stopping at the **nearest** committed boundary, rather than widening to the furthest boundary that still fits `max_compaction_size`.
## Why
Widening makes job length swing up to the size cap instead of tracking a single compaction window, and `max_compaction_size` is the knob meant to tune that. The synthesis backend also splits large requests to stay under its own request-size limits, so a wider window gets split apart again downstream. This matches the decision made when #7621 was backed out of production on Aug 21.
## Scope
Two files: the boundary-selection code and the tests that assert it.
#7621 also carried cleanups unrelated to window selection — a `chunks_exact(N)` → `as_chunks::<N>()` migration across the index and types crates, and two `drain(..).collect()` → `std::mem::take` call sites. Those stay. The diff here is the exact inverse of #7621 for the two files it touches, and nothing else has modified them since it merged.
## Production context
This behavior is already absent from production. It was hotfixed onto `release/2026-08-18` and then reverted there, but the matching revert never landed on `main` — so it is present in `release/2026-08-31`, and promoting that branch would reintroduce it.
Once this merges, a cherry-pick onto `release/2026-08-31` (via the `apply-hotfix` workflow) closes the second path, ahead of the production promotion.
## Validation
`cargo check -p worker --tests` passes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Expose the input collection UUIDs for each active fn-consumer job.
The fn-consumer now retains the collection IDs from each dispatched
batch and returns them through the existing ListInProgressJobs RPC as a
backward-compatible repeated field.
## Testing
- cargo fmt --all --check
- git diff --check
- focused worker test build started locally; full validation is
delegated to CI
## Compatibility
The new protobuf field uses tag 3, so existing clients remain
wire-compatible. No migration or deployment configuration changes are
required.
## Description
`tilt-setup-prebuild` uses `medyagh/setup-minikube@latest` with no
`minikube-version`, so the minikube version resolves at run time.
minikube **1.39.0** (picked up starting 2026-09-02) fails to start the
`none` driver on GitHub runners: it selects the containerd runtime and
setup dies with
```
X Exiting due to RUNTIME_ENABLE: Failed to start container runtime: Temporary Error: sudo /usr/local/bin/crictl version: exit status 1
validate CRI v1 runtime API for endpoint "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService
```
This broke every hosted-chroma tilt CI job today (this repo's own tilt
jobs consume the same action). The last job to pass used **1.38.1**
(2026-09-01); the first to fail used **1.39.0** — reruns reproduce it
deterministically.
## Change
Pin `minikube-version: 1.38.1` (last known good). We can unpin or bump
once minikube's none-driver regression is fixed upstream.
## Validation
- Failing hosted runs:
[js](https://github.com/chroma-core/hosted-chroma/actions/runs/33689852042/job/100458949827),
[JS
(E2E)](https://github.com/chroma-core/hosted-chroma/actions/runs/33689855589/job/100445983133)
— both die in minikube setup before any test runs.
- Passing 1.38.1 run for contrast: [JS (E2E) on hosted
#8324](https://github.com/chroma-core/hosted-chroma/actions/runs/33554882211/job/100013319088).
- This PR's own CI exercises the pinned action.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5 <[email protected]>
## Description of changes
The live fn-consumer can defer an item while this test runs, moving
it to the back of the queue and breaking the previous FIFO-order
assertion. Verify each retrieved item by matching its fn_id to the
expected completion offset instead of relying on push order.
Rename the test to test_k8s_integration_work_queue_enqueue_and_filtering
to reflect that it checks enqueue and filtering rather than FIFO
ordering, which remains covered by the QueueState unit tests.
## Test plan
CI
## Migration plan
N/A
## Observability plan
N/A
## Documentation Changes
N/A
Co-authored-by: AI
## Summary
Adds ListInProgressJobs to the fn-consumer gRPC service so operators can
inspect work running on an individual consumer.
- returns the attached function ID and expiry time for each active job
- reads the in-progress map through the FnConsumerManager message queue
- exposes no mutation or cancellation behavior
## Testing
- cargo test -p worker --lib fn_consumer::fn_consumer_manager::tests
