run-llama/semtools
1
0
Fork 0
mirror of https://github.com/run-llama/semtools.git synced 2026-07-21 11:15:24 -04:00
Code Issues 4 Packages Projects Releases 10 Wiki Activity

[PR #24] [MERGED] beta workspaces for persistance/performance on large datasets #34

New Issue
Closed
opened 2026-02-16 04:15:17 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 04:15:17 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/run-llama/semtools/pull/24
Author: @logan-markewich
Created: 9/13/2025
Status: ✅ Merged
Merged: 9/13/2025
Merged by: @logan-markewich

Base: main ← Head: logan/beta_workspaces

📝 Commits (9)

📊 Changes

13 files changed (+6974 additions, -595 deletions)

View changed files

📝 .github/workflows/lint.yml (+6 -0)
📝 .github/workflows/release.yml (+5 -0)
📝 .github/workflows/test.yml (+3 -0)
📝 Cargo.lock (+5034 -565)
📝 Cargo.toml (+18 -4)
➕ cli/workspace.js (+19 -0)
📝 package.json (+3 -2)
📝 scripts/install.js (+10 -1)
📝 src/bin/search.rs (+513 -23)
➕ src/bin/workspace.rs (+126 -0)
📝 src/lib.rs (+3 -0)
➕ src/workspace/mod.rs (+286 -0)
➕ src/workspace/store.rs (+948 -0)

📄 Description

Fixes https://github.com/run-llama/semtools/issues/16

Adds an initial (beta for now) version of persisted workspaces.

The general idea is something like this

workspaces select my-proj --doc-top-k 200
export SEMTOOLS_WORKSPACE=my-proj

search ./large_dir/*.txt --top-k 5

If a workpsace is active, search will automatically back up and persist embeddings into the workspace (if the requested list of documents is larger that the workspace top-k). It does this by generating an averaged document-level embedding and storing it in the vector db.

Then at search time, we can select the doc-top-k documents, and do the original line-by-line embedding search from there. For large datasets, this means huge speedups in runtime because after the initial ingestion, subsequent runs will only re-embed the selected doc-top-k documents

Workspace help:

$ workspace --help       
Manage semtools workspaces

Usage: workspace <COMMAND>

Commands:
  select  Select or create a workspace (prints export command to run)
  status  Show active workspace and basic stats
  prune   Remove stale or missing files from store
  help    Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/run-llama/semtools/pull/24 **Author:** [@logan-markewich](https://github.com/logan-markewich) **Created:** 9/13/2025 **Status:** ✅ Merged **Merged:** 9/13/2025 **Merged by:** [@logan-markewich](https://github.com/logan-markewich) **Base:** `main` ← **Head:** `logan/beta_workspaces` --- ### 📝 Commits (9) - [`26b637f`](https://github.com/run-llama/semtools/commit/26b637f58d92089ea396d8f81faa3e30782ec97e) add beta workspace feature - [`2d558eb`](https://github.com/run-llama/semtools/commit/2d558ebb48be9fa881513f73d4e5a737cc8eed8f) add release scripts for workspaces - [`cf63a04`](https://github.com/run-llama/semtools/commit/cf63a04746e03b5ebe5dfd05724477eda45e5994) npm version - [`1ff6927`](https://github.com/run-llama/semtools/commit/1ff6927a35b7a1286a37b3dc572a3ba15ce2b9c7) nits - [`306868c`](https://github.com/run-llama/semtools/commit/306868cc2ff13c56df2ed1c5180d43f022b0583c) fix workflows - [`8538146`](https://github.com/run-llama/semtools/commit/8538146cbde57832a3e48b9b35441544aec7f318) updates - [`ae1b236`](https://github.com/run-llama/semtools/commit/ae1b236f5248e48462d62d730c728970b37618c3) clippy - [`54e6db7`](https://github.com/run-llama/semtools/commit/54e6db793e233f3e6f039055cf3bc4e22b0b204b) fix tests - [`aacb998`](https://github.com/run-llama/semtools/commit/aacb998ce63c15252672db1035b7e2b9119cb1dc) appease the clippy gods ### 📊 Changes **13 files changed** (+6974 additions, -595 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/lint.yml` (+6 -0) 📝 `.github/workflows/release.yml` (+5 -0) 📝 `.github/workflows/test.yml` (+3 -0) 📝 `Cargo.lock` (+5034 -565) 📝 `Cargo.toml` (+18 -4) ➕ `cli/workspace.js` (+19 -0) 📝 `package.json` (+3 -2) 📝 `scripts/install.js` (+10 -1) 📝 `src/bin/search.rs` (+513 -23) ➕ `src/bin/workspace.rs` (+126 -0) 📝 `src/lib.rs` (+3 -0) ➕ `src/workspace/mod.rs` (+286 -0) ➕ `src/workspace/store.rs` (+948 -0) </details> ### 📄 Description Fixes https://github.com/run-llama/semtools/issues/16 Adds an initial (beta for now) version of persisted workspaces. The general idea is something like this ```bash workspaces select my-proj --doc-top-k 200 export SEMTOOLS_WORKSPACE=my-proj search ./large_dir/*.txt --top-k 5 ``` If a workpsace is active, `search` will automatically back up and persist embeddings into the workspace (if the requested list of documents is larger that the workspace top-k). It does this by generating an averaged document-level embedding and storing it in the vector db. Then at search time, we can select the `doc-top-k` documents, and do the original line-by-line embedding search from there. For large datasets, this means huge speedups in runtime because after the initial ingestion, subsequent runs will only re-embed the selected `doc-top-k` documents Workspace help: ```bash $ workspace --help Manage semtools workspaces Usage: workspace <COMMAND> Commands: select Select or create a workspace (prints export command to run) status Show active workspace and basic stats prune Remove stale or missing files from store help Print this message or the help of the given subcommand(s) Options: -h, --help Print help -V, --version Print version ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-16 04:15:17 -05:00
yindo closed this issue 2026-02-16 04:15:17 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
enhancement
pull-request
Mirrored from GitHub Pull Request
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/semtools#34
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?