`git rev-parse --show-toplevel` names the outermost repository, so with atuin
checked out as a subdirectory of another one, `list` and `update` looked for
vendor/.repositories.json in the wrong place.
hub keeps atuin's history in its monorepo and syncs it both ways. This
trigger starts that sync on every push to main; the build it starts takes no
input from this one.
A few thing here in prep for session resume working
1. Incremental builds for ai sessions too. Similar to how we build
history records incrementally. Tracks a watermark, so that on daemon
boot we don't end up having to rebuild the entire harness db again
2. Groups sessions. Previously subagents etc all showed up as brand new
sessions, which was pretty spammy. Fix that up so that subagents are
grouped under the primary session
3. Reduces search params complexity
Switch from [`directories`](https://docs.rs/directories) to
[`dirs`](https://docs.rs/dirs). The two crates are both based on
[`dirs_sys`](https://docs.rs/dirs_sys) and return the same results.[^1]
Benefits:
* Faster. `directories` queries *every* potential directory even if you
only need one. For example,
[`dirs::home_dir()`](https://docs.rs/dirs/7/dirs/fn.home_dir.html) is
9.5x faster than [`directories::BaseDirs::new().map(|d|
d.home_dir())`](https://docs.rs/directories/latest/directories/struct.BaseDirs.html#method.home_dir).
* Simpler. Given that we weren't using higher-level features of
`directories` like `ProjectDirs`, the API of `dirs` is simpler.
* Fewer dependencies. We were already transitively pulling in `dirs`;
switching away from `directories` deletes a dependency from our lockfile
without adding any.
[^1]: `dirs` v6 has a [bug](https://codeberg.org/dirs/dirs-rs#7) that
causes `preference_dir` on Windows to diverge from `directories`. We do
not use this function and are thus not impacted. It's not worth
upgrading to v7 just yet, as that would pull in two different versions
of `dirs` in our crate (`shellexpand` depends on v6).
Main rejects a whole synced message if its parent_kind has a value the
build doesn't recognise, so as soon as a newer build adds a kind, older
builds would silently drop those messages; this reads an unknown kind as
"none" and keeps the message.
<!-- Thank you for making a PR! Bug fixes are always welcome, but if
you're adding a new feature or changing an existing one, we'd really
appreciate if you open an issue, post on the forum, or drop in on
Discord -->
## Checks
- [ ] I am happy for maintainers to push small adjustments to this PR,
to speed up the review cycle
- [ ] I have checked that there are no existing pull requests for the
same thing
Currently we use OSC 133 escape sequences to mark the start and end of
command output for output capture in pty-proxy. However, this has some
problems:
* OSC 133 markers added by other programs or plugins can interfere with
pty-proxy. Atuin's shell integration only emits OSC 133 markers when the
shell is running in the foreground process group of the terminal (e.g.,
not nested inside a multiplexer). However, if another program adds its
own OSC 133 markers, pty-proxy will get confused and start capturing
incorrect data.
* We overload OSC 133 with nonstandard Atuin-specific parameters. We add
a `history_id=<history-id>` parameter to the OSC 133 `D` marker, but
this is nonstandard and could potentially confuse other programs.
* OSC 133 sequences could potentially alter the behavior of other
programs the user is using. Switching to pty-proxy should feel seamless
and should not cause changes beyond the Atuin features it enables.
Therefore, this PR switches away from OSC 133 to a custom Atuin-specific
escape sequence, `OSC 18188735`. This has been chosen to minimize the
chance of conflicts with any existing program or terminal:
* 18188735 is `atuin` interpreted as a base-36 number. It is quite long
and therefore unlikely to be in use.
* 18188735 fits in a signed 32-bit integer, which many terminals use for
OSC numbers. Truncating to 16 or 8 bits produces results that aren't in
use either (35263, 191).
* No prefix of 18188735 of length 3 or greater is in use either, for
terminals that only read a fixed number of digits.
Credit goes to @markovejnovic for the idea to use a custom escape
sequence.
### Testing
I tested the following:
* Update Atuin to this branch and spawn new pty-proxy. Output capture
still works.
* Spawn old pty-proxy, then update Atuin to this branch and spawn
subshell. Output capture still works.
* Tested the above with both the old daemon (v18.23.0) and new daemon
from this branch.
Compilation fails on 32-bit systems with:
```
error: literal out of range for `usize`
--> crates/atuin-daemon/tests/common/corpus.rs:164:49
|
164 | let duration = i64::conv(self.rng.below(5_000_000_000));
| ^^^^^^^^^^^^^
|
= note: the literal `5_000_000_000` does not fit into the type `usize` whose range is `0..=4294967295`
= note: `#[deny(overflowing_literals)]` on by default
```
The usize argument to `below` must be less than 4_294_967_295 on 32-bit
systems.
## Checks
- [x] I am happy for maintainers to push small adjustments to this PR,
to speed up the review cycle
- [x] I have checked that there are no existing pull requests for the
same thing
Removing information about dotfiles from support matrix to not confuse
new users.
<!-- Thank you for making a PR! Bug fixes are always welcome, but if
you're adding a new feature or changing an existing one, we'd really
appreciate if you open an issue, post on the forum, or drop in on
Discord -->
## Checks
- [x] I am happy for maintainers to push small adjustments to this PR,
to speed up the review cycle
- [x] I have checked that there are no existing pull requests for the
same thing
If you run `atuin output search` when output capture is disabled, you
get the following error:
```
Error: output capture is disabled. enable [output] in your config to search command output.
```
It is not obvious what it means to `enable [output]`. Now that we have a
command to enable it, it's better to suggest that.
We cannot rely on GitHub to actually be up. I just deployed a github
release proxy, allowlisted on certain repos only, to handle this.
## Checks
- [ ] I am happy for maintainers to push small adjustments to this PR,
to speed up the review cycle
- [ ] I have checked that there are no existing pull requests for the
same thing
Adds atuin_ai_session_list and atuin_ai_session_read alongside a
reworked atuin_ai_session_search, so agents can find a past session and
actually read it.
## Checks
- [ ] I am happy for maintainers to push small adjustments to this PR,
to speed up the review cycle
- [ ] I have checked that there are no existing pull requests for the
same thing
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Previously we used the full size session db, which added up to a lot of
disk
## Checks
- [ ] I am happy for maintainers to push small adjustments to this PR,
to speed up the review cycle
- [ ] I have checked that there are no existing pull requests for the
same thing
Remove all workflows that have been replaced with buildkite
We have intentionally moved slow tests (illumos, nix, windows) to be
main only. If fast runners can be found or setup, we can bring them back
again.
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Follow-up to #4274, needed before cutting CI over from GitHub Actions.
It fixes the hang on the first `main` build and addresses the review
feedback on #4274 that was never answered before it merged. One commit
per fix.
### Fixes
- **Installer steps hang on `main`** (`ce325a28`). `install.sh` reads
its prompts (import history, `atuin setup`) from `/dev/tty`. GitHub
Actions jobs have no controlling terminal, so it took its
non-interactive path there. The Buildkite agent runs commands in a pty,
so on the first `main` build (#1375) `install` and `install-macos`
waited at "import your existing shell history? [Y/n]" until they timed
out. The installer now runs in a new session with no controlling
terminal, so it detects non-interactive exactly as it did on GitHub
Actions. Reproduced locally under a pty (hangs before the fix, passes
after, for both bash and zsh).
- **P1: the Test Engine token was visible to test code** (`96357dc3`).
The nextest steps had `ATUIN_TEST_ENGINE_TOKEN` in their environment,
and any test, build script or dependency could read it and print it to
this pipeline's public logs. The test steps now only save their JUnit
reports as artifacts, renamed per job by
`.buildkite/hooks/post-command`. A separate `test-engine` step, which
runs no repository code, uploads them with the token.
- **P1: the `.deb` cache could hide a missing package** (`2ff57522`).
The cache is saved once per step, so it can predate a newly added
package, and `dpkg` would then succeed without it. apt_install now
checks every requested package after the fast path, and falls back to
`apt` if one is missing. Tested with stub `dpkg`/`apt-get`. Also
silences apt's "unsandboxed as root" warning.
- **P2: Vale failures could pass** (`18496526`). The step now fails on
Vale's own non-zero exit (config or runtime errors), not only on
reported alerts. Tested: clean docs pass, a planted prose error fails
with exit 1, and a broken config fails with exit 2 where it used to
pass.
- **P2: test Postgres listened on every interface** (`459a50f4`). It now
binds to `127.0.0.1`, and `ATUIN_DB_URI` uses `127.0.0.1`.
- **P2: test coverage for bulk output-index removal** (`8e128df9`). The
chunked statements are split into `Schema::remove_chunked`, and a test
with small chunk sizes checks that removing a mix of indexed and
never-indexed ids drops exactly the indexed ones from both tables. I
confirmed it fails if only the first chunk runs.
**Validated on this PR's build (#1377)**, with the `main`-only installer
steps temporarily enabled on this branch (`fc5186cb`, since reverted in
`ecc93945`):
- `install` and `install-macos` both pass. Each installer took its own
"Non-interactive environment detected" path.
- `test-engine` found the 9 per-job reports (every test shard) and
uploaded all 9 to the atuin suite. No other job has the token.
**Checks:** `atuin-daemon`: 377 tests pass; clippy `-D warnings` and
nightly rustfmt are clean; ShellCheck passes on all scripts and hooks.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Move atuin's CI over to Buildkite. I've removed a few jobs from branches
and pushed them onto main, mostly things like the nixos build, illumos
etc - they were really slow, and difficult to optimise.
The goal is to be on buildkit, and have CI take around a minute per
commit. Once we have more tests, we can keep parallelizing
I tested out both the nvme disks and the buildkite cache. The former is
much faster, but is trickier to isolate across branches. I would prefer
that there is 0 chance of a branch messing with a build on main, or with
another branch.
---------
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Unlikely to be hit, as bash will almost certainly run before this does.
But good to have it just in case.
<!-- Thank you for making a PR! Bug fixes are always welcome, but if
you're adding a new feature or changing an existing one, we'd really
appreciate if you open an issue, post on the forum, or drop in on
Discord -->
## Checks
- [ ] I am happy for maintainers to push small adjustments to this PR,
to speed up the review cycle
- [ ] I have checked that there are no existing pull requests for the
same thing
This PR refactors `Shell::default_shell`. The motivating factor is that
it was causing a test to hang and fail when run under Nextest, and
worse, would leave an orphaned process consuming 100% CPU.
Specifically, if `cargo nextest run -E
'test(=doctor_runs_on_fresh_install)'` is run while attached to a
terminal:
* `atuin doctor` calls `Shell::default_shell`.
* `default_shell` runs `sh -ic "getent passwd $LOGNAME"`.
* `-i` makes `sh` (in my case, `dash`) interactive, which turns on job
control, causing `sh` to try to control the terminal.
* Nextest runs every test in its own process group, and that process
group is not the terminal's foreground process group. When a background
process tries to control the terminal, it receives SIGTTIN, which causes
it to stop.
* `sh`, now stopped, waits forever until Nextest times out after 30s.
* After 30s, Nextest kills `atuin doctor` but not `sh`, orphaning `sh`.
* The orphaned `sh` no longer receives SIGTTIN; instead, reads from the
terminal immediately fail.
* In the case of `dash`, it keeps retrying the failing terminal read in
an infinite loop, using 100% CPU.
The following improvements have been made to `Shell::default_shell`:
* It no longer spawns interactive shells, or any shells at all.
* It correctly handles non-UTF-8 output from the query commands it runs.
* It no longer has to query the system name at runtime, relying on the
compile-time configuration instead.
* On Linux, it uses the UID rather than username for lookups, which is
more robust.
* Error messages are more descriptive.
* Allocations have been reduced.
Unused code from `atuin-dotfiles` has been removed as well; it was the
sole user of `Shell::run_interactive`, now also removed by this commit.
<!-- Thank you for making a PR! Bug fixes are always welcome, but if
you're adding a new feature or changing an existing one, we'd really
appreciate if you open an issue, post on the forum, or drop in on
Discord -->
## Checks
- [ ] I am happy for maintainers to push small adjustments to this PR,
to speed up the review cycle
- [ ] I have checked that there are no existing pull requests for the
same thing
Historically we have had several issues related to a mismatch between
what the client thinks the daemon socket path should be and what it
actually is, generally due to environment variable differences (e.g.,
daemon launched without `TMPDIR` set but the client runs with it).
To prevent these kinds of issues going forward, this PR makes the daemon
store the path to its socket in the pidfile. The client, when connecting
to the daemon, uses the socket path in the pidfile if present. This
means that even if you change `daemon.socket_path` in `config.toml`
after launching the daemon, the client will still be able to connect to
it.
### Implementation
Historically, the pidfile had this format:
```
<pid>
<atuin version>
```
This PR changes the format to:
```
<pid>
v1
<PidfileInfo serialized as JSON>
```
where `PidfileInfo` contains the socket path, and version as before.
This change is backward-compatible: the PID is still in the same place,
and we never actually used the version for anything. The version always
starts with a number since it comes from `CARGO_PKG_VERSION`, so `v1`
unambiguously identifies the file as the new format.
To avoid reading the pidfile in a half-written state, we use a new file,
the *pidfile lock* (same path as the pidfile with `.lock` appended) to
guard reads and writes of the pidfile. This mirrors the startup lock
file (`.startup.lock` appended) that we already use.
### New files
* `crates/atuin-daemon/src/pidfile.rs`: New module containing all the
pidfile-related code.
* `crates/atuin-common/src/fs/lock.rs`: New utility module for managing
file locks.
* `crates/atuin-common/src/string/auto_os_string.rs`: Better
serialization of `OsString` with serde.
### Testing
In addition to the unit tests, I performed some smoke testing, including
starting the daemon with a custom `socket_path`, changing the
`socket_path` in `config.toml`, and then making sure the client can
still connect. I also made sure the client can still connect to an older
daemon that doesn't write its path in the pidfile.
`Backoff::Exponential` waits 1ms, 2ms, 4ms, 8ms, etc. – an exponential
curve. This means that `Backoff::Linear` should, in theory, wait 1ms,
2ms, 3ms, and so on – linear growth – but instead it polls with a fixed,
unchanging delay. This makes *constant* a more appropriate name than
linear (a constant function is technically linear, with a slope of 0,
but using the more specific name better communicates what the algorithm
does).
Fixes#3866. When the daemon is enabled but autostart is off, and the
daemon either isn't running or has the incorrect version, history is
silently discarded. This PR falls back to saving history locally in the
client instead.
`cargo clippy` currently generates this warning:
```
warning: `time::Duration::new` does not refer to a reachable function
--> /home/taylor/code/atuin/clippy.toml:6:1
|
6 | / [[disallowed-methods]]
7 | | path = "time::Duration::new"
8 | | reason = "panics on overflow; use atuin_common::time::DurationExt::try_new"
| |___________________________________________________________________________^
|
```
`time` 0.3.54 made `Duration` an alias of `SignedDuration` instead of a
struct in its own right, and we're on 0.3.55 now. Clippy can't resolve
the type alias, so this PR updates the path in clippy.toml to point to
`SignedDuration`.
After a team discussion, we've decided to limit the installation methods
mentioned in the docs only to those that will always install the latest
version (potentially lagging by a few days).
This PR removes mentions of installation methods that do not meet this
requirement:
* XBPS (Void Linux): currently v18.16.1.
* Nixpkgs: although unstable is relatively up-to-date (v18.21.0), stable
is currently v18.15.2. Only the Nixpkgs mention is removed; installation
of the repository as a flake is still listed.
Changes:
* Don't strip OSC 133 sequences from the prompt if the PTY proxy isn't
in use. This allows other integrations that rely on OSC 133 to function
as usual.
* Only strip Atuin's OSC 133 sequences from the prompt if, after doing
so, there are no OSC 133 sequences remaining. Otherwise, this likely
indicates that some other program has added its own OSC 133 sequences,
and we don't want to end up half-removing them (e.g., if our end pattern
matches but not the start, we'll strip the "B" marker but not the "A"
one).
Fixes#4149.
Marko Vejnovic