feat(proc): expose tracked root wait completion

Problem: A foreground command can return bounded cancellation while cmd.Wait
is still blocked, so call return cannot establish resource-lifetime completion.

Root cause: The tracked wait owner exposes its logical result and cleanup
diagnostics but no separate observation of the still-running root Wait.

Fix: Register a stable root-wait channel before the waiter starts and close it
after Wait returns, before result publication. Expose it with the explicit
RootWaitCompleted name. Preserve existing execution, cancellation, retry and
error behavior. No production lease consumer or whole-scope claim is added.

Verification: Deterministic cancellation/retry/publication tests, portable Go
child execution and a real blocked-output process/lease regression pass. Root
Wait remains distinct from descendant or external-scope completion; nil means
no registered observation. Full local/static and platform evidence is recorded
in the PR without inheriting previous heads' native results.
This commit is contained in:
SivanCola committed 2026-10-25 20:00:34 +08:00
1 parent 540c14998b
commit 5d484c4fe2
6 files changed
+475 -6

No files matched your search

+118
View File
@@ -0,0 +1,118 @@
---
owner: @SivanCola
backup: @esengine
status: active
reviewed: 2026-10-25
---
# Root wait observation / 根进程等待观察
## Cause and scope / 根因与范围
Tracked foreground execution deliberately returns after a cancellation grace
period even when `cmd.Wait` has not returned. That bounded result is useful for
responsive cancellation; it cannot also prove physical completion.
`TrackedCommand.RootWaitCompleted` exposes only the registered root wait. Its
channel closes after the wait function returns and before its result is sent.
Cancellation and cleanup-retry expiry do not close it. No production lease,
tool, hook or resource consumer uses the new method in this unit.
前台取消采用有界返回以保持响应,但返回时 `cmd.Wait` 仍可能未完成。本次保留
这一行为,新增明确命名的根等待完成信号,不把调用返回等同于物理执行结束。
## Source, sink and authority / 输入、副作用与权威
- The existing tracked-command owner registers a channel before starting its
wait goroutine. Access to channel identity uses its existing mutex.
- The goroutine closes the channel only after `wait()` returns. It closes
before publishing the buffered result, so receiving the result cannot race
ahead of notification. A blocked result receiver cannot delay notification.
- The buffered result remains available to the existing foreground or retry
consumer. The signal adds no goroutine, timer, command, path or OS handle.
- Root completion preserves success, exit errors and cancellation error
identities. Existing launch, tracking, kill and retry policies stay unchanged.
- Nil handles and handles without a registered wait return nil. Nil is not
completion evidence or permission to release a resource.
信号身份沿现有 mutex 发布,在等待返回后、结果发送前关闭。既有取消、重试、
错误分类和进程操作保持原样,没有新命令、路径、句柄或后台 goroutine。
## Meaning of each boundary / 各边界的含义
| Observation | What it proves | What it does not prove |
| --- | --- | --- |
| Foreground cancellation returns | Grace policy returned control | Root wait or descendants ended |
| Retry window expires | Automatic retry budget ended | Root wait completed |
| RootWaitCompleted closes | Registered root `cmd.Wait` returned | Descendants or external scope ended |
| Method returns nil | No wait was registered on this handle | Command never ran or all effects ended |
| ProcessState exists | os/exec observed a root process result | Output copies or descendants ended |
A root wait may return an error and still be completed. Conversely a failed
launch can return a tracked handle without registering a wait. In particular,
Windows launch preparation may start and reap a suspended child before returning
an error. The nil signal makes no stronger claim about that path.
完成可以伴随错误;失败启动也可能产生未登记等待的句柄。Windows 的启动准备
可能先创建再回收挂起子进程,因此 nil 不能被解释为“肯定未执行”。
## Verification / 验证
- Deterministic synthetic-time tests keep Wait blocked through cancellation,
grace expiry and retry expiry, then release it and observe the same channel.
- An unbuffered receiver control proves completion notification precedes result
publication without depending on goroutine scheduling speed.
- Actual portable Go child commands cover success, nonzero exit, missing
executable, cancellation before launch and unchanged untracked behavior.
- An actual child writes into a deliberately blocked output sink. Bounded
cancellation returns while `cmd.Wait` remains pending. A test-only consumer
binds the signal to an existing mutation lease and ends it logically.
- Conflicting admission remains busy, disjoint admission proceeds, and releasing
the output sink lets the lease close. Repeating old cleanup does not release
the next transaction. All fixture processes and waits are joined.
真实子进程测试固定阻塞输出处理,因此不靠时间快慢猜测 Wait 是否完成。将信号
接到租约的调用仅存在于回归测试,不代表运行时已启用写事务。
## Compatibility / 兼容性
| Contract | Before | After |
| --- | --- | --- |
| Tracked foreground return/error | Existing bounded cancellation | Unchanged |
| Untracked RunCommand | Nil tracked handle | Unchanged |
| Root wait observation method | Absent | Additive, local-only method |
| Diagnostics/wire/session records | Existing fields | Unchanged |
| OS tracking and sandbox policy | Existing backend behavior | Unchanged |
There is no new persisted field, migration, model-visible schema or prompt
prefix. This method has no preceding API implementation to compare. Deliberately
weakened completion ordering or cancellation closure controls validate the new
contract; they are not claims that an older API already promised this signal.
本次没有持久化或 wire 变更。新增接口此前不存在,故意削弱信号规则的对照用于
验证新测试,不能冒称旧版已经承诺这个接口却违反它。
## Limits and next integration / 边界与后续接入
Do not use this signal alone to release a lease for an unresolved shell or hook.
The command may spawn descendants, detach, delegate work to a service or return
`exec.ErrWaitDelay`. Root Wait is not a process-scope termination protocol.
Full write-lifetime integration must separately establish the backend's supported
scope, distinguish unknown termination, retain ownership when completion cannot
be proved, and provide attributable diagnostics and recovery. Closing every
channel is not a valid substitute for proving completion.
Until those boundaries are implemented, this unit does not activate shared
transactions, settle unknown scopes, recover process crashes or guarantee every
descendant is gone. Existing cancellation cleanup remains best-effort where the
backend is best-effort. Native platform evidence is recorded separately.
不能仅凭根等待完成就释放未知 shell/hook 写入租约。后代、脱离进程、服务委托和
输出超时都有独立边界。无法证明范围终止时必须保留占用并给出诊断及恢复路径,
不能为了关闭 channel 而伪造完成。本次不启用共享事务,也不完成未知范围恢复。
This is the author's security assessment, not independent review or disposition
of the original large PR's CodeQL findings.
本说明为作者评估,不替代独立安全审查或原大 PR 的告警逐项判断。
+18
View File
@@ -0,0 +1,18 @@
package proc
func publishRootWait(done chan struct{}, results chan<- error, err error) {
close(done)
results <- err
}
// RootWaitCompleted closes after the registered cmd.Wait returns, before its
// result is received. Nil means no registered wait, not completed execution.
// It does not prove that descendants or an external execution scope have ended.
func (p *TrackedCommand) RootWaitCompleted() <-chan struct{} {
if p == nil {
return nil
}
p.mu.Lock()
defer p.mu.Unlock()
return p.rootWait
}
+194
View File
@@ -0,0 +1,194 @@
package proc
import (
"context"
"errors"
"os"
"os/exec"
"sync"
"testing"
"testing/synctest"
"time"
)
func assertRootWaitClosed(t *testing.T, done <-chan struct{}) {
t.Helper()
if done == nil {
t.Fatal("registered root wait has no completion signal")
}
select {
case <-done:
default:
t.Fatal("root wait completed without closing its signal")
}
}
func TestRootWaitCompletionPrecedesResultAndRetainsWaitError(t *testing.T) {
for _, errWait := range []error{nil, errors.New("root wait failed")} {
tracked := &TrackedCommand{}
err := waitForTrackedCommand(t.Context(), tracked, func() error { return errWait }, time.Second, time.Second, time.Second)
if !errors.Is(err, errWait) {
t.Fatalf("wait error changed: %v", err)
}
done := tracked.RootWaitCompleted()
assertRootWaitClosed(t, done)
if tracked.RootWaitCompleted() != done {
t.Fatal("completion identity changed")
}
}
}
func TestRootWaitPublicationDoesNotDependOnReceiver(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
done, results := make(chan struct{}), make(chan error)
expected := errors.New("root wait result")
go publishRootWait(done, results, expected)
defer func() {
if err := <-results; !errors.Is(err, expected) {
t.Fatalf("result changed: %v", err)
}
}()
synctest.Wait()
assertRootWaitClosed(t, done)
})
}
func TestRootWaitCompletionSurvivesCancellationAndRetryExpiry(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
tracked := &TrackedCommand{}
started, unblock := make(chan struct{}), make(chan struct{})
release := sync.OnceFunc(func() { close(unblock) })
defer release()
returned := make(chan error, 1)
go func() {
returned <- waitForTrackedCommand(ctx, tracked, func() error {
close(started)
<-unblock
return nil
}, time.Second, time.Second, 3*time.Second)
}()
<-started
done := tracked.RootWaitCompleted()
if done == nil {
t.Fatal("wait signal was not published before entering Wait")
}
cancel()
if err := <-returned; !errors.Is(err, context.Canceled) {
t.Fatalf("logical cancellation result changed: %v", err)
}
synctest.Wait()
select {
case <-done:
t.Fatal("bounded cancellation was mistaken for root completion")
default:
}
time.Sleep(4 * time.Second)
synctest.Wait()
select {
case <-done:
t.Fatal("retry expiry was mistaken for root completion")
default:
}
release()
synctest.Wait()
assertRootWaitClosed(t, done)
if done != tracked.RootWaitCompleted() {
t.Fatal("late completion replaced the registered signal")
}
})
}
func TestRootWaitCompletionDoesNotChangeCancelledWaitErrors(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
tracked := &TrackedCommand{}
entered, release := make(chan struct{}), make(chan struct{})
errWait := errors.New("root exited after cancellation")
returned := make(chan error, 1)
go func() {
returned <- waitForTrackedCommand(ctx, tracked, func() error {
close(entered)
<-release
return errWait
}, time.Hour, time.Hour, time.Hour)
}()
<-entered
cancel()
synctest.Wait()
close(release)
if err := <-returned; !errors.Is(err, context.Canceled) || !errors.Is(err, errWait) {
t.Fatalf("cancel/wait identities changed: %v", err)
}
assertRootWaitClosed(t, tracked.RootWaitCompleted())
})
}
func TestRootWaitUnregisteredIsNotCompleted(t *testing.T) {
for _, tracked := range []*TrackedCommand{nil, {}} {
if done := tracked.RootWaitCompleted(); done != nil {
t.Fatal("unregistered wait fabricated completion")
}
}
}
func TestRootWaitCommandHelper(t *testing.T) {
if mode := os.Getenv("REASONIX_ROOT_WAIT_HELPER"); mode != "" {
if mode == "error" {
os.Exit(7)
}
os.Exit(0)
}
}
func TestRootWaitCompletionOnRealCommands(t *testing.T) {
for _, mode := range []string{"success", "error"} {
t.Run(mode, func(t *testing.T) {
cmd := exec.CommandContext(t.Context(), os.Args[0], "-test.run=^TestRootWaitCommandHelper$")
cmd.Env = append(os.Environ(), "REASONIX_ROOT_WAIT_HELPER="+mode)
tracked, err := RunCommand(t.Context(), cmd, RunOptions{Track: true})
if mode == "success" && err != nil {
t.Fatal(err)
}
if mode == "error" {
var exited *exec.ExitError
if !errors.As(err, &exited) || exited.ExitCode() != 7 {
t.Fatalf("exit error changed: %v", err)
}
}
if cmd.ProcessState == nil {
t.Fatal("real command was not reaped")
}
assertRootWaitClosed(t, tracked.RootWaitCompleted())
})
}
}
func TestRootWaitDoesNotRegisterBeforeStartOrForUntrackedCommands(t *testing.T) {
for _, scenario := range []string{"cancelled_before_start", "missing_executable", "untracked"} {
t.Run(scenario, func(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
cmd := exec.CommandContext(ctx, os.Args[0], "-test.run=^TestRootWaitCommandHelper$")
cmd.Env = append(os.Environ(), "REASONIX_ROOT_WAIT_HELPER=success")
if scenario == "cancelled_before_start" {
cancel()
}
if scenario == "missing_executable" {
cmd.Path = t.TempDir() + string(os.PathSeparator) + "missing"
}
tracked, err := RunCommand(ctx, cmd, RunOptions{Track: scenario != "untracked"})
if scenario != "untracked" && err == nil {
t.Fatal("unexpected successful launch")
}
if scenario == "untracked" && (tracked != nil || err != nil) {
t.Fatalf("untracked contract changed: %v", err)
}
if tracked.RootWaitCompleted() != nil {
t.Fatal("unregistered wait has completion proof")
}
})
}
}
+13 -6
View File
@@ -46,11 +46,12 @@ type RunDiagnostics struct {
type TrackedCommand struct {
cmd *exec.Cmd
mu sync.Mutex
job *TrackedJob
tree *TreeTracker
killed bool
diag RunDiagnostics
mu sync.Mutex
job *TrackedJob
tree *TreeTracker
killed bool
diag RunDiagnostics
rootWait <-chan struct{}
}
// RunCommand starts cmd and waits for it. When tracking is enabled, cancellation
@@ -138,7 +139,13 @@ func normalizeRunOptions(opts RunOptions) RunOptions {
func waitForTrackedCommand(ctx context.Context, tracked *TrackedCommand, wait func() error, grace, retryEvery, retryFor time.Duration) error {
waitCh := make(chan error, 1)
go func() { waitCh <- wait() }()
done := make(chan struct{})
tracked.mu.Lock()
tracked.rootWait = done
tracked.mu.Unlock()
go func() {
publishRootWait(done, waitCh, wait())
}()
select {
case err := <-waitCh:
+126
View File
@@ -0,0 +1,126 @@
package mutation
import (
"context"
"errors"
"fmt"
"os"
"os/exec"
"sync"
"testing"
"time"
"reasonix/internal/base/proc"
)
func TestMutationRootWaitHelper(t *testing.T) {
if os.Getenv("REASONIX_MUTATION_ROOT_WAIT_HELPER") == "1" {
fmt.Print("root-output")
os.Exit(0)
}
}
type rootWaitOutput struct {
once sync.Once
entered chan struct{}
release chan struct{}
}
func (w *rootWaitOutput) Write(p []byte) (int, error) {
w.once.Do(func() { close(w.entered) })
<-w.release
return len(p), nil
}
func TestRootWaitRetainsLeaseAfterRealBoundedCancellation(t *testing.T) {
root, state := t.TempDir(), t.TempDir()
released := make(chan struct{}, 1)
c, err := New(root, state, func(n Notice) {
if n.State == "released" && n.CallID == "root-wait" {
released <- struct{}{}
}
})
if err != nil {
t.Fatal(err)
}
request := intent("held")
request.CallID = "root-wait"
lease, err := c.Acquire(t.Context(), request)
if err != nil {
t.Fatal(err)
}
defer lease.End(NotExecuted)
ctx, cancel := context.WithTimeout(t.Context(), 20*time.Second)
defer cancel()
output := &rootWaitOutput{entered: make(chan struct{}), release: make(chan struct{})}
unblock := sync.OnceFunc(func() { close(output.release) })
cmd := exec.CommandContext(ctx, os.Args[0], "-test.run=^TestMutationRootWaitHelper$")
cmd.Env = append(os.Environ(), "REASONIX_MUTATION_ROOT_WAIT_HELPER=1")
cmd.Stdout = output
type result struct {
tracked *proc.TrackedCommand
err error
}
returned, joined := make(chan result, 1), make(chan struct{})
go func() {
defer close(joined)
tracked, err := proc.RunCommand(ctx, cmd, proc.RunOptions{Track: true,
CancelWaitGrace: time.Millisecond, CancelRetryInterval: time.Millisecond, CancelRetryFor: time.Millisecond})
returned <- result{tracked, err}
}()
t.Cleanup(func() { unblock(); cancel(); <-joined })
select {
case <-output.entered:
case <-ctx.Done():
t.Fatal(ctx.Err())
}
cancel()
res := <-returned
if !errors.Is(res.err, context.Canceled) {
t.Fatalf("bounded cancellation result: %v", res.err)
}
done := res.tracked.RootWaitCompleted()
if done == nil {
t.Fatal("real tracked wait was not registered")
}
t.Cleanup(func() { unblock(); <-done })
select {
case <-done:
t.Fatal("blocked output copy was not a pending cmd.Wait")
default:
}
if err := lease.HoldUntil(done); err != nil {
t.Fatal(err)
}
lease.End(Unknown)
other := testCoordinator(t, root, state)
if next, err := other.TryAcquire(t.Context(), request); !errors.Is(err, ErrBusy) {
if next != nil {
next.End(NotExecuted)
}
t.Fatalf("logical return released physical ownership: %v", err)
}
disjoint, err := other.TryAcquire(t.Context(), intent("disjoint"))
if err != nil {
t.Fatalf("unrelated file blocked: %v", err)
}
disjoint.End(NotExecuted)
unblock()
select {
case <-released:
case <-time.After(10 * time.Second):
t.Fatal("completed root wait did not release the lease")
}
next, err := other.TryAcquire(t.Context(), request)
if err != nil {
t.Fatal(err)
}
defer next.End(NotExecuted)
lease.End(NotExecuted)
if probe, err := c.TryAcquire(t.Context(), request); !errors.Is(err, ErrBusy) {
if probe != nil {
probe.End(NotExecuted)
}
t.Fatalf("old cleanup released a new transaction: %v", err)
}
}
@@ -0,0 +1,6 @@
## 变更
- 为受跟踪命令增加根进程 Wait 完成信号,明确区分取消返回与等待真正完成;
不改变取消策略,不把该信号当成整个进程范围结束的证明。Refs #14025
- Add a root-wait completion observation for tracked commands, separate from
bounded cancellation return; it does not prove descendant completion. Refs #14025