Wire the plan executor to the start-phase vocabulary the reconciler
already produces (#14200): OpWaitCondition, OpRunPreStart, the enriched
OpStartContainer, and OpRunPostStart all had no case in executeNode
and hit "unknown operation type".
Every operation reuses the exact imperative primitive it mirrors —
waitDependency, runPreStart, runHook, injectSecrets/injectConfigs — so
behavior and events cannot drift between the two engines: execWaitCondition
resolves the dependency's live containers and hands off to the same
polling/condition-check functions waitDependencies uses; the enriched
OpStartContainer resolves its target either from the observed container
or, for a replica the plan itself creates, from the CreateContainer
node's result (the mechanism OpRenameContainer already relies on), then
injects secrets/configs before ContainerStart; OpRunPreStart re-checks
the daemon first so a replica that started in the observe-to-execute
window skips a redundant hook run.
A replica's pre_start/start/post_start chain shares one event group,
reporting a single Starting→Started progression exactly like
startServiceContainer does today — Starting fires specifically at
StartContainer, not at the (silent) pre_start hooks preceding it.
pkg/api/event.go documents this per-resource progression contract.
Still inert in production: create() (executePlan's only caller never
builds a plan with start-phase scope, so none of this runs outside
tests until a later brick in epic #14081 wires a real caller.
EOF
)
Signed-off-by: Nicolas De Loof <[email protected]>
The legal values of DownOptions.Images lived in pkg/compose
(image_pruner.go) while the field itself was a bare string in pkg/api,
and the only validation fired inside ImagesToPrune — after down had
already removed the project's containers, leaving the teardown half
done on a typo. The CLI validated its --rmi flag separately, so only
SDK callers were exposed.
ImagePruneMode and its three values now live in pkg/api next to the
field they constrain (type-aliased in pkg/compose for existing
consumers), down() rejects any other value before touching a single
resource, and the CLI check reuses the same definition. A unit test
runs Down with expectation-free mocks: one daemon call would fail it.
Epic #14074, B.6.
Signed-off-by: Nicolas De Loof <[email protected]>
Binding the announced gateway alone restricts by destination address, not
by the interface traffic arrives on, so another bridge can still reach it
by IP. A provider that pins the socket to the relay-link bridge's own
interface (SO_BINDTODEVICE on Linux) keeps the endpoint relay-only.
Signed-off-by: Nicolas De Loof <[email protected]>
Unify the up and down paths on one removal helper that tolerates both
NotFound and Conflict and emits the same progress events as removeNetwork,
build one down op per relay-link network gated on the shared limiter (so
the empty-ops warning is reachable again), and share the label filter.
Reword the isolation claim: the relay-link network is joined by nothing
else, but its gateway is a host address that local containers can still
reach by IP; only a provider binding exclusively that address keeps the
endpoint off loopback and LAN-facing addresses.
Signed-off-by: Nicolas De Loof <[email protected]>
get-relay-info answers pointed a locally-bound provider at one of the
dependents' own networks: shared infrastructure every other service on
it could also reach, and no more isolated than any project bridge.
Provision a dedicated, internal:true bridge network scoped to that one
provider service instead, created only when the provider itself sends
get-relay-info -- the provider's own declaration that it binds locally,
never for a provider backing a remote resource. The relay container
joins it alongside the dependents' networks it already joins; it is
torn down with the relay on endpoint changes and on down, including
the no-compose-file down path.
The network's lifecycle is hardened against the races a shared,
project-scoped resource runs into:
- a NetworkCreate conflict on the deterministic name re-checks by
label (not just by name) before adopting the existing network, so a
genuine name clash with an unrelated, user-declared network fails
loudly instead of being silently adopted into the relay's isolation
boundary;
- a NotFound gateway lookup after losing that conflict falls back to
an unresolved answer instead of propagating the error;
- removeRelayLinkNetwork and ensureRelayLinkNetworksDown both tolerate
a NetworkRemove conflict from the daemon's asynchronous endpoint
disconnect, which can race the relay container's own removal;
- ensureRelayLinkNetworksDown accumulates failures with errors.Join
across every relay-link network of a project instead of aborting on
the first one, and skips (warn-only) a network that still has active
endpoints, mirroring removeNetwork's existing tolerance for regular
project networks.
docs/extension.md documents that the "desktop" network name in the
get-relay-info answer (under Docker Desktop) is an opaque placeholder,
never a real Docker network a provider can inspect or connect to.
Signed-off-by: Nicolas De Loof <[email protected]>
pre_start hook containers become first-class resources of the
reconciliation plan. The reconciler plans one OpCreateHookContainer per
declared hook — gated on the same predicate the start phase uses to run
the hooks: a replica to start and no running replica SURVIVING the plan
(a recreated replica is not running at start time, so its runners are
planned too). Runner creation depends on every replica node (the
executor resolves the VolumesFrom target against a final live view),
on the infrastructure, and on the purge of previously observed runners:
names are deterministic (<project>-<service>-pre_start-<i>) so repeated
plans converge on the same container instead of accumulating anonymous
ones, and a new HookIndexLabel ties each runner to its hook.
runPreStart becomes pure execution: it looks up the created-state
runner for each hook by label and never creates one. A declared hook
without a prepared runner is an actionable error naming the
reconciliation command (docker compose up <service>) — the accepted
consequence is that stop-then-start of a hooked service errors, since
runners are consumed on success (locked by e2e). Success removal,
failure retention and cancellation cleanup are unchanged.
Runners stay invisible to the start/ps container listings — they carry
no ConfigHashLabel, which getDefaultFilters requires — and down keeps
removing them by hook label.
Rebased onto main, which diverged substantially since this branch was
opened (the unrelated container-spec-layering refactor, #14093,
restructured compose-go's ServiceConfig/PreStartHook types this PR also
touches). Reconciling pre_start.go/pre_start_test.go against that
refactor surfaced a real bug: createPreStartContainer builds the hook's
config via the shared getCreateConfigs, which unconditionally stamps a
ConfigHashLabel — silently contradicting this same commit's own
"invisible to ps/start" guarantee above, since that label is exactly
what getDefaultFilters requires. Fixed by deleting the label after
getCreateConfigs returns, with a test asserting its absence (an
independent local review, via the GitHub Copilot CLI, and a
fresh-context review agent both converged on catching this during
rebase verification). Also restored two security/label tests that only
existed on one side of the rebase (SecuritySensitiveFieldsHonored,
HookLabelsMerged), ported to call createPreStartContainer directly.
Also addresses glours' three review comments on this PR: hook-container
create operations for a service no longer chain serially (each depends
on the same shared containerNodes+infraDeps+purges set, letting the
executor create them in parallel), the corresponding plan-string test
assertion, and a clearer missing-runner error message hinting this
isn't necessarily a planning bug.
Signed-off-by: Nicolas De Loof <[email protected]>
compose-spec/compose-go#866 is merged: jobs top-level element, container
specification layered as ContainerSpec/WorkloadSpec, pre_start hooks
carrying the full container specification resolved at load time. Bump
to the merged head and adapt in the same movement — composite literals
setting moved fields wrap them into the embedded ContainerSpec /
WorkloadSpec (promoted field access was already source-compatible, so
this is literal-only, no behavior change), and pre_start handling is
typed against PreStartHook.
Co-Authored-By: Claude Fable 5 <[email protected]>
Signed-off-by: Nicolas De Loof <[email protected]>
A provider's resource lives outside the compose network: consumers could
only reach it through injected variables carrying a host-published
address — nothing like the compose-native experience of addressing a
service by name at its well-known port.
A provider may now publish where each endpoint of its resource actually
listens:
{"type": "publish-endpoint", "message": "80=localhost:49152"}
The endpoint is announced as seen from the provider's host: the relay —
the component that knows it runs inside a container — rewrites loopback
or unspecified upstream hosts to host.docker.internal (resolved through
its injected host-gateway extra_host); routable addresses pass through.
When at least one endpoint is published, compose deploys a relay
container in place of the service: a minimal TCP forwarder (new relay/
directory, published as docker/compose-relay, overridable with
COMPOSE_RELAY_IMAGE for internal registries) joining the networks of the
services that depend on the provider service, aliased with the service
name. Consumers then use http://<service>:<port> as if the service were
a regular container.
The relay is a first-class project container — canonical name, standard
compose labels including config-hash (label-driven commands run without
the compose file keep seeing the service: ps, logs, stop, down) — plus
the com.docker.compose.relay label declaring its role:
- the reconciler already leaves provider services' containers alone, and
the relay's identity hash (image + routes) makes up idempotent: kept
when routes are unchanged, recreated otherwise;
- process-level commands (exec, cp) refuse a relay — there is no service
process in it to act on;
- the up monitor excludes relays from the containers whose termination
ends an attached up: they are long-lived infrastructure and would
otherwise keep 'up' waiting forever.
The example provider demonstrates the flow behind PROVIDER_DEMO_ENDPOINT
(a detached helper serving a fixed HTTP response), backed by an e2e
scenario asserting the compose-native address works and exec is refused.
Signed-off-by: Nicolas De Loof <[email protected]>
`docker compose port SERVICE PORT` picks the first matching publisher
as its answer, but containerPublishers()'s sort.Slice does not
guarantee tie order, so a dual-stack publish (same target port on
both an IPv4 and an IPv6 address) could non-deterministically return
either one. Switch to sort.SliceStable to preserve the daemon's
original order for ties.
Also drops a dead length check and a duplicated host:port formatter.
Signed-off-by: Guillaume Lours <[email protected]>
Compose silently ignored several spec-valid attributes it doesn't
honor outside Swarm mode: deploy.update_config, credential_spec,
ports[].mode: host, and others had no visible effect on a plain
`docker compose up`.
Detection is wired into compose-go's loader as a fail-closed
allowlist: every attribute path the schema declares, minus the
handful this runtime deliberately doesn't implement, plus a few
value-conditional checks for attributes only unsupported for one
specific value. A newly-specified attribute gets flagged
automatically, with no corresponding code change needed here.
Depends on an unreleased compose-go change; go.mod carries a
temporary replace to a fork branch until it's released.
Signed-off-by: Guillaume Lours <[email protected]>
A failing pre_start hook container is now retained (AutoRemove: false)
so operators can run 'docker logs <id>' and 'docker ps -a' to diagnose
the failure. On success the container is removed explicitly, mirroring
the old AutoRemove behaviour including anonymous volumes.
Before each pre_start run, stale hook containers from a previous failed
run are detected via project+service+HookLabel filters and force-removed
so they do not accumulate.
Changes:
- pkg/api/labels.go: add HookLabel (com.docker.compose.hook)
- pkg/compose/filters.go: add hookFilter helper
- pkg/compose/pre_start.go:
- AutoRemove: false in createPreStartContainer
- HookLabel added to container labels
- runPreStartHook: explicit ContainerRemove on success; retain on failure
- removeOrphanPreStartContainers: new helper called in runPreStart
- Tests: update all existing pre_start tests for the new flow; add
feature tests (success removes, failure retains, orphan cleanup) and
coverage-gap tests (lowestNumberedContainer, waitPreStart cancel,
preStartResultErr, streamPreStartLogs error paths, old-API network
paths, ExecCreate/Attach/Inspect errors, hookExitError branches)
Coverage after: hook.go 100%, pre_start.go most functions 100%
(was 89% and 64% respectively per Codecov delta).
Signed-off-by: Guillaume Lours <[email protected]>
RunOptions embeds CreateOptions, which suggests every creation option
applies to the one-off's dependencies; in reality only Build,
IgnoreOrphans, RemoveOrphans and QuietPull are propagated
(startDependencies builds a fresh CreateOptions from exactly those
four). Setting Recreate or Inherit on a RunOptions was a silent no-op.
The same struct also serves Exec, which honors only the exec-relevant
subset of its ~20 fields.
Both facts are now stated on the type, the embedding and the
Exec-only Index field.
Part of #14074 (B: pkg/api promises things the implementation does not
honor).
Signed-off-by: Nicolas De Loof <[email protected]>
StartOptions fuses two contracts: the fields s.start honors (Project,
AttachTo-as-project-source, Wait, WaitTimeout) and the fields only Up's
foreground session reads (Attach, OnExit, ExitCodeFrom, Watch,
NavigationMenu). Passing the latter to Start compiled, returned no
error, and did nothing.
Each session field is now marked 'honored by Up's foreground session
only; ignored by Start'. Two implicit behaviors become explicit along
the way: Attach doubles as the detached/interactive mode switch (nil
means Up returns once containers are started), and AttachTo has two
unrelated meanings (project reconstruction source for Start, log
scoping for Up).
Part of #14074 (B: pkg/api promises things the implementation does not
honor).
Signed-off-by: Nicolas De Loof <[email protected]>
Three structs carry a Services field with three different meanings:
ProjectLoadOptions.Services narrows the loaded project (the only real
filter), CreateOptions.Services only selects which recreation policy
applies to which service while the WHOLE project converges, and
StartOptions.Services only scopes Up's log monitor and is ignored by
Start. Nothing said so: an SDK consumer calling
Create(project, CreateOptions{Services: []string{"web"}}) reasonably
expected to create web alone, and got the full project.
The interface doc now states the underlying invariant — methods act on
the whole project they receive; scoping is done by narrowing the
project, and downstream Services fields are intent markers, not
filters — and each of the three fields documents its actual effect.
Part of #14074 (B: pkg/api promises things the implementation does not
honor).
Signed-off-by: Nicolas De Loof <[email protected]>
The Compose interface promised 'Scale manages numbers of container
instances running per service', but ScaleOptions had no replica count:
callers had to know they must mutate project.Services[x] via SetScale
before calling, and ScaleOptions.Services only tuned the recreation
policy. An SDK consumer following the interface doc got a no-op
convergence of the whole project.
ScaleOptions gains Replicas (service -> count); the backend applies it
to the model itself before converging, and derives the targeted
services from its keys when Services is not set. The CLI now does what
a CLI should: parse SERVICE=REPLICAS tuples and hand them to the
backend, instead of pre-mutating the model. Passing a pre-mutated
project with an empty Replicas map still behaves as before, so
existing SDK callers are unaffected.
Part of #14074 (B: pkg/api promises things the implementation does not
honor).
Signed-off-by: Nicolas De Loof <[email protected]>
Show an ENGINE column in `compose ps` default table output when the
com.docker.compose.engine label is present. Expose {{.Engine}} for custom
--format templates.
Rename api.EngineLabel to api.ContainerEngineLabel to avoid the
name collision with desktop.EngineLabel (com.docker.desktop.address).
Signed-off-by: Nick Sieger <[email protected]>
With the containerd image store, ImageSummary.ID holds the digest of the
platform-specific manifest so ServiceHash stays stable across attested
rebuilds (see contentDigest). resolveImageVolumes reused that same value
as the `type: image` mount Source, but the daemon only resolves a mount
Source by name/tag or top-level image ID, not by manifest digest — so
`compose up` failed with "No such image" whenever the volume's source
image was already present locally (always for a built image; on a
second run for a pulled one).
Keep Source as the resolved image name, and track the digest separately
via a new com.docker.compose.image-volume-digest label so mustRecreate
can still detect a rebuilt/updated source image independently of Source.
Fixes#14005
Signed-off-by: Ricardo Branco <[email protected]>
Signed-off-by: Guillaume Lours <[email protected]>
Address review feedback on pre_start hook image resolution:
1. GetDependentImages now skips a hook image equal to the service image
(resolved via GetImageNameOrDefault), so `config --images` no longer
prints a duplicate line and pullRequiredImages no longer schedules a
redundant pull for it.
2. pullRequiredImages (up/create path) now dedups dependent images by
reference via a `scheduled` set, so several hooks/services sharing the
same missing image don't schedule concurrent redundant pulls. The hook
pass moved to a helper (addPreStartHookPulls) to keep complexity in check.
3. The `pull` command no longer skips hook images under `pull_policy: build`.
A hook image is a registry image that can't be built, so only `never`
justifies skipping it — making `pull` consistent with the `up` path.
Signed-off-by: Nicolas De Loof <[email protected]>
pre_start hooks run as ephemeral init containers with their own image
(ServiceHook.Image), but that image was ignored by image resolution:
`config --images` didn't list it, `pull` didn't fetch it, and `up` failed
at runtime with "No such image" when it wasn't already present locally.
Add a GetDependentImages helper that returns a service's pre_start hook
images, and use it wherever service images are collected/pulled:
getLocalImagesDigests, pullRequiredImages (up path), the pull command, and
`config --images`. Hook images inherit the parent service pull policy.
post_start/pre_stop hooks run via ExecCreate inside the service container
and never use hook.Image, so they are intentionally out of scope.
Digest resolution/locking (--resolve-image-digests / --lock-image-digests)
is not covered: compose-go's WithImagesResolved only resolves service.Image
(needs an upstream change), and the --lock-image-digests override merges
pre_start by concatenation, which would duplicate hooks.
Signed-off-by: Nicolas De Loof <[email protected]>
Provider info and error messages containing newlines broke the TTY
progress display (timer drifting to a new line, broken cursor
movement). Extract only the first line for progress events via
firstLine(). Full messages remain available through the provider's
own debug message type.
Skip provider services during watch rebuild convergence by adding a
SkipProviders flag to CreateOptions, set only by the watch rebuild
path. This prevents unnecessary re-invocation of providers on every
file change while preserving normal provider execution for all other
commands (up, create, run, scale).
Signed-off-by: Guillaume Lours <[email protected]>
Also update TestDefaultNetworkSettings:
Test that the network with the highest priority is returned as
"primary" network, and other networks as extra networks.
Signed-off-by: Sebastiaan van Stijn <[email protected]>
Currently when using models, the final message is 'confugiring' which could let users think the DMR configuration is still pending
Signed-off-by: Guillaume Lours <[email protected]>
# Conflicts:
# pkg/api/event.go
Most files already grouped imports into "stdlib -> other -> local",
but some files didn't. The gci formatter is similar to goimports, but
has better options to make sure imports are grouped in the expected
order (and to make sure no additional groups are present).
This formatter has a 'fix' function, so code can be re-formatted auto-
matically;
golangci-lint run -v --fix
Signed-off-by: Sebastiaan van Stijn <[email protected]>
This commit adds a new LoadProject method to the Compose service API,
allowing SDK users to programmatically load Compose projects with full
control over the loading process.
Changes:
1. New API method (pkg/api/api.go):
- LoadProject(ctx, ProjectLoadOptions) (*types.Project, error)
- ProjectLoadOptions struct with all loader configuration
- LoadListener callback for event notifications (metrics, etc.)
- ProjectOptionsFns field for compose-go loader options
2. Implementation (pkg/compose/loader.go):
- createRemoteLoaders: Git and OCI remote loader setup
- buildProjectOptions: Translates ProjectLoadOptions to compose-go options
- postProcessProject: Service filtering, labels, resource pruning
3. Unit test (pkg/compose/loader_test.go):
- Tests basic project loading functionality
- Verifies ProjectOptionsFns with cli.WithoutEnvironmentResolution
4. Mock update (pkg/mocks/mock_docker_compose_api.go):
- Added LoadProject to mock interface
Key design decisions:
- LoadListener pattern keeps metrics collection in CLI, not SDK
- ProjectOptionsFns exposes compose-go options directly (e.g., cli.WithInterpolation(false))
- Post-processing in SDK: labels, service filtering, resource pruning
- Environment resolution NOT in SDK (command responsibility)
- Compatibility mode handling (api.Separator)
Signed-off-by: Guillaume Lours <[email protected]>
This commit prepares the Compose service for SDK usage by abstracting away
the hard dependency on command.Cli. The Docker CLI remains the standard path
for the CLI tool, but SDK users can now provide custom implementations of
streams and context information.
Signed-off-by: Guillaume Lours <[email protected]>
This commit introduces WithMaxConcurrency and WithDryRun to replace direct mutators on composeService
commands and flags are translated into a set of functional parameters which are eventually applied
as a ComposeService is created just before being actually used by a command
Signed-off-by: Nicolas De Loof <[email protected]>
This warning was added in [moby@4a8b3ca] to print a warning when building
Linux images from a Windows client. Window's filesystem does not have an
"executable" bit, which mean that, for example, copying a shell script
to an image during build would lose the executable bit. So for Windows
clients, the executable bit would be set on all files, unconditionally.
Originally this was detected in the client, which had direct access to
the API response headers, but when refactoring the client to use a common
library in [moby@535c4c9], this was refactored into a `ImageBuildResponse`
wrapper, deconstructing the API response into an `io.Reader` and a string
field containing only the `OSType` header.
This was the only use and only purpose of the `OSType` field, and now that
BuildKit is the default builder for Linux images, this warning didn't get
printed unless BuildKit was explicitly disabled.
This patch removes the warning, so that we can potentially remove the
field, or the `ImageBuildResponse` type altogether.
[moby@4a8b3ca]: https://github.com/moby/moby/commit/4a8b3cad6096854027151dfbcfb4b2cd8841ad95
[moby@535c4c9]: https://github.com/moby/moby/commit/535c4c9a59b1e58c897677d6948a595cb3d28639
Signed-off-by: Sebastiaan van Stijn <[email protected]>
Nicolas De Loof