This was racy with the sleep, so the Compose file has been
tweaked to make it pass reliably.
Now, there's 3 services:
* `running` - sleeps forever
* `exit` - exits _successfully_ immediately
* depends on `running` started
* `fail` - exits _with error_ immediately
* depends on `exit` succeeding
Now, the test can ensure that the containers are all run/
started in the expected order the assertions will be reliable.
Before, it was possible for `fail` to run & exit before `exit`,
for example. The `running` service also ensures there's always
at least one other "running" container when we do an abort.
Signed-off-by: Milas Bowman <[email protected]>
When creating automatic file shares, ignore any non-directory
bind mounts, e.g. for an individual (normal) file or a special
type like a Unix socket (`/var/run/docker.sock`).
Additionally, there's no need to create a directory if one
does not exist, the API will handle that. However, the check
for existence remains so that `create_host_path: false`
mounts can be ignored.
Signed-off-by: Milas Bowman <[email protected]>
Unfortunately, the feature flag mechanism for experimental features
isn't adequate. To avoid some edge cases where Compose might try to
use Synchronized file shares with Desktop when the feature isn't
available, we need to query a new endpoint.
Before we move any of this out of experimental, we need to improve
the integration here with Desktop & Compose so that we get all the
necessary feature/experiment state up-front to reduce the quantity
of IPC calls needed up-front.
For now, there's some intentional redundancy to avoid making this
extra call if we can avoid it. The actual endpoint is very cheap/
fast, but every IPC call is a potential point of of failure, so
it's worth it.
Signed-off-by: Milas Bowman <[email protected]>
Use environment variable for global opt-out and Docker Desktop (if
available) to determine specific experiment states.
In the future, we'll allow per-feature opt-in/opt-out via env vars
as well, but currently there is a single `COMPOSE_EXPERIMENTAL` env
var that can be used to opt-out of all experimental features
independently of Docker Desktop configuration.
This has not been the default for quite a while and required
setting an environment variable to revert back.
The tar implementation is more performant and addresses several
edge cases with the original `docker cp` version, so it's time
to fully retire it.
The scaffolding for multiple sync implementations remains to
support future experimentation here.
Signed-off-by: Milas Bowman <[email protected]>
This wasn't always getting passed, so adding it to the wrapper
function where it'll pass the `WithContext()` loader method at
the last moment.
Signed-off-by: Milas Bowman <[email protected]>
This was a bad configuration (my fault) that meant each span was
exported synchronously, as it ended. That can cause weird behavior
such as stuttering/blocking.
There's really no reason to NOT use the batch processor, it's the
recommended way to configure it. In the future, it might make sense
to tune the intervals based on the fact that Compose is a CLI vs
a long-running server app, but we handle flushing out on exit
already, so it's not a huge deal.
Signed-off-by: Milas Bowman <[email protected]>
When using the Moby/Docker Engine API client, we do not have a
useful user agent value being reported. Ideally, in the future,
the Docker CLI will set this appropriately for plugins when it
initializes the client.
For now, manually set it, which is a bit hacky because it
requires some casting & manually invoking an option function
that's technically meant for initialization. In practice, this
is pretty safe - the cast is checked defensively and we ignore
any errors (which shouldn't be possible anyway).
Signed-off-by: Milas Bowman <[email protected]>
Currently, we publish Compose artifacts following the OCI 1.1
specification, which is still in the RC state.
As a result, not all registries support it yet. Most notably,
AWS ECR will reject certain OCI 1.1-compliant requests with
`405 Method Not Supported` with cryptic `Invalid JSON` errors.
This adds initial support for Compose to generate either an
OCI 1.0 or OCI 1.1 compatible manifest. Notably, the OCI 1.0
manifest will be missing the `application/vnd.docker.compose.project`
artifact type, as that does not exist in that version of the
spec. (Less importantly, it uses an empty `ImageConfig`
instead of the newer `application/vnd.oci.empty.v1+json` media
type for the config.)
Currently, this is not exposed as an option (via CLI flags or
env vars). By default, OCI 1.1 is used unless the registry
domain is `amazonaws.com`, which indicates an ECR registry, so
Compose will instead use OCI 1.0.
Moving forward, we should decide how much we want to expose/
support different OCI versions and investigate if there's a
more generic way to feature probe the registry to avoid
maintaining a hardcoded list of domains, which is both tedious
and insufficient.
Signed-off-by: Milas Bowman <[email protected]>
* Eliminate direct dependency on gopkg.in/yaml.v2
* Add gopkg.in/yaml.v2 as a restricted import
* Add github.com/distribution/distribution as a restricted dependency in favor of distribution/reference which is the subset of functionality that Compose needs
* Remove an unused exclusion
NOTE: This does change the `compose config` output slightly but does NOT change the semantics:
* YAML indentation is slightly different for lists (this is a `v2` / `v3` thing)
* JSON is now "minified" instead of pretty-printed (I think this generally desirable and more consistent with other JSON command outputs)
Signed-off-by: Milas Bowman <[email protected]>
I missed this during a refactor and there wasn't test coverage.
Instead of adding more heavy-weight integration tests, I tried
to use `gomock` here to assert on the options objects after CLI
flag parsing. I think with a few more helpers, this could be a
good way to get a lot more combinations covered without adding
a ton of slow E2E tests.
Signed-off-by: Milas Bowman <[email protected]>
The `alpha watch` command current "attaches" to an already-running
Compose project, so it's necessary to run something like
`docker compose up --wait` first.
Now, we'll do the equivalent of an `up --build` before starting the
watch, so that we know the project is up-to-date and running.
Additionally, unlike an interactive `up`, the services are not stopped
when `watch` exits (e.g. via `Ctrl-C`). This prevents the need to start
from scratch each time the command is run - if some services are already
running and up-to-date, they can be used as-is. A `down` can always be
used to destroy everything, and we can consider introducing a flag like
`--down-on-exit` to `watch` or changing the default.
Signed-off-by: Milas Bowman <[email protected]>
The big change here is to pass around an explicit `*BuildOptions` object
as part of Compose operations like `up` & `run` that may or may not do
builds. If the options object is `nil`, no builds whatsoever will be
attempted.
Motivation is to allow for partial rebuilds in the context of an `up`
for watch. This was broken and tricky to accomplish because various parts
of the Compose APIs mutate the `*Project` for convenience in ways that
make it unusable afterwards. (For example, it might set `service.Build = nil`
because it's not going to build that service right _then_. But we might
still want to build it later!)
NOTE: This commit does not actually touch the watch logic. This is all
in preparation to make it possible.
As part of this, a bunch of code moved around and I eliminated a bunch
of partially redundant logic, mostly around multi-platform. Several
edge cases have been addressed as part of this:
* `DOCKER_DEFAULT_PLATFORM` was _overriding_ explicitly set platforms
in some cases, this is no longer true, and it behaves like the Docker
CLI now
* It was possible for Compose to build an image for one platform and
then try to run it for a different platform (and fail)
* Errors are no longer returned if a local image exists but for the
wrong platform - the correct platform will be fetched/built (if
possible).
Because there's a LOT of subtlety and tricky logic here, I've also tried
to add an excessive amount of explanatory comments.
Signed-off-by: Milas Bowman <[email protected]>
If running `up` in foreground mode (i.e. not `-d`),
when exiting via `Ctrl-C`, Compose stops all the
services it launched directly as part of that `up`
command.
In one of the E2E tests (`TestUpDependenciesNotStopped`),
this was occasionally flaking because the stop
behavior was racy: the return might not block on
the stop operation because it gets added to the
error group in a goroutine. As a result, it was
possible for no services to get terminated on exit.
There were a few other related pieces here that
I uncovered and tried to fix while stressing this.
For example, the printer could cause a deadlock if
an event was sent to it after it stopped.
Also, an error group wasn't really appropriate here;
each goroutine is a different operation for printing,
signal-handling, etc. If one part fails, we don't
actually want printing to stop, for example. This has
been switched to a `multierror.Group`, which has the
same API but coalesces errors instead of canceling a
context the moment the first one fails and returning
that single error.
Signed-off-by: Milas Bowman <[email protected]>
* Use unique project name prefixes (some of these tests assert
on output using the project name as a magic string, so could
be impacted by other tests with the same project name prefix)
* Tear down port range project before starting to try and avoid
race conditions with the engine and port assignment
Signed-off-by: Milas Bowman <[email protected]>
This is a good place to start introducing (local) exclusivity
to Compose. Now, when `alpha watch` launches, it will check for
the existence of a PID file in the user XDG runtime directory,
and create one if the existing one is stale or does not exist.
If the PID file exists and is valid, an error is returned and
Compose exits.
A slight tweak to the experimental remote Git loader has been
made to use the XDG package for consistency.
Signed-off-by: Milas Bowman <[email protected]>
By default, `compose up` attaches to all services (i.e.
shows log output from every associated container). If
a service is specified, e.g. `compose up foo`, then
only `foo`'s logs are tailed. The `--attach-dependencies`
flag can also be used, so that if `foo` depended upon
`bar`, then `bar`'s logs would also be followed. It's
also possible to use `--no-attach` to filter out one
or more services explicitly, e.g. `compose up --no-attach=noisy`
would launch all services, including `noisy`, and would
show log output from every service _except_ `noisy`.
Lastly, it's possible to use `up --attach` to explicitly
restrict to a subset of services (or their dependencies).
How these flags interact with each other is also worth
thinking through.
There were a few different connected issues here, but
the primary issue was that running `compose up foo` was
always attaching dependencies regardless of `--attach-dependencies`.
The filtering logic here has been updated so that it
behaves predictably both when launching all services
(`compose up`) or a subset (`compose up foo`) as well
as various flag combinations on top of those.
Notably, this required making some changes to how it
watches containers. The logic here between attaching
for logs and monitoring for lifecycle changes is
tightly coupled, so some changes were needed to ensure
that the full set of services being `up`'d are _watched_
and the subset that should have logs shown are _attached_.
(This does mean faking the attach with an event but not
actually doing it.)
While handling that, I adjusted the context lifetimes
here, which improves error handling that gets shown to
the user and should help avoid potential leaks by getting
rid of a `context.Background()`.
Signed-off-by: Milas Bowman <[email protected]>
Refactor to use a consistent code path for determining the build
args for a service image regardless of whether BuildKit or the
classic builder is being used.
After recent changes, these code paths had diverged, so the classic
builder was missing the proxy variables from the Docker client
config.
Signed-off-by: Milas Bowman <[email protected]>
This was left over from debugging, but we should not block.
OTel will handle the connection in the background.
Signed-off-by: Milas Bowman <[email protected]>
Swap the default implementation now that batching is merged.
Keeping the `docker cp` based implementation around for the
moment, but it needs to be _explicitly_ disabled now by setting
`COMPOSE_EXPERIMENTAL_WATCH_TAR=0`.
After the next release, we should remove the `docker cp`
implementation entirely.
Signed-off-by: Milas Bowman <[email protected]>
If an optional dependency exits successfully (exit code of 0),
with a service condition of `service_completed_successfully`,
don't log a warning.
Signed-off-by: Milas Bowman <[email protected]>
* When waiting for dependencies, `select` on the context as well
as the ticker
* Write multiple progress events "transactionally" (i.e. hold the
lock for the duration to avoid other events being interleaved)
* Do not change "finished" steps back to "in progress" to prevent
flickering
Signed-off-by: Milas Bowman <[email protected]>
Adjust the debouncing logic so that it applies to all inbound file
events, regardless of whether they match a sync or rebuild rule.
When the batch is flushed out, if any event for the service is a
rebuild event, then the service is rebuilt and all sync events for
the batch are ignored. If _all_ events in the batch are sync events,
then a sync is triggered, passing the entire batch at once. This
provides a substantial performance win for the new `tar`-based
implementation, as it can efficiently transfer the changes in bulk.
Additionally, this helps with jitter, e.g. it's not uncommon for
there to be double-writes in quick succession to a file, so even if
there's not many files being modified at once, it can still prevent
some unnecessary transfers.
Signed-off-by: Milas Bowman <[email protected]>
Support services with scale > 1 for the tar watch sync.
Add a "lossy" multi-writer specific to pipes that writes the
tar data to each `io.PipeWriter`, which is connected to `stdin`
for the `tar` process being exec'd in the container.
The data is written serially to each writer. This could be
adjusted to do concurrent writes but that will rapidly increase
the I/O load, so is not done here - in general, 99% of the
time you'll be developing (and thus using watch/sync) with a
single replica of a service.
If a write fails, the corresponding `io.PipeWriter` is removed
from the active set and closed with an error.
This means that a single container copy failing won't stop
writes to the others that are succeeding. Of course, they will
be in an inconsistent state afterwards still, but that's a
different problem.
Signed-off-by: Milas Bowman <[email protected]>
We cannot guarantee the exact value of `CapEff` across
environments, and this test has started failing some places,
e.g. Docker Desktop, and now GitHub Actions (likely due to
a kernel upgrade on the runners or similar).
By setting `privileged: true` on the build, we're asking for
the `security.insecure` entitlement on the build. A safe
assumption is that will include `CAP_SYS_ADMIN`, which won't
be present otherwise, so mask the `CapEff` value and check
for that.
It's worth noting that realistically, the build won't even
be able to complete without the correct entitlement, since the
`Dockerfile` uses `RUN --security=insecure`, so this is really
an additional sanity check.
Signed-off-by: Milas Bowman <[email protected]>
Just moving some code around in preparation for an alternative
sync implementation that can do bulk transfers by using `tar`.
Signed-off-by: Milas Bowman <[email protected]>
When building, if images are being pushed, ensure that only
named images (i.e. services with a populated `image` field)
are attempted to be pushed.
Services without `image` get an auto-generated name, which
will be a "Docker library" reference since they're in the
format `$project-$service`, which is implicitly the same as
`docker.io/library/$project-$service`. A push for that is
never desirable / will always fail.
The key here is that we cannot overwrite the `<svc>.image`
field when doing builds, as we need to be able to check for
its presence to determine whether a push makes sense.
Fixes#10813.
Signed-off-by: Milas Bowman <[email protected]>
Fix forthcoming via https://github.com/compose-spec/compose-go/pull/436
which addresses some symlink limitations. These can
actually effect other platforms but are most common
on macOS because the test creates temporary directories,
which are symlinked on macOS.
Signed-off-by: Milas Bowman <[email protected]>
Lots of our phony Compose files launch pointless long-lived processes
so we can assert on state. However, this means they often don't respond
well to signals on their own, requiring Compose to timeout and kill
them when doing a `down`.
Add in lots of `init: true` where appropriate so that we don't block
for no reason while running E2E tests all over the place.
Additionally, a couple tests have gotten a cleanup so they don't leave
behind containers. I still want to build this into the framework in
the future, but this is easier for the moment and won't cause any
trouble in the future.
Signed-off-by: Milas Bowman <[email protected]>
The local Go package module path was missing from a couple of jobs,
which made them slower than needed since they were re-downloading
a bunch of dependencies.
In particular, this makes `make lint` waaaay faster!
Signed-off-by: Milas Bowman <[email protected]>
* Run `down` before and after test to not leave around containers
* Kill the `wait` process that's waiting on `infinity`
* NOTE: If the test is actually working, this should exit once
the `down` happens, but this ensures that we kill everything
we start
I'd like to generalize more of this into the framework, but this
is a quick fix to prevent filling up CI machines with tons of
processes over time.
Signed-off-by: Milas Bowman <[email protected]>
As part of the fix for #10668, the logic was adjusted so that the
default (highest-priority) network is used in the `ContainerCreate`,
and then the remaining networks are connected via calls to
`NetworkConnect` before starting the container.
Unfortunately, `ServiceConfig::NetworksByPriority` is neither
deterministic nor stable when networks have the same priority.
It's non-deterministic because the order of networks from parsing
YAML is random, since they are loaded into a Go map (which have
random iteration order). Additionally, it's not using a `SortStable`
in `compose-go`, so even if the load order was predictable, it
still might produce different results.
While I look at improving `compose-go` here to prevent this from
tripping us up in the future, this fix looks at _all_ networks for
a service and ignores the "default" one now. Before, it would
always skip the first one in the slice since that _should_ have
been the "default".
Signed-off-by: Milas Bowman <[email protected]>
Engine API only allows at most one network to be connected as
part of the ContainerCreate API request. Compose will pick the
highest priority network.
Afterwards, the remaining networks (if any) are connected before
the container is actually started.
The big change here is that, previously, the highest-priority
network was connected in the create, and then disconnected and
immediately reconnected along with all the others. This was
racy because evidently connecting the container to the network
as part of the create isn't synchronous, so sometimes when Compose
tried to disconnect it, the API would return an error like:
```
container <id> is not connected to the network <network>
```
To avoid needing to disconnect and immediately reconnect, the
network config logic has been refactored to ensure that it sets
up the network config correctly the first time.
Signed-off-by: Milas Bowman <[email protected]>
* Move all the initialization code out of `main.go`
* Ensure spans are reported when there's an error with the
command
* Attach the Compose version & active Docker context to the
resource instead of the span
* Name the root CLI span `cli/<cmd>` for clarity and grab
the full subcommand path (e.g. `alpha-viz` instead of just
`viz`)
Signed-off-by: Milas Bowman <[email protected]>
* If there's no `otel` key (or the value is `null`) in the config,
don't return an error
* Propagate error from the exporter instead of panicking
Signed-off-by: Milas Bowman <[email protected]>
This is a bunch of OTEL initialization code. It's all in
`internal/` because there are re-usable parts here, but Compose
isn't the right spot. Once we've stabilized the interfaces a bit
and the need arises, we can move it to a separate module.
Currently, a single span is produced to wrap the root Compose
command.
Compose will respect the standard OTEL environment variables
as well as OTEL metadata from the Docker context. Both can be
used simultaneously. The latter is intended for local system
integration and is restricted to Unix sockets / named pipes.
None of this is enabled by default. It requires setting the
`COMPOSE_EXPERIMENTAL_OTEL=1` environment variable to
gate it during development.
Signed-off-by: Milas Bowman <[email protected]>
Attempting to fix the state of codecov action checks right now,
which are behaving very erratically.
Using the new functionality in Go 1.20 to merge multiple reports,
so now the unit & E2E coverage data reports are stored as artifacts
and then downloaded, merged, and finally uploaded to codecov as a
new job.
Additionally, add a `codecov.yml` config and try to turn down the
aggressiveness of it for CI checks.
Signed-off-by: Milas Bowman <[email protected]>
Write the warning using `logrus.Warn`. The function being used was
coming from `cfssl`'s log package, which was presumably the result
of auto-import being _slightly_ too aggressive.
(Note: `cfssl` is still an indirect dependency after this.)
Signed-off-by: Milas Bowman <[email protected]>
Some error messages have been tweaked slightly, this adapts the
assertions to work on both Engine v20.10.x and v23.x.
Signed-off-by: Milas Bowman <[email protected]>
If we go to inspect a container that we got an event for and it
no longer exists on the server, handle clean up without erroring
out.
Fixes#10373.
Signed-off-by: Milas Bowman <[email protected]>
Use latest Go minor release. Note: this release included fixes for
several CVEs, but they do not impact Compose.
Small errors have been fixed to keep the linter happy.
Signed-off-by: Milas Bowman <[email protected]>
Instead of trying to make this work nicely cross-platform,
just push the Coverage logic into the GitHub Actions job,
as that's really where we care about it.
(It's surprisingly difficult to make this nicely portable;
to make PowerShell not error out if the path does not exist
you have to ignore ALL errors and the way that env vars are
passed to processes is not the same.)
Signed-off-by: Milas Bowman <[email protected]>
This was running two tests in parallel that would build/delete the
same images. Run in serial instead since that's not safe.
Signed-off-by: Milas Bowman <[email protected]>
Milas Bowman