## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## Rationale for this change
Sometimes CI failed because of docker rates limits.
```
thread 'test_s3_url_fallback' (11052) panicked at datafusion-cli/tests/cli_integration.rs:116:13:
Failed to start MinIO container. Ensure Docker is running and accessible: failed to pull the image 'minio/minio:RELEASE.2025-02-28T09-55-16Z', error: Docker responded with status code 500: toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit
stack backtrace:
```
Example
https://github.com/apache/datafusion/actions/runs/22262073722/job/64401977127
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
Ignore the tests if rates limit hit only
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes#20225.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
When there is not enough memory to continue external sort, either
increasing the `memory limit` or decreasing
`sort_spill_reservation_bytes` need to be applied. It can be useful to
guide the user with clearer error message by highlighting required
configs for the consistency because expected settings are as follows:
```
SET datafusion.runtime.memory_limit = '10G'
SET datafusion.execution.sort_spill_reservation_bytes = 10485760
```
Current:
```
Not enough memory to continue external sort. Consider increasing the memory limit,
or decreasing sort_spill_reservation_bytes
```
New:
```
Not enough memory to continue external sort. Consider increasing the memory limit config:
'datafusion.runtime.memory_limit', or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.
```
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
A new UT case has been added to cover this use-case and it has been
tested locally successfully.
**For updated snapshot files:**
Following integration tests have been successful by DataFusion Test
Pipeline:
```
test test_cli_top_memory_consumers::case_1 ... ok
test test_cli_top_memory_consumers::case_2 ... ok
test test_cli_top_memory_consumers::case_3 ... ok
```
**Ref:**
https://github.com/apache/datafusion/actions/runs/21811797863/job/62925363536?pr=20226
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
Yes, improving legacy `ExternalSorter` `ResourcesExhausted` Error
Message.
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Supersedes #19010
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
We don't actually need the dependency on `testcontainers` per their
README:
> **Note**: you don't need to explicitly depend on `testcontainers` as
it's re-exported dependency of `testcontainers-modules` with aligned
version between these crates.
-
https://github.com/testcontainers/testcontainers-rs-modules-community/blob/331abcc6e61d9d76e5f8e6ec91566ce874d8fc32/README.md
It was causing some version conflict issues when we tried to bump
`testcontainers-modules` so remove it and keep only modules, fixing the
code to use the re-exports.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
Remove `testcontainers` dependency, amend imports to use re-exports of
`testcontainers-modules`.
Bump `testcontainers-modules` from 0.13 to 0.14.
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
CLI test suite compiles and runs. Ran the SLT postgres compat mode
locally successfully:
```sh
datafusion (testcontainers-0.14)$ PG_COMPAT=true PG_URI="postgresql://postgres@127.0.0.1/postgres" cargo test --features=postgres --test sqllogictests
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.19s
Running bin/sqllogictests.rs (/Users/jeffrey/.cargo_target_cache/debug/deps/sqllogictests-223c346f75c524e2)
Completed 6 test files in 0 seconds
```
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
No.
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes#19380.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
Snapshot test passes but the existing value is in a legacy format.
Updated insta snapshots to new format by running
`cargo insta test --force-update-snapshots`
## What changes are included in this PR?
Snapshots in various directories.
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
Yes
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
No
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
No
## Which issue does this PR close?
This addresses part of https://github.com/apache/datafusion/issues/15804
but does not close it.
## Rationale for this change
Now that we are on MSRV 1.88 we can use rust edition 2024, which brings
let chains and other nice features. It also improves `unsafe` checking.
In order to introduce these changes in slower way instead of one massive
PR that is too difficult to manage we are updating a few crates at a
time.
## What changes are included in this PR?
Updates these crates to 2024.
- datafusion-cli
- datafusion-proto
- datafusion-substrait
- datafusion-wasmtest
## Are these changes tested?
Existing unit tests. There are no functional code changes.
## Are there any user-facing changes?
None.
## Note
It is recommended to review with the ignore whitespace setting:
https://github.com/apache/datafusion/pull/19357/files?w=1
## Which issue does this PR close?
This does not fully close, but is an incremental building block
component for:
- https://github.com/apache/datafusion/issues/17207
The full context of how this code is likely to progress can be seen in
the POC for this effort:
- https://github.com/apache/datafusion/pull/17266
## Rationale for this change
Further fills out method instrumentation
## What changes are included in this PR?
- Adds instrumentation to head requests in the instrumented object store
- Adds instrumentatin to delete requests in the instrumented object
store
- Adds tests for new code
## Are these changes tested?
Yes. New unit tests have been added.
## Are there any user-facing changes?
No-ish
##
cc @alamb
## Which issue does this PR close?
- part of https://github.com/apache/datafusion/issues/17207
## Rationale for this change
As suggested by @BlakeOrth in
https://github.com/apache/datafusion/pull/18045#issuecomment-3403692516
here is an attempt to improve the output of datafusion object store
trace profiling:
## What changes are included in this PR?
Update the output format when `\object_store_profiling summary` is set
Current format (on main, before this PR):
```sql
Summaries:
Get
count: 2
duration min: 0.024603s
duration max: 0.031946s
duration avg: 0.028274s
size min: 8 B
size max: 34322 B
size avg: 17165 B
size sum: 34330 B
```
New format (after this PR):
```sql
DataFusion CLI v50.2.0
> \object_store_profiling summary
ObjectStore Profile mode set to Summary
> select count(*) from 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet';
+----------+
| count(*) |
+----------+
| 1000000 |
+----------+
1 row(s) fetched.
Elapsed 6.754 seconds.
Object Store Profiling
Instrumented Object Store: instrument_mode: Summary, inner: HttpStore
Summaries:
+-----------+----------+-----------+-----------+-----------+-----------+-------+
| Operation | Metric | min | max | avg | sum | count |
+-----------+----------+-----------+-----------+-----------+-----------+-------+
| Get | duration | 0.031645s | 0.047780s | 0.039713s | 0.079425s | 2 |
| Get | size | 8 B | 34322 B | 17165 B | 34330 B | 2 |
+-----------+----------+-----------+-----------+-----------+-----------+-------+
```
## Are these changes tested?
Yes
## Are there any user-facing changes?
Nicer datafusion-cli output
- Adds the ability for a user to choose a summary only output for an
instrumented object store when using the CLI
- The existing "enabled" setting that displays both a summary and a
detailed usage for each object store call has been renamed to `Trace`
to improve clarity
- Adds additional test cases for summary only and modifies existing
tests to use trace
- Updates user guide docs to reflect the CLI flag and command line
changes
* Adds summary output to CLI instrumented object stores
- Adds a `RequestSummary` type for the instrumented object store to
display summary statistics about instrumented requests
- Adds a generic Stats type to track the statistics for the summary
- Adds tests for the new code
- Adds a basic summary output to the user-facing display when profiling
is enabled
- Adds docs for new and newly exported public items
* - Updates integration test and validation snapshot
* Adds instrumentation to get requests for datafusion-cli
- Adds request instrumentation to GET operations when using the
InstrumentedObjectStore in datafusion-cli
- Adds RequestDetails type to store and format details about
instrumented requests
- Adds tests for new functionality
* Add cli-integration test
* Verify the contents of the reqest
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* Improved experience when remote object store URL does not end in /
* refactor
* fix code style
* fix
* fix
* Simplify implementation, add more tests
* tweak
* Add comments
* fix and test non existing
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* Differentiate 0-row and 1-row EmptyRelation in EXPLAIN
The `LogicalPlan::EmptyRelation` can produce no rows or exactly 1 null
(placeholder) row of the requested schema. When viewing EXPLAIN output
of a LogicalPlan it's good to know which one is the case.
* empty: re-run CI
* style: simplify some strings for readability
* fix: formatting in `datafusion/` directory
* refactor: replace long `format!` string
* refactor: replace `format!` with `assert_eq!`
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* Do not normalize values
* Fix tests & update docs
* Prettier
* Lowercase config params
* Add snap to CLI & set up AWS mock
* Refactor tests
* Unify transform and parse
* Fix tests
* Setup CLI
* Show minio output
* Format Cargo.toml
* Do not hardcode AWS params
* Test options parsing
* Add allow http
* Fix aws build
* Fix ip
* Remove slash ☠️
* Format cargo toml
* Remove integration_setup.bash
* Update docs
* Do not hardcode test names
* Relock cargo
* Remove aws sdk and set up minio in-place
* Nit: Add missing ready local to the docs
* Fix backslash test
* Add missing backslash
* put integration scripts in a separate folder
* Move s3 tests from extended to rust flow
* Reorganise the docs
* Prettier
* Do not use rust container to get docker
* Add missing protobuf
* revert change to extended.yml
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* Revert "Fix: New Datafusion-cli streaming printing way should handle corner case for only one small batch which lines are less than max_rows (#14921)"
This reverts commit 463ef3b8bc.
* Revert "Datafusion-cli: Redesign the datafusion-cli execution and print, make it totally streaming printing without memory overhead (#14877)"
This reverts commit 53fc94f22f.
* feat: add `-c option` to pass the SQL query directly as an argument on datafusion-cli
* feat: add integration tests of datafusion-cli
* chore: exclude test data files of datafusion-cli from RAT check
Oleks V