mirror of
https://github.com/langchain-ai/datafusion.git
synced 2026-07-18 13:15:59 -04:00
e937cadbcceff6a42bee2c5fc8d03068fa0eb30c
728 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oleks V
|
df8f818b29 |
chore: Avoid build fails on MinIO rate limits (#20472)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change Sometimes CI failed because of docker rates limits. ``` thread 'test_s3_url_fallback' (11052) panicked at datafusion-cli/tests/cli_integration.rs:116:13: Failed to start MinIO container. Ensure Docker is running and accessible: failed to pull the image 'minio/minio:RELEASE.2025-02-28T09-55-16Z', error: Docker responded with status code 500: toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit stack backtrace: ``` Example https://github.com/apache/datafusion/actions/runs/22262073722/job/64401977127 <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? Ignore the tests if rates limit hit only <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Dmitrii Blaginin
|
60457d0b0a |
Runs-on for more actions (#20274)
Follow up on https://github.com/apache/datafusion/pull/20107: switch more actions to the new flow | Job | OLD | NEW | Delta | |---|---|---|---| | **linux build test** (from #20107) | 3m 55s | 1m 46s | -2m 09s (55% faster) | | **cargo test (amd64)** (from #20107) | 11m 34s | 3m 13s | -8m 21s (72% faster) | | **cargo check datafusion features** | 11m 18s | 6m 21s | -4m 57s (44% faster) | | **cargo examples (amd64)** | 9m 13s | 4m 35s | -4m 38s (50% faster) | | **verify benchmark results (amd64)** | 11m 48s | 4m 22s | -7m 26s (63% faster) | | **cargo check datafusion-substrait features** | 10m 20s | 3m 56s | -6m 24s (62% faster) | | **cargo check datafusion-proto features** | 4m 48s | 2m 25s | -2m 23s (50% faster) | | **cargo test datafusion-cli (amd64)** | 5m 42s | 1m 58s | -3m 44s (65% faster) | | **cargo test doc (amd64)** | 8m 07s | 3m 16s | -4m 51s (60% faster) | | **cargo doc** | 5m 10s | 1m 56s | -3m 14s (63% faster) | | **Run sqllogictest with Postgres runner** | 6m 06s | 2m 46s | -3m 20s (55% faster) | | **Run sqllogictest in Substrait round-trip mode** | 6m 42s | 2m 38s | -4m 04s (61% faster) | | **clippy** | 6m 01s | 2m 10s | -3m 51s (64% faster) | | **check configs.md and \*\*\*_functions.md is up-to-date** | 6m 54s | 2m 12s | -4m 42s (68% faster) | |
||
|
dependabot[bot]
|
043f908b60 |
chore(deps): bump the all-other-cargo-deps group with 6 updates (#20470)
Bumps the all-other-cargo-deps group with 6 updates: | Package | From | To | | --- | --- | --- | | [async-compression](https://github.com/Nullus157/async-compression) | `0.4.39` | `0.4.40` | | [clap](https://github.com/clap-rs/clap) | `4.5.59` | `4.5.60` | | [wasm-bindgen-test](https://github.com/wasm-bindgen/wasm-bindgen) | `0.3.58` | `0.3.61` | | [aws-credential-types](https://github.com/smithy-lang/smithy-rs) | `1.2.12` | `1.2.13` | | [tonic](https://github.com/hyperium/tonic) | `0.14.4` | `0.14.5` | | [syn](https://github.com/dtolnay/syn) | `2.0.116` | `2.0.117` | Updates `async-compression` from 0.4.39 to 0.4.40 <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/Nullus157/async-compression/commit/9d848a02f13f3a56542e4123be8947a8da06097e"><code>9d848a0</code></a> chore: release (<a href="https://redirect.github.com/Nullus157/async-compression/issues/452">#452</a>)</li> <li><a href="https://github.com/Nullus157/async-compression/commit/9df508b037dafb9a2d80bfd60fcd6679891abef1"><code>9df508b</code></a> Fix update of bytes read in the encoder state machine. (<a href="https://redirect.github.com/Nullus157/async-compression/issues/456">#456</a>)</li> <li><a href="https://github.com/Nullus157/async-compression/commit/0370b470db4dbe8f92a178320438e3094495a99a"><code>0370b47</code></a> Stop consuming input on errors in codecs. (<a href="https://redirect.github.com/Nullus157/async-compression/issues/451">#451</a>)</li> <li><a href="https://github.com/Nullus157/async-compression/commit/9a4b0961f988cdc2b70dae0f4310046c7fedc307"><code>9a4b096</code></a> chore(deps): update rand requirement from 0.9 to 0.10 (<a href="https://redirect.github.com/Nullus157/async-compression/issues/449">#449</a>)</li> <li>See full diff in <a href="https://github.com/Nullus157/async-compression/compare/async-compression-v0.4.39...async-compression-v0.4.40">compare view</a></li> </ul> </details> <br /> Updates `clap` from 4.5.59 to 4.5.60 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/releases">clap's releases</a>.</em></p> <blockquote> <h2>v4.5.60</h2> <h2>[4.5.60] - 2026-02-19</h2> <h3>Fixes</h3> <ul> <li><em>(help)</em> Quote empty default values, possible values</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's changelog</a>.</em></p> <blockquote> <h2>[4.5.60] - 2026-02-19</h2> <h3>Fixes</h3> <ul> <li><em>(help)</em> Quote empty default values, possible values</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/clap-rs/clap/commit/33d24d844b11c0e926ae132e1af338ff070bdf4a"><code>33d24d8</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/9332409f4a6c1d5c22064e839ec8e9bc040f3be7"><code>9332409</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/b7adce5a17089596eecb2af6985e6503f2ffcd38"><code>b7adce5</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/6166">#6166</a> from fabalchemy/fix-dynamic-powershell-completion</li> <li><a href="https://github.com/clap-rs/clap/commit/009bba44ec3d182028ec3a72f5b6f3e507827768"><code>009bba4</code></a> fix(clap_complete): Improve powershell registration</li> <li><a href="https://github.com/clap-rs/clap/commit/d89d57dfb4bdd18930a40c6d7f4fadb23ee9c5b3"><code>d89d57d</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/f18b67ec3d4ce6ac1acf115adaab2f16ab2ed3c7"><code>f18b67e</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/9d218eb418526143c9110f734f78a608b8cf6440"><code>9d218eb</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/6165">#6165</a> from epage/shirt</li> <li><a href="https://github.com/clap-rs/clap/commit/126440ca846613671e1dac98198b2ceb17dab2b0"><code>126440c</code></a> fix(help): Correctly calculate padding for short-only args</li> <li><a href="https://github.com/clap-rs/clap/commit/9e3c05ef3800a3e638b8224a7881a81517a4f4db"><code>9e3c05e</code></a> test(help): Show panic with short, valueless arg</li> <li><a href="https://github.com/clap-rs/clap/commit/c9898d0fece98d8520d3dd954cf457b685b3308f"><code>c9898d0</code></a> test(help): Verify short with value</li> <li>Additional commits viewable in <a href="https://github.com/clap-rs/clap/compare/clap_complete-v4.5.59...clap_complete-v4.5.60">compare view</a></li> </ul> </details> <br /> Updates `wasm-bindgen-test` from 0.3.58 to 0.3.61 <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/wasm-bindgen/wasm-bindgen/commits">compare view</a></li> </ul> </details> <br /> Updates `aws-credential-types` from 1.2.12 to 1.2.13 <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> Updates `tonic` from 0.14.4 to 0.14.5 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/hyperium/tonic/releases">tonic's releases</a>.</em></p> <blockquote> <h2>v0.14.5</h2> <h2>What's Changed</h2> <ul> <li>Add max connections setting</li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/hyperium/tonic/compare/v0.14.4...v0.14.5">https://github.com/hyperium/tonic/compare/v0.14.4...v0.14.5</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/hyperium/tonic/commit/3f7caf3171393734ef19e12d010bd9c945c9e242"><code>3f7caf3</code></a> chore: prepare v0.14.5 release (<a href="https://redirect.github.com/hyperium/tonic/issues/2516">#2516</a>)</li> <li><a href="https://github.com/hyperium/tonic/commit/3f56644955162b344ce4a2641823776574ae98e4"><code>3f56644</code></a> grpc(chore): add missing copyright notices (<a href="https://redirect.github.com/hyperium/tonic/issues/2513">#2513</a>)</li> <li><a href="https://github.com/hyperium/tonic/commit/1769c91a96f054416e0d11c84fcc26284262dda2"><code>1769c91</code></a> feat(xds): implement xDS subscription worker (<a href="https://redirect.github.com/hyperium/tonic/issues/2478">#2478</a>)</li> <li><a href="https://github.com/hyperium/tonic/commit/56f8c6db4718c32e8cb1732438b87c85a3a8c1f6"><code>56f8c6d</code></a> feat(grpc): Add TCP listener API in the Runtime trait + tests for server cred...</li> <li><a href="https://github.com/hyperium/tonic/commit/149f3668f0514bd79f12524778ca76eb6341a3f5"><code>149f366</code></a> feat(grpc) Add channel credentials API + Insecure credentials (<a href="https://redirect.github.com/hyperium/tonic/issues/2495">#2495</a>)</li> <li>See full diff in <a href="https://github.com/hyperium/tonic/compare/v0.14.4...v0.14.5">compare view</a></li> </ul> </details> <br /> Updates `syn` from 2.0.116 to 2.0.117 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/dtolnay/syn/releases">syn's releases</a>.</em></p> <blockquote> <h2>2.0.117</h2> <ul> <li>Fix parsing of <code>self::</code> pattern in first function argument (<a href="https://redirect.github.com/dtolnay/syn/issues/1970">#1970</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/dtolnay/syn/commit/7bcb37cdb3399977658c8b52d2441d37e42e48f2"><code>7bcb37c</code></a> Release 2.0.117</li> <li><a href="https://github.com/dtolnay/syn/commit/9c6e7d3b8df7b30909d60395f88a6ca07688e1c1"><code>9c6e7d3</code></a> Merge pull request <a href="https://redirect.github.com/dtolnay/syn/issues/1970">#1970</a> from dtolnay/receiver</li> <li><a href="https://github.com/dtolnay/syn/commit/019a84847eded0cdb1f7856e0752ba618155cfc9"><code>019a848</code></a> Fix self:: pattern in first function argument</li> <li><a href="https://github.com/dtolnay/syn/commit/23f54f3cf61ddedd5daea4f347eca2d4b84c8abb"><code>23f54f3</code></a> Update test suite to nightly-2026-02-18</li> <li><a href="https://github.com/dtolnay/syn/commit/b99b9a627c46580343398472e7b08a131357a994"><code>b99b9a6</code></a> Unpin CI miri toolchain</li> <li>See full diff in <a href="https://github.com/dtolnay/syn/compare/2.0.116...2.0.117">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
833635557b |
chore(deps): bump clap from 4.5.57 to 4.5.59 (#20404)
Bumps [clap](https://github.com/clap-rs/clap) from 4.5.57 to 4.5.59. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/releases">clap's releases</a>.</em></p> <blockquote> <h2>v4.5.59</h2> <h2>[4.5.59] - 2026-02-16</h2> <h3>Fixes</h3> <ul> <li><code>Command::ignore_errors</code> no longer masks help/version on subcommands</li> </ul> <h2>v4.5.58</h2> <h2>[4.5.58] - 2026-02-11</h2> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's changelog</a>.</em></p> <blockquote> <h2>[4.5.59] - 2026-02-16</h2> <h3>Fixes</h3> <ul> <li><code>Command::ignore_errors</code> no longer masks help/version on subcommands</li> </ul> <h2>[4.5.58] - 2026-02-11</h2> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/clap-rs/clap/commit/0bb3ad7e12e729be9f152391558689ac4fdd31ec"><code>0bb3ad7</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/5cb5ce3873a882ba2a7d619864202eadef21fffa"><code>5cb5ce3</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/245c8ba75a481250a48170f1add11532a7b7fd33"><code>245c8ba</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/6149">#6149</a> from epage/wrap</li> <li><a href="https://github.com/clap-rs/clap/commit/dd17a418a9e6665c98ff6e0ba2a039fd1921988e"><code>dd17a41</code></a> fix(help): Correctly calculate wrap points with ANSI escape codes</li> <li><a href="https://github.com/clap-rs/clap/commit/2cc4e350b9ea8955a9cf229405407426921e7871"><code>2cc4e35</code></a> test(ui): Avoid override term width</li> <li><a href="https://github.com/clap-rs/clap/commit/93e3559b1e4c8c81377f3598f7249b7708f4c379"><code>93e3559</code></a> refactor(help): Clarify that we're carrying over indentation</li> <li><a href="https://github.com/clap-rs/clap/commit/b1c46e633c04f5cb0d819b15f25c1fde1a6e42c4"><code>b1c46e6</code></a> refactor(help): Clarify var name</li> <li><a href="https://github.com/clap-rs/clap/commit/03b9b38df059c1a9a529f295e038f81de295627a"><code>03b9b38</code></a> test(help): Show styled wrapping behavior</li> <li><a href="https://github.com/clap-rs/clap/commit/c9a39a534c3e95926be272765bec48a80e5ea9e7"><code>c9a39a5</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/6146">#6146</a> from clap-rs/renovate/actions-checkout-5.x</li> <li><a href="https://github.com/clap-rs/clap/commit/58599fb7bf865f8ec0a7a021dea8111f5dffe6d2"><code>58599fb</code></a> chore(deps): Update actions/checkout action to v5</li> <li>Additional commits viewable in <a href="https://github.com/clap-rs/clap/compare/clap_complete-v4.5.57...clap_complete-v4.5.59">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
9acf387a21 |
chore(deps): bump aws-config from 1.8.13 to 1.8.14 (#20407)
Bumps [aws-config](https://github.com/smithy-lang/smithy-rs) from 1.8.13 to 1.8.14. <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dario curreri
|
f471aaf275 |
feat(datafusion-cli): enhance CLI helper with default hint (#20310)
# Pull Request Description ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> When the CLI prompt is empty, users have no indication of how to get help or exit. Showing a subtle gray hint (e.g. `\? for help, \q to quit`) on an empty line improves discoverability without changing any existing behavior. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Implement `Hinter::hint` on `CliHelper` so that when the current line is empty, a default hint is shown (e.g. `\? for help, \q to quit`) in gray. - Add a `DEFAULT_HINT_SUGGESTION` constant and use it for the hint text. - Expose the highlighter’s `Color` type and its methods as `pub(crate)` and add `Color::gray()` for styling the hint. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> No new tests were added; the change is a small UI hint. Manual testing: start the CLI and confirm the hint appears on an empty line and disappears when typing. Happy to add tests if the project expects them for this behavior. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Yes. When the input line is empty, users now see a gray hint: `\? for help, \q to quit`. No API or CLI flag changes. Documentation could optionally mention this hint in the CLI docs if the project documents REPL UX. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Eren Avsarogullari
|
4d06c40d8d |
Improve ExternalSorter ResourcesExhausted Error Message (#20226)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #20225. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> When there is not enough memory to continue external sort, either increasing the `memory limit` or decreasing `sort_spill_reservation_bytes` need to be applied. It can be useful to guide the user with clearer error message by highlighting required configs for the consistency because expected settings are as follows: ``` SET datafusion.runtime.memory_limit = '10G' SET datafusion.execution.sort_spill_reservation_bytes = 10485760 ``` Current: ``` Not enough memory to continue external sort. Consider increasing the memory limit, or decreasing sort_spill_reservation_bytes ``` New: ``` Not enough memory to continue external sort. Consider increasing the memory limit config: 'datafusion.runtime.memory_limit', or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'. ``` ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> A new UT case has been added to cover this use-case and it has been tested locally successfully. **For updated snapshot files:** Following integration tests have been successful by DataFusion Test Pipeline: ``` test test_cli_top_memory_consumers::case_1 ... ok test test_cli_top_memory_consumers::case_2 ... ok test test_cli_top_memory_consumers::case_3 ... ok ``` **Ref:** https://github.com/apache/datafusion/actions/runs/21811797863/job/62925363536?pr=20226 ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Yes, improving legacy `ExternalSorter` `ResourcesExhausted` Error Message. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
dependabot[bot]
|
b790ed313e |
chore(deps): bump aws-config from 1.8.12 to 1.8.13 (#20263)
Bumps [aws-config](https://github.com/smithy-lang/smithy-rs) from 1.8.12 to 1.8.13. <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
cbe878ccdd |
chore(deps): bump clap from 4.5.56 to 4.5.57 (#20265)
Bumps [clap](https://github.com/clap-rs/clap) from 4.5.56 to 4.5.57. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/releases">clap's releases</a>.</em></p> <blockquote> <h2>v4.5.57</h2> <h2>[4.5.57] - 2026-02-03</h2> <h3>Fixes</h3> <ul> <li>Regression from 4.5.55 where having an argument with <code>.value_terminator("--")</code> caused problems with an argument with <code>.last(true)</code></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's changelog</a>.</em></p> <blockquote> <h2>[4.5.57] - 2026-02-03</h2> <h3>Fixes</h3> <ul> <li>Regression from 4.5.55 where having an argument with <code>.value_terminator("--")</code> caused problems with an argument with <code>.last(true)</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/clap-rs/clap/commit/69c0ddbbfb56db1bccbb5954b62bb89a567a3c8d"><code>69c0ddb</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/8206bba73fd6c5d567cb95949fd1c3c6c48e4e20"><code>8206bba</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/c109d67ea493823727411f60f354edb3d83117ee"><code>c109d67</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/6104">#6104</a> from epage/hide</li> <li><a href="https://github.com/clap-rs/clap/commit/9d7f2128f77023941b53b7cfc311120a2ead75a2"><code>9d7f212</code></a> fix(complete): Hide dot files on dynamic completer</li> <li><a href="https://github.com/clap-rs/clap/commit/77b3fdbbea64ae0b0b3a51309bcbb861360de8d1"><code>77b3fdb</code></a> test(complete): Show dot file behavior</li> <li><a href="https://github.com/clap-rs/clap/commit/f89b9b8d1b818a2eb3863745be48725ace2d8f12"><code>f89b9b8</code></a> test(derive): Make stable across upgrade</li> <li><a href="https://github.com/clap-rs/clap/commit/58eb8a937ac6ca4a59614dc26deedb6cfe16c424"><code>58eb8a9</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/10a2a7559b0663143d56c850c0c40ed31620cb5b"><code>10a2a75</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/a42eebf56bf20d587347abb03105f95c98bfda51"><code>a42eebf</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/6103">#6103</a> from epage/mut_subcommands</li> <li><a href="https://github.com/clap-rs/clap/commit/5335f54d73eef9276c13313661fcfffb720c87cf"><code>5335f54</code></a> feat: Add Command::mut_subcommands</li> <li>Additional commits viewable in <a href="https://github.com/clap-rs/clap/compare/clap_complete-v4.5.56...clap_complete-v4.5.57">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dario curreri
|
aeeb0ba07a |
fix(datafusion-cli): solve row count bug addingsaturating_add to prevent potential overflow (#20185)
Currently `print_options::MaxRows::Unlimited` basically always panics
with `attempt to add with overflow`, because we are summing `usize::MAX`
with a positive number.
`saturating_add` solves this issue
# MRE
```rs
fn main() {
let max = usize::MAX;
println!("max: {}", max);
let max = max + 1;
println!("max: {}", max);
}
```
|
||
|
dependabot[bot]
|
4557033c30 |
chore(deps): bump clap from 4.5.54 to 4.5.56 (#20131)
Bumps [clap](https://github.com/clap-rs/clap) from 4.5.54 to 4.5.56. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/releases">clap's releases</a>.</em></p> <blockquote> <h2>v4.5.56</h2> <h2>[4.5.56] - 2026-01-29</h2> <h3>Fixes</h3> <ul> <li>On conflict error, don't show conflicting arguments in the usage</li> </ul> <h2>v4.5.55</h2> <h2>[4.5.55] - 2026-01-27</h2> <h3>Fixes</h3> <ul> <li>Fix inconsistency in precedence between positionals with a <code>value_terminator("--")</code> and escapes (<code>--</code>) where <code>./foo -- bar</code> means the first arg is empty, rather than escaping future args</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's changelog</a>.</em></p> <blockquote> <h2>[4.5.56] - 2026-01-29</h2> <h3>Fixes</h3> <ul> <li>On conflict error, don't show conflicting arguments in the usage</li> </ul> <h2>[4.5.55] - 2026-01-27</h2> <h3>Fixes</h3> <ul> <li>Fix inconsistency in precedence between positionals with a <code>value_terminator("--")</code> and escapes (<code>--</code>) where <code>./foo -- bar</code> means the first arg is empty, rather than escaping future args</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/clap-rs/clap/commit/9cec1007acdc3cd990feded4322a4bccd2fd471c"><code>9cec100</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/00e72e06f46e2c21e5bb4dd82aa5fca02a9e5c16"><code>00e72e0</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/c7848ff6fc3f8e0f7b66eaee10d44b43eea54538"><code>c7848ff</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/6094">#6094</a> from epage/home</li> <li><a href="https://github.com/clap-rs/clap/commit/60184fb76a3d88277f89430402d01a121feb858c"><code>60184fb</code></a> feat(complete): Expand ~ in native completions</li> <li><a href="https://github.com/clap-rs/clap/commit/09969d3c1af9dd22fb944c09f8b1c27274cad824"><code>09969d3</code></a> chore(deps): Update Rust Stable to v1.89 (<a href="https://redirect.github.com/clap-rs/clap/issues/6093">#6093</a>)</li> <li><a href="https://github.com/clap-rs/clap/commit/520beb5ec2d2bb5dd11912d27127df4e97027965"><code>520beb5</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/2bd8ab3c009fc975db28209c3c3fb526364342ae"><code>2bd8ab3</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/220875b58511028ba9cd38f7195b8b3315b72d0d"><code>220875b</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/6091">#6091</a> from epage/possible</li> <li><a href="https://github.com/clap-rs/clap/commit/e5eb6c9d84efac5d660322e92dbbc0158266602d"><code>e5eb6c9</code></a> fix(help): Integrate 'Possible Values:' into 'Arg::help'</li> <li><a href="https://github.com/clap-rs/clap/commit/594a771030e43df8c806ea1a029862339739a0f3"><code>594a771</code></a> refactor(help): Make empty tracking more consistent</li> <li>Additional commits viewable in <a href="https://github.com/clap-rs/clap/compare/clap_complete-v4.5.54...clap_complete-v4.5.56">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Gabriel
|
b6f7521752 |
Do not require mut in memory reservation methods (#19759)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Prerequisite for the following PRs: - https://github.com/apache/datafusion/pull/19760 - https://github.com/apache/datafusion/pull/19761 Even if the api on the `MemoryPool` does not require `&mut self` for growing/shrinking the reserved size, the api in `MemoryReservation` does, making simple implementations irrepresentable without synchronization primitives. For example, the following would require a `Mutex` for concurrent access to the `MemoryReservation` in different threads, even though the `MemoryPool` doesn't: ```rust let mut stream: SendableRecordBatchStream = SendableRecordBatchStream::new(); let mem: Arc<MemoryReservation> = Arc::new(MemoryReservation::new_empty()); let mut builder = ReceiverStreamBuilder::new(10); let tx = builder.tx(); { let mem = mem.clone(); builder.spawn(async move { while let Some(msg) = stream.next().await { mem.try_grow(msg.unwrap().get_array_memory_size()); // ❌ `mem` is not mutable tx.send(msg).unwrap(); } }); } builder .build() .inspect_ok(|msg| mem.shrink(msg.get_array_memory_size())); // ❌ `mem` is not mutable ``` ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Make the methods in `MemoryReservation` require `&self` instead of `&mut self` for allowing concurrent shrink/grows from different tasks for the same reservation. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> yes, by current tests ## Are there any user-facing changes? Users can now safely call methods of `MemoryReservation` from different tasks without synchronization primitives. This is a backwards compatible API change, as it will work out of the box for current users, however, depending on their clippy configuration, they might see some new warnings about "unused muts" in their codebase. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Nick
|
8afccc9510 |
Allow dropping qualified columns (#19549)
## Which issue does this PR close? - Closes #19548 ## Rationale for this change Explanation in the issue. Motivation coming more concretely from datafusion-python https://github.com/apache/datafusion-python/issues/1305#issuecomment-3649891506 ## What changes are included in this PR? * Adds the test from the issue to highlight expected behavior * Expands drop_columns to coerce things into a fully qualified column to support the range of column varieties * This further adds a helper to extract the table name associated with the dataframe to simplify use of qualified drop columns support * This is potentially the most controversial part. I could see a nicer api being `df.col(<name>)` to match the expr version but then we probably do repeated checks for the underlying table name unless there is some caching somewhere. Maybe that performance impact isn't significant. ## Are these changes tested? Yes some additional tests are provided. ## Are there any user-facing changes? I had to update the `drop_columns(&[])` test since the type can no longer be inferred. I'm not sure if that is representative of any actual use cases though since I expect the more common is a vector that might be empty in which case the type would be specified. It now requires specifying columns with dots in them similar to other places `"\"f.col1\""` to disambiguate from `"f.col1"`. |
||
|
Andrew Lamb
|
0808f3a8d2 |
Improvements to list_files_cache table function (#19703)
## Which issue does this PR close? - Follow on to https://github.com/apache/datafusion/pull/19616 ## Rationale for this change I had a few minor comments / suggestions while reviewing https://github.com/apache/datafusion/pull/19616 from @jizezhang but they weren't needed to do the initial merge, so I would like to propose them in a follow up PR ## What changes are included in this PR? 1. Improve documentation 2. Improve handling of `table_ref` in ListingTableURL 3. use Null rather than `"NULL"` in `list_files_cache` table function I can break this into separate PRs if that would help ## Are these changes tested? Yes by CI ## Are there any user-facing changes? The `list_files_cache` function now might return null |
||
|
Adrian Garcia Badaracco
|
0cf45cae9c |
Refactor cache APIs to support ordering information (#19597)
## Which issue does this PR close? Part of #19433 ## Rationale for this change In preparation for ordering inference from Parquet metadata, the cache system needs refactoring to: 1. Support storing ordering information alongside statistics 2. Simplify the `CacheAccessor` trait by removing the `Extra` associated type and `*_with_extra` methods 3. Move validation logic into typed wrapper structs with explicit `is_valid_for()` methods ## What changes are included in this PR? ### Simplify `CacheAccessor` trait **Before:** ```rust pub trait CacheAccessor<K, V>: Send + Sync { type Extra: Clone; fn get(&self, k: &K) -> Option<V>; fn get_with_extra(&self, k: &K, e: &Self::Extra) -> Option<V>; fn put(&self, key: &K, value: V) -> Option<V>; fn put_with_extra(&self, key: &K, value: V, e: &Self::Extra) -> Option<V>; // ... other methods } ``` **After:** ```rust pub trait CacheAccessor<K, V>: Send + Sync { fn get(&self, key: &K) -> Option<V>; fn put(&self, key: &K, value: V) -> Option<V>; // ... other methods (no Extra type, no *_with_extra methods) } ``` ### Introduce typed wrapper structs for cached values Instead of passing validation metadata separately via `Extra`, embed it in the cached value type: - **`CachedFileMetadata`** - contains `meta: ObjectMeta`, `statistics: Arc<Statistics>`, `ordering: Option<LexOrdering>` - **`CachedFileList`** - contains `files: Arc<Vec<ObjectMeta>>` with `filter_by_prefix()` helper - **`CachedFileMetadataEntry`** - contains `meta: ObjectMeta`, `file_metadata: Arc<dyn FileMetadata>` Each wrapper has an `is_valid_for(&ObjectMeta)` method that checks if the cached entry is still valid (size and last_modified match). ### New validation pattern The typical usage pattern changes from: ```rust // Before: validation hidden in get_with_extra if let Some(stats) = cache.get_with_extra(&path, &object_meta) { // use stats } ``` To: ```rust // After: explicit validation if let Some(cached) = cache.get(&path) { if cached.is_valid_for(&object_meta) { // use cached.statistics } } ``` ### Add ordering support - `CachedFileMetadata` has new `ordering: Option<LexOrdering>` field - `FileStatisticsCacheEntry` has new `has_ordering: bool` field for introspection ## Are these changes tested? Yes, existing cache tests pass plus new tests for ordering support. ## Are there any user-facing changes? Breaking change to cache traits. Users with custom cache implementations will need to: 1. Update `CacheAccessor` impl to remove `Extra` type and `*_with_extra` methods 2. Update cached value types to the new wrappers (`CachedFileMetadata`, etc.) 3. Update callsites to use the new validation pattern with `is_valid_for()` 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
jizezhang
|
e6049de5a7 |
Make default ListingFilesCache table scoped (#19616)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Builds on https://github.com/apache/datafusion/pull/19388 - Closes https://github.com/apache/datafusion/issues/19573 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> This PR explores one way to make `ListFilesCache` table scoped. A session level cache is still used, but the cache key is made a "table-scoped" path, for which a new struct ``` pub struct TableScopedPath(pub Option<TableReference>, pub Path); ``` is defined. `TableReference` comes from `CreateExternalTable` passed to `ListingTableFactory::create`. Additionally, when a table is dropped, all entries related to a table is dropped by modifying `SessionContext::find_and_deregister` method. Testing (change on adding `list_files_cache()` for cli is included for easier testing). - Testing cache reuse on a single table. ``` > \object_store_profiling summary ObjectStore Profile mode set to Summary > create external table test stored as parquet location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/'; 0 row(s) fetched. Elapsed 14.878 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Summary, inner: AmazonS3(overturemaps-us-west-2) Summaries: +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Operation | Metric | min | max | avg | sum | count | +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Get | duration | 0.030597s | 0.209235s | 0.082396s | 36.254189s | 440 | | Get | size | 204782 B | 857230 B | 497304.88 B | 218814144 B | 440 | | List | duration | 0.192037s | 0.192037s | 0.192037s | 0.192037s | 1 | | List | size | | | | | 1 | +-----------+----------+-----------+-----------+-------------+-------------+-------+ > select table, path, unnest(metadata_list) from list_files_cache() limit 1; +-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | table | path | UNNEST(list_files_cache().metadata_list) | +-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test | release/2025-12-17.0/theme=base | {file_path: release/2025-12-17.0/theme=base/type=bathymetry/part-00000-dd0f2f50-b436-4710-996f-f1b06181a3a1-c000.zstd.parquet, file_modified: 2025-12-17T22:32:50, file_size_bytes: 40280159, e_tag: "15090401f8f936c3f83bb498cb99a41d-3", version: NULL} | +-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row(s) fetched. Elapsed 0.058 seconds. Object Store Profiling > select count(*) from test where type = 'infrastructure'; +-----------+ | count(*) | +-----------+ | 142969564 | +-----------+ 1 row(s) fetched. Elapsed 0.028 seconds. Object Store Profiling ``` - Test separate cache entries are created for two tables with same path ``` > create external table test2 stored as parquet location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/'; 0 row(s) fetched. Elapsed 14.798 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Summary, inner: AmazonS3(overturemaps-us-west-2) Summaries: +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Operation | Metric | min | max | avg | sum | count | +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Get | duration | 0.030238s | 0.350465s | 0.073670s | 32.414654s | 440 | | Get | size | 204782 B | 857230 B | 497304.88 B | 218814144 B | 440 | | List | duration | 0.133334s | 0.133334s | 0.133334s | 0.133334s | 1 | | List | size | | | | | 1 | +-----------+----------+-----------+-----------+-------------+-------------+-------+ > select count(*) from test2 where type = 'bathymetry'; +----------+ | count(*) | +----------+ | 59963 | +----------+ 1 row(s) fetched. Elapsed 0.009 seconds. Object Store Profiling > select table, path from list_files_cache(); +-------+---------------------------------+ | table | path | +-------+---------------------------------+ | test | release/2025-12-17.0/theme=base | | test2 | release/2025-12-17.0/theme=base | +-------+---------------------------------+ 2 row(s) fetched. Elapsed 0.004 seconds. ``` - Test cache associated with a table is dropped when table is dropped, and the other table with same path is unaffected. ``` > drop table test; 0 row(s) fetched. Elapsed 0.015 seconds. Object Store Profiling > select table, path from list_files_cache(); +-------+---------------------------------+ | table | path | +-------+---------------------------------+ | test2 | release/2025-12-17.0/theme=base | +-------+---------------------------------+ 1 row(s) fetched. Elapsed 0.005 seconds. Object Store Profiling > select count(*) from list_files_cache() where table = 'test'; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row(s) fetched. Elapsed 0.014 seconds. > select count(*) from test2 where type = 'infrastructure'; +-----------+ | count(*) | +-----------+ | 142969564 | +-----------+ 1 row(s) fetched. Elapsed 0.013 seconds. Object Store Profiling ``` - Test that dropping a view does not remove cache ``` > create view test2_view as (select * from test2 where type = 'infrastructure'); 0 row(s) fetched. Elapsed 0.103 seconds. Object Store Profiling > select count(*) from test2_view; +-----------+ | count(*) | +-----------+ | 142969564 | +-----------+ 1 row(s) fetched. Elapsed 0.094 seconds. Object Store Profiling > drop view test2_view; 0 row(s) fetched. Elapsed 0.002 seconds. Object Store Profiling > select table, path from list_files_cache(); +-------+---------------------------------+ | table | path | +-------+---------------------------------+ | test2 | release/2025-12-17.0/theme=base | +-------+---------------------------------+ 1 row(s) fetched. Elapsed 0.007 seconds. ``` ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
jizezhang
|
1037f0aa20 |
feat: add list_files_cache table function for datafusion-cli (#19388)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes https://github.com/apache/datafusion/issues/19055. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ``` > CREATE EXTERNAL TABLE nyc_taxi_rides STORED AS PARQUET LOCATION 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/' ; 0 row(s) fetched. Elapsed 10.061 seconds. > SELECT metadata_size_bytes, expires_in, unnest(metadata_list) FROM list_files_cache(); +---------------------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | metadata_size_bytes | expires_in | UNNEST(list_files_cache().metadata_list) | +---------------------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200901.parquet, file_modified: 2025-05-30T09:44:23, file_size_bytes: 222192983, e_tag: "e8d016c3c7af80bf911d96387febe2c1-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200902.parquet, file_modified: 2025-05-30T09:46:00, file_size_bytes: 211023080, e_tag: "1021626ff5ef606422aa7121edd69f3b-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200903.parquet, file_modified: 2025-05-30T09:47:20, file_size_bytes: 229202874, e_tag: "96e7494b217099c6a07e9c4298cbe783-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200904.parquet, file_modified: 2025-05-30T09:44:37, file_size_bytes: 225659965, e_tag: "728c45fabdcd8e40bdef4dfc28df9b0f-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200905.parquet, file_modified: 2025-05-30T09:46:12, file_size_bytes: 232847306, e_tag: "f59e45bd8bd1d77cd7ae8ab6ab468bcc-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200906.parquet, file_modified: 2025-05-30T09:47:26, file_size_bytes: 224226575, e_tag: "8ebb698eea85f9af87065ac333efc449-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200907.parquet, file_modified: 2025-05-30T09:44:52, file_size_bytes: 217168413, e_tag: "7d7ee77f6cac4adc18aa3a9e74600dd3-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200908.parquet, file_modified: 2025-05-30T09:46:23, file_size_bytes: 217303109, e_tag: "e9883055d92a33b941aab971423e681b-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200909.parquet, file_modified: 2025-05-30T09:47:28, file_size_bytes: 223333499, e_tag: "6f0917e6003b38df9060d71c004eb961-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200910.parquet, file_modified: 2025-05-30T09:44:54, file_size_bytes: 246300471, e_tag: "8928b29da44e041021e10077683b7817-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200911.parquet, file_modified: 2025-05-30T09:46:37, file_size_bytes: 227920860, e_tag: "4cd26a1a7f82af080c33e890dc1fef27-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200912.parquet, file_modified: 2025-05-30T09:44:24, file_size_bytes: 233873308, e_tag: "23f4584e494e3c065c777c270c9eedbc-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201001.parquet, file_modified: 2025-05-30T09:45:18, file_size_bytes: 235166925, e_tag: "effcc8cc41b40cf7ac466f911d7b9459-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201002.parquet, file_modified: 2025-05-30T09:46:59, file_size_bytes: 177367931, e_tag: "ce8b7817ecc47da86ccbfa6b51ffa06b-10", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201003.parquet, file_modified: 2025-05-30T09:44:26, file_size_bytes: 205857224, e_tag: "94a078b61e3b652387e6f2a673dc3f4e-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201004.parquet, file_modified: 2025-05-30T09:45:04, file_size_bytes: 243024246, e_tag: "a1efbebfdabc204e0041d8714aaec01a-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201005.parquet, file_modified: 2025-05-30T09:46:47, file_size_bytes: 248130090, e_tag: "d3cf585e00ce627a807348c84a42d0a6-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201006.parquet, file_modified: 2025-05-30T09:44:25, file_size_bytes: 237068130, e_tag: "831db33281a5c017f8ffc466bd47546b-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201007.parquet, file_modified: 2025-05-30T09:45:35, file_size_bytes: 234826090, e_tag: "790e05983e6592e4920c88fbd2bfe774-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201008.parquet, file_modified: 2025-05-30T09:47:14, file_size_bytes: 197990272, e_tag: "d87ddb446e5cbc0f6831fafd95cfd027-11", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201009.parquet, file_modified: 2025-05-30T09:44:27, file_size_bytes: 243408943, e_tag: "abfbe3b29942bcd68d131d95540278d3-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201010.parquet, file_modified: 2025-05-30T09:45:47, file_size_bytes: 225277041, e_tag: "f768c7b77497b2bf3efd5cb2a4362977-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201011.parquet, file_modified: 2025-05-30T09:47:23, file_size_bytes: 220010577, e_tag: "c6830cbe1f3ae918f9280db3aa847b03-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201012.parquet, file_modified: 2025-05-30T09:44:24, file_size_bytes: 219773352, e_tag: "264f7ea433076690a3bbe5566168e5c5-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201101.parquet, file_modified: 2025-05-30T09:45:52, file_size_bytes: 212535107, e_tag: "ca3bdc2707b29667c78c39517781eac4-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201102.parquet, file_modified: 2025-05-30T09:47:23, file_size_bytes: 223138164, e_tag: "e2b3c0fd0c0d66ac6363600de0c8b2ad-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201103.parquet, file_modified: 2025-05-30T09:44:26, file_size_bytes: 252843261, e_tag: "fd5d4e01568cd6e7ef1e00de76441e5b-15", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201104.parquet, file_modified: 2025-05-30T09:46:10, file_size_bytes: 233123935, e_tag: "2b510cc2c0c73d9ec7374c9e6d56c388-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201105.parquet, file_modified: 2025-05-30T09:44:24, file_size_bytes: 246843111, e_tag: "abc2f58bd520b2013aa1a333d317c70c-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201106.parquet, file_modified: 2025-05-30T09:44:58, file_size_bytes: 238786647, e_tag: "0e456698dc42a850ff7b764506cb511d-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201107.parquet, file_modified: 2025-05-30T09:46:40, file_size_bytes: 233249259, e_tag: "28177227cbff94a6a819a0568a14e9b2-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201108.parquet, file_modified: 2025-05-30T09:44:25, file_size_bytes: 212681184, e_tag: "fdcb442e1010630c0553a7018762a8ba-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201109.parquet, file_modified: 2025-05-30T09:45:13, file_size_bytes: 232399266, e_tag: "ccca37be5a3579a8bc644490226ed29a-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201110.parquet, file_modified: 2025-05-30T09:46:52, file_size_bytes: 248471033, e_tag: "eebe34c1bb74f63433eb607810969553-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201111.parquet, file_modified: 2025-05-30T09:44:26, file_size_bytes: 231103826, e_tag: "7c76b9fc111462b76336d63bce3253c7-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201112.parquet, file_modified: 2025-05-30T09:45:40, file_size_bytes: 236102882, e_tag: "26c10d1d85c4565cbb9e8fc6a7bc745c-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201201.parquet, file_modified: 2025-05-30T09:47:21, file_size_bytes: 236184052, e_tag: "8cdc15a22462579dcf90d669cea0f04b-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201202.parquet, file_modified: 2025-05-30T09:44:27, file_size_bytes: 238377570, e_tag: "4e6734c5c2e77c68dde5155a45dac81c-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201203.parquet, file_modified: 2025-05-30T09:46:06, file_size_bytes: 258226172, e_tag: "b7b07fa0f4fefcf0ba0fc69ba344b5c8-15", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201204.parquet, file_modified: 2025-05-30T09:44:25, file_size_bytes: 248190698, e_tag: "968c13850fa9a7cb46337bc8fc9d13fa-14", version: NULL} | | . | | . | | . | +---------------------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 96 row(s) fetched. (First 40 displayed. Use --maxrows to adjust) Elapsed 0.057 seconds. ``` ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> This will enable a new user-facing table function to datafusion cli. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Jeffrey Vo
|
09455f1814 |
chore: bump testcontainers-modules to 0.14 and remove testcontainers dep (#19620)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Supersedes #19010 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> We don't actually need the dependency on `testcontainers` per their README: > **Note**: you don't need to explicitly depend on `testcontainers` as it's re-exported dependency of `testcontainers-modules` with aligned version between these crates. - https://github.com/testcontainers/testcontainers-rs-modules-community/blob/331abcc6e61d9d76e5f8e6ec91566ce874d8fc32/README.md It was causing some version conflict issues when we tried to bump `testcontainers-modules` so remove it and keep only modules, fixing the code to use the re-exports. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Remove `testcontainers` dependency, amend imports to use re-exports of `testcontainers-modules`. Bump `testcontainers-modules` from 0.13 to 0.14. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> CLI test suite compiles and runs. Ran the SLT postgres compat mode locally successfully: ```sh datafusion (testcontainers-0.14)$ PG_COMPAT=true PG_URI="postgresql://postgres@127.0.0.1/postgres" cargo test --features=postgres --test sqllogictests Finished `test` profile [unoptimized + debuginfo] target(s) in 0.19s Running bin/sqllogictests.rs (/Users/jeffrey/.cargo_target_cache/debug/deps/sqllogictests-223c346f75c524e2) Completed 6 test files in 0 seconds ``` ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
dependabot[bot]
|
d9d55cfc6d |
chore(deps): bump aws-config from 1.8.11 to 1.8.12 (#19453)
Bumps [aws-config](https://github.com/smithy-lang/smithy-rs) from 1.8.11 to 1.8.12. <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
kosiew
|
ead8209803 |
update insta snapshots (#19381)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19380. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Snapshot test passes but the existing value is in a legacy format. Updated insta snapshots to new format by running `cargo insta test --force-update-snapshots` ## What changes are included in this PR? Snapshots in various directories. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> No |
||
|
Peter Lee
|
2c9077a8bf |
[datafusion-cli] Implement average LIST duration for object store profiling (#19127)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes https://github.com/apache/datafusion/issues/18138 ## Rationale for this change The `list` operation returns a stream, so it previously recorded `duration: None`, missing performance insights. Time-to-first-item is a useful metric for list operations, indicating how quickly results start. This adds duration tracking by measuring time until the first item is yielded (or the stream ends). ## What changes are included in this PR? 1. Added `TimeToFirstItemStream`: A stream wrapper that measures elapsed time from creation until the first item is yielded (or the stream ends if empty). 2. Updated `instrumented_list`: Wraps the inner stream with `TimeToFirstItemStream` to record duration. 3. Changed `requests` field: Switched from `Mutex<Vec<RequestDetails>>` to `Arc<Mutex<Vec<RequestDetails>>>` to allow sharing across async boundaries (needed for the stream wrapper). 4. Updated tests: Modified `instrumented_store_list` to consume at least one stream item and verify that `duration` is now `Some(Duration)` instead of `None`. ## Are these changes tested? Yes. The existing test `instrumented_store_list` was updated to: - Consume at least one item from the stream using `stream.next().await` - Assert that `request.duration.is_some()` (previously `is_none()`) All tests pass, including the updated list test and other instrumented operation tests. ## Are there any user-facing changes? Users with profiling enabled will see duration values for `list` operations instead of nothing. |
||
|
Tim Saucer
|
59dcc362ca |
Update remaining crates to rust 2024 edition (#19361)
## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/15804 ## Rationale for this change Now that we are on MSRV 1.88 we can use rust edition 2024, which brings let chains and other nice features. It also improves `unsafe` checking. In order to introduce these changes in slower way instead of one massive PR that is too difficult to manage we are updating a few crates at a time. ## What changes are included in this PR? Updates these crates to 2024. - datafusion-benchmarks - datafusion-ffi - datafusion-sqllogictest - datafusion-examples ## Are these changes tested? Existing unit tests. There are no functional code changes. ## Are there any user-facing changes? None. ## Note It is recommended to review with the ignore whitespace setting: https://github.com/apache/datafusion/pull/19361/files?w=1 |
||
|
Tim Saucer
|
1f26716280 |
Update 4 crates to rust 2024 edition (#19357)
## Which issue does this PR close? This addresses part of https://github.com/apache/datafusion/issues/15804 but does not close it. ## Rationale for this change Now that we are on MSRV 1.88 we can use rust edition 2024, which brings let chains and other nice features. It also improves `unsafe` checking. In order to introduce these changes in slower way instead of one massive PR that is too difficult to manage we are updating a few crates at a time. ## What changes are included in this PR? Updates these crates to 2024. - datafusion-cli - datafusion-proto - datafusion-substrait - datafusion-wasmtest ## Are these changes tested? Existing unit tests. There are no functional code changes. ## Are there any user-facing changes? None. ## Note It is recommended to review with the ignore whitespace setting: https://github.com/apache/datafusion/pull/19357/files?w=1 |
||
|
dependabot[bot]
|
ea8087d77e |
chore(deps): bump clap from 4.5.50 to 4.5.53 (#19326)
Bumps [clap](https://github.com/clap-rs/clap) from 4.5.50 to 4.5.53. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/releases">clap's releases</a>.</em></p> <blockquote> <h2>v4.5.53</h2> <h2>[4.5.53] - 2025-11-19</h2> <h3>Features</h3> <ul> <li>Add <code>default_values_if</code>, <code>default_values_ifs</code></li> </ul> <h2>v4.5.52</h2> <h2>[4.5.52] - 2025-11-17</h2> <h3>Fixes</h3> <ul> <li>Don't panic when <code>args_conflicts_with_subcommands</code> conflicts with an <code>ArgGroup</code></li> </ul> <h2>v4.5.51</h2> <h2>[4.5.51] - 2025-10-29</h2> <h3>Fixes</h3> <ul> <li><em>(help)</em> Correctly calculate padding for short flags that take a value</li> <li><em>(help)</em> Don't panic on short flags using <code>ArgAction::Count</code></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's changelog</a>.</em></p> <blockquote> <h2>[4.5.53] - 2025-11-19</h2> <h3>Features</h3> <ul> <li>Add <code>default_values_if</code>, <code>default_values_ifs</code></li> </ul> <h2>[4.5.52] - 2025-11-17</h2> <h3>Fixes</h3> <ul> <li>Don't panic when <code>args_conflicts_with_subcommands</code> conflicts with an <code>ArgGroup</code></li> </ul> <h2>[4.5.51] - 2025-10-29</h2> <h3>Fixes</h3> <ul> <li><em>(help)</em> Correctly calculate padding for short flags that take a value</li> <li><em>(help)</em> Don't panic on short flags using <code>ArgAction::Count</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/clap-rs/clap/commit/3716f9f4289594b43abec42b2538efd1a90ff897"><code>3716f9f</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/613b69a6b7bff729b7a363fa0c91fd03f48d12c3"><code>613b69a</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/d117f7acdeedebaf5fd7847debb15c834423f159"><code>d117f7a</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/6028">#6028</a> from epage/arg</li> <li><a href="https://github.com/clap-rs/clap/commit/cb8255d2f3c7f12ebf07ec1c55ac98b6848599a9"><code>cb8255d</code></a> feat(builder): Allow quoted id's for arg macro</li> <li><a href="https://github.com/clap-rs/clap/commit/1036060f1319412d3d50d821a7b39a0a0122f0f7"><code>1036060</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/6025">#6025</a> from AldaronLau/typos-in-faq</li> <li><a href="https://github.com/clap-rs/clap/commit/2fcafc0aee6380e1f0c44a3e927cef1bfc88930e"><code>2fcafc0</code></a> docs: Fix minor grammar issues in FAQ</li> <li><a href="https://github.com/clap-rs/clap/commit/a380b65fe9eceade90bce8aeb13c205265fcceee"><code>a380b65</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/6023">#6023</a> from epage/template</li> <li><a href="https://github.com/clap-rs/clap/commit/4d7ab1483cd0f0849668d274aa2fb6358872eca9"><code>4d7ab14</code></a> chore: Update from _rust/main template</li> <li><a href="https://github.com/clap-rs/clap/commit/b8a7ea49d973a35bb6b3f43506b8319f340a20a4"><code>b8a7ea4</code></a> chore(deps): Update Rust Stable to v1.87 (<a href="https://redirect.github.com/clap-rs/clap/issues/18">#18</a>)</li> <li><a href="https://github.com/clap-rs/clap/commit/f9842b3b3f920ef64c5fc06298b4762018d88809"><code>f9842b3</code></a> chore: Avoid MSRV problems out of the box</li> <li>Additional commits viewable in <a href="https://github.com/clap-rs/clap/compare/clap_complete-v4.5.50...clap_complete-v4.5.53">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Nuno Faria
|
dc4e3ab473 |
feat: Implement the statistics_cache function (#19054)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #18953. ## Rationale for this change Allow a way to check the contents of the file statistics cache. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Added the `statistics_cache` function to `datafusion-cli`. - Converted `FileStatisticsCache` to a trait and implemented the `list_entries` method. - Added unit tests. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Yes, `FileStatisticsCache` has been changed to a trait. Previous implementations need to implement the `list_entries` method. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Andrew Lamb
|
a06073913b |
Update to arrow, parquet to 57.1.0 (#18820)
## Which issue does this PR close? - related to https://github.com/apache/arrow-rs/issues/8464 ## Rationale for this change Get latest and greatest code from arrow ## What changes are included in this PR? 1. Update to Arrow 57.1.0 2. Update for API changes (comments inline) ## Are these changes tested? Yes, by CI ## Are there any user-facing changes? No --------- Co-authored-by: Raz Luvaton <16746759+rluvaton@users.noreply.github.com> |
||
|
dependabot[bot]
|
ec51b94eeb |
chore(deps): bump aws-config from 1.8.7 to 1.8.11 (#18896)
Bumps [aws-config](https://github.com/smithy-lang/smithy-rs) from 1.8.7 to 1.8.11. <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Congxian Qiu
|
6d9ab45399 |
Enhance the help message for invalid command in datafusion-cli (#18603)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #18375 . ## What changes are included in this PR? Enhance the help message for invalid command, add the `help` command in the help message. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Manual test <img width="737" height="190" alt="image" src="https://github.com/user-attachments/assets/073d4b96-db6f-448b-a7a4-3f9fa456a72e" /> <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? No <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Andrew Lamb
|
c6ad17cf2b |
Upgrade DataFusion to arrow/parquet 57.0.0 (#17888)
## Which issue does this PR close? - Related to https://github.com/apache/arrow-rs/issues/7835 - Closes #3666 Note while this PR looks massive, a large portion is display updates due to better display of Fields and DataTypes ## Rationale for this change Upgrade to the latest arrow Also, there are several new features in arrow-57 that I want to be able to test including Variant, arrow-avro, and a new parquet metadata reader. ## What changes are included in this PR? 1. Update arrow/parquet 2. Update prost 3. Update substrait 4. Update pbjson 5. Make API changes to avoid deprecated APIs ## Are these changes tested? By CI ## Are there any user-facing changes? New arrow |
||
|
dependabot[bot]
|
1feb80f164 |
chore(deps): bump clap from 4.5.48 to 4.5.50 (#18292)
Bumps [clap](https://github.com/clap-rs/clap) from 4.5.48 to 4.5.50. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/releases">clap's releases</a>.</em></p> <blockquote> <h2>v4.5.50</h2> <h2>[4.5.50] - 2025-10-20</h2> <h3>Features</h3> <ul> <li>Accept <code>Cow</code> where <code>String</code> and <code>&str</code> are accepted</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's changelog</a>.</em></p> <blockquote> <h2>[4.5.50] - 2025-10-20</h2> <h3>Features</h3> <ul> <li>Accept <code>Cow</code> where <code>String</code> and <code>&str</code> are accepted</li> </ul> <h2>[4.5.49] - 2025-10-13</h2> <h3>Fixes</h3> <ul> <li><em>(help)</em> Correctly wrap when ANSI escape codes are present</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/clap-rs/clap/commit/d8acd4729878ca72d305e6cf7adf7acc0da36738"><code>d8acd47</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/7c2b8d9ad4d22650f969bd455d80b4181a7e25ff"><code>7c2b8d9</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/e69a2ea55bc9076d95caf60d79e481581f688724"><code>e69a2ea</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/5987">#5987</a> from mernen/fix-bash-comp-words-loop</li> <li><a href="https://github.com/clap-rs/clap/commit/e03cc2e798183e9528f53d42d8b2699f034fc667"><code>e03cc2e</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/5988">#5988</a> from cordx56/fix-builder-custom-version-docs</li> <li><a href="https://github.com/clap-rs/clap/commit/5ab2579844a47a26b4567f77a7b9d198be006f0a"><code>5ab2579</code></a> fix: Minor fix for builder docs about version</li> <li><a href="https://github.com/clap-rs/clap/commit/2f66432721bd24602455dc3e31765195c6107c34"><code>2f66432</code></a> fix(complete): Only parse arguments before current</li> <li><a href="https://github.com/clap-rs/clap/commit/4d9d2100f75693645ea68180ed4b6b3ecacb9923"><code>4d9d210</code></a> test(complete): Illustrate current behavior in Bash</li> <li><a href="https://github.com/clap-rs/clap/commit/6abe2f8c61e31d8d43fee42c18414926c60893be"><code>6abe2f8</code></a> chore: Release</li> <li><a href="https://github.com/clap-rs/clap/commit/d5c74542ce628b57424caec88efee1a231c436a0"><code>d5c7454</code></a> docs: Update changelog</li> <li><a href="https://github.com/clap-rs/clap/commit/5b2e960267b94d4811c9c3b99c62899a87505413"><code>5b2e960</code></a> Merge pull request <a href="https://redirect.github.com/clap-rs/clap/issues/5985">#5985</a> from mernen/bash-cur</li> <li>Additional commits viewable in <a href="https://github.com/clap-rs/clap/compare/clap_complete-v4.5.48...clap_complete-v4.5.50">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Aldrin M
|
167baf7182 |
Fix: Do not normalize table names when deserializing from protobuf (#18187)
## Which issue does this PR close? Closes #18122 ## Rationale for this change Existing behavior is to use the `relation` field of `ColumnRelation` message to construct a `TableReference` ([mod.rs#L146][fromcol_proto], [mod.rs#L171][tryfrom_proto]). However, the `relation` field is a string and `From<String> for TableReference` always calls parse_identifiers_normalized with `ignore_case: False`, which always normalizes the identifier to lower case ([TableReference::parse_str][parse_str]). For a description of the bug at a bit of a higher level, see #18122. ## What changes are included in this PR? This PR introduces the following: 1. An implementation `From<protobuf::ColumnRelation>` and `From<&protobuf::ColumnRelation>` for `TableReference`. 2. Updated logic in `TryFrom<&protobuf::DFSchema>` for `DFSchema` and in `From<protobuf::Column>` for `Column` that correctly leads to the new `From` impls for `TableReference` to be invoked. 3. A new method, `TableReference::parse_str_normalized`, that parses an identifier without normalizing it, with some logic from `TableReference::parse_str` being refactored to accommodate code reuse. ## Are these changes tested? Commit a355196 adds a new test case, `roundtrip_mixed_case_table_reference`, that tests the desired behavior. The existing behavior (without the fix in 0616df2 and with the extra line `println!("{}", server_logical_plan.display_indent_schema());`): ``` cargo test "roundtrip_mixed_case_table_reference" --test proto_integration -- --nocapture Compiling datafusion-proto v48.0.1 (/Users/aldrinm/code/bauplanlabs/datafusion/octalene-datafusion/datafusion/proto) Finished `test` profile [unoptimized + debuginfo] target(s) in 1.56s Running tests/proto_integration.rs (target/debug/deps/proto_integration-775454d70979734b) running 1 test thread 'cases::roundtrip_logical_plan::roundtrip_mixed_case_table_reference' panicked at datafusion/proto/tests/cases/roundtrip_logical_plan.rs:2690:5: assertion `left == right` failed left: "Filter: TestData.a = Int64(1) [a:Int64;N]\n TableScan: TestData projection=[a], partial_filters=[TestData.a = Int64(1)] [a:Int64;N]" right: "Filter: testdata.a = Int64(1) [a:Int64;N]\n TableScan: TestData projection=[a], partial_filters=[testdata.a = Int64(1)] [a:Int64;N]" note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace test cases::roundtrip_logical_plan::roundtrip_mixed_case_table_reference ... FAILED failures: failures: cases::roundtrip_logical_plan::roundtrip_mixed_case_table_reference test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 112 filtered out; finished in 0.09s ``` With the fix implemented (0616df2): ``` running 1 test Filter: TestData.a = Int64(1) [a:Int64;N] TableScan: TestData projection=[a], partial_filters=[TestData.a = Int64(1)] [a:Int64;N] test cases::roundtrip_logical_plan::roundtrip_mixed_case_table_reference ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 112 filtered out; finished in 0.06s ``` ## Are there any user-facing changes? None. <!-- Resources --> [fromcol_proto]: https://github.com/apache/datafusion/blob/50.2.0/datafusion/proto-common/src/from_proto/mod.rs#L146 [tryfrom_proto]: https://github.com/apache/datafusion/blob/50.2.0/datafusion/proto-common/src/from_proto/mod.rs#L171 [parse_str]: https://github.com/apache/datafusion/blob/50.2.0/datafusion/common/src/table_reference.rs#L273 |
||
|
Blake Orth
|
774b6fee0b |
Adds instrumentation to COPY operations in the CLI (#18227)
## Which issue does this PR close? - Closes #17207 ## Rationale for this change Completes the instrumentation of the object store used in datafusion-cli 🎉 ## What changes are included in this PR? - Adds instrumentation to copy - Adds instrumentation to copy_if_not_exists - Adds tests for new code ## Are these changes tested? Yes. New unit tests have been added. ## Are there any user-facing changes? No-ish ## cc @alamb This PR completes the final methods for the instrumented object store! It seems like this should allow us to close the issue we've had tied to all of these PRs and additional work in this area can probably be done under new/targeted issues. |
||
|
Jonathan Chen
|
531af8e43a |
feat: Add deregister_object_store (#17999)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #17854 . ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Blake Orth
|
d5ea5e9876 |
Adds DELETE and HEAD instrumentation to CLI (#18206)
## Which issue does this PR close? This does not fully close, but is an incremental building block component for: - https://github.com/apache/datafusion/issues/17207 The full context of how this code is likely to progress can be seen in the POC for this effort: - https://github.com/apache/datafusion/pull/17266 ## Rationale for this change Further fills out method instrumentation ## What changes are included in this PR? - Adds instrumentation to head requests in the instrumented object store - Adds instrumentatin to delete requests in the instrumented object store - Adds tests for new code ## Are these changes tested? Yes. New unit tests have been added. ## Are there any user-facing changes? No-ish ## cc @alamb |
||
|
Blake Orth
|
93f136c06d |
Adds instrumentation to PUT ops in the CLI (#18139)
## Which issue does this PR close? This does not fully close, but is an incremental building block component for: - https://github.com/apache/datafusion/issues/17207 The full context of how this code is likely to progress can be seen in the POC for this effort: - https://github.com/apache/datafusion/pull/17266 ## Rationale for this change Further fills out the missing methods that have yet to be instrumented in the instrumented object store. ## What changes are included in this PR? - Adds instrumentation around put_opts - Adds instrumentation around put_multipart - Adds tests for newly instrumented methods ## Are these changes tested? Yes. Unit tests have been added for the new methods Example output: ```sql DataFusion CLI v50.2.0 > CREATE EXTERNAL TABLE test(a bigint, b bigint) STORED AS parquet LOCATION '../../test_table/'; 0 row(s) fetched. Elapsed 0.003 seconds. > \object_store_profiling trace ObjectStore Profile mode set to Trace > INSERT INTO test values (1, 2), (3, 4); +-------+ | count | +-------+ | 2 | +-------+ 1 row(s) fetched. Elapsed 0.007 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Trace, inner: LocalFileSystem(file:///) 2025-10-17T19:02:15.440246215+00:00 operation=List path=home/blake/open_source_src/datafusion-BlakeOrth/test_table 2025-10-17T19:02:15.444096012+00:00 operation=Put duration=0.000249s size=815 path=home/blake/open_source_src/datafusion-BlakeOrth/test_table/a9pjKBxSOtXZobJO_0.parquet Summaries: List count: 1 Put count: 1 duration min: 0.000249s duration max: 0.000249s duration avg: 0.000249s size min: 815 B size max: 815 B size avg: 815 B size sum: 815 B > ``` (note: I have no idea how to exercise/show a multi-part put operation, or if DataFusion even utilizes multipart puts for large files) ## Are there any user-facing changes? No-ish cc @alamb --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Andrew Lamb
|
5a074ea4c8 |
Improve datafusion-cli object store profiling summary display (#18085)
## Which issue does this PR close? - part of https://github.com/apache/datafusion/issues/17207 ## Rationale for this change As suggested by @BlakeOrth in https://github.com/apache/datafusion/pull/18045#issuecomment-3403692516 here is an attempt to improve the output of datafusion object store trace profiling: ## What changes are included in this PR? Update the output format when `\object_store_profiling summary` is set Current format (on main, before this PR): ```sql Summaries: Get count: 2 duration min: 0.024603s duration max: 0.031946s duration avg: 0.028274s size min: 8 B size max: 34322 B size avg: 17165 B size sum: 34330 B ``` New format (after this PR): ```sql DataFusion CLI v50.2.0 > \object_store_profiling summary ObjectStore Profile mode set to Summary > select count(*) from 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet'; +----------+ | count(*) | +----------+ | 1000000 | +----------+ 1 row(s) fetched. Elapsed 6.754 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Summary, inner: HttpStore Summaries: +-----------+----------+-----------+-----------+-----------+-----------+-------+ | Operation | Metric | min | max | avg | sum | count | +-----------+----------+-----------+-----------+-----------+-----------+-------+ | Get | duration | 0.031645s | 0.047780s | 0.039713s | 0.079425s | 2 | | Get | size | 8 B | 34322 B | 17165 B | 34330 B | 2 | +-----------+----------+-----------+-----------+-----------+-----------+-------+ ``` ## Are these changes tested? Yes ## Are there any user-facing changes? Nicer datafusion-cli output |
||
|
Blake Orth
|
ec3d20bff6 |
Adds instrumentation to delimited LIST operations in CLI (#18134)
## Which issue does this PR close? This does not fully close, but is an incremental building block component for: - https://github.com/apache/datafusion/issues/17207 The full context of how this code is likely to progress can be seen in the POC for this effort: - https://github.com/apache/datafusion/pull/17266 ## Rationale for this change Continued progress filling out methods that are instrumented by the instrumented object store ## What changes are included in this PR? - Adds instrumentation around delimited list operations into the instrumented object store - Adds test cases for the new code ## Are these changes tested? Yes, unit tests have been added. Example output: ```sql DataFusion CLI v50.2.0 > CREATE EXTERNAL TABLE overture_partitioned STORED AS PARQUET LOCATION 's3://overturemaps-us-west-2/release/2025-09-24.0/theme=addresses/'; 0 row(s) fetched. Elapsed 2.307 seconds. > \object_store_profiling trace ObjectStore Profile mode set to Trace > select count(*) from overture_partitioned; +-----------+ | count(*) | +-----------+ | 446544475 | +-----------+ 1 row(s) fetched. Elapsed 1.932 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Trace, inner: AmazonS3(overturemaps-us-west-2) 2025-10-17T17:05:27.922724180+00:00 operation=List duration=0.132154s path=release/2025-09-24.0/theme=addresses 2025-10-17T17:05:28.054894440+00:00 operation=List duration=0.049048s path=release/2025-09-24.0/theme=addresses/type=address 2025-10-17T17:05:28.104233937+00:00 operation=Get duration=0.053522s size=8 range: bytes=1070778162-1070778169 path=release/2025-09-24.0/theme=addresses/type=address/part-00000-52872134-68de-44a6-822d-15fa29a0f606-c000.zstd.parquet 2025-10-17T17:05:28.106862343+00:00 operation=Get duration=0.108103s size=8 range: bytes=1017940335-1017940342 path=release/2025-09-24.0/theme=addresses/type=address/part-00003-52872134-68de-44a6-822d-15fa29a0f606-c000.zstd.parquet ... 2025-10-17T17:05:28.589084204+00:00 operation=Get duration=0.084737s size=836971 range: bytes=1112791717-1113628687 path=release/2025-09-24.0/theme=addresses/type=address/part-00009-52872134-68de-44a6-822d-15fa29a0f606-c000.zstd.parquet Summaries: List count: 2 duration min: 0.049048s duration max: 0.132154s duration avg: 0.090601s Get count: 33 duration min: 0.045500s duration max: 0.162114s duration avg: 0.089775s size min: 8 B size max: 917946 B size avg: 336000 B size sum: 11088026 B > ``` Note that a `LIST` report showing a duration must be a `list_with_delimiter()` call because a standard `list` call does not currently report a duration. ## Are there any user-facing changes? No-ish cc @alamb |
||
|
Blake Orth
|
dce59f850c |
Adds instrumentation to LIST operations in CLI (#18103)
## Which issue does this PR close? This does not fully close, but is an incremental building block component for: - https://github.com/apache/datafusion/issues/17207 The full context of how this code is likely to progress can be seen in the POC for this effort: - https://github.com/apache/datafusion/pull/17266 ## Rationale for this change Continued progress filling out the methods that are instrumented for the instrumented object store. ## What changes are included in this PR? - Adds instrumentation around basic list operations into the instrumented object store - Adds test cases for new code ## Are these changes tested? Yes. Example output: ```sql DataFusion CLI v50.2.0 > \object_store_profiling trace ObjectStore Profile mode set to Trace > CREATE EXTERNAL TABLE nyc_taxi_rides STORED AS PARQUET LOCATION 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet'; 0 row(s) fetched. Elapsed 2.679 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Trace, inner: AmazonS3(altinity-clickhouse-data) 2025-10-16T18:53:09.512970085+00:00 operation=List path=nyc_taxi_rides/data/tripdata_parquet Summaries: List count: 1 Instrumented Object Store: instrument_mode: Trace, inner: AmazonS3(altinity-clickhouse-data) 2025-10-16T18:53:09.929709943+00:00 operation=List path=nyc_taxi_rides/data/tripdata_parquet 2025-10-16T18:53:10.106757629+00:00 operation=List path=nyc_taxi_rides/data/tripdata_parquet 2025-10-16T18:53:10.220555058+00:00 operation=Get duration=0.230604s size=8 range: bytes=222192975-222192982 path=nyc_taxi_rides/data/tripdata_parquet/data-200901.parquet 2025-10-16T18:53:10.226399832+00:00 operation=Get duration=0.263826s size=8 range: bytes=233123927-233123934 path=nyc_taxi_rides/data/tripdata_parquet/data-201104.parquet 2025-10-16T18:53:10.226194195+00:00 operation=Get duration=0.269754s size=8 range: bytes=252843253-252843260 path=nyc_taxi_rides/data/tripdata_parquet/data-201103.parquet . . . 2025-10-16T18:53:11.928787014+00:00 operation=Get duration=0.072248s size=18278 range: bytes=201384109-201402386 path=nyc_taxi_rides/data/tripdata_parquet/data-201509.parquet 2025-10-16T18:53:11.933475464+00:00 operation=Get duration=0.068880s size=17175 range: bytes=195411804-195428978 path=nyc_taxi_rides/data/tripdata_parquet/data-201601.parquet 2025-10-16T18:53:11.949629591+00:00 operation=Get duration=0.065645s size=19872 range: bytes=214807880-214827751 path=nyc_taxi_rides/data/tripdata_parquet/data-201603.parquet Summaries: List count: 2 Get count: 288 duration min: 0.060930s duration max: 0.444601s duration avg: 0.133339s size min: 8 B size max: 44247 B size avg: 18870 B size sum: 5434702 B > ``` ## Are there any user-facing changes? No-ish ## cc @alamb |
||
|
dario curreri
|
4e03c92059 |
refactor: add dialect enum (#18043)
## Which issue does this PR close? - Closes #18042 ## Rationale for this change This PR introduces a new dialect enum to improve type safety and code maintainability when handling different SQL dialects in DataFusion 1. Provide compile-time guarantees for dialect handling 2. Improve code readability and self-documentation 3. Enable better IDE support and autocomplete ## What changes are included in this PR? - Added a new `Dialect` enum to represent supported SQL dialects - Refactored existing code to use the new enum instead of previous representations - Modified tests to work with the new enum-based approach ## Are these changes tested? Yes ## Are there any user-facing changes? Yes, this is an API change: the type of the `dialect` field changed from `String` to `Dialect` |
||
|
Blake Orth
|
3bca1bb6eb |
Adds Trace and Summary to CLI instrumented stores (#18064)
- Adds the ability for a user to choose a summary only output for an instrumented object store when using the CLI - The existing "enabled" setting that displays both a summary and a detailed usage for each object store call has been renamed to `Trace` to improve clarity - Adds additional test cases for summary only and modifies existing tests to use trace - Updates user guide docs to reflect the CLI flag and command line changes |
||
|
Blake Orth
|
ca2585e752 |
Adds summary output to CLI instrumented object stores (#18045)
* Adds summary output to CLI instrumented object stores - Adds a `RequestSummary` type for the instrumented object store to display summary statistics about instrumented requests - Adds a generic Stats type to track the statistics for the summary - Adds tests for the new code - Adds a basic summary output to the user-facing display when profiling is enabled - Adds docs for new and newly exported public items * - Updates integration test and validation snapshot |
||
|
dependabot[bot]
|
f6c6fe0f25 |
chore(deps): bump aws-config from 1.8.6 to 1.8.7 (#18038)
Bumps [aws-config](https://github.com/smithy-lang/smithy-rs) from 1.8.6 to 1.8.7. - [Release notes](https://github.com/smithy-lang/smithy-rs/releases) - [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/smithy-lang/smithy-rs/commits) --- updated-dependencies: - dependency-name: aws-config dependency-version: 1.8.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Blake Orth
|
a1acf775a9 |
Adds instrumentation to get requests for datafusion-cli (#18016)
* Adds instrumentation to get requests for datafusion-cli - Adds request instrumentation to GET operations when using the InstrumentedObjectStore in datafusion-cli - Adds RequestDetails type to store and format details about instrumented requests - Adds tests for new functionality * Add cli-integration test * Verify the contents of the reqest --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Blake Orth
|
f210939ecc |
Adds Object Store Profiling options/commands to CLI (#18004)
* Adds Object Store Profiling options/commands to CLI - Adds a CLI option and command to datafusion-cli to enable or disabled object store profiling - Integrates the command with the instrumented object stores to allow the user input to change the mode of the instrumented stores - Adds tests to exercise the expected behavior of the commands - Adds user docs for the commands/CLI options - Updates visibility of `InstrumentedObjectStore` now that it needs to be interacted with outside of its module * Improves InstrumentedObjectStoreRegistry ergonomics - Adds better methods to build an InstrumentedObjectStoreRegistry to reduce code duplication in common usage - Enhances test success criteria - Normalizes method names |
||
|
Blake Orth
|
73b1f2bd72 |
Adds Instrument Mode for InstrumentedObjectStore in datafusion-cli (#18000)
- Adds mode type to allow changing the mode of an InstrumentedObjectStore to datafusion-cli - Implements string parsing and u8 conversion for InstrumentedObjectStoreMode - Adds tests to validate trait implementations |
||
|
Blake Orth
|
40b72330bc |
Adds Instrumented Object Store to CLI (#17984)
* Adds Instrumented Object Store to CLI - Adds the InstrumentedObjectStore to datafusion-cli to support building instrumented output for queries executed via the cli - Integrates the new InstrumentedObjectStore into the InstrumentedObjectStoreRegistry to ensure any ObjectStore instance registered gets instrumentation * Fixes doc comment that referenced private items |
||
|
Blake Orth
|
f8ff82a187 |
feat: Adds Instrumented Object Store Registry to datafusion-cli (#17953)
* feat: Adds Instrumented Object Store Registry to datafusion-cli - Adds a new Object Store Registry wrapper to datafusion-cli to support interacting with instrumented object store instances - Adds basic tests for the new registry wrapper * Adds doc comments to new public types and methods |
||
|
Oleks V
|
149d3e9533 |
chore: Extend backtrace coverage for Execution and Internal errors (#17921)
* chore: Extend backtrace coverage * fmt * part2 * doc * feedback * feedback * clippy * feedback |
||
|
dependabot[bot]
|
bfd5e5582a |
chore(deps): bump aws-credential-types from 1.2.6 to 1.2.7 (#17936)
Bumps [aws-credential-types](https://github.com/smithy-lang/smithy-rs) from 1.2.6 to 1.2.7. - [Release notes](https://github.com/smithy-lang/smithy-rs/releases) - [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md) - [Commits](https://github.com/smithy-lang/smithy-rs/commits) --- updated-dependencies: - dependency-name: aws-credential-types dependency-version: 1.2.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Andy Grove
|
a9d04606b4 |
minor: Include consumer name in OOM message (#17870)
* include consumer name in oom message * update more tests * more test updates |