mirror of
https://github.com/langchain-ai/datafusion.git
synced 2026-07-19 15:24:21 -04:00
e937cadbcceff6a42bee2c5fc8d03068fa0eb30c
691 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Neil Conway
|
585bbf35d3 |
perf: Optimize array_has_any() with scalar arg (#20385)
## Which issue does this PR close? - Closes #20384. - See #18181 for related context. ## Rationale for this change When `array_has_any` is passed a scalar for either of its arguments, we can use a much faster algorithm: rather than doing O(N*M) comparisons for each row of the columnar arg, we can build a hash table on the scalar argument and probe it instead. ## What changes are included in this PR? * Add benchmark to cover the one-scalar-arg case * Implement optimization as described above Note that we fallback to a linear scan when the scalar arg is smaller than a threshold (<= 8 elements), because benchmarks suggested probing a HashSet is not profitable for very small arrays. ## Are these changes tested? Yes. Tests pass and benchmarked. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Oleks V
|
ed0323a2bb |
feat: support arrays_zip function (#20440)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change Summary - Adds a new arrays_zip scalar function that combines multiple arrays into a single array of structs, where each struct field corresponds to an input array - Shorter arrays within a row are padded with NULLs to match the longest array's length - Compatible with Spark's arrays_zip behavior <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? ``` arrays_zip takes N list arrays and produces a List<Struct<c0, c1, ..., cN>> where each struct contains the elements at the same index from each input array. > SELECT arrays_zip([1, 2, 3], ['a', 'b', 'c']); [{c0: 1, c1: a}, {c0: 2, c1: b}, {c0: 3, c1: c}] > SELECT arrays_zip([1, 2], [3, 4, 5]); [{c0: 1, c1: 3}, {c0: 2, c1: 4}, {c0: NULL, c1: 5}] Implementation details: - Implemented in set_ops.rs following existing array function patterns - Uses MutableArrayData builders per column with row-by-row processing for efficient memory handling - For each row, computes the max array length, copies values from each input array, and pads shorter arrays with NULLs - Supports variadic arguments (2 or more arrays) - Handles NULL list entries, NULL elements, empty arrays, mixed types, and Null-typed arguments - Registered as arrays_zip with alias list_zip - Uses Signature::variadic_any with Volatility::Immutable ``` <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Adrian Garcia Badaracco
|
1ee782f783 |
Migrate Python usage to uv workspace (#20414)
I was having trouble getting benchmarks to gen data. ## Summary - Replace three independent `requirements.txt` files with a uv workspace (`benchmarks`, `dev`, `docs` projects) - Single `uv.lock` lockfile for reproducible dependency resolution - Simplify `bench.sh` by removing all ad-hoc venv/pip logic in favor of `uv run` ## Test plan - [ ] `uv sync` resolves all deps from repo root - [ ] `uv run --project benchmarks python3 benchmarks/compare.py` works - [ ] `uv run --project docs sphinx-build docs/source docs/build` builds docs - [ ] Run a benchmark from `bench.sh` that uses Python (e.g., h2o data gen or compare flow) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Neil Conway
|
ace9cd44b7 |
perf: Optimize trim UDFs for single-character trims (#20328)
## Which issue does this PR close? - Closes #20327 ## Rationale for this change By default, btrim(), ltrim(), and rtrim() trim space characters; it is also reasonably common for queries to specify a non-default trim pattern that is still a single ASCII character. We can optimize for this case by doing a byte-level scan, rather than invoking the more heavyweight std::string machinery used for more complex trim scenarios. ## What changes are included in this PR? Add a benchmark for trimming spaces, and implement the optimization described above. Also fixed an error in the documentation. ## Are these changes tested? Yes, and benchmarked. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> |
||
|
Neil Conway
|
b7f6090874 |
perf: Optimize lpad, rpad for ASCII strings (#20278)
The previous implementation incurred the overhead of Unicode machinery,
even for the common case that both the input string and the fill string
consistent only of ASCII characters. For the ASCII-only case, we can
assume that the length in bytes equals the length in characters, and
avoid expensive graphene-based segmentation. This follows similar
optimizations applied elsewhere in the codebase.
Benchmarks indicate this is a significant performance win for ASCII-only
input (4x-10x faster) but only a mild regression for Unicode input (2-5%
slower).
Along the way:
* Combine: a few instances of `write_str(str)? + append_value("")` with
`append_value(str)`, which saves a few cycles
* Add a missing test case for truncating the input string
* Add benchmarks for Unicode input
## Which issue does this PR close?
- Closes #20277.
## Are these changes tested?
Covered by existing tests. Added new benchmarks for Unicode inputs.
## Are there any user-facing changes?
No.
---------
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
|
||
|
Neil Conway
|
c3f080774c |
perf: Optimize translate() UDF for scalar inputs (#20305)
## Which issue does this PR close? - Closes #20302. ## Rationale for this change `translate()` is commonly invoked with constant values for its second and third arguments. We can take advantage of that to significantly optimize its performance by precomputing the translation lookup table, rather than recomputing it for every row. For ASCII-only inputs, we can further replace the hashmap lookup table with a fixed-size array that maps ASCII byte values directly. For scalar ASCII inputs, this yields roughly a 10x performance improvement. For scalar UTF8 inputs, the performance improvement is more like 50%, although less so for long strings. Along the way, add support for `translate()` on `LargeUtf8` input, along with an SLT test, and improve the docs. ## What changes are included in this PR? * Add a benchmark for scalar/constant input to translate * Add a missing test case * Improve translate() docs * Support translate() on LargeUtf8 input * Optimize translate() for scalar inputs by precomputing lookup hashmap * Optimize translate() for ASCII inputs by precomputing ASCII byte-wise lookup table ## Are these changes tested? Yes. Added an extra test case and did a bunch of benchmarking. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Adrian Garcia Badaracco
|
7d217b1ec3 |
Reduce ExtractLeafExpressions optimizer overhead with fast pre-scan (#20341)
## Summary Follow-up to #20117 which added the `ExtractLeafExpressions` and `PushDownLeafProjections` optimizer rules for get_field pushdown. Benchmarking revealed that these rules added 5-31% overhead on *all* queries (including those with no struct/get_field expressions) because they unconditionally allocated column HashSets, extractors, and walked every expression tree for every Filter/Sort/Limit/Aggregate/Join node. This PR adds: - **`has_extractable_expr()` pre-scan**: A lightweight check using `Expr::exists()` that short-circuits before any expensive allocations when no `MoveTowardsLeafNodes` expressions are present - **Config option** `datafusion.optimizer.enable_leaf_expression_pushdown` to disable the rules entirely ### Benchmark Results (vs no-rules baseline) | Benchmark | Before Fix | After Fix | |---|---|---| | physical_select_aggregates_from_200 | +31.1% | +3.7% | | physical_many_self_joins | +12.9% | +2.2% | | physical_join_consider_sort | +12.9% | +1.0% | | physical_unnest_to_join | +12.5% | +1.4% | | physical_select_one_from_700 | +12.2% | +2.6% | | physical_theta_join_consider_sort | +8.7% | +0.2% | | physical_plan_tpch_q18 | +9.3% | +1.4% | | physical_plan_tpch_all | +4.8% | +2.1% | | physical_plan_tpcds_all | +5.6% | +2.2% | ## Test plan - [x] All 47 `extract_leaf_expressions` unit tests pass - [x] Benchmarked with `cargo bench -p datafusion --bench sql_planner` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Neil Conway
|
75428f1e0a |
fix: Avoid integer overflow in split_part() (#20198)
Along the way, improve the docs slightly.
## Rationale for this change
Evaluating `SELECT SPLIT_PART('', '', -9223372036854775808);` yields (in
a debug build):
```
thread 'main' (41405991) panicked at datafusion/functions/src/string/split_part.rs:236:47:
attempt to negate with overflow
```
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## Are these changes tested?
Yes, added unit test.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
|
||
|
karuppuchamysuresh
|
9de192af29 |
docs: update data_types.md to reflect current Arrow type mappings (#20072)
## Which issue does this PR close? - Closes #18314 ## Rationale for this change The documentation in `data_types.md` was outdated and showed `Utf8` as the default mapping for character types (CHAR, VARCHAR, TEXT, STRING), but the current implementation defaults to `Utf8View`. This caused confusion for users reading the documentation as it didn't match the actual behavior. Additionally, the "Supported Arrow Types" section at the end was redundant since `arrow_typeof` now supports all Arrow types, making the comprehensive list unnecessary. ## What changes are included in this PR? 1. **Updated Character Types table**: Changed the Arrow DataType column from `Utf8` to `Utf8View` for CHAR, VARCHAR, TEXT, and STRING types 2. **Added configuration note**: Documented the `datafusion.sql_parser.map_string_types_to_utf8view` setting that allows users to switch back to `Utf8` if needed 3. **Removed outdated section**: Deleted the "Supported Arrow Types" section (39 lines) as it's no longer necessary ## Are these changes tested? This is a documentation-only change. The documentation accurately reflects the current behavior of DataFusion: - The default mapping to `Utf8View` is the current implementation behavior - The `datafusion.sql_parser.map_string_types_to_utf8view` configuration option exists and works as documented ## Are there any user-facing changes? Yes, documentation changes only. Users will now see accurate information about: - The correct default Arrow type mappings for character types - How to configure the string type mapping behavior if they need the old `Utf8` behavior --------- Co-authored-by: Claude (claude-sonnet-4.5) <noreply@anthropic.com> |
||
|
Nuno Faria
|
92f60ad513 |
docs: Automatically update DataFusion version in docs (#20001)
## Which issue does this PR close? - N/A. ## Rationale for this change I was looking at the download page (https://datafusion.apache.org/download.html) when I noticed the version there was not updated in a while. Likewise, there are other places in the docs where the version is not updated. This PR changes the `update_datafusion_versions.py` script to automatically update those files, in addition to the Cargo files. ## What changes are included in this PR? - Updated `update_datafusion_versions.py` to check some doc files. - Updated the doc files to the current major version. ## Are these changes tested? Yes, script runs correctly. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> |
||
|
Andrew Lamb
|
0a68b81ade |
[main] Update version to 52.1.0 (#19878) (#20028)
## Which issue does this PR close? - part of https://github.com/apache/datafusion/issues/19784 ## Rationale for this change Forward port changes from branch-52 to main ## What changes are included in this PR? Update release version to 52.1.0 and add changelo by cherry-picking - 9f3ddcecd6033a9d55161175d5dbe29697a9a922 ## Are these changes tested? By CI ## Are there any user-facing changes? New version |
||
|
Nuno Faria
|
073174b034 |
feat: Show the number of matched Parquet pages in DataSourceExec (#19977)
## Which issue does this PR close? - Closes #19875. ## Rationale for this change Show the number of matched (and pruned) pages in the explain analyze plan to help make decisions about file optimization. Example: ```sql DataSourceExec: ..., metrics=[ ... page_index_rows_pruned=1.00 K total → 100 matched, page_index_pages_pruned=100 total → 10 matched, ... ] ``` ## What changes are included in this PR? - Added `page_index_pages_pruned` metric to DataSourceExec. - Updated and extended existing tests. ## Are these changes tested? Yes. ## Are there any user-facing changes? New metric in the explain plans. |
||
|
cht42
|
8efc2b605e |
feat(spark): add base64 and unbase64 functions (#19968)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19967 - Part of #15914 ## Rationale for this change Add spark compatible base64/unbase64 functions ## What changes are included in this PR? - new encoding mode in DF encoding UDF for padded base64 - spark udfs for base64/unbase64 ## Are these changes tested? yes in SLT ## Are there any user-facing changes? yes |
||
|
Andrew Lamb
|
f8a22a51bf |
docs: Fix some broken / missing links in the DataFusion documentation (#19958)
## Which issue does this PR close? ## Rationale for this change While reading the DataFusion documentation, I noticed a broken link here: <img width="787" height="885" alt="Screenshot 2026-01-23 at 9 34 43 AM" src="https://github.com/user-attachments/assets/9013151d-599b-4973-968e-0033c6614643" /> ## What changes are included in this PR? Fix several broken and missing links (this was done by codex and I reviewed the PR) ## Are these changes tested? Yes, by CI ## Are there any user-facing changes? Better docs. No functional changes intended |
||
|
Andrew Lamb
|
d1eea0755a |
Fix broken links in the documentation (#19964)
## Which issue does this PR close? - Similarly to https://github.com/apache/datafusion/pull/19958 ## Rationale for this change I found some broken links in our docs using codex ] <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? Update broken links (will comment inline) ## Are these changes tested? By CI ## Are there any user-facing changes? Better docs / fewer broken links --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> |
||
|
Nuno Faria
|
e062ae5053 |
minor: Fix doc about write_batch_size (#19979)
## Which issue does this PR close? - N/A. ## Rationale for this change In the docs, `write_batch_size` was defined as representing in bytes, while it actually represents number of rows. ## What changes are included in this PR? - Updates documentation about `write_batch_size`. ## Are these changes tested? Yes. ## Are there any user-facing changes? No. |
||
|
kosiew
|
0c82adec71 |
Fix struct casts to align fields by name (prevent positional mis-casts) (#19674)
## Which issue does this PR close? * Closes #17285. ## Rationale for this change DataFusion’s struct casting and some coercion paths were effectively positional: when two structs had the same field types but different field *orders*, casting could silently swap values. This is surprising to users and can lead to silent data corruption (e.g. `{b: 3, a: 4}::STRUCT(a INT, b INT)` yielding `{a: 3, b: 4}`). The goal of this PR is to make struct casting behavior match user expectations by matching fields by **name** (case-sensitive) and recursively applying the same logic to nested structs, while keeping a compatible fallback for structs with **no** shared field names. ## What changes are included in this PR? * **Name-based struct casting implementation** in `datafusion_common::nested_struct`: * Match struct fields by **name**, reorder to match target schema, recursively cast nested structs. * Fill **missing target fields** with null arrays. * Ignore **extra source fields**. * **Positional mapping fallback** when there is *no name overlap* **and** field counts match (avoids breaking `struct(1, 'x')::STRUCT(a INT, b VARCHAR)` style casts). * Improved handling for **NULL / all-null struct inputs** by producing a correctly typed null struct array. * Centralized validation via `validate_field_compatibility` and helper `fields_have_name_overlap`. * **Ensure struct casting paths use the name-based logic**: * `ScalarValue::cast_to_with_options`: route `Struct` casts through `nested_struct::cast_column`. * `ColumnarValue::cast_to`: for `Struct` targets, cast via `nested_struct::cast_column`; non-struct casts still use Arrow’s standard casting. * **Type coercion improvements for structs in binary operators / CASE**: * When two structs have at least one overlapping name, coerce **by name**. * Otherwise, preserve prior behavior by coercing **positionally**. * **Planning-time cast validation for struct-to-struct**: * `physical-expr` CAST planning now validates struct compatibility using the same rules as runtime (`validate_struct_compatibility`) to fail fast. * `ExprSchemable` allows struct-to-struct casts to pass type checking; detailed compatibility is enforced by the runtime / planning-time validator. * **Optimizer safety**: * Avoid const-folding struct casts when field counts differ. * Avoid const-folding casts of **0-row** struct literals due to evaluation batch dimension mismatches. * **Tests and SQL logic tests**: * New unit tests covering: * name-based reordering * missing fields (nullable vs non-nullable) * null struct fields and nested nulls * positional fallback with no overlap * coercion behavior and simplifier behavior * Updated/added `.slt` cases to reflect the new semantics and to add coverage for struct casts and nested struct reordering. * **Minor docs/maintenance**: * Adjusted doc comment referencing `ParquetWriterOptions` so it doesn’t break when the `parquet` feature is disabled. ## Are these changes tested? Yes. * Added/updated Rust unit tests in: * `datafusion/common/src/nested_struct.rs` * `datafusion/expr-common/src/columnar_value.rs` * `datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs` * Added/updated SQL logic tests in: * `datafusion/sqllogictest/test_files/case.slt` * `datafusion/sqllogictest/test_files/struct.slt` These tests cover: * correct value mapping when struct field order differs * nested struct reordering * insertion of nulls for missing nullable fields * erroring on missing non-nullable target fields * positional mapping fallback when there is no name overlap * planning-time validation vs runtime behavior alignment ## Are there any user-facing changes? Yes. * **Struct casts are now name-based** (case-sensitive): fields are matched by name, reordered to the target schema, missing fields are null-filled (if nullable), and extra fields are ignored. * **Fallback behavior**: if there is *no* name overlap and field counts match, casting proceeds **positionally**. * **Potential behavior change** in queries relying on the prior positional behavior when structs shared names but were out of order (previously could yield swapped values). This PR changes that to the safer, expected behavior. No public API changes are introduced, but this is a semantic change in struct casting. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested. |
||
|
Jeffrey Vo
|
ac67ae4af5 |
Ensure null inputs to array setop functions return null output (#19683)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19682 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Explained in issue. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Change array_except, array_intersect and array_union UDFs to return null if either input is null. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Added & fixed tests. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Behaviour change to a function output. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Andrew Lamb
|
c91dcfdb6a |
Docs: add additional links to blog posts (#19833)
## Which issue does this PR close? - Part of #7013 ## Rationale for this change We have written some good blogs recently that provide additional context and backstory. Let's make sure they are available for others to read ## What changes are included in this PR? Add links to select doc pages ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
cht42
|
f3f6dec7c9 |
feat: Add support for 'isoyear' in date_part function (#19821)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19820. ## Rationale for this change `isoyear` part is available in both PG and Spark EXTRACT functions. https://www.postgresql.org/docs/current/functions-datetime.html#:~:text=the%20week%20numbering.-,isoyear,-The%20ISO%208601 https://github.com/apache/spark/blob/a03bedb6c1281c5263a42bfd20608d2ee005ab05/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L3360 ## What changes are included in this PR? Support for part `isoyear` in date_part function. ## Are these changes tested? yes in SLT ## Are there any user-facing changes? yes <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
xudong.w
|
ca904b3086 |
Row group limit pruning for row groups that entirely match predicates (#18868)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of https://github.com/apache/datafusion/issues/18860 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> See https://github.com/apache/datafusion/issues/18860#issuecomment-3563442093 ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> 1. How to decide if we can do limit pruning without messing up the sql semantics. 2. Add logic to decide if a row group is fully matched, all rows in the row group are matched the predicated. 3. Use the fully matched row groups to return limit rows. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 3. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> No, no new configs, or API change |
||
|
Miao
|
5edda9b309 |
fix: calculate total seconds from interval fields for extract(epoch) (#19807)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19799. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> The epoch function incorrectly used `date_part(array, DatePart::Second)` for intervals which only extracts the seconds component of the interval structure (0 for "15 minutes"), not the total seconds from all components. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Modified epoch() function in date_part.rs to properly handle interval types by extracting struct fields and calculating total seconds - Added regression tests in date_part.slt ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. Added tests verifying correct conversion for all interval types and precision levels. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Yes. `extract(epoch from interval)` now returns correct total seconds instead of 0. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
xudong.w
|
383e673ac3 |
Update 52.0.0 release version number and changelog (#19767)
The 52.0.0 has released, the PR updates version number and changlog on main branch |
||
|
Kumar Ujjawal
|
d103d8886f |
chore: remove LZO Parquet compression (#19726)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19720. ## Rationale for this change - Choosing LZO compression errors, I think it might never get supported so the best option moving forward is to remove it algother and update the docs. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Removed LZO from parse_compression_string() function - Removed docs - Updated exptected test output <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? User choosing LZO as compression will get a clear error message: ``` Unknown or unsupported parquet compression: lzo. Valid values are: uncompressed, snappy, gzip(level), brotli(level), lz4, zstd(level), and lz4_raw. ``` <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Adam Gutglick
|
8ba46466d3 |
docs: Fix two small issues in introduction.md (#19712)
This is a small maintanence PR to fix a couple of small issues: 1. The `datafusion-tui` isn't used anywhere, and as far as I can tell its now basically a part of `datafusion-dft` (which is already referenced). 2. The call for action wasn't rendered into the HTML page. I was there up until #4903, which included this small mistake which every subsequent change just maintained. |
||
|
UBarney
|
c98fa5616e |
perfect hash join (#19411)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #17635. ## Rationale for this change This PR introduces a Perfect Hash Join optimization by using an array-based direct mapping(`ArrayMap`) instead of a HashMap. The array-based approach outperforms the standard Hash Join when the build-side keys are **_dense_** (i.e., the ratio of `count / (max - min+1)` is high) or when the key range `(max - min)` is sufficiently **small**. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> The following results from the hj.rs benchmark suite. The benchmark was executed with the optimization enabled by setting `DATAFUSION_EXECUTION_PERFECT_HASH_JOIN_MIN_KEY_DENSITY=0.1` ``` ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Query ┃ base_hj ┃ density=0.1 ┃ Change ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ QQuery 1_density=1_prob_hit=1_25*1.5M │ 5.50 ms │ 4.54 ms │ +1.21x faster │ │ QQuery 2_density=0.026_prob_hit=1_25*1.5M │ 6.13 ms │ 5.43 ms │ +1.13x faster │ │ QQuery 3_density=1_prob_hit=1_100K*60M │ 132.59 ms │ 97.42 ms │ +1.36x faster │ │ QQuery 4_density=1_prob_hit=0.1_100K*60M │ 146.66 ms │ 97.75 ms │ +1.50x faster │ │ QQuery 5_density=0.75_prob_hit=1_100K*60M │ 139.85 ms │ 103.82 ms │ +1.35x faster │ │ QQuery 6_density=0.75_prob_hit=0.1_100K*60M │ 256.62 ms │ 192.15 ms │ +1.34x faster │ │ QQuery 7_density=0.5_prob_hit=1_100K*60M │ 136.27 ms │ 91.64 ms │ +1.49x faster │ │ QQuery 8_density=0.5_prob_hit=0.1_100K*60M │ 234.89 ms │ 185.35 ms │ +1.27x faster │ │ QQuery 9_density=0.2_prob_hit=1_100K*60M │ 132.76 ms │ 98.44 ms │ +1.35x faster │ │ QQuery 10_density=0.2_prob_hit=0.1_100K*60M │ 240.04 ms │ 184.93 ms │ +1.30x faster │ │ QQuery 11_density=0.1_prob_hit=1_100K*60M │ 133.02 ms │ 108.11 ms │ +1.23x faster │ │ QQuery 12_density=0.1_prob_hit=0.1_100K*60M │ 235.44 ms │ 209.10 ms │ +1.13x faster │ │ QQuery 13_density=0.01_prob_hit=1_100K*60M │ 135.64 ms │ 132.52 ms │ no change │ │ QQuery 14_density=0.01_prob_hit=0.1_100K*60M │ 235.88 ms │ 234.62 ms │ no change │ │ QQuery 15_density=0.2_prob_hit=0.1_100K_(20%_dups)*60M │ 178.49 ms │ 147.55 ms │ +1.21x faster │ └────────────────────────────────────────────────────────┴───────────┴─────────────┴───────────────┘ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Benchmark Summary ┃ ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ Total Time (base_hj) │ 2349.79ms │ │ Total Time (density=0.1) │ 1893.37ms │ │ Average Time (base_hj) │ 156.65ms │ │ Average Time (density=0.1) │ 126.22ms │ │ Queries Faster │ 13 │ │ Queries Slower │ 0 │ │ Queries with No Change │ 2 │ │ Queries with Failure │ 0 │ └────────────────────────────┴───────────┘ ``` The following results from tpch-sf10 ``` ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Query ┃ base ┃ perfect_hj ┃ Change ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ QQuery 1 │ 739.66 ms │ 743.84 ms │ no change │ │ QQuery 2 │ 315.94 ms │ 317.53 ms │ no change │ │ QQuery 3 │ 655.79 ms │ 669.24 ms │ no change │ │ QQuery 4 │ 215.48 ms │ 218.79 ms │ no change │ │ QQuery 5 │ 1131.42 ms │ 1146.03 ms │ no change │ │ QQuery 6 │ 202.32 ms │ 190.83 ms │ +1.06x faster │ │ QQuery 7 │ 1734.06 ms │ 1710.50 ms │ no change │ │ QQuery 8 │ 1185.05 ms │ 1173.90 ms │ no change │ │ QQuery 9 │ 2036.76 ms │ 1994.30 ms │ no change │ │ QQuery 10 │ 907.32 ms │ 893.20 ms │ no change │ │ QQuery 11 │ 306.63 ms │ 275.46 ms │ +1.11x faster │ │ QQuery 12 │ 404.00 ms │ 381.95 ms │ +1.06x faster │ │ QQuery 13 │ 531.67 ms │ 498.58 ms │ +1.07x faster │ │ QQuery 14 │ 317.63 ms │ 303.04 ms │ no change │ │ QQuery 15 │ 602.24 ms │ 572.18 ms │ no change │ │ QQuery 16 │ 200.00 ms │ 201.68 ms │ no change │ │ QQuery 17 │ 1848.67 ms │ 1790.60 ms │ no change │ │ QQuery 18 │ 2130.63 ms │ 2179.84 ms │ no change │ │ QQuery 19 │ 501.81 ms │ 529.85 ms │ 1.06x slower │ │ QQuery 20 │ 637.91 ms │ 661.72 ms │ no change │ │ QQuery 21 │ 1882.43 ms │ 1917.10 ms │ no change │ │ QQuery 22 │ 130.68 ms │ 141.76 ms │ 1.08x slower │ └──────────────┴────────────┴────────────┴───────────────┘ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓ ┃ Benchmark Summary ┃ ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩ │ Total Time (base) │ 18618.10ms │ │ Total Time (perfect_hj) │ 18511.93ms │ │ Average Time (base) │ 846.28ms │ │ Average Time (perfect_hj) │ 841.45ms │ │ Queries Faster │ 4 │ │ Queries Slower │ 2 │ │ Queries with No Change │ 16 │ │ Queries with Failure │ 0 │ └───────────────────────────┴────────────┘ ``` ## What changes are included in this PR? - During the `collect_left_input` (build) phase, we now conditionally use an `ArrayMap` instead of a standard `JoinHashMapType`. This optimization is triggered only when the following conditions are met: - There is exactly one join key. - The join key can be any integer type convertible to u64 (excluding i128 and u128). - The key distribution is sufficiently dense or the key range (max - min) is small enough to justify an array-based allocation. - build_side.num_rows() < `u32::MAX` - The `ArrayMap` works by storing the minimum key as an offset and using a Vec to directly map a key `k` to its build-side index via `data[k- offset]`. - Rewrite Hash Join micro-benchmarks in benchmarks/src/hj.rs to evaluate ArrayMap and HashMap performance across varying key densities and probe hit rates <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? Yes, this PR introduces two new session configuration parameters to control the behavior of the Perfect Hash Join optimization: - `perfect_hash_join_small_build_threshold`: This parameter defines the maximum key range (max_key - min_key) for the build side to be considered "small." If the key range is below this threshold, the array-based join will be triggered regardless of key density. - `perfect_hash_join_min_key_density`: This parameter sets the minimum density (row_count / key_range) required to enable the perfect hash join optimization for larger key ranges <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
jizezhang
|
e6049de5a7 |
Make default ListingFilesCache table scoped (#19616)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Builds on https://github.com/apache/datafusion/pull/19388 - Closes https://github.com/apache/datafusion/issues/19573 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> This PR explores one way to make `ListFilesCache` table scoped. A session level cache is still used, but the cache key is made a "table-scoped" path, for which a new struct ``` pub struct TableScopedPath(pub Option<TableReference>, pub Path); ``` is defined. `TableReference` comes from `CreateExternalTable` passed to `ListingTableFactory::create`. Additionally, when a table is dropped, all entries related to a table is dropped by modifying `SessionContext::find_and_deregister` method. Testing (change on adding `list_files_cache()` for cli is included for easier testing). - Testing cache reuse on a single table. ``` > \object_store_profiling summary ObjectStore Profile mode set to Summary > create external table test stored as parquet location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/'; 0 row(s) fetched. Elapsed 14.878 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Summary, inner: AmazonS3(overturemaps-us-west-2) Summaries: +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Operation | Metric | min | max | avg | sum | count | +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Get | duration | 0.030597s | 0.209235s | 0.082396s | 36.254189s | 440 | | Get | size | 204782 B | 857230 B | 497304.88 B | 218814144 B | 440 | | List | duration | 0.192037s | 0.192037s | 0.192037s | 0.192037s | 1 | | List | size | | | | | 1 | +-----------+----------+-----------+-----------+-------------+-------------+-------+ > select table, path, unnest(metadata_list) from list_files_cache() limit 1; +-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | table | path | UNNEST(list_files_cache().metadata_list) | +-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test | release/2025-12-17.0/theme=base | {file_path: release/2025-12-17.0/theme=base/type=bathymetry/part-00000-dd0f2f50-b436-4710-996f-f1b06181a3a1-c000.zstd.parquet, file_modified: 2025-12-17T22:32:50, file_size_bytes: 40280159, e_tag: "15090401f8f936c3f83bb498cb99a41d-3", version: NULL} | +-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row(s) fetched. Elapsed 0.058 seconds. Object Store Profiling > select count(*) from test where type = 'infrastructure'; +-----------+ | count(*) | +-----------+ | 142969564 | +-----------+ 1 row(s) fetched. Elapsed 0.028 seconds. Object Store Profiling ``` - Test separate cache entries are created for two tables with same path ``` > create external table test2 stored as parquet location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/'; 0 row(s) fetched. Elapsed 14.798 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Summary, inner: AmazonS3(overturemaps-us-west-2) Summaries: +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Operation | Metric | min | max | avg | sum | count | +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Get | duration | 0.030238s | 0.350465s | 0.073670s | 32.414654s | 440 | | Get | size | 204782 B | 857230 B | 497304.88 B | 218814144 B | 440 | | List | duration | 0.133334s | 0.133334s | 0.133334s | 0.133334s | 1 | | List | size | | | | | 1 | +-----------+----------+-----------+-----------+-------------+-------------+-------+ > select count(*) from test2 where type = 'bathymetry'; +----------+ | count(*) | +----------+ | 59963 | +----------+ 1 row(s) fetched. Elapsed 0.009 seconds. Object Store Profiling > select table, path from list_files_cache(); +-------+---------------------------------+ | table | path | +-------+---------------------------------+ | test | release/2025-12-17.0/theme=base | | test2 | release/2025-12-17.0/theme=base | +-------+---------------------------------+ 2 row(s) fetched. Elapsed 0.004 seconds. ``` - Test cache associated with a table is dropped when table is dropped, and the other table with same path is unaffected. ``` > drop table test; 0 row(s) fetched. Elapsed 0.015 seconds. Object Store Profiling > select table, path from list_files_cache(); +-------+---------------------------------+ | table | path | +-------+---------------------------------+ | test2 | release/2025-12-17.0/theme=base | +-------+---------------------------------+ 1 row(s) fetched. Elapsed 0.005 seconds. Object Store Profiling > select count(*) from list_files_cache() where table = 'test'; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row(s) fetched. Elapsed 0.014 seconds. > select count(*) from test2 where type = 'infrastructure'; +-----------+ | count(*) | +-----------+ | 142969564 | +-----------+ 1 row(s) fetched. Elapsed 0.013 seconds. Object Store Profiling ``` - Test that dropping a view does not remove cache ``` > create view test2_view as (select * from test2 where type = 'infrastructure'); 0 row(s) fetched. Elapsed 0.103 seconds. Object Store Profiling > select count(*) from test2_view; +-----------+ | count(*) | +-----------+ | 142969564 | +-----------+ 1 row(s) fetched. Elapsed 0.094 seconds. Object Store Profiling > drop view test2_view; 0 row(s) fetched. Elapsed 0.002 seconds. Object Store Profiling > select table, path from list_files_cache(); +-------+---------------------------------+ | table | path | +-------+---------------------------------+ | test2 | release/2025-12-17.0/theme=base | +-------+---------------------------------+ 1 row(s) fetched. Elapsed 0.007 seconds. ``` ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Kumar Ujjawal
|
646213ec7c |
feat: add Time type support to date_trunc function (#19640)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of #19025. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Added Time64/Time32 signatures to date_trunc - Added time truncation logic (hour, minute, second, millisecond, microsecond) - Error for invalid granularities (day, week, month, quarter, year) <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
jizezhang
|
1037f0aa20 |
feat: add list_files_cache table function for datafusion-cli (#19388)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes https://github.com/apache/datafusion/issues/19055. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ``` > CREATE EXTERNAL TABLE nyc_taxi_rides STORED AS PARQUET LOCATION 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/' ; 0 row(s) fetched. Elapsed 10.061 seconds. > SELECT metadata_size_bytes, expires_in, unnest(metadata_list) FROM list_files_cache(); +---------------------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | metadata_size_bytes | expires_in | UNNEST(list_files_cache().metadata_list) | +---------------------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200901.parquet, file_modified: 2025-05-30T09:44:23, file_size_bytes: 222192983, e_tag: "e8d016c3c7af80bf911d96387febe2c1-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200902.parquet, file_modified: 2025-05-30T09:46:00, file_size_bytes: 211023080, e_tag: "1021626ff5ef606422aa7121edd69f3b-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200903.parquet, file_modified: 2025-05-30T09:47:20, file_size_bytes: 229202874, e_tag: "96e7494b217099c6a07e9c4298cbe783-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200904.parquet, file_modified: 2025-05-30T09:44:37, file_size_bytes: 225659965, e_tag: "728c45fabdcd8e40bdef4dfc28df9b0f-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200905.parquet, file_modified: 2025-05-30T09:46:12, file_size_bytes: 232847306, e_tag: "f59e45bd8bd1d77cd7ae8ab6ab468bcc-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200906.parquet, file_modified: 2025-05-30T09:47:26, file_size_bytes: 224226575, e_tag: "8ebb698eea85f9af87065ac333efc449-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200907.parquet, file_modified: 2025-05-30T09:44:52, file_size_bytes: 217168413, e_tag: "7d7ee77f6cac4adc18aa3a9e74600dd3-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200908.parquet, file_modified: 2025-05-30T09:46:23, file_size_bytes: 217303109, e_tag: "e9883055d92a33b941aab971423e681b-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200909.parquet, file_modified: 2025-05-30T09:47:28, file_size_bytes: 223333499, e_tag: "6f0917e6003b38df9060d71c004eb961-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200910.parquet, file_modified: 2025-05-30T09:44:54, file_size_bytes: 246300471, e_tag: "8928b29da44e041021e10077683b7817-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200911.parquet, file_modified: 2025-05-30T09:46:37, file_size_bytes: 227920860, e_tag: "4cd26a1a7f82af080c33e890dc1fef27-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200912.parquet, file_modified: 2025-05-30T09:44:24, file_size_bytes: 233873308, e_tag: "23f4584e494e3c065c777c270c9eedbc-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201001.parquet, file_modified: 2025-05-30T09:45:18, file_size_bytes: 235166925, e_tag: "effcc8cc41b40cf7ac466f911d7b9459-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201002.parquet, file_modified: 2025-05-30T09:46:59, file_size_bytes: 177367931, e_tag: "ce8b7817ecc47da86ccbfa6b51ffa06b-10", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201003.parquet, file_modified: 2025-05-30T09:44:26, file_size_bytes: 205857224, e_tag: "94a078b61e3b652387e6f2a673dc3f4e-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201004.parquet, file_modified: 2025-05-30T09:45:04, file_size_bytes: 243024246, e_tag: "a1efbebfdabc204e0041d8714aaec01a-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201005.parquet, file_modified: 2025-05-30T09:46:47, file_size_bytes: 248130090, e_tag: "d3cf585e00ce627a807348c84a42d0a6-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201006.parquet, file_modified: 2025-05-30T09:44:25, file_size_bytes: 237068130, e_tag: "831db33281a5c017f8ffc466bd47546b-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201007.parquet, file_modified: 2025-05-30T09:45:35, file_size_bytes: 234826090, e_tag: "790e05983e6592e4920c88fbd2bfe774-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201008.parquet, file_modified: 2025-05-30T09:47:14, file_size_bytes: 197990272, e_tag: "d87ddb446e5cbc0f6831fafd95cfd027-11", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201009.parquet, file_modified: 2025-05-30T09:44:27, file_size_bytes: 243408943, e_tag: "abfbe3b29942bcd68d131d95540278d3-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201010.parquet, file_modified: 2025-05-30T09:45:47, file_size_bytes: 225277041, e_tag: "f768c7b77497b2bf3efd5cb2a4362977-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201011.parquet, file_modified: 2025-05-30T09:47:23, file_size_bytes: 220010577, e_tag: "c6830cbe1f3ae918f9280db3aa847b03-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201012.parquet, file_modified: 2025-05-30T09:44:24, file_size_bytes: 219773352, e_tag: "264f7ea433076690a3bbe5566168e5c5-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201101.parquet, file_modified: 2025-05-30T09:45:52, file_size_bytes: 212535107, e_tag: "ca3bdc2707b29667c78c39517781eac4-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201102.parquet, file_modified: 2025-05-30T09:47:23, file_size_bytes: 223138164, e_tag: "e2b3c0fd0c0d66ac6363600de0c8b2ad-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201103.parquet, file_modified: 2025-05-30T09:44:26, file_size_bytes: 252843261, e_tag: "fd5d4e01568cd6e7ef1e00de76441e5b-15", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201104.parquet, file_modified: 2025-05-30T09:46:10, file_size_bytes: 233123935, e_tag: "2b510cc2c0c73d9ec7374c9e6d56c388-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201105.parquet, file_modified: 2025-05-30T09:44:24, file_size_bytes: 246843111, e_tag: "abc2f58bd520b2013aa1a333d317c70c-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201106.parquet, file_modified: 2025-05-30T09:44:58, file_size_bytes: 238786647, e_tag: "0e456698dc42a850ff7b764506cb511d-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201107.parquet, file_modified: 2025-05-30T09:46:40, file_size_bytes: 233249259, e_tag: "28177227cbff94a6a819a0568a14e9b2-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201108.parquet, file_modified: 2025-05-30T09:44:25, file_size_bytes: 212681184, e_tag: "fdcb442e1010630c0553a7018762a8ba-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201109.parquet, file_modified: 2025-05-30T09:45:13, file_size_bytes: 232399266, e_tag: "ccca37be5a3579a8bc644490226ed29a-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201110.parquet, file_modified: 2025-05-30T09:46:52, file_size_bytes: 248471033, e_tag: "eebe34c1bb74f63433eb607810969553-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201111.parquet, file_modified: 2025-05-30T09:44:26, file_size_bytes: 231103826, e_tag: "7c76b9fc111462b76336d63bce3253c7-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201112.parquet, file_modified: 2025-05-30T09:45:40, file_size_bytes: 236102882, e_tag: "26c10d1d85c4565cbb9e8fc6a7bc745c-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201201.parquet, file_modified: 2025-05-30T09:47:21, file_size_bytes: 236184052, e_tag: "8cdc15a22462579dcf90d669cea0f04b-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201202.parquet, file_modified: 2025-05-30T09:44:27, file_size_bytes: 238377570, e_tag: "4e6734c5c2e77c68dde5155a45dac81c-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201203.parquet, file_modified: 2025-05-30T09:46:06, file_size_bytes: 258226172, e_tag: "b7b07fa0f4fefcf0ba0fc69ba344b5c8-15", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201204.parquet, file_modified: 2025-05-30T09:44:25, file_size_bytes: 248190698, e_tag: "968c13850fa9a7cb46337bc8fc9d13fa-14", version: NULL} | | . | | . | | . | +---------------------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 96 row(s) fetched. (First 40 displayed. Use --maxrows to adjust) Elapsed 0.057 seconds. ``` ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> This will enable a new user-facing table function to datafusion cli. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Bruce Ritchie
|
ada0923a39 |
Respect execution timezone in to_timestamp and related functions (#19078)
## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/17998. Continuation of PR #18025 by @kosiew ## Rationale for this change Previously, the to_timestamp() family of functions (to_timestamp, to_timestamp_seconds, to_timestamp_millis, to_timestamp_micros, to_timestamp_nanos) always interpreted timezone-free (naïve) timestamps as UTC, ignoring the datafusion.execution.time_zone configuration option. This behavior caused inconsistencies when users configured a specific execution timezone and expected timestamp conversions to respect it. This PR introduces full timezone awareness to these functions so that: - Naïve timestamp strings are interpreted as being in the configured execution timezone, or UTC if the configured execution timezone is `None`. - All returned timestamps are in the execution timezone. ## What changes are included in this PR? Code, tests. ## Are these changes tested? Yes, via code tests and slt tests. ## Are there any user-facing changes? Yes: - to_timestamp() and its precision variants now respect datafusion.execution.time_zone when parsing timezone-free timestamps and return timestamps in the execution time zone.. These changes make timestamp functions consistent with session timezone semantics and improve correctness for global workloads. --------- Co-authored-by: Siew Kam Onn <kosiew@gmail.com> |
||
|
Kumar Ujjawal
|
818706ab78 |
feat: to_time function (#19540)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of #19025. ## Rationale for this change Previously we needed to create timestamp and then extract the time component from that. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Added the function for `to_time` - Relevant slt test - Relevant unit tests - Updated docs <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? - All tests pass <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Jeffrey Vo
|
13f38435a2 |
Introduce TypeSignatureClass::Any (#19485)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19438 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> See issue. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Add new `TypeSignatureClass` variant `Any` and refactor `arrow_typeof` and `arrow_metadata` function signatures to use this. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Existing tests. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Kumar Ujjawal
|
62740802f6 |
Update to_unixtime udf function to support a consistent set of argument types (#19442)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19119 ## Rationale for this change to_unixtime lacks the support for several data types. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? Expanded `to_unixtime` support to all signed ints (Int8/16/32/64), all unsigned ints (UInt8/16/32/64), all floats (Float16/32/64), all UTF8 variants (Utf8/Utf8View/LargeUtf8), <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Added sqllogictest <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> |
||
|
Adrian Garcia Badaracco
|
a405d3fe4a |
Support nested field access in get_field with multiple path arguments (#19389)
## Summary This PR extends `get_field` to accept multiple field name arguments for nested struct/map access, enabling `get_field(col, 'a', 'b', 'c')` as equivalent to `col['a']['b']['c']`. **The primary motivation is to make it easier for downstream optimizations to match on and optimize struct/map field access patterns.** By representing `col['a']['b']['c']` as a single `get_field(col, 'a', 'b', 'c')` call rather than nested `get_field(get_field(get_field(col, 'a'), 'b'), 'c')` calls, optimization rules can more easily identify and transform field access patterns. This is related / maybe prep work for #19387 but I think is a good improvement in its own right. ## Changes - **Variadic signature**: `get_field` now accepts 2+ arguments (base + one or more field names) - **Type validation at planning time**: Accessing a field on a non-struct/map type (e.g., `get_field({a: 1}, 'a', 'b')`) fails during planning with a clear error message indicating which argument position caused the failure - **Bracket syntax optimization**: The `FieldAccessPlanner` now merges consecutive bracket accesses into a single `get_field` call (e.g., `s['a']['b']` → `get_field(s, 'a', 'b')`) - **Mixed access handling**: Array index access correctly breaks the batching (e.g., `s['a'][0]['b']` → `get_field(array_element(get_field(s, 'a'), 0), 'b')`) ## Example ```sql -- Direct function call with nested access SELECT get_field(my_struct, 'outer', 'inner', 'value'); -- Equivalent bracket syntax (now optimized to single get_field) SELECT my_struct['outer']['inner']['value']; -- EXPLAIN shows single get_field call EXPLAIN SELECT s['a']['b'] FROM t; -- Projection: get_field(t.s, Utf8("a"), Utf8("b")) ``` ## Backwards Compatibility - The original 2-argument form `get_field(struct, 'field')` continues to work unchanged - Existing queries using bracket syntax will automatically benefit from the optimization ## Test plan - [x] Backwards compatibility test for 2-argument form - [x] Multi-level get_field with 2, 3, and 5 levels of nesting - [x] Type validation error tests at argument positions 2, 3, 4 - [x] Non-existent field error tests - [x] Null handling (null at base, null in middle of chain) - [x] Mixed array/struct access (verifies array index breaks batching) - [x] Nullable parent propagation - [x] EXPLAIN test verifying single get_field call for bracket syntax - [x] Minimum argument validation (0 and 1 argument cases) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Huaijin
|
0bd880931e |
fix: csv schema_infer_max_records set to 0 return null datatype (#19432)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - close https://github.com/apache/datafusion/issues/19417 ## Rationale for this change - see https://github.com/apache/datafusion/issues/19417 - related to https://github.com/apache/datafusion/pull/17796 ## What changes are included in this PR? when schema_infer_max_records set to 0 in csv, return datatype as string ## Are these changes tested? add test case for schema_infer_max_records equal to 0 ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Bruce Ritchie
|
4a1f69f9bc |
Update date_bin to support Time32 and Time64 data types (#19341)
## Which issue does this PR close? Part of #19025 ## Rationale for this change Expand support for binning time data types. ## What changes are included in this PR? Code, tests. ## Are these changes tested? Yes, slt tests. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> |
||
|
xonx
|
d844f8687a |
Add:arrow_metadata() UDF (#19435)
## Which issue does this PR close? Closes #19356 ## Rationale for this change This PR implements the arrow_metadata UDF as requested in issue #19356. ## What changes are included in this PR? Added arrow_metadata UDF Refactored Tests ## Are these changes tested? Yes. ## Are there any user-facing changes? Yes. |
||
|
Gene Bordegaray
|
5419ff5902 |
feat: hash partitioning satisfies subset (#19304)
## Which issue does this PR close? - Closes #19269. ## Rationale for this change See to issue #19269 for deeper rationale. DF did not have the notion that being partitioned on a superset of the required partitioning satisfied the condition. Having this logic will eliminate unnecessary repartitions and in turn other operators like partial aggregations. I introduced this behavior with the `repartition_subset_satisfactions` flag (default false) as there are some cases where repartitioning may still be wanted when we satisfy partitioning via this subset property. In particular, if when partitioned via Hash(a) there is data skew but when partitioned on Hash(a, b) there is better distribution, a user may want to turn this optimization off. I also made it the case such that if we satisfy repartitioning via a subset but the current amount of partitions < target_partitions, then we will still repartition to maintain and increase parallelism in the system when possible. ## What changes are included in this PR? - Modified `satisfy()` logic to check for subsets and return an enum of type of match: exact, subset, none - Do in `EnforceDistribution`, where `satisfy()` is called, do not allow subset logic for partitioned join operators as partitioning on each side much match exactly, thus need to repartition if subset logic is true - Created unit and sqllogictests ## Are these changes tested? - Unit test - sqllogictest - tpch correctness ### Benchmarks I did not see any drastic changes in benches, but the shuffle eliminations will be great improvements for distributed DF. <img width="628" height="762" alt="Screenshot 2025-12-12 at 8 28 15 PM" src="https://github.com/user-attachments/assets/4b42945f-34e0-46c9-a4ce-e7ccdd0c0603" /> <img width="490" height="746" alt="Screenshot 2025-12-12 at 8 30 15 PM" src="https://github.com/user-attachments/assets/846aef1b-8c5d-462d-83e7-7fa1e2a9372e" /> ## Are there any user-facing changes? Yes, users will now have the `repartition_subset_satisfications` option as described in this PR --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Kumar Ujjawal
|
d493f3d441 |
Add Decimal support to Ceil and Floor (#18979)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #7689. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Added dedicated ceil/floor UDF implementations that keep existing float/int behavior but operate directly on Decimal128 arrays, including overflow checks and metadata preservation. - Updated the math module wiring plus sqllogictest coverage so decimal cases are executed and validated end to end. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? - All existing tests pass - Added new tests for the changes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Qi Zhu
|
3a41cc6078 |
Establish the high level API for sort pushdown and the optimizer rule and support reverse files and row groups (#19064)
## Which issue does this PR close? Establish the high level API for sort pushdown and the optimizer rule. Only re-arrange files and row groups and return Inexact, now support reverse order case, and we don't need to cache anything for this implementation, so it's no memory overhead. It will have huge performance improvement with dynamic topk pushdown to skip row groups. Details: Performance results on ClickBench sorted data: 13ms vs 300ms baseline (23x faster), close to aggressive caching approach (9.8ms) but with much better memory stability [details](https://github.com/apache/datafusion/pull/18817#issuecomment-3605978882) - Closes https://github.com/apache/datafusion/issues/19059 - Closes https://github.com/apache/datafusion/issues/10433 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
delamarch3
|
79cfe8e921 |
Add runtime config options for list_files_cache_limit and list_files_cache_ttl (#19108)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19056 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Make the list file cache memory limit and TTL configurable via runtime config. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> * Add ability to `SET` and `RESET` `list_files_cache_limit` and `list_files_cache_ttl` * `list_files_cache_ttl` will expect the duration to look like either `1m30s` or `30` (I'm wondering if it would be simpler for it to just accept a single unit?) * Add `update_cache_ttl()` to the `ListFilesCache` trait so we can update it from `RuntimeEnvBuilder::build()` * Add config entries ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> `update_cache_ttl()` has been added to the `ListFilesCache` trait <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Bruce Ritchie
|
5a01e68643 |
Update to_date udf function to support a consistent set of argument types (#19134)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19120 ## Rationale for this change Improved type support for to_date function. ## What changes are included in this PR? Code, slt, updated docs. ## Are these changes tested? Yes ## Are there any user-facing changes? More types supported. |
||
|
Bruce Ritchie
|
044a4a7369 |
Add make_time function (#19183)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of #19025 ## Rationale for this change There wasn't a good way to make a time from component parts. ## What changes are included in this PR? Code, test, docs ## Are these changes tested? Yes ## Are there any user-facing changes? New function. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Gene Bordegaray
|
bde16083ad |
feat: Preserve File Partitioning From File Scans (#19124)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19090. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Datafusion does not have the option to preserve file partitioning from file scans, rather it always returns `UknownPartitioning`. Some queries and datasets would see great benefits by preserving their explicit partitioning to avoid shuffles. An example of this would be the following scenario: Say have data partitioned by `f_dkey` and ordered by `(f_dkey, timestamp)`, which is hive-style partitioned: ``` f_dkey=A/data.parquet f_dkey=B/data.parquet' f_dkey=C/data.parquet' ... Each table (partitioned by f_dkey and sorted internally sorted by timestamp): | f_dkey | timestamp | value | |--------|------------------------|--------| | A | 2023-01-01T09:00:00 | 95.5 | | A | 2023-01-01T09:00:10 | 102.3 | | A | 2023-01-01T09:00:20 | 98.7 | | A | 2023-01-01T09:12:20 | 105.1 | | A | 2023-01-01T09:12:30 | 100.0 | | A | 2023-01-01T09:12:40 | 150.0 | | A | 2023-01-01T09:12:50 | 120.8 | ``` Runnuing the query: ```sql EXPLAIN SELECT f_dkey, count(*), avg(value) FROM fact_table_ordered GROUP BY f_dkey ORDER BY f_dkey; ``` Prior to this PR would produce the plan: ```text 01)SortPreservingMergeExec: [f_dkey@0 ASC NULLS LAST] 02)--ProjectionExec: expr=[f_dkey@0 as f_dkey, count(Int64(1))@1 as count(*), avg(fact_table_ordered.value)@2 as avg(fact_table_ordered.value)] 03)----AggregateExec: mode=FinalPartitioned, gby=[f_dkey@0 as f_dkey], aggr=[count(Int64(1)), avg(fact_table_ordered.value)], ordering_mode=Sorted 04)------SortExec: expr=[f_dkey@0 ASC NULLS LAST], preserve_partitioning=[true] 05)--------CoalesceBatchesExec: target_batch_size=8192 06)----------RepartitionExec: partitioning=Hash([f_dkey@0], 3), input_partitions=3 07)------------AggregateExec: mode=Partial, gby=[f_dkey@1 as f_dkey], aggr=[count(Int64(1)), avg(fact_table_ordered.value)], ordering_mode=Sorted 08)--------------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=A/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=C/data.parquet]]}, projection=[value, f_dkey], output_ordering=[f_dkey@1 ASC NULLS LAST], file_type=parquet ``` This can be improved. Our data is ordered on `(f_dkey, timestamp)`, when we hash repartition by `f_dkey` we lose our sort ordering thus forcing a `SortExec` to be inserted after the repartition. You could set `datafusion.optimizer.prefer_existing_sort = true;` to preserve the ordering through the repartition and thus preserve the ordering, but with the tradeoff of a more expensive shuffle. Since our data is partitioned by `f_dkey` at file scan time we can eliminate both the hash repartitioning, the eliminating the `SortExec` in the process. This would result in a plan that looks like: ```text 01)SortPreservingMergeExec: [f_dkey@0 ASC NULLS LAST] 02)--ProjectionExec: expr=[f_dkey@0 as f_dkey, count(Int64(1))@1 as count(*), avg(fact_table_ordered.value)@2 as avg(fact_table_ordered.value)] 03)----AggregateExec: mode=SinglePartitioned, gby=[f_dkey@1 as f_dkey], aggr=[count(Int64(1)), avg(fact_table_ordered.value)], ordering_mode=Sorted 04)------DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=A/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=C/data.parquet]]}, projection=[value, f_dkey], output_ordering=[f_dkey@1 ASC NULLS LAST], file_type=parquet ``` ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> I have extended the `FileScanConfig`'s implementation of `DataSource` to have the ability to preserve hive-style partitioning by declaring it's `output_partitioning` as hash partitoned on the hive columns. When the user sets the option `preserve_file_partitions > 0` (0 by default, which is disabled) Datafusion will take advantage of partitioned files. Specifically, when `preserve_file_partitions` is enabled: 1. if `preserve_file_partitions` > the number distinct partitions found -> we fallback to partitioning by byte ranges 2. if `preserve_file_partitions` <= the number distinct partitions found -> we will keep the file partitioning Because we can have fewer file partition groups than `target_partitions`, forcing a partition group (with possibly large amounts of data) to be read in a single partition can increase file I/O. This configuration choice was made to be able to control the amount of I/O overhead a user is willing to have in order to eliminate shuffles. (This was recommended by @gabotechs and is a great approach to have more granularity over this behavior rather than a boolean flag, thank you) Reusing hash repartitioning has rippling effects throughout query plans, such as propagating through joins and windows as well as preserving order, which is great to see. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 5. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> - Added unit tests for new functionality - Added sqllogictests to confirm end to end behavior - Added new benchmark that compares queries with: no optimization, preserver order through repartitions, and preserve partitioning from file scans (this PR). We see nice speed ups and this scales linearly as data grows (table with results below) ### Small Data: - Normal config: 10 partitions × 1000000 rows = 10000000 total rows - High-cardinality config: 25 partitions × 500000 rows = 12500000 total rows | Plan | preserve_order (ms) | Speedup vs not opt | Speedup vs psr | preserve_order_join (ms) | Speedup vs not opt | Speedup vs psr | preserve_order_window (ms) | Speedup vs not opt | Speedup vs psr | |---------------------------|---------------------|---------------------------|---------------------------|---------------------------|---------------------------|----------------------------|-----------------------------|-----------------------------|----------------------------| | not optimized | 17.268 | - | - | 41.521 | - | - | 28.796 | - | - | | preserve sort repartition | 15.908 | - | - | 40.580 | - | - | 17.669 | - | - | | optimized (this PR) | 15.000 | 13.1% (1.15x) | 5.7% (1.06x) | 40.977 | 1.3% (1.01x) | -1.0% (0.99x) | 4.301 | 85.1% (6.70x) | 75.7% (4.11x) | The optimized plan is roughly on par with the other plans for preserve_order and preserve_order_join, but it makes preserve_order_window 6.7× faster than not optimized and 4.1× faster than preserve-sort-repartition. --- ### Medium Dataset: - Normal config: 30 partitions × 3000000 rows = 90000000 total rows - High-cardinality config: 75 partitions × 1500000 rows = 112500000 total rows | Plan | preserve_order (ms) | Speedup vs not opt | Speedup vs psr | preserve_order_join (ms) | Speedup vs not opt | Speedup vs psr | preserve_order_window (ms) | Speedup vs not opt | Speedup vs psr | |---------------------------|---------------------|---------------------------|---------------------------|---------------------------|---------------------------|----------------------------|-----------------------------|-----------------------------|----------------------------| | not optimized | 752.130 | - | - | 451.300 | - | - | 193.210 | - | - | | preserve sort repartition | 392.050 | - | - | 477.400 | - | - | 115.320 | - | - | | optimized (this PR) | 93.818 | 87.5% (8.02x) | 76.1% (4.18x) | 203.540 | 54.9% (2.22x) | 57.4% (2.35x) | 9.841 | 94.9% (19.63x) | 91.5% (11.72x) | The optimized plan makes preserve_order about 8.0× faster than not optimized (4.2× vs PSR), preserve_order_join about 2.2× faster than not optimized (2.35× vs PSR), and preserve_order_window a huge 19.6× faster than not optimized (11.7× vs PSR). --- ### Large Dataset: - Normal config: 50 partitions × 6000000 rows = 300000000 total rows - High-cardinality config: 125 partitions × 3000000 rows = 375000000 total rows | Plan | preserve_order (ms) | Speedup vs not opt | Speedup vs psr | preserve_order_join (ms) | Speedup vs not opt | Speedup vs psr | preserve_order_window (ms) | Speedup vs not opt | Speedup vs psr | |---------------------------|---------------------|---------------------------|---------------------------|---------------------------|---------------------------|----------------------------|-----------------------------|-----------------------------|----------------------------| | not optimized | 2699.700 | - | - | 1563.800 | - | - | 614.440 | - | - | | preserve sort repartition | 1244.200 | - | - | 1594.300 | - | - | 371.300 | - | - | | optimized (this PR) | 290.740 | 89.2% (9.29x) | 76.6% (4.28x) | 645.180 | 58.7% (2.42x) | 59.5% (2.47x) | 11.538 | 98.1% (53.25x) | 96.9% (32.18x) | The optimized plan makes preserve_order about 9.3× faster than not optimized (4.3× vs PSR), preserve_order_join about 2.4× faster than not optimized (2.5× vs PSR), and preserve_order_window an extreme 53× faster than not optimized (32× vs PSR). ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Yes users now can use the `preserve_file_partitions` option to define the amount of partitions they want to preserve file partitioning for (0 disabled). If enabled and triggered, users will see elimination of repartitions on their file partition key when appropriate. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> ## Follow-Up Work - **Superset Partitioning:** currently, `Hash(a)` doesn't satisfy `Hash(a, b)` although it should. This is because `Hash(a)` guarantees that all of `a` is contained in a single partition. Thus, since `Hash(a, b)` is a subset of `Hash(a)`, anything that is `Hash(a)` is also `Hash(a, b)`. - **Reduce File I/O with Preserve File Partitioning:** In the current implementation, when a partition value has many files all of this file I/O will go to one task. This is a tradeoff that increases I/O overhead to eliminate shuffle and sort overhead. There could be ways to increase I/O while still maintaining partitioning. - **Sort Satisfaction for Monotonic Functions:** If we are sorted by `timestamp` and then try to order by `date_bin('1 hour', timestamp)`, Datafusion will not recognize that this is implicitly satisfied. Thus, for monotonic functions: `date_bin`, `CAST`, `FLOOR`, etc. we should maintain ordering, eliminating unnecessary sorts. |
||
|
mag1c1an1
|
5496c30431 |
fix: typo in sql/ddl (#19276)
## Which issue does this PR close? None, this is a tiny typo fix. <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Make datafusion doc great! ## What changes are included in this PR? Change doc. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? No need to test code. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? NO <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Nuno Faria
|
dc4e3ab473 |
feat: Implement the statistics_cache function (#19054)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #18953. ## Rationale for this change Allow a way to check the contents of the file statistics cache. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Added the `statistics_cache` function to `datafusion-cli`. - Converted `FileStatisticsCache` to a trait and implemented the `list_entries` method. - Added unit tests. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Yes, `FileStatisticsCache` has been changed to a trait. Previous implementations need to implement the `list_entries` method. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Adrian Garcia Badaracco
|
c0e8bb501a |
Push down InList or hash table references from HashJoinExec depending on the size of the build side (#18393)
This PR is part of an EPIC to push down hash table references from HashJoinExec into scans. The EPIC is tracked in https://github.com/apache/datafusion/issues/17171. A "target state" is tracked in https://github.com/apache/datafusion/pull/18393 (*this PR*). There is a series of PRs to get us to this target state in smaller more reviewable changes that are still valuable on their own: - https://github.com/apache/datafusion/pull/18448 - https://github.com/apache/datafusion/pull/18449 (depends on https://github.com/apache/datafusion/pull/18448) - https://github.com/apache/datafusion/pull/18451 As those are merged I will rebase this PR to keep track of the "remaining work", and we can use this PR to explore big picture ideas or benchmarks of the final state. |
||
|
Andrew Lamb
|
8dac8f1c41 |
Minor: fix link errors in docs (#19088)
I had an AI tool (`codex`) look for broken links in the docs and it found two. I tested both changes locally |
||
|
Andrew Lamb
|
9af6858ad4 |
Add force_filter_selections to restore pushdown_filters behavior prior to parquet 57.1.0 upgrade (#19003)
~Draft until https://github.com/apache/datafusion/pull/18820 is merged~ ## Which issue does this PR close? - Follow on to https://github.com/apache/datafusion/pull/18820 ## Rationale for this change The parquet 57.1.0 upgrade includes a new adaptive filter from @hhhizzz : - https://github.com/apache/arrow-rs/pull/8733 Our testing shows this is faster in all cases, but I want to have an escape valve for people to turn it off if they hit some issue. I had originally included this in #18820 but @rluvaton suggested it would be easier to understand as its own PR in https://github.com/apache/datafusion/pull/18820#pullrequestreview-3509993052 ## What changes are included in this PR? 1. Add a `force_filter_selections` config setting 2. Add configuration guide 3. Add tests ## Are these changes tested? Yes ## Are there any user-facing changes? A new boolean flag |
||
|
Yongting You
|
653caa143a |
doc: add FilterExec metrics to user-guide/metrics.md (#19043)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> part of https://github.com/apache/datafusion/issues/16602 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Start adding each operator's metrics to the user-guide page ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> |