mirror of
https://github.com/langchain-ai/datafusion.git
synced 2026-07-19 23:35:20 -04:00
e937cadbcceff6a42bee2c5fc8d03068fa0eb30c
993 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Neil Conway
|
585bbf35d3 |
perf: Optimize array_has_any() with scalar arg (#20385)
## Which issue does this PR close? - Closes #20384. - See #18181 for related context. ## Rationale for this change When `array_has_any` is passed a scalar for either of its arguments, we can use a much faster algorithm: rather than doing O(N*M) comparisons for each row of the columnar arg, we can build a hash table on the scalar argument and probe it instead. ## What changes are included in this PR? * Add benchmark to cover the one-scalar-arg case * Implement optimization as described above Note that we fallback to a linear scan when the scalar arg is smaller than a threshold (<= 8 elements), because benchmarks suggested probing a HashSet is not profitable for very small arrays. ## Are these changes tested? Yes. Tests pass and benchmarked. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Albert Skalt
|
34dad2ccee |
Cache PlanProperties, add fast-path for with_new_children (#19792)
- closes https://github.com/apache/datafusion/issues/19796 This patch aims to implement a fast-path for the ExecutionPlan::with_new_children function for some plans, moving closer to a physical plan re-use implementation and improving planning performance. If the passed children properties are the same as in self, we do not actually recompute self's properties (which could be costly if projection mapping is required). Instead, we just replace the children and re-use self's properties as-is. To be able to compare two different properties -- ExecutionPlan::properties(...) signature is modified and now returns `&Arc<PlanProperties>`. If `children` properties are the same in `with_new_children` -- we clone our properties arc and then a parent plan will consider our properties as unchanged, doing the same. - Return `&Arc<PlanProperties>` from `ExecutionPlan::properties(...)` instead of a reference. - Implement `with_new_children` fast-path if there is no children properties changes for all major plans. Note: currently, `reset_plan_states` does not allow to re-use plan in general: it is not supported for dynamic filters and recursive queries features, as in this case state reset should update pointers in the children plans. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Oleks V
|
ed0323a2bb |
feat: support arrays_zip function (#20440)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change Summary - Adds a new arrays_zip scalar function that combines multiple arrays into a single array of structs, where each struct field corresponds to an input array - Shorter arrays within a row are padded with NULLs to match the longest array's length - Compatible with Spark's arrays_zip behavior <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? ``` arrays_zip takes N list arrays and produces a List<Struct<c0, c1, ..., cN>> where each struct contains the elements at the same index from each input array. > SELECT arrays_zip([1, 2, 3], ['a', 'b', 'c']); [{c0: 1, c1: a}, {c0: 2, c1: b}, {c0: 3, c1: c}] > SELECT arrays_zip([1, 2], [3, 4, 5]); [{c0: 1, c1: 3}, {c0: 2, c1: 4}, {c0: NULL, c1: 5}] Implementation details: - Implemented in set_ops.rs following existing array function patterns - Uses MutableArrayData builders per column with row-by-row processing for efficient memory handling - For each row, computes the max array length, copies values from each input array, and pads shorter arrays with NULLs - Supports variadic arguments (2 or more arrays) - Handles NULL list entries, NULL elements, empty arrays, mixed types, and Null-typed arguments - Registered as arrays_zip with alias list_zip - Uses Signature::variadic_any with Volatility::Immutable ``` <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Dmitrii Blaginin
|
60457d0b0a |
Runs-on for more actions (#20274)
Follow up on https://github.com/apache/datafusion/pull/20107: switch more actions to the new flow | Job | OLD | NEW | Delta | |---|---|---|---| | **linux build test** (from #20107) | 3m 55s | 1m 46s | -2m 09s (55% faster) | | **cargo test (amd64)** (from #20107) | 11m 34s | 3m 13s | -8m 21s (72% faster) | | **cargo check datafusion features** | 11m 18s | 6m 21s | -4m 57s (44% faster) | | **cargo examples (amd64)** | 9m 13s | 4m 35s | -4m 38s (50% faster) | | **verify benchmark results (amd64)** | 11m 48s | 4m 22s | -7m 26s (63% faster) | | **cargo check datafusion-substrait features** | 10m 20s | 3m 56s | -6m 24s (62% faster) | | **cargo check datafusion-proto features** | 4m 48s | 2m 25s | -2m 23s (50% faster) | | **cargo test datafusion-cli (amd64)** | 5m 42s | 1m 58s | -3m 44s (65% faster) | | **cargo test doc (amd64)** | 8m 07s | 3m 16s | -4m 51s (60% faster) | | **cargo doc** | 5m 10s | 1m 56s | -3m 14s (63% faster) | | **Run sqllogictest with Postgres runner** | 6m 06s | 2m 46s | -3m 20s (55% faster) | | **Run sqllogictest in Substrait round-trip mode** | 6m 42s | 2m 38s | -4m 04s (61% faster) | | **clippy** | 6m 01s | 2m 10s | -3m 51s (64% faster) | | **check configs.md and \*\*\*_functions.md is up-to-date** | 6m 54s | 2m 12s | -4m 42s (68% faster) | |
||
|
Eren Avsarogullari
|
a936d0de95 |
test: Extend Spark Array functions: array_repeat , shuffle and slice test coverage (#20420)
## Which issue does this PR close? - Closes #20419. ## Rationale for this change This PR adds new positive test cases for `datafusion-spark` array functions: `array_repeat `, `shuffle`, `slice` for the following use-cases: ``` - nested function execution, - different datatypes such as timestamp, - casting before function execution ``` Also, being updated contributor-guide testing documentation with minor addition. ## What changes are included in this PR? Being added new positive test cases to `datafusion-spark` array functions: `array_repeat `, `shuffle`, `slice`. ## Are these changes tested? Yes, adding new positive test cases. ## Are there any user-facing changes? No |
||
|
Adrian Garcia Badaracco
|
1ee782f783 |
Migrate Python usage to uv workspace (#20414)
I was having trouble getting benchmarks to gen data. ## Summary - Replace three independent `requirements.txt` files with a uv workspace (`benchmarks`, `dev`, `docs` projects) - Single `uv.lock` lockfile for reproducible dependency resolution - Simplify `bench.sh` by removing all ad-hoc venv/pip logic in favor of `uv run` ## Test plan - [ ] `uv sync` resolves all deps from repo root - [ ] `uv run --project benchmarks python3 benchmarks/compare.py` works - [ ] `uv run --project docs sphinx-build docs/source docs/build` builds docs - [ ] Run a benchmark from `bench.sh` that uses Python (e.g., h2o data gen or compare flow) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Neil Conway
|
ace9cd44b7 |
perf: Optimize trim UDFs for single-character trims (#20328)
## Which issue does this PR close? - Closes #20327 ## Rationale for this change By default, btrim(), ltrim(), and rtrim() trim space characters; it is also reasonably common for queries to specify a non-default trim pattern that is still a single ASCII character. We can optimize for this case by doing a byte-level scan, rather than invoking the more heavyweight std::string machinery used for more complex trim scenarios. ## What changes are included in this PR? Add a benchmark for trimming spaces, and implement the optimization described above. Also fixed an error in the documentation. ## Are these changes tested? Yes, and benchmarked. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> |
||
|
Neil Conway
|
b7f6090874 |
perf: Optimize lpad, rpad for ASCII strings (#20278)
The previous implementation incurred the overhead of Unicode machinery,
even for the common case that both the input string and the fill string
consistent only of ASCII characters. For the ASCII-only case, we can
assume that the length in bytes equals the length in characters, and
avoid expensive graphene-based segmentation. This follows similar
optimizations applied elsewhere in the codebase.
Benchmarks indicate this is a significant performance win for ASCII-only
input (4x-10x faster) but only a mild regression for Unicode input (2-5%
slower).
Along the way:
* Combine: a few instances of `write_str(str)? + append_value("")` with
`append_value(str)`, which saves a few cycles
* Add a missing test case for truncating the input string
* Add benchmarks for Unicode input
## Which issue does this PR close?
- Closes #20277.
## Are these changes tested?
Covered by existing tests. Added new benchmarks for Unicode inputs.
## Are there any user-facing changes?
No.
---------
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
|
||
|
Neil Conway
|
c3f080774c |
perf: Optimize translate() UDF for scalar inputs (#20305)
## Which issue does this PR close? - Closes #20302. ## Rationale for this change `translate()` is commonly invoked with constant values for its second and third arguments. We can take advantage of that to significantly optimize its performance by precomputing the translation lookup table, rather than recomputing it for every row. For ASCII-only inputs, we can further replace the hashmap lookup table with a fixed-size array that maps ASCII byte values directly. For scalar ASCII inputs, this yields roughly a 10x performance improvement. For scalar UTF8 inputs, the performance improvement is more like 50%, although less so for long strings. Along the way, add support for `translate()` on `LargeUtf8` input, along with an SLT test, and improve the docs. ## What changes are included in this PR? * Add a benchmark for scalar/constant input to translate * Add a missing test case * Improve translate() docs * Support translate() on LargeUtf8 input * Optimize translate() for scalar inputs by precomputing lookup hashmap * Optimize translate() for ASCII inputs by precomputing ASCII byte-wise lookup table ## Are these changes tested? Yes. Added an extra test case and did a bunch of benchmarking. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Adrian Garcia Badaracco
|
7d217b1ec3 |
Reduce ExtractLeafExpressions optimizer overhead with fast pre-scan (#20341)
## Summary Follow-up to #20117 which added the `ExtractLeafExpressions` and `PushDownLeafProjections` optimizer rules for get_field pushdown. Benchmarking revealed that these rules added 5-31% overhead on *all* queries (including those with no struct/get_field expressions) because they unconditionally allocated column HashSets, extractors, and walked every expression tree for every Filter/Sort/Limit/Aggregate/Join node. This PR adds: - **`has_extractable_expr()` pre-scan**: A lightweight check using `Expr::exists()` that short-circuits before any expensive allocations when no `MoveTowardsLeafNodes` expressions are present - **Config option** `datafusion.optimizer.enable_leaf_expression_pushdown` to disable the rules entirely ### Benchmark Results (vs no-rules baseline) | Benchmark | Before Fix | After Fix | |---|---|---| | physical_select_aggregates_from_200 | +31.1% | +3.7% | | physical_many_self_joins | +12.9% | +2.2% | | physical_join_consider_sort | +12.9% | +1.0% | | physical_unnest_to_join | +12.5% | +1.4% | | physical_select_one_from_700 | +12.2% | +2.6% | | physical_theta_join_consider_sort | +8.7% | +0.2% | | physical_plan_tpch_q18 | +9.3% | +1.4% | | physical_plan_tpch_all | +4.8% | +2.1% | | physical_plan_tpcds_all | +5.6% | +2.2% | ## Test plan - [x] All 47 `extract_leaf_expressions` unit tests pass - [x] Benchmarked with `cargo bench -p datafusion --bench sql_planner` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Michael Kleen
|
8fc9681b74 |
feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations (#19930)
## Which issue does this PR close? Cleaned-up version of https://github.com/apache/datafusion/pull/18806 with: - Removed `outer_queries_schema` from PlannerContext - Planning logic only (optimizer modifications removed) - sql logic tests moved to sql_integration.rs - Closes https://github.com/apache/datafusion/issues/19816 ## Rationale for this change See https://github.com/apache/datafusion/pull/18806 ## What changes are included in this PR? See https://github.com/apache/datafusion/pull/18806 ## Are these changes tested? Yes ## Are there any user-facing changes? `outer_queries_schema` is removed from PlannerContext. --------- Co-authored-by: Duong Cong Toai <duongcongtoai.fjc@gmail.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
mishop-15
|
15eea084fb |
Break upgrade guides into separate pages (#20183)
Fixes #20155 ## Which issue does this PR close? Closes #20155 ## Rationale for this change The upgrade guide was one giant page with all versions. When you search for something like "Aggregate", you get results from multiple versions and can't tell if you're looking at v46 or v44 changes. ## What changes are included in this PR? Split upgrading.md into separate files - one per version. Created an upgrading/ directory with an index similar to how user-guide/sql/ is organized. ## Are these changes tested? Built the docs locally with `make html`. Each version now gets its own page and ctrl-F works within that version only. ## Are there any user-facing changes? Yes - upgrade guides are now separate pages instead of one long page. Makes it easier to find version-specific info. |
||
|
Neil Conway
|
75428f1e0a |
fix: Avoid integer overflow in split_part() (#20198)
Along the way, improve the docs slightly.
## Rationale for this change
Evaluating `SELECT SPLIT_PART('', '', -9223372036854775808);` yields (in
a debug build):
```
thread 'main' (41405991) panicked at datafusion/functions/src/string/split_part.rs:236:47:
attempt to negate with overflow
```
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## Are these changes tested?
Yes, added unit test.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
|
||
|
kosiew
|
8c478e9452 |
Disallow positional struct casting when field names don’t overlap (#19955)
## Which issue does this PR close? * Closes #19841. ## Rationale for this change Struct-to-struct casting previously fell back to **positional mapping** when there was **no field-name overlap** and the number of fields matched. That behavior is ambiguous and can silently produce incorrect results when source/target schemas have different field naming conventions or ordering. This PR makes struct casting **strictly name-based**: when there is no overlap in field names between the source and target structs, the cast is rejected with a clear planning error. This prevents accidental, hard-to-detect data corruption and forces callers to provide explicit field names (or align schemas) when casting. ## What changes are included in this PR? * Removed the positional fallback logic for struct casting in `cast_struct_column`; child fields are now resolved **only by name**. * Updated `validate_struct_compatibility` to **error out** when there is **no field name overlap**, instead of allowing positional compatibility checks. * Updated unit tests to reflect the new behavior (no-overlap casts now fail with an appropriate error). * Updated SQLLogicTest files to construct structs using **explicit field names** (e.g. `{id: 1}` / `{a: 1, b: 'x'}` or `struct(… AS field)`), avoiding reliance on positional behavior. * Improved error messaging to explicitly mention the lack of field name overlap. ## Are these changes tested? Yes. * Updated existing Rust unit tests in `nested_struct.rs` to assert the new failure mode and error message. * Updated SQLLogicTest coverage (`struct.slt`, `joins.slt`) to use named struct literals so tests continue to validate struct behavior without positional casting. ## Are there any user-facing changes? Yes — behavioral change / potential breaking change. * Casting between two `STRUCT`s with **no overlapping field names** now fails (previously it could succeed via positional mapping if field counts matched). * Users must provide explicit field names (e.g. `{a: 1, b: 'x'}` or `struct(expr AS a, expr AS b)`) or ensure schemas share field names. * Error messages are more explicit: casts are rejected when there is “no field name overlap”. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Dmitrii Blaginin
|
96a6bd71ed |
Runs-on for linux-build-lib and linux-test (2X faster CI) (#20107)
## Which issue does this PR close? Related to https://github.com/apache/datafusion/issues/13813 Thanks to the infra team and @gmcdonald specifically, we now have the ability to use more powerful AWS-provided runners in our CI 🥳 DataFusion has one of the largest runtimes across Apache projects - that's why we're bringing those runners here first. Since we're first to test this, I think it's reasonable to do a gradual transition, so I updated the two most frequently failing actions to be hosted in AWS. The plan is to test that everything works fine and then transition the remaining actions. ## What changes are included in this PR? If the org is `apache`, we'll now use ASF-provisioned runners in the ASF infra AWS account. Forks will not have access to those runners, so they will fall back to GitHub-provisioned ones. ## Are these changes tested? Yes. - Forks in non-apache orgs: https://github.com/blaginin/datafusion/pull/10 - Apache org: https://github.com/apache/datafusion-sandbox/pull/163 ## Are there any user-facing changes? No |
||
|
karuppuchamysuresh
|
9de192af29 |
docs: update data_types.md to reflect current Arrow type mappings (#20072)
## Which issue does this PR close? - Closes #18314 ## Rationale for this change The documentation in `data_types.md` was outdated and showed `Utf8` as the default mapping for character types (CHAR, VARCHAR, TEXT, STRING), but the current implementation defaults to `Utf8View`. This caused confusion for users reading the documentation as it didn't match the actual behavior. Additionally, the "Supported Arrow Types" section at the end was redundant since `arrow_typeof` now supports all Arrow types, making the comprehensive list unnecessary. ## What changes are included in this PR? 1. **Updated Character Types table**: Changed the Arrow DataType column from `Utf8` to `Utf8View` for CHAR, VARCHAR, TEXT, and STRING types 2. **Added configuration note**: Documented the `datafusion.sql_parser.map_string_types_to_utf8view` setting that allows users to switch back to `Utf8` if needed 3. **Removed outdated section**: Deleted the "Supported Arrow Types" section (39 lines) as it's no longer necessary ## Are these changes tested? This is a documentation-only change. The documentation accurately reflects the current behavior of DataFusion: - The default mapping to `Utf8View` is the current implementation behavior - The `datafusion.sql_parser.map_string_types_to_utf8view` configuration option exists and works as documented ## Are there any user-facing changes? Yes, documentation changes only. Users will now see accurate information about: - The correct default Arrow type mappings for character types - How to configure the string type mapping behavior if they need the old `Utf8` behavior --------- Co-authored-by: Claude (claude-sonnet-4.5) <noreply@anthropic.com> |
||
|
Nuno Faria
|
92f60ad513 |
docs: Automatically update DataFusion version in docs (#20001)
## Which issue does this PR close? - N/A. ## Rationale for this change I was looking at the download page (https://datafusion.apache.org/download.html) when I noticed the version there was not updated in a while. Likewise, there are other places in the docs where the version is not updated. This PR changes the `update_datafusion_versions.py` script to automatically update those files, in addition to the Cargo files. ## What changes are included in this PR? - Updated `update_datafusion_versions.py` to check some doc files. - Updated the doc files to the current major version. ## Are these changes tested? Yes, script runs correctly. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> |
||
|
Kumar Ujjawal
|
36c0cda206 |
fix: respect DataFrameWriteOptions::with_single_file_output for paths without extensions (#19931)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #13323. ## Rationale for this change When using `DataFrameWriteOptions::with_single_file_output(true)`, the setting was being ignored if the output path didn't have a file extension. For example: ```rust df.write_parquet("/path/to/output", DataFrameWriteOptions::new().with_single_file_output(true), None).await?; ``` Would create a directory /path/to/output/ with files inside instead of a single file at /path/to/output. This happened because the demuxer used a heuristic based solely on file extension, ignoring the explicit user setting. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - New FileOutputMode enum: Uses explicit modes (Automatic, SingleFile, Directory) in FileSinkConfig for clearer output path handling. - The demuxer now uses the user's explicit setting instead of always relying on extension-based heuristics. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? - New unit test test_single_file_output_without_extension tests the fixed behavior - All sqllogictest pass <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? Breaking for direct FileSinkConfig construction: The struct now requires file_output_mode: FileOutputMode field. Use FileOutputMode::Automatic to preserve existing behavior. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Andrew Lamb
|
0a68b81ade |
[main] Update version to 52.1.0 (#19878) (#20028)
## Which issue does this PR close? - part of https://github.com/apache/datafusion/issues/19784 ## Rationale for this change Forward port changes from branch-52 to main ## What changes are included in this PR? Update release version to 52.1.0 and add changelo by cherry-picking - 9f3ddcecd6033a9d55161175d5dbe29697a9a922 ## Are these changes tested? By CI ## Are there any user-facing changes? New version |
||
|
Tim Saucer
|
41d48b3f6c |
feat: implement protobuf converter trait to allow control over serialization and deserialization processes (#19437)
## Which issue does this PR close? - Closes https://github.com/apache/datafusion/issues/18477 ## Rationale for this change This PR adds a new trait for converting to and from Protobuf objects and Physical expressions and plans. ## What changes are included in this PR? - Add `PhysicalExtensionProtoCodec` and default implementation. - Update all methods in the physical encoding/decoding methods to use this trait. - Added two examples - Added unit test ## Are these changes tested? Two examples and round trip unit test are added. ## Are there any user-facing changes? If users are going through the recommended interfaces in the documentation, `logical_plan_to_bytes` and `logical_plan_from_bytes` they will have no user facing change. If they are instead calling into the inner methods `PhysicalPlanNode::try_from_physical_plan` and so on, then they will need to provide a proto converter. A default implementation is provided. --------- Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> |
||
|
Nuno Faria
|
073174b034 |
feat: Show the number of matched Parquet pages in DataSourceExec (#19977)
## Which issue does this PR close? - Closes #19875. ## Rationale for this change Show the number of matched (and pruned) pages in the explain analyze plan to help make decisions about file optimization. Example: ```sql DataSourceExec: ..., metrics=[ ... page_index_rows_pruned=1.00 K total → 100 matched, page_index_pages_pruned=100 total → 10 matched, ... ] ``` ## What changes are included in this PR? - Added `page_index_pages_pruned` metric to DataSourceExec. - Updated and extended existing tests. ## Are these changes tested? Yes. ## Are there any user-facing changes? New metric in the explain plans. |
||
|
Nuno Faria
|
adddd4c32b |
fix: Make generate_series return an empty set with invalid ranges (#19999)
## Which issue does this PR close? - Closes #19998. ## Rationale for this change Make the `generate_series` table function follow the Postgres's convention. ## What changes are included in this PR? - Removed the error when the range is invalid. - Updated existing tests. ## Are these changes tested? Yes. ## Are there any user-facing changes? `generate_series`/`range` return an empty set with invalid ranges. |
||
|
cht42
|
8efc2b605e |
feat(spark): add base64 and unbase64 functions (#19968)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19967 - Part of #15914 ## Rationale for this change Add spark compatible base64/unbase64 functions ## What changes are included in this PR? - new encoding mode in DF encoding UDF for padded base64 - spark udfs for base64/unbase64 ## Are these changes tested? yes in SLT ## Are there any user-facing changes? yes |
||
|
Andrew Lamb
|
f8a22a51bf |
docs: Fix some broken / missing links in the DataFusion documentation (#19958)
## Which issue does this PR close? ## Rationale for this change While reading the DataFusion documentation, I noticed a broken link here: <img width="787" height="885" alt="Screenshot 2026-01-23 at 9 34 43 AM" src="https://github.com/user-attachments/assets/9013151d-599b-4973-968e-0033c6614643" /> ## What changes are included in this PR? Fix several broken and missing links (this was done by codex and I reviewed the PR) ## Are these changes tested? Yes, by CI ## Are there any user-facing changes? Better docs. No functional changes intended |
||
|
Nuno Faria
|
4d63f8c927 |
minor: Add favicon (#20000)
## Which issue does this PR close? - N/A. ## Rationale for this change There is no favicon currently set for the DataFusion site. For example, when searching on Google: <img width="228" height="139" alt="image" src="https://github.com/user-attachments/assets/c0d2cd43-26fb-4715-a10c-0e60b7853a0a" /> When a tab is opened: <img width="257" height="42" alt="image" src="https://github.com/user-attachments/assets/4deb74b1-f29f-4bb5-a18b-d634e8ac0b16" /> Adding the logo makes it look nicer. ## What changes are included in this PR? Added favicon. ## Are these changes tested? Manual checks. ## Are there any user-facing changes? No. |
||
|
Andrew Lamb
|
d1eea0755a |
Fix broken links in the documentation (#19964)
## Which issue does this PR close? - Similarly to https://github.com/apache/datafusion/pull/19958 ## Rationale for this change I found some broken links in our docs using codex ] <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? Update broken links (will comment inline) ## Are these changes tested? By CI ## Are there any user-facing changes? Better docs / fewer broken links --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> |
||
|
Nuno Faria
|
e062ae5053 |
minor: Fix doc about write_batch_size (#19979)
## Which issue does this PR close? - N/A. ## Rationale for this change In the docs, `write_batch_size` was defined as representing in bytes, while it actually represents number of rows. ## What changes are included in this PR? - Updates documentation about `write_batch_size`. ## Are these changes tested? Yes. ## Are there any user-facing changes? No. |
||
|
kosiew
|
0c82adec71 |
Fix struct casts to align fields by name (prevent positional mis-casts) (#19674)
## Which issue does this PR close? * Closes #17285. ## Rationale for this change DataFusion’s struct casting and some coercion paths were effectively positional: when two structs had the same field types but different field *orders*, casting could silently swap values. This is surprising to users and can lead to silent data corruption (e.g. `{b: 3, a: 4}::STRUCT(a INT, b INT)` yielding `{a: 3, b: 4}`). The goal of this PR is to make struct casting behavior match user expectations by matching fields by **name** (case-sensitive) and recursively applying the same logic to nested structs, while keeping a compatible fallback for structs with **no** shared field names. ## What changes are included in this PR? * **Name-based struct casting implementation** in `datafusion_common::nested_struct`: * Match struct fields by **name**, reorder to match target schema, recursively cast nested structs. * Fill **missing target fields** with null arrays. * Ignore **extra source fields**. * **Positional mapping fallback** when there is *no name overlap* **and** field counts match (avoids breaking `struct(1, 'x')::STRUCT(a INT, b VARCHAR)` style casts). * Improved handling for **NULL / all-null struct inputs** by producing a correctly typed null struct array. * Centralized validation via `validate_field_compatibility` and helper `fields_have_name_overlap`. * **Ensure struct casting paths use the name-based logic**: * `ScalarValue::cast_to_with_options`: route `Struct` casts through `nested_struct::cast_column`. * `ColumnarValue::cast_to`: for `Struct` targets, cast via `nested_struct::cast_column`; non-struct casts still use Arrow’s standard casting. * **Type coercion improvements for structs in binary operators / CASE**: * When two structs have at least one overlapping name, coerce **by name**. * Otherwise, preserve prior behavior by coercing **positionally**. * **Planning-time cast validation for struct-to-struct**: * `physical-expr` CAST planning now validates struct compatibility using the same rules as runtime (`validate_struct_compatibility`) to fail fast. * `ExprSchemable` allows struct-to-struct casts to pass type checking; detailed compatibility is enforced by the runtime / planning-time validator. * **Optimizer safety**: * Avoid const-folding struct casts when field counts differ. * Avoid const-folding casts of **0-row** struct literals due to evaluation batch dimension mismatches. * **Tests and SQL logic tests**: * New unit tests covering: * name-based reordering * missing fields (nullable vs non-nullable) * null struct fields and nested nulls * positional fallback with no overlap * coercion behavior and simplifier behavior * Updated/added `.slt` cases to reflect the new semantics and to add coverage for struct casts and nested struct reordering. * **Minor docs/maintenance**: * Adjusted doc comment referencing `ParquetWriterOptions` so it doesn’t break when the `parquet` feature is disabled. ## Are these changes tested? Yes. * Added/updated Rust unit tests in: * `datafusion/common/src/nested_struct.rs` * `datafusion/expr-common/src/columnar_value.rs` * `datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs` * Added/updated SQL logic tests in: * `datafusion/sqllogictest/test_files/case.slt` * `datafusion/sqllogictest/test_files/struct.slt` These tests cover: * correct value mapping when struct field order differs * nested struct reordering * insertion of nulls for missing nullable fields * erroring on missing non-nullable target fields * positional mapping fallback when there is no name overlap * planning-time validation vs runtime behavior alignment ## Are there any user-facing changes? Yes. * **Struct casts are now name-based** (case-sensitive): fields are matched by name, reordered to the target schema, missing fields are null-filled (if nullable), and extra fields are ignored. * **Fallback behavior**: if there is *no* name overlap and field counts match, casting proceeds **positionally**. * **Potential behavior change** in queries relying on the prior positional behavior when structs shared names but were out of order (previously could yield swapped values). This PR changes that to the safer, expected behavior. No public API changes are introduced, but this is a semantic change in struct casting. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested. |
||
|
Adrian Garcia Badaracco
|
936f959ddf |
Add FilterExecBuilder to avoid recomputing properties multiple times (#19854)
- Closes #19608, - replaces #19619 --------- Co-authored-by: Ganesh Patil <7030871503ganeshpatil@gmail.com> |
||
|
Jeffrey Vo
|
ac67ae4af5 |
Ensure null inputs to array setop functions return null output (#19683)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19682 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Explained in issue. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Change array_except, array_intersect and array_union UDFs to return null if either input is null. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Added & fixed tests. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Behaviour change to a function output. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Andrew Lamb
|
c91dcfdb6a |
Docs: add additional links to blog posts (#19833)
## Which issue does this PR close? - Part of #7013 ## Rationale for this change We have written some good blogs recently that provide additional context and backstory. Let's make sure they are available for others to read ## What changes are included in this PR? Add links to select doc pages ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
cht42
|
f3f6dec7c9 |
feat: Add support for 'isoyear' in date_part function (#19821)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19820. ## Rationale for this change `isoyear` part is available in both PG and Spark EXTRACT functions. https://www.postgresql.org/docs/current/functions-datetime.html#:~:text=the%20week%20numbering.-,isoyear,-The%20ISO%208601 https://github.com/apache/spark/blob/a03bedb6c1281c5263a42bfd20608d2ee005ab05/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L3360 ## What changes are included in this PR? Support for part `isoyear` in date_part function. ## Are these changes tested? yes in SLT ## Are there any user-facing changes? yes <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
xudong.w
|
ca904b3086 |
Row group limit pruning for row groups that entirely match predicates (#18868)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of https://github.com/apache/datafusion/issues/18860 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> See https://github.com/apache/datafusion/issues/18860#issuecomment-3563442093 ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> 1. How to decide if we can do limit pruning without messing up the sql semantics. 2. Add logic to decide if a row group is fully matched, all rows in the row group are matched the predicated. 3. Use the fully matched row groups to return limit rows. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 3. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> No, no new configs, or API change |
||
|
Miao
|
5edda9b309 |
fix: calculate total seconds from interval fields for extract(epoch) (#19807)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19799. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> The epoch function incorrectly used `date_part(array, DatePart::Second)` for intervals which only extracts the seconds component of the interval structure (0 for "15 minutes"), not the total seconds from all components. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Modified epoch() function in date_part.rs to properly handle interval types by extracting struct fields and calculating total seconds - Added regression tests in date_part.slt ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. Added tests verifying correct conversion for all interval types and precision levels. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Yes. `extract(epoch from interval)` now returns correct total seconds instead of 0. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Andrew Lamb
|
de40f0c9f5 |
Docs: Fix some links in docs (#19834)
Small doc tweaks |
||
|
Yongting You
|
567ba75840 |
doc: Add an auto-generated dependency graph for internal crates (#19280)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> A dependency graph for workspace member crates are often needed when doing refactors, I want it to be included in the doc, and have a script to update it automatically. Here is the preview: <img width="1203" height="951" alt="image" src="https://github.com/user-attachments/assets/527c18fc-258e-465f-a150-f2aafe3e6db9" /> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - adds a script to generate the dependency graph `deps.svg`, and verify if the existing one is up to date. - adds a documentation page in `Contributor Guide` to show this graph - adds a CI job to check if the generated dependency graph is up to date with the code. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> I tested the dependency graph display locally, see above. Is it possible to see the preview from this PR's change online? I also included a dummy crate in the initial commit, to test if the CI can catch it and throw understandable error message. ## Are there any user-facing changes? No <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
xudong.w
|
2fb9fb3fb9 |
Update the upgrading.md (#19769)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - part of 18566 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
xudong.w
|
383e673ac3 |
Update 52.0.0 release version number and changelog (#19767)
The 52.0.0 has released, the PR updates version number and changlog on main branch |
||
|
Kumar Ujjawal
|
d103d8886f |
chore: remove LZO Parquet compression (#19726)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19720. ## Rationale for this change - Choosing LZO compression errors, I think it might never get supported so the best option moving forward is to remove it algother and update the docs. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Removed LZO from parse_compression_string() function - Removed docs - Updated exptected test output <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? User choosing LZO as compression will get a clear error message: ``` Unknown or unsupported parquet compression: lzo. Valid values are: uncompressed, snappy, gzip(level), brotli(level), lz4, zstd(level), and lz4_raw. ``` <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Andrew Lamb
|
013efb4fe1 |
docs: Refine Communication documentation to highlight Discord (#19714)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of #7013 ## Rationale for this change I was speaking with someone who wanted to get more involved with DataFusion recently, and it was not clear from the website that discord is far more active than Slack. I think it would help new new users to make this clearer in the documentation to lower the barrier to joining the communitu ## What changes are included in this PR? Improve the communication documentation here: https://datafusion.apache.org/contributor-guide/communication.html and mention that discord is more active than slack <img width="1191" height="856" alt="Screenshot 2026-01-09 at 8 54 39 AM" src="https://github.com/user-attachments/assets/9bbef5d4-645d-45f0-85aa-6b12a2a7eea5" /> While I was reviewing the communication documentation, I noticed some other minor issues that could be improved which I will call out inline ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Adam Gutglick
|
8ba46466d3 |
docs: Fix two small issues in introduction.md (#19712)
This is a small maintanence PR to fix a couple of small issues: 1. The `datafusion-tui` isn't used anywhere, and as far as I can tell its now basically a part of `datafusion-dft` (which is already referenced). 2. The call for action wasn't rendered into the HTML page. I was there up until #4903, which included this small mistake which every subsequent change just maintained. |
||
|
UBarney
|
c98fa5616e |
perfect hash join (#19411)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #17635. ## Rationale for this change This PR introduces a Perfect Hash Join optimization by using an array-based direct mapping(`ArrayMap`) instead of a HashMap. The array-based approach outperforms the standard Hash Join when the build-side keys are **_dense_** (i.e., the ratio of `count / (max - min+1)` is high) or when the key range `(max - min)` is sufficiently **small**. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> The following results from the hj.rs benchmark suite. The benchmark was executed with the optimization enabled by setting `DATAFUSION_EXECUTION_PERFECT_HASH_JOIN_MIN_KEY_DENSITY=0.1` ``` ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Query ┃ base_hj ┃ density=0.1 ┃ Change ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ QQuery 1_density=1_prob_hit=1_25*1.5M │ 5.50 ms │ 4.54 ms │ +1.21x faster │ │ QQuery 2_density=0.026_prob_hit=1_25*1.5M │ 6.13 ms │ 5.43 ms │ +1.13x faster │ │ QQuery 3_density=1_prob_hit=1_100K*60M │ 132.59 ms │ 97.42 ms │ +1.36x faster │ │ QQuery 4_density=1_prob_hit=0.1_100K*60M │ 146.66 ms │ 97.75 ms │ +1.50x faster │ │ QQuery 5_density=0.75_prob_hit=1_100K*60M │ 139.85 ms │ 103.82 ms │ +1.35x faster │ │ QQuery 6_density=0.75_prob_hit=0.1_100K*60M │ 256.62 ms │ 192.15 ms │ +1.34x faster │ │ QQuery 7_density=0.5_prob_hit=1_100K*60M │ 136.27 ms │ 91.64 ms │ +1.49x faster │ │ QQuery 8_density=0.5_prob_hit=0.1_100K*60M │ 234.89 ms │ 185.35 ms │ +1.27x faster │ │ QQuery 9_density=0.2_prob_hit=1_100K*60M │ 132.76 ms │ 98.44 ms │ +1.35x faster │ │ QQuery 10_density=0.2_prob_hit=0.1_100K*60M │ 240.04 ms │ 184.93 ms │ +1.30x faster │ │ QQuery 11_density=0.1_prob_hit=1_100K*60M │ 133.02 ms │ 108.11 ms │ +1.23x faster │ │ QQuery 12_density=0.1_prob_hit=0.1_100K*60M │ 235.44 ms │ 209.10 ms │ +1.13x faster │ │ QQuery 13_density=0.01_prob_hit=1_100K*60M │ 135.64 ms │ 132.52 ms │ no change │ │ QQuery 14_density=0.01_prob_hit=0.1_100K*60M │ 235.88 ms │ 234.62 ms │ no change │ │ QQuery 15_density=0.2_prob_hit=0.1_100K_(20%_dups)*60M │ 178.49 ms │ 147.55 ms │ +1.21x faster │ └────────────────────────────────────────────────────────┴───────────┴─────────────┴───────────────┘ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Benchmark Summary ┃ ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ Total Time (base_hj) │ 2349.79ms │ │ Total Time (density=0.1) │ 1893.37ms │ │ Average Time (base_hj) │ 156.65ms │ │ Average Time (density=0.1) │ 126.22ms │ │ Queries Faster │ 13 │ │ Queries Slower │ 0 │ │ Queries with No Change │ 2 │ │ Queries with Failure │ 0 │ └────────────────────────────┴───────────┘ ``` The following results from tpch-sf10 ``` ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Query ┃ base ┃ perfect_hj ┃ Change ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ QQuery 1 │ 739.66 ms │ 743.84 ms │ no change │ │ QQuery 2 │ 315.94 ms │ 317.53 ms │ no change │ │ QQuery 3 │ 655.79 ms │ 669.24 ms │ no change │ │ QQuery 4 │ 215.48 ms │ 218.79 ms │ no change │ │ QQuery 5 │ 1131.42 ms │ 1146.03 ms │ no change │ │ QQuery 6 │ 202.32 ms │ 190.83 ms │ +1.06x faster │ │ QQuery 7 │ 1734.06 ms │ 1710.50 ms │ no change │ │ QQuery 8 │ 1185.05 ms │ 1173.90 ms │ no change │ │ QQuery 9 │ 2036.76 ms │ 1994.30 ms │ no change │ │ QQuery 10 │ 907.32 ms │ 893.20 ms │ no change │ │ QQuery 11 │ 306.63 ms │ 275.46 ms │ +1.11x faster │ │ QQuery 12 │ 404.00 ms │ 381.95 ms │ +1.06x faster │ │ QQuery 13 │ 531.67 ms │ 498.58 ms │ +1.07x faster │ │ QQuery 14 │ 317.63 ms │ 303.04 ms │ no change │ │ QQuery 15 │ 602.24 ms │ 572.18 ms │ no change │ │ QQuery 16 │ 200.00 ms │ 201.68 ms │ no change │ │ QQuery 17 │ 1848.67 ms │ 1790.60 ms │ no change │ │ QQuery 18 │ 2130.63 ms │ 2179.84 ms │ no change │ │ QQuery 19 │ 501.81 ms │ 529.85 ms │ 1.06x slower │ │ QQuery 20 │ 637.91 ms │ 661.72 ms │ no change │ │ QQuery 21 │ 1882.43 ms │ 1917.10 ms │ no change │ │ QQuery 22 │ 130.68 ms │ 141.76 ms │ 1.08x slower │ └──────────────┴────────────┴────────────┴───────────────┘ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓ ┃ Benchmark Summary ┃ ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩ │ Total Time (base) │ 18618.10ms │ │ Total Time (perfect_hj) │ 18511.93ms │ │ Average Time (base) │ 846.28ms │ │ Average Time (perfect_hj) │ 841.45ms │ │ Queries Faster │ 4 │ │ Queries Slower │ 2 │ │ Queries with No Change │ 16 │ │ Queries with Failure │ 0 │ └───────────────────────────┴────────────┘ ``` ## What changes are included in this PR? - During the `collect_left_input` (build) phase, we now conditionally use an `ArrayMap` instead of a standard `JoinHashMapType`. This optimization is triggered only when the following conditions are met: - There is exactly one join key. - The join key can be any integer type convertible to u64 (excluding i128 and u128). - The key distribution is sufficiently dense or the key range (max - min) is small enough to justify an array-based allocation. - build_side.num_rows() < `u32::MAX` - The `ArrayMap` works by storing the minimum key as an offset and using a Vec to directly map a key `k` to its build-side index via `data[k- offset]`. - Rewrite Hash Join micro-benchmarks in benchmarks/src/hj.rs to evaluate ArrayMap and HashMap performance across varying key densities and probe hit rates <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? Yes, this PR introduces two new session configuration parameters to control the behavior of the Perfect Hash Join optimization: - `perfect_hash_join_small_build_threshold`: This parameter defines the maximum key range (max_key - min_key) for the build side to be considered "small." If the key range is below this threshold, the array-based join will be triggered regardless of key density. - `perfect_hash_join_min_key_density`: This parameter sets the minimum density (row_count / key_range) required to enable the perfect hash join optimization for larger key ranges <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
jizezhang
|
e6049de5a7 |
Make default ListingFilesCache table scoped (#19616)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Builds on https://github.com/apache/datafusion/pull/19388 - Closes https://github.com/apache/datafusion/issues/19573 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> This PR explores one way to make `ListFilesCache` table scoped. A session level cache is still used, but the cache key is made a "table-scoped" path, for which a new struct ``` pub struct TableScopedPath(pub Option<TableReference>, pub Path); ``` is defined. `TableReference` comes from `CreateExternalTable` passed to `ListingTableFactory::create`. Additionally, when a table is dropped, all entries related to a table is dropped by modifying `SessionContext::find_and_deregister` method. Testing (change on adding `list_files_cache()` for cli is included for easier testing). - Testing cache reuse on a single table. ``` > \object_store_profiling summary ObjectStore Profile mode set to Summary > create external table test stored as parquet location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/'; 0 row(s) fetched. Elapsed 14.878 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Summary, inner: AmazonS3(overturemaps-us-west-2) Summaries: +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Operation | Metric | min | max | avg | sum | count | +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Get | duration | 0.030597s | 0.209235s | 0.082396s | 36.254189s | 440 | | Get | size | 204782 B | 857230 B | 497304.88 B | 218814144 B | 440 | | List | duration | 0.192037s | 0.192037s | 0.192037s | 0.192037s | 1 | | List | size | | | | | 1 | +-----------+----------+-----------+-----------+-------------+-------------+-------+ > select table, path, unnest(metadata_list) from list_files_cache() limit 1; +-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | table | path | UNNEST(list_files_cache().metadata_list) | +-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test | release/2025-12-17.0/theme=base | {file_path: release/2025-12-17.0/theme=base/type=bathymetry/part-00000-dd0f2f50-b436-4710-996f-f1b06181a3a1-c000.zstd.parquet, file_modified: 2025-12-17T22:32:50, file_size_bytes: 40280159, e_tag: "15090401f8f936c3f83bb498cb99a41d-3", version: NULL} | +-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row(s) fetched. Elapsed 0.058 seconds. Object Store Profiling > select count(*) from test where type = 'infrastructure'; +-----------+ | count(*) | +-----------+ | 142969564 | +-----------+ 1 row(s) fetched. Elapsed 0.028 seconds. Object Store Profiling ``` - Test separate cache entries are created for two tables with same path ``` > create external table test2 stored as parquet location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/'; 0 row(s) fetched. Elapsed 14.798 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Summary, inner: AmazonS3(overturemaps-us-west-2) Summaries: +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Operation | Metric | min | max | avg | sum | count | +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Get | duration | 0.030238s | 0.350465s | 0.073670s | 32.414654s | 440 | | Get | size | 204782 B | 857230 B | 497304.88 B | 218814144 B | 440 | | List | duration | 0.133334s | 0.133334s | 0.133334s | 0.133334s | 1 | | List | size | | | | | 1 | +-----------+----------+-----------+-----------+-------------+-------------+-------+ > select count(*) from test2 where type = 'bathymetry'; +----------+ | count(*) | +----------+ | 59963 | +----------+ 1 row(s) fetched. Elapsed 0.009 seconds. Object Store Profiling > select table, path from list_files_cache(); +-------+---------------------------------+ | table | path | +-------+---------------------------------+ | test | release/2025-12-17.0/theme=base | | test2 | release/2025-12-17.0/theme=base | +-------+---------------------------------+ 2 row(s) fetched. Elapsed 0.004 seconds. ``` - Test cache associated with a table is dropped when table is dropped, and the other table with same path is unaffected. ``` > drop table test; 0 row(s) fetched. Elapsed 0.015 seconds. Object Store Profiling > select table, path from list_files_cache(); +-------+---------------------------------+ | table | path | +-------+---------------------------------+ | test2 | release/2025-12-17.0/theme=base | +-------+---------------------------------+ 1 row(s) fetched. Elapsed 0.005 seconds. Object Store Profiling > select count(*) from list_files_cache() where table = 'test'; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row(s) fetched. Elapsed 0.014 seconds. > select count(*) from test2 where type = 'infrastructure'; +-----------+ | count(*) | +-----------+ | 142969564 | +-----------+ 1 row(s) fetched. Elapsed 0.013 seconds. Object Store Profiling ``` - Test that dropping a view does not remove cache ``` > create view test2_view as (select * from test2 where type = 'infrastructure'); 0 row(s) fetched. Elapsed 0.103 seconds. Object Store Profiling > select count(*) from test2_view; +-----------+ | count(*) | +-----------+ | 142969564 | +-----------+ 1 row(s) fetched. Elapsed 0.094 seconds. Object Store Profiling > drop view test2_view; 0 row(s) fetched. Elapsed 0.002 seconds. Object Store Profiling > select table, path from list_files_cache(); +-------+---------------------------------+ | table | path | +-------+---------------------------------+ | test2 | release/2025-12-17.0/theme=base | +-------+---------------------------------+ 1 row(s) fetched. Elapsed 0.007 seconds. ``` ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Kumar Ujjawal
|
646213ec7c |
feat: add Time type support to date_trunc function (#19640)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of #19025. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Added Time64/Time32 signatures to date_trunc - Added time truncation logic (hour, minute, second, millisecond, microsecond) - Error for invalid granularities (day, week, month, quarter, year) <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Adrian Garcia Badaracco
|
35ff4ab0a0 |
Allow logical optimizer to be run without evaluating now() & refactor SimplifyInfo (#19505)
In trying to fix #19418 I kept getting turned around about what was needed where. The `SimplifyInfo` trait made it extra hard to understand. I ended up realizing that the main reason for the trait to exist was tests. Removing the trait and adding a builder style API to `SimplifyContext` made it IMO more ergonomic for tests and other call sites, easier to track the code (no trait opaqueness) and clearer what simplification capabilities are available in each site. This got rid of e.g. some places where we were calling `ExecutionProps::new()` just to pass that into `SimplifyContext` which in turn would hand out references to the default query time, a default `ContigOptions`, etc; or in `datafusion/core/src/execution/session_state.rs` where we did `let dummy_schema = DFSchema::empty()`. This let me solve several problems: - Can store optimized logical plans for prepared statements - Users can optionally run an optimizer pass on logical plans without evaluating time functions Compared to #19426 this avoids adding a config option and is actually less lines of code (negative diff). Fixes #19418, closes #19426 (replaces it). --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
jizezhang
|
1037f0aa20 |
feat: add list_files_cache table function for datafusion-cli (#19388)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes https://github.com/apache/datafusion/issues/19055. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ``` > CREATE EXTERNAL TABLE nyc_taxi_rides STORED AS PARQUET LOCATION 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/' ; 0 row(s) fetched. Elapsed 10.061 seconds. > SELECT metadata_size_bytes, expires_in, unnest(metadata_list) FROM list_files_cache(); +---------------------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | metadata_size_bytes | expires_in | UNNEST(list_files_cache().metadata_list) | +---------------------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200901.parquet, file_modified: 2025-05-30T09:44:23, file_size_bytes: 222192983, e_tag: "e8d016c3c7af80bf911d96387febe2c1-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200902.parquet, file_modified: 2025-05-30T09:46:00, file_size_bytes: 211023080, e_tag: "1021626ff5ef606422aa7121edd69f3b-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200903.parquet, file_modified: 2025-05-30T09:47:20, file_size_bytes: 229202874, e_tag: "96e7494b217099c6a07e9c4298cbe783-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200904.parquet, file_modified: 2025-05-30T09:44:37, file_size_bytes: 225659965, e_tag: "728c45fabdcd8e40bdef4dfc28df9b0f-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200905.parquet, file_modified: 2025-05-30T09:46:12, file_size_bytes: 232847306, e_tag: "f59e45bd8bd1d77cd7ae8ab6ab468bcc-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200906.parquet, file_modified: 2025-05-30T09:47:26, file_size_bytes: 224226575, e_tag: "8ebb698eea85f9af87065ac333efc449-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200907.parquet, file_modified: 2025-05-30T09:44:52, file_size_bytes: 217168413, e_tag: "7d7ee77f6cac4adc18aa3a9e74600dd3-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200908.parquet, file_modified: 2025-05-30T09:46:23, file_size_bytes: 217303109, e_tag: "e9883055d92a33b941aab971423e681b-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200909.parquet, file_modified: 2025-05-30T09:47:28, file_size_bytes: 223333499, e_tag: "6f0917e6003b38df9060d71c004eb961-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200910.parquet, file_modified: 2025-05-30T09:44:54, file_size_bytes: 246300471, e_tag: "8928b29da44e041021e10077683b7817-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200911.parquet, file_modified: 2025-05-30T09:46:37, file_size_bytes: 227920860, e_tag: "4cd26a1a7f82af080c33e890dc1fef27-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-200912.parquet, file_modified: 2025-05-30T09:44:24, file_size_bytes: 233873308, e_tag: "23f4584e494e3c065c777c270c9eedbc-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201001.parquet, file_modified: 2025-05-30T09:45:18, file_size_bytes: 235166925, e_tag: "effcc8cc41b40cf7ac466f911d7b9459-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201002.parquet, file_modified: 2025-05-30T09:46:59, file_size_bytes: 177367931, e_tag: "ce8b7817ecc47da86ccbfa6b51ffa06b-10", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201003.parquet, file_modified: 2025-05-30T09:44:26, file_size_bytes: 205857224, e_tag: "94a078b61e3b652387e6f2a673dc3f4e-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201004.parquet, file_modified: 2025-05-30T09:45:04, file_size_bytes: 243024246, e_tag: "a1efbebfdabc204e0041d8714aaec01a-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201005.parquet, file_modified: 2025-05-30T09:46:47, file_size_bytes: 248130090, e_tag: "d3cf585e00ce627a807348c84a42d0a6-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201006.parquet, file_modified: 2025-05-30T09:44:25, file_size_bytes: 237068130, e_tag: "831db33281a5c017f8ffc466bd47546b-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201007.parquet, file_modified: 2025-05-30T09:45:35, file_size_bytes: 234826090, e_tag: "790e05983e6592e4920c88fbd2bfe774-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201008.parquet, file_modified: 2025-05-30T09:47:14, file_size_bytes: 197990272, e_tag: "d87ddb446e5cbc0f6831fafd95cfd027-11", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201009.parquet, file_modified: 2025-05-30T09:44:27, file_size_bytes: 243408943, e_tag: "abfbe3b29942bcd68d131d95540278d3-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201010.parquet, file_modified: 2025-05-30T09:45:47, file_size_bytes: 225277041, e_tag: "f768c7b77497b2bf3efd5cb2a4362977-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201011.parquet, file_modified: 2025-05-30T09:47:23, file_size_bytes: 220010577, e_tag: "c6830cbe1f3ae918f9280db3aa847b03-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201012.parquet, file_modified: 2025-05-30T09:44:24, file_size_bytes: 219773352, e_tag: "264f7ea433076690a3bbe5566168e5c5-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201101.parquet, file_modified: 2025-05-30T09:45:52, file_size_bytes: 212535107, e_tag: "ca3bdc2707b29667c78c39517781eac4-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201102.parquet, file_modified: 2025-05-30T09:47:23, file_size_bytes: 223138164, e_tag: "e2b3c0fd0c0d66ac6363600de0c8b2ad-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201103.parquet, file_modified: 2025-05-30T09:44:26, file_size_bytes: 252843261, e_tag: "fd5d4e01568cd6e7ef1e00de76441e5b-15", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201104.parquet, file_modified: 2025-05-30T09:46:10, file_size_bytes: 233123935, e_tag: "2b510cc2c0c73d9ec7374c9e6d56c388-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201105.parquet, file_modified: 2025-05-30T09:44:24, file_size_bytes: 246843111, e_tag: "abc2f58bd520b2013aa1a333d317c70c-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201106.parquet, file_modified: 2025-05-30T09:44:58, file_size_bytes: 238786647, e_tag: "0e456698dc42a850ff7b764506cb511d-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201107.parquet, file_modified: 2025-05-30T09:46:40, file_size_bytes: 233249259, e_tag: "28177227cbff94a6a819a0568a14e9b2-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201108.parquet, file_modified: 2025-05-30T09:44:25, file_size_bytes: 212681184, e_tag: "fdcb442e1010630c0553a7018762a8ba-12", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201109.parquet, file_modified: 2025-05-30T09:45:13, file_size_bytes: 232399266, e_tag: "ccca37be5a3579a8bc644490226ed29a-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201110.parquet, file_modified: 2025-05-30T09:46:52, file_size_bytes: 248471033, e_tag: "eebe34c1bb74f63433eb607810969553-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201111.parquet, file_modified: 2025-05-30T09:44:26, file_size_bytes: 231103826, e_tag: "7c76b9fc111462b76336d63bce3253c7-13", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201112.parquet, file_modified: 2025-05-30T09:45:40, file_size_bytes: 236102882, e_tag: "26c10d1d85c4565cbb9e8fc6a7bc745c-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201201.parquet, file_modified: 2025-05-30T09:47:21, file_size_bytes: 236184052, e_tag: "8cdc15a22462579dcf90d669cea0f04b-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201202.parquet, file_modified: 2025-05-30T09:44:27, file_size_bytes: 238377570, e_tag: "4e6734c5c2e77c68dde5155a45dac81c-14", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201203.parquet, file_modified: 2025-05-30T09:46:06, file_size_bytes: 258226172, e_tag: "b7b07fa0f4fefcf0ba0fc69ba344b5c8-15", version: NULL} | | 18138 | NULL | {file_path: nyc_taxi_rides/data/tripdata_parquet/data-201204.parquet, file_modified: 2025-05-30T09:44:25, file_size_bytes: 248190698, e_tag: "968c13850fa9a7cb46337bc8fc9d13fa-14", version: NULL} | | . | | . | | . | +---------------------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 96 row(s) fetched. (First 40 displayed. Use --maxrows to adjust) Elapsed 0.057 seconds. ``` ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> This will enable a new user-facing table function to datafusion cli. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
mag1c1an1
|
9b2505ce6a |
fix(doc): close #19393, make upgrading guide match v51 api (#19648)
Change-Id: Id62d32d14fa19b34b592e186e7962bb96a6a6964 ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19393 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Make datafusion doc great ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Move “Refactoring of `FileSource` constructors and `FileScanConfigBuilder` to accept schemas upfront" section to right place. ## Are these changes tested? No need for code test. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> Signed-off-by: mag1cian <mag1cian@icloud.com> |
||
|
Bruce Ritchie
|
ada0923a39 |
Respect execution timezone in to_timestamp and related functions (#19078)
## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/17998. Continuation of PR #18025 by @kosiew ## Rationale for this change Previously, the to_timestamp() family of functions (to_timestamp, to_timestamp_seconds, to_timestamp_millis, to_timestamp_micros, to_timestamp_nanos) always interpreted timezone-free (naïve) timestamps as UTC, ignoring the datafusion.execution.time_zone configuration option. This behavior caused inconsistencies when users configured a specific execution timezone and expected timestamp conversions to respect it. This PR introduces full timezone awareness to these functions so that: - Naïve timestamp strings are interpreted as being in the configured execution timezone, or UTC if the configured execution timezone is `None`. - All returned timestamps are in the execution timezone. ## What changes are included in this PR? Code, tests. ## Are these changes tested? Yes, via code tests and slt tests. ## Are there any user-facing changes? Yes: - to_timestamp() and its precision variants now respect datafusion.execution.time_zone when parsing timezone-free timestamps and return timestamps in the execution time zone.. These changes make timestamp functions consistent with session timezone semantics and improve correctness for global workloads. --------- Co-authored-by: Siew Kam Onn <kosiew@gmail.com> |
||
|
Clement de Groc
|
1320069246 |
Fix typo in contributor guide architecture section (#19613)
Fix a typo in the contributor guide architecture section. |
||
|
Kumar Ujjawal
|
818706ab78 |
feat: to_time function (#19540)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of #19025. ## Rationale for this change Previously we needed to create timestamp and then extract the time component from that. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Added the function for `to_time` - Relevant slt test - Relevant unit tests - Updated docs <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? - All tests pass <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |