mirror of
https://github.com/langchain-ai/datafusion.git
synced 2026-08-27 02:21:16 -04:00
main
335 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Neil Conway
|
d6fb3608b0 |
perf: Optimize array_position for scalar needle (#20532)
## Which issue does this PR close? - Closes #20530 ## Rationale for this change The previous implementation of `array_position` used `compare_element_to_list` for every input row. When the needle is a scalar (quite common), we can do much better by searching over the entire flat haystack values array with a single call to `arrow_ord::cmp::not_distinct`. We can then iterate over the resulting set bits to determine per-row results. This is ~5-10x faster than the previous implementation for typical inputs. ## What changes are included in this PR? * Implement new fast path for `array_position` with scalar needle * Improve docs for `array_position` * Don't use `internal_err` to report a user-visible error ## Are these changes tested? Yes, and benchmarked. Additional tests added in a separate PR (#20531) ## Are there any user-facing changes? No. |
||
|
Neil Conway
|
585bbf35d3 |
perf: Optimize array_has_any() with scalar arg (#20385)
## Which issue does this PR close? - Closes #20384. - See #18181 for related context. ## Rationale for this change When `array_has_any` is passed a scalar for either of its arguments, we can use a much faster algorithm: rather than doing O(N*M) comparisons for each row of the columnar arg, we can build a hash table on the scalar argument and probe it instead. ## What changes are included in this PR? * Add benchmark to cover the one-scalar-arg case * Implement optimization as described above Note that we fallback to a linear scan when the scalar arg is smaller than a threshold (<= 8 elements), because benchmarks suggested probing a HashSet is not profitable for very small arrays. ## Are these changes tested? Yes. Tests pass and benchmarked. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Oleks V
|
ed0323a2bb |
feat: support arrays_zip function (#20440)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change Summary - Adds a new arrays_zip scalar function that combines multiple arrays into a single array of structs, where each struct field corresponds to an input array - Shorter arrays within a row are padded with NULLs to match the longest array's length - Compatible with Spark's arrays_zip behavior <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? ``` arrays_zip takes N list arrays and produces a List<Struct<c0, c1, ..., cN>> where each struct contains the elements at the same index from each input array. > SELECT arrays_zip([1, 2, 3], ['a', 'b', 'c']); [{c0: 1, c1: a}, {c0: 2, c1: b}, {c0: 3, c1: c}] > SELECT arrays_zip([1, 2], [3, 4, 5]); [{c0: 1, c1: 3}, {c0: 2, c1: 4}, {c0: NULL, c1: 5}] Implementation details: - Implemented in set_ops.rs following existing array function patterns - Uses MutableArrayData builders per column with row-by-row processing for efficient memory handling - For each row, computes the max array length, copies values from each input array, and pads shorter arrays with NULLs - Supports variadic arguments (2 or more arrays) - Handles NULL list entries, NULL elements, empty arrays, mixed types, and Null-typed arguments - Registered as arrays_zip with alias list_zip - Uses Signature::variadic_any with Volatility::Immutable ``` <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Neil Conway
|
ace9cd44b7 |
perf: Optimize trim UDFs for single-character trims (#20328)
## Which issue does this PR close? - Closes #20327 ## Rationale for this change By default, btrim(), ltrim(), and rtrim() trim space characters; it is also reasonably common for queries to specify a non-default trim pattern that is still a single ASCII character. We can optimize for this case by doing a byte-level scan, rather than invoking the more heavyweight std::string machinery used for more complex trim scenarios. ## What changes are included in this PR? Add a benchmark for trimming spaces, and implement the optimization described above. Also fixed an error in the documentation. ## Are these changes tested? Yes, and benchmarked. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> |
||
|
Neil Conway
|
b7f6090874 |
perf: Optimize lpad, rpad for ASCII strings (#20278)
The previous implementation incurred the overhead of Unicode machinery,
even for the common case that both the input string and the fill string
consistent only of ASCII characters. For the ASCII-only case, we can
assume that the length in bytes equals the length in characters, and
avoid expensive graphene-based segmentation. This follows similar
optimizations applied elsewhere in the codebase.
Benchmarks indicate this is a significant performance win for ASCII-only
input (4x-10x faster) but only a mild regression for Unicode input (2-5%
slower).
Along the way:
* Combine: a few instances of `write_str(str)? + append_value("")` with
`append_value(str)`, which saves a few cycles
* Add a missing test case for truncating the input string
* Add benchmarks for Unicode input
## Which issue does this PR close?
- Closes #20277.
## Are these changes tested?
Covered by existing tests. Added new benchmarks for Unicode inputs.
## Are there any user-facing changes?
No.
---------
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
|
||
|
Neil Conway
|
c3f080774c |
perf: Optimize translate() UDF for scalar inputs (#20305)
## Which issue does this PR close? - Closes #20302. ## Rationale for this change `translate()` is commonly invoked with constant values for its second and third arguments. We can take advantage of that to significantly optimize its performance by precomputing the translation lookup table, rather than recomputing it for every row. For ASCII-only inputs, we can further replace the hashmap lookup table with a fixed-size array that maps ASCII byte values directly. For scalar ASCII inputs, this yields roughly a 10x performance improvement. For scalar UTF8 inputs, the performance improvement is more like 50%, although less so for long strings. Along the way, add support for `translate()` on `LargeUtf8` input, along with an SLT test, and improve the docs. ## What changes are included in this PR? * Add a benchmark for scalar/constant input to translate * Add a missing test case * Improve translate() docs * Support translate() on LargeUtf8 input * Optimize translate() for scalar inputs by precomputing lookup hashmap * Optimize translate() for ASCII inputs by precomputing ASCII byte-wise lookup table ## Are these changes tested? Yes. Added an extra test case and did a bunch of benchmarking. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Neil Conway
|
75428f1e0a |
fix: Avoid integer overflow in split_part() (#20198)
Along the way, improve the docs slightly.
## Rationale for this change
Evaluating `SELECT SPLIT_PART('', '', -9223372036854775808);` yields (in
a debug build):
```
thread 'main' (41405991) panicked at datafusion/functions/src/string/split_part.rs:236:47:
attempt to negate with overflow
```
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## Are these changes tested?
Yes, added unit test.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
|
||
|
karuppuchamysuresh
|
9de192af29 |
docs: update data_types.md to reflect current Arrow type mappings (#20072)
## Which issue does this PR close? - Closes #18314 ## Rationale for this change The documentation in `data_types.md` was outdated and showed `Utf8` as the default mapping for character types (CHAR, VARCHAR, TEXT, STRING), but the current implementation defaults to `Utf8View`. This caused confusion for users reading the documentation as it didn't match the actual behavior. Additionally, the "Supported Arrow Types" section at the end was redundant since `arrow_typeof` now supports all Arrow types, making the comprehensive list unnecessary. ## What changes are included in this PR? 1. **Updated Character Types table**: Changed the Arrow DataType column from `Utf8` to `Utf8View` for CHAR, VARCHAR, TEXT, and STRING types 2. **Added configuration note**: Documented the `datafusion.sql_parser.map_string_types_to_utf8view` setting that allows users to switch back to `Utf8` if needed 3. **Removed outdated section**: Deleted the "Supported Arrow Types" section (39 lines) as it's no longer necessary ## Are these changes tested? This is a documentation-only change. The documentation accurately reflects the current behavior of DataFusion: - The default mapping to `Utf8View` is the current implementation behavior - The `datafusion.sql_parser.map_string_types_to_utf8view` configuration option exists and works as documented ## Are there any user-facing changes? Yes, documentation changes only. Users will now see accurate information about: - The correct default Arrow type mappings for character types - How to configure the string type mapping behavior if they need the old `Utf8` behavior --------- Co-authored-by: Claude (claude-sonnet-4.5) <noreply@anthropic.com> |
||
|
cht42
|
8efc2b605e |
feat(spark): add base64 and unbase64 functions (#19968)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19967 - Part of #15914 ## Rationale for this change Add spark compatible base64/unbase64 functions ## What changes are included in this PR? - new encoding mode in DF encoding UDF for padded base64 - spark udfs for base64/unbase64 ## Are these changes tested? yes in SLT ## Are there any user-facing changes? yes |
||
|
Andrew Lamb
|
f8a22a51bf |
docs: Fix some broken / missing links in the DataFusion documentation (#19958)
## Which issue does this PR close? ## Rationale for this change While reading the DataFusion documentation, I noticed a broken link here: <img width="787" height="885" alt="Screenshot 2026-01-23 at 9 34 43 AM" src="https://github.com/user-attachments/assets/9013151d-599b-4973-968e-0033c6614643" /> ## What changes are included in this PR? Fix several broken and missing links (this was done by codex and I reviewed the PR) ## Are these changes tested? Yes, by CI ## Are there any user-facing changes? Better docs. No functional changes intended |
||
|
Nuno Faria
|
e062ae5053 |
minor: Fix doc about write_batch_size (#19979)
## Which issue does this PR close? - N/A. ## Rationale for this change In the docs, `write_batch_size` was defined as representing in bytes, while it actually represents number of rows. ## What changes are included in this PR? - Updates documentation about `write_batch_size`. ## Are these changes tested? Yes. ## Are there any user-facing changes? No. |
||
|
kosiew
|
0c82adec71 |
Fix struct casts to align fields by name (prevent positional mis-casts) (#19674)
## Which issue does this PR close? * Closes #17285. ## Rationale for this change DataFusion’s struct casting and some coercion paths were effectively positional: when two structs had the same field types but different field *orders*, casting could silently swap values. This is surprising to users and can lead to silent data corruption (e.g. `{b: 3, a: 4}::STRUCT(a INT, b INT)` yielding `{a: 3, b: 4}`). The goal of this PR is to make struct casting behavior match user expectations by matching fields by **name** (case-sensitive) and recursively applying the same logic to nested structs, while keeping a compatible fallback for structs with **no** shared field names. ## What changes are included in this PR? * **Name-based struct casting implementation** in `datafusion_common::nested_struct`: * Match struct fields by **name**, reorder to match target schema, recursively cast nested structs. * Fill **missing target fields** with null arrays. * Ignore **extra source fields**. * **Positional mapping fallback** when there is *no name overlap* **and** field counts match (avoids breaking `struct(1, 'x')::STRUCT(a INT, b VARCHAR)` style casts). * Improved handling for **NULL / all-null struct inputs** by producing a correctly typed null struct array. * Centralized validation via `validate_field_compatibility` and helper `fields_have_name_overlap`. * **Ensure struct casting paths use the name-based logic**: * `ScalarValue::cast_to_with_options`: route `Struct` casts through `nested_struct::cast_column`. * `ColumnarValue::cast_to`: for `Struct` targets, cast via `nested_struct::cast_column`; non-struct casts still use Arrow’s standard casting. * **Type coercion improvements for structs in binary operators / CASE**: * When two structs have at least one overlapping name, coerce **by name**. * Otherwise, preserve prior behavior by coercing **positionally**. * **Planning-time cast validation for struct-to-struct**: * `physical-expr` CAST planning now validates struct compatibility using the same rules as runtime (`validate_struct_compatibility`) to fail fast. * `ExprSchemable` allows struct-to-struct casts to pass type checking; detailed compatibility is enforced by the runtime / planning-time validator. * **Optimizer safety**: * Avoid const-folding struct casts when field counts differ. * Avoid const-folding casts of **0-row** struct literals due to evaluation batch dimension mismatches. * **Tests and SQL logic tests**: * New unit tests covering: * name-based reordering * missing fields (nullable vs non-nullable) * null struct fields and nested nulls * positional fallback with no overlap * coercion behavior and simplifier behavior * Updated/added `.slt` cases to reflect the new semantics and to add coverage for struct casts and nested struct reordering. * **Minor docs/maintenance**: * Adjusted doc comment referencing `ParquetWriterOptions` so it doesn’t break when the `parquet` feature is disabled. ## Are these changes tested? Yes. * Added/updated Rust unit tests in: * `datafusion/common/src/nested_struct.rs` * `datafusion/expr-common/src/columnar_value.rs` * `datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs` * Added/updated SQL logic tests in: * `datafusion/sqllogictest/test_files/case.slt` * `datafusion/sqllogictest/test_files/struct.slt` These tests cover: * correct value mapping when struct field order differs * nested struct reordering * insertion of nulls for missing nullable fields * erroring on missing non-nullable target fields * positional mapping fallback when there is no name overlap * planning-time validation vs runtime behavior alignment ## Are there any user-facing changes? Yes. * **Struct casts are now name-based** (case-sensitive): fields are matched by name, reordered to the target schema, missing fields are null-filled (if nullable), and extra fields are ignored. * **Fallback behavior**: if there is *no* name overlap and field counts match, casting proceeds **positionally**. * **Potential behavior change** in queries relying on the prior positional behavior when structs shared names but were out of order (previously could yield swapped values). This PR changes that to the safer, expected behavior. No public API changes are introduced, but this is a semantic change in struct casting. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested. |
||
|
Jeffrey Vo
|
ac67ae4af5 |
Ensure null inputs to array setop functions return null output (#19683)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19682 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Explained in issue. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Change array_except, array_intersect and array_union UDFs to return null if either input is null. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Added & fixed tests. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Behaviour change to a function output. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Andrew Lamb
|
c91dcfdb6a |
Docs: add additional links to blog posts (#19833)
## Which issue does this PR close? - Part of #7013 ## Rationale for this change We have written some good blogs recently that provide additional context and backstory. Let's make sure they are available for others to read ## What changes are included in this PR? Add links to select doc pages ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
cht42
|
f3f6dec7c9 |
feat: Add support for 'isoyear' in date_part function (#19821)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19820. ## Rationale for this change `isoyear` part is available in both PG and Spark EXTRACT functions. https://www.postgresql.org/docs/current/functions-datetime.html#:~:text=the%20week%20numbering.-,isoyear,-The%20ISO%208601 https://github.com/apache/spark/blob/a03bedb6c1281c5263a42bfd20608d2ee005ab05/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L3360 ## What changes are included in this PR? Support for part `isoyear` in date_part function. ## Are these changes tested? yes in SLT ## Are there any user-facing changes? yes <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Miao
|
5edda9b309 |
fix: calculate total seconds from interval fields for extract(epoch) (#19807)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19799. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> The epoch function incorrectly used `date_part(array, DatePart::Second)` for intervals which only extracts the seconds component of the interval structure (0 for "15 minutes"), not the total seconds from all components. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Modified epoch() function in date_part.rs to properly handle interval types by extracting struct fields and calculating total seconds - Added regression tests in date_part.slt ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. Added tests verifying correct conversion for all interval types and precision levels. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Yes. `extract(epoch from interval)` now returns correct total seconds instead of 0. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Kumar Ujjawal
|
d103d8886f |
chore: remove LZO Parquet compression (#19726)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19720. ## Rationale for this change - Choosing LZO compression errors, I think it might never get supported so the best option moving forward is to remove it algother and update the docs. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Removed LZO from parse_compression_string() function - Removed docs - Updated exptected test output <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? User choosing LZO as compression will get a clear error message: ``` Unknown or unsupported parquet compression: lzo. Valid values are: uncompressed, snappy, gzip(level), brotli(level), lz4, zstd(level), and lz4_raw. ``` <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Kumar Ujjawal
|
646213ec7c |
feat: add Time type support to date_trunc function (#19640)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of #19025. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Added Time64/Time32 signatures to date_trunc - Added time truncation logic (hour, minute, second, millisecond, microsecond) - Error for invalid granularities (day, week, month, quarter, year) <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Bruce Ritchie
|
ada0923a39 |
Respect execution timezone in to_timestamp and related functions (#19078)
## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/17998. Continuation of PR #18025 by @kosiew ## Rationale for this change Previously, the to_timestamp() family of functions (to_timestamp, to_timestamp_seconds, to_timestamp_millis, to_timestamp_micros, to_timestamp_nanos) always interpreted timezone-free (naïve) timestamps as UTC, ignoring the datafusion.execution.time_zone configuration option. This behavior caused inconsistencies when users configured a specific execution timezone and expected timestamp conversions to respect it. This PR introduces full timezone awareness to these functions so that: - Naïve timestamp strings are interpreted as being in the configured execution timezone, or UTC if the configured execution timezone is `None`. - All returned timestamps are in the execution timezone. ## What changes are included in this PR? Code, tests. ## Are these changes tested? Yes, via code tests and slt tests. ## Are there any user-facing changes? Yes: - to_timestamp() and its precision variants now respect datafusion.execution.time_zone when parsing timezone-free timestamps and return timestamps in the execution time zone.. These changes make timestamp functions consistent with session timezone semantics and improve correctness for global workloads. --------- Co-authored-by: Siew Kam Onn <kosiew@gmail.com> |
||
|
Kumar Ujjawal
|
818706ab78 |
feat: to_time function (#19540)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of #19025. ## Rationale for this change Previously we needed to create timestamp and then extract the time component from that. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Added the function for `to_time` - Relevant slt test - Relevant unit tests - Updated docs <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? - All tests pass <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Jeffrey Vo
|
13f38435a2 |
Introduce TypeSignatureClass::Any (#19485)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19438 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> See issue. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Add new `TypeSignatureClass` variant `Any` and refactor `arrow_typeof` and `arrow_metadata` function signatures to use this. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Existing tests. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Kumar Ujjawal
|
62740802f6 |
Update to_unixtime udf function to support a consistent set of argument types (#19442)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19119 ## Rationale for this change to_unixtime lacks the support for several data types. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? Expanded `to_unixtime` support to all signed ints (Int8/16/32/64), all unsigned ints (UInt8/16/32/64), all floats (Float16/32/64), all UTF8 variants (Utf8/Utf8View/LargeUtf8), <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Added sqllogictest <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> |
||
|
Adrian Garcia Badaracco
|
a405d3fe4a |
Support nested field access in get_field with multiple path arguments (#19389)
## Summary This PR extends `get_field` to accept multiple field name arguments for nested struct/map access, enabling `get_field(col, 'a', 'b', 'c')` as equivalent to `col['a']['b']['c']`. **The primary motivation is to make it easier for downstream optimizations to match on and optimize struct/map field access patterns.** By representing `col['a']['b']['c']` as a single `get_field(col, 'a', 'b', 'c')` call rather than nested `get_field(get_field(get_field(col, 'a'), 'b'), 'c')` calls, optimization rules can more easily identify and transform field access patterns. This is related / maybe prep work for #19387 but I think is a good improvement in its own right. ## Changes - **Variadic signature**: `get_field` now accepts 2+ arguments (base + one or more field names) - **Type validation at planning time**: Accessing a field on a non-struct/map type (e.g., `get_field({a: 1}, 'a', 'b')`) fails during planning with a clear error message indicating which argument position caused the failure - **Bracket syntax optimization**: The `FieldAccessPlanner` now merges consecutive bracket accesses into a single `get_field` call (e.g., `s['a']['b']` → `get_field(s, 'a', 'b')`) - **Mixed access handling**: Array index access correctly breaks the batching (e.g., `s['a'][0]['b']` → `get_field(array_element(get_field(s, 'a'), 0), 'b')`) ## Example ```sql -- Direct function call with nested access SELECT get_field(my_struct, 'outer', 'inner', 'value'); -- Equivalent bracket syntax (now optimized to single get_field) SELECT my_struct['outer']['inner']['value']; -- EXPLAIN shows single get_field call EXPLAIN SELECT s['a']['b'] FROM t; -- Projection: get_field(t.s, Utf8("a"), Utf8("b")) ``` ## Backwards Compatibility - The original 2-argument form `get_field(struct, 'field')` continues to work unchanged - Existing queries using bracket syntax will automatically benefit from the optimization ## Test plan - [x] Backwards compatibility test for 2-argument form - [x] Multi-level get_field with 2, 3, and 5 levels of nesting - [x] Type validation error tests at argument positions 2, 3, 4 - [x] Non-existent field error tests - [x] Null handling (null at base, null in middle of chain) - [x] Mixed array/struct access (verifies array index breaks batching) - [x] Nullable parent propagation - [x] EXPLAIN test verifying single get_field call for bracket syntax - [x] Minimum argument validation (0 and 1 argument cases) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Huaijin
|
0bd880931e |
fix: csv schema_infer_max_records set to 0 return null datatype (#19432)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - close https://github.com/apache/datafusion/issues/19417 ## Rationale for this change - see https://github.com/apache/datafusion/issues/19417 - related to https://github.com/apache/datafusion/pull/17796 ## What changes are included in this PR? when schema_infer_max_records set to 0 in csv, return datatype as string ## Are these changes tested? add test case for schema_infer_max_records equal to 0 ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Bruce Ritchie
|
4a1f69f9bc |
Update date_bin to support Time32 and Time64 data types (#19341)
## Which issue does this PR close? Part of #19025 ## Rationale for this change Expand support for binning time data types. ## What changes are included in this PR? Code, tests. ## Are these changes tested? Yes, slt tests. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> |
||
|
xonx
|
d844f8687a |
Add:arrow_metadata() UDF (#19435)
## Which issue does this PR close? Closes #19356 ## Rationale for this change This PR implements the arrow_metadata UDF as requested in issue #19356. ## What changes are included in this PR? Added arrow_metadata UDF Refactored Tests ## Are these changes tested? Yes. ## Are there any user-facing changes? Yes. |
||
|
Kumar Ujjawal
|
d493f3d441 |
Add Decimal support to Ceil and Floor (#18979)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #7689. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Added dedicated ceil/floor UDF implementations that keep existing float/int behavior but operate directly on Decimal128 arrays, including overflow checks and metadata preservation. - Updated the math module wiring plus sqllogictest coverage so decimal cases are executed and validated end to end. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? - All existing tests pass - Added new tests for the changes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Bruce Ritchie
|
5a01e68643 |
Update to_date udf function to support a consistent set of argument types (#19134)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #19120 ## Rationale for this change Improved type support for to_date function. ## What changes are included in this PR? Code, slt, updated docs. ## Are these changes tested? Yes ## Are there any user-facing changes? More types supported. |
||
|
Bruce Ritchie
|
044a4a7369 |
Add make_time function (#19183)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of #19025 ## Rationale for this change There wasn't a good way to make a time from component parts. ## What changes are included in this PR? Code, test, docs ## Are these changes tested? Yes ## Are there any user-facing changes? New function. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
mag1c1an1
|
5496c30431 |
fix: typo in sql/ddl (#19276)
## Which issue does this PR close? None, this is a tiny typo fix. <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Make datafusion doc great! ## What changes are included in this PR? Change doc. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? No need to test code. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? NO <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Andrew Lamb
|
8dac8f1c41 |
Minor: fix link errors in docs (#19088)
I had an AI tool (`codex`) look for broken links in the docs and it found two. I tested both changes locally |
||
|
Jeffrey Vo
|
4eb2933445 |
Refactor crypto functions code (#18664)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> N/A ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Deduplicate & simplify code in the crypto functions. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Fold Sha224/Sha256/Sha384/Sha512 into a common struct. Cleanup signature & return types. Simplify code in `datafusion/functions/src/crypto/basic.rs` ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Existing tests. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Some public methods were removed, though I don't believe they were intended to be used outside of other DataFusion crates. <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Andrew Lamb
|
838e1dea83 |
Update links in documentation to point at new example locations (#18931)
## Which issue does this PR close? - part of #18142 ## Rationale for this change @cj-zhukov has been consolidating the examples, but that also means that the links to the examples are now out of date ## What changes are included in this PR? Update links (I used openai codex, with the prompt below) and reviewed the results <details><summary>Prompt</summary> <p> ``` We have moved the examples around in datafusion-examples/ There are some links in doc comments such as in `datafusion/src/lib.rs`: [`expr_api`.rs]: https://github.com/apache/datafusion/blob/main/datafusion-examples/ examples/expr_api.rs This example has been moved to ./datafusion-examples/examples/query_planning/ expr_api.rs Please look for all other links to datafusion-examples and update them if necessary to point at the new path. ``` </p> </details> ## Are these changes tested? By CI ## Are there any user-facing changes? Fixed docs |
||
|
kosiew
|
b477816c86 |
Enforce explicit opt-in for WITHIN GROUP syntax in aggregate UDAFs (#18607)
## Which issue does this PR close? Closes #18109. ## Rationale for this change Previously, the SQL planner accepted `WITHIN GROUP` clauses for all aggregate UDAFs, even those that did not explicitly support ordered-set semantics. This behavior was too permissive and inconsistent with PostgreSQL. For example, queries such as `SUM(x) WITHIN GROUP (ORDER BY x)` were allowed, even though `SUM` is not an ordered-set aggregate. This PR enforces stricter validation so that only UDAFs that explicitly return `true` from `supports_within_group_clause()` may use `WITHIN GROUP`. All other aggregates now produce a clear planner error when this syntax is used. ## What changes are included in this PR? * Added type alias `WithinGroupExtraction` to simplify complex tuple return types used by helper functions. * Introduced a new helper method `extract_and_prepend_within_group_args` to centralize logic for handling `WITHIN GROUP` argument rewriting. * Updated the planner to: * Validate that only UDAFs with `supports_within_group_clause()` can accept `WITHIN GROUP`. * Prepend `WITHIN GROUP` ordering expressions to function arguments only for supported ordered-set aggregates. * Produce clear error messages when `WITHIN GROUP` is used incorrectly. * Added comprehensive unit tests verifying correct behavior and failure cases: * `WITHIN GROUP` rejected for non-ordered-set aggregates (`MIN`, `SUM`, etc.). * `WITHIN GROUP` accepted for ordered-set aggregates such as `percentile_cont`. * Validation for named arguments, multiple ordering expressions, and semantic conflicts with `OVER` clauses. * Updated SQL logic tests (`aggregate.slt`) to reflect new rejection behavior. * Updated documentation: * `aggregate_functions.md` and developer docs to clarify when and how `WITHIN GROUP` can be used. * `upgrading.md` to inform users of this stricter enforcement and migration guidance. ## Are these changes tested? ✅ Yes. * New tests in `sql_integration.rs` validate acceptance, rejection, and argument behavior of `WITHIN GROUP` for both valid and invalid cases. * SQL logic tests (`aggregate.slt`) include negative test cases confirming planner rejections. ## Are there any user-facing changes? ✅ Yes. * Users attempting to use `WITHIN GROUP` with regular aggregates (e.g. `SUM`, `AVG`, `MIN`, `MAX`) will now see a planner error: > `WITHIN GROUP is only supported for ordered-set aggregate functions` * Documentation has been updated to clearly describe `WITHIN GROUP` semantics and provide examples of valid and invalid usage. No API-breaking changes were introduced; only stricter planner validation and improved error messaging. |
||
|
Gene Bordegaray
|
552dbe4f19 |
fix: Eliminate consecutive repartitions (#18521)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #18341. - Closes https://github.com/apache/datafusion/issues/9370 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Cases where two RepartitionExec operators appear consecutively in the plan. This is unneeded overhead that eliminating provides speed ups. Full Report: [The Physical Optimizer and Fixing Consecutive Repartitions In the Enforce Distribution Rule.pdf](https://github.com/user-attachments/files/23420831/The.Physical.Optimizer.and.Fixing.Consecutive.Repartitions.In.the.Enforce.Distribution.Rule.pdf) Issue Report: [Fixing Consecutive Repartitions In the Enforce Distribution Rule.pdf](https://github.com/user-attachments/files/23420880/Fixing.Consecutive.Repartitions.In.the.Enforce.Distribution.Rule.pdf) ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Change to repartition adding logic in `enforce_distribution.rs` A ton of test and bench updates to mirror new behavior ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes benchmarked and tested, check report for benchmarks ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Cora Sutton
|
e4f6a144ac |
Fix instances of "the the" to be "the" in comments/docs (#18478)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> There's no issue for this, just some simple text fixes. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> While "the the" *can* be grammatically correct, in these instances it was not. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> "the the" -> "the" ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Not at as such, no. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> Yes, in both code docs and in the website. |
||
|
Bruce Ritchie
|
9ea67f538c |
Change default time_zone to None (was "+00:00") (#18359)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes - #18204 - #18081 - fixes #18219 as a side effect ## Rationale for this change Default timezone was previously zulu however with the recent change to support default tz in now(), current_date(), etc which used to have no default tz the choice was made to unset the system wide timezone. ## What changes are included in this PR? Code, tests, upgrading doc. ## Are these changes tested? Yes, with existing tests. ## Are there any user-facing changes? Yes. Any query that used to use the default timezone would return a timestamp with a timezone of 'Z' will now return a timestamp without a timezone. This can be changed back to the previous behaviour with the sql ```sql SET TIMEZONE = '+00:00'; ``` --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Jeffrey Vo
|
8de8621aef |
minor: doc fixes for timestamp output format (#18315)
Followup some doc fixes missed in #17888 |
||
|
Jeffrey Vo
|
60904e4f1a |
Deduplicate range/gen_series nested functions code (#18198)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Doing some prework for #15881 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> `Range` and `GenSeries` are essentially the same except for whether they include upper bounds or not; unify their function code to reduce duplication, making future changes easier. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Remove `GenSeries` struct, folding it into `Range`. Do some more minor refactoring to their code. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Existing tests (updated some error messages). ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> Not really (updated some error messages). <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> |
||
|
Andrew Lamb
|
c6ad17cf2b |
Upgrade DataFusion to arrow/parquet 57.0.0 (#17888)
## Which issue does this PR close? - Related to https://github.com/apache/arrow-rs/issues/7835 - Closes #3666 Note while this PR looks massive, a large portion is display updates due to better display of Fields and DataTypes ## Rationale for this change Upgrade to the latest arrow Also, there are several new features in arrow-57 that I want to be able to test including Variant, arrow-avro, and a new parquet metadata reader. ## What changes are included in this PR? 1. Update arrow/parquet 2. Update prost 3. Update substrait 4. Update pbjson 5. Make API changes to avoid deprecated APIs ## Are these changes tested? By CI ## Are there any user-facing changes? New arrow |
||
|
Sriram Sundar
|
6d52e54bc8 |
Docs: Update SQL example for current_time() and current_date(). (#18200)
## Which issue does this PR close? - Closes #18199 ## What changes are included in this PR? - Added a SQL example in scalar functions md to demonstrate setting execution time zone (optional) for current_time() and current_date(). ## Are these changes tested? ## Are there any user-facing changes? |
||
|
Sriram Sundar
|
37aad28424 |
Feat: Make current_time aware of execution timezone. (#18040)
## Which issue does this PR close? - Closes #17996. ## Rationale for this change - The current_time() function currently uses UTC tz. This PR updates current_time() to use the tz set in 'datafusion.execution.time_zone' ## What changes are included in this PR? - current_time() returns a tz aware date via the 'datafusion.execution.time_zone' config option. ## Are these changes tested? - Tested with Datafusion CLI with slt covering popular scenarios added. |
||
|
Yongting You
|
ec2402aee9 |
feat: Support configurable EXPLAIN ANALYZE detail level (#18098)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> `EXPLAIN ANALYZE` can be used for profiling and displays the results alongside the EXPLAIN plan. The issue is that it currently shows too many low-level details. It would provide a better user experience if only the most commonly used metrics were shown by default, with more detailed metrics available through specific configuration options. ### Example In `datafusion-cli`: ``` > CREATE EXTERNAL TABLE IF NOT EXISTS lineitem STORED AS parquet LOCATION '/Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem'; 0 row(s) fetched. Elapsed 0.000 seconds. explain analyze select * from lineitem where l_orderkey = 3000000; ``` The parquet reader includes a large number of low-level details: ``` metrics=[output_rows=19813, elapsed_compute=14ns, batches_split=0, bytes_scanned=2147308, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=18, num_predicate_creation_errors=0, page_index_rows_matched=19813, page_index_rows_pruned=729088, predicate_cache_inner_records=0, predicate_cache_records=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=1, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=21.997µs, metadata_load_time=273.83µs, page_index_eval_time=29.915µs, row_pushdown_eval_time=42ns, statistics_eval_time=76.248µs, time_elapsed_opening=4.02146ms, time_elapsed_processing=24.787461ms, time_elapsed_scanning_total=24.17671ms, time_elapsed_scanning_until_data=23.103665ms] ``` I believe only a subset of it is commonly used, for example `output_rows`, `metadata_load_time`, and how many file/row-group/pages are pruned, and it would better to only display the most common ones by default. ### Existing `VERBOSE` keyword There is a existing verbose keyword in `EXPLAIN ANALYZE VERBOSE`, however it's turning on per-partition metrics instead of controlling detail level. I think it would be hard to mix this partition control and the detail level introduced in this PR, so they're separated: the following config will be used for detail level and the semantics of `EXPLAIN ANALYZE VERBOSE` keep unchanged. ### This PR: configurable explain analyze level 1. Introduced a new config option `datafusion.explain.analyze_level`. When set to `dev` (default value), all existing metrics will be shown. If set to `summary`, only `BaselineMetrics` will be displayed (i.e. `output_rows` and `elapsed_compute`). Note now we only include `BaselineMetrics` for simplicity, in the follow-up PRs we can figure out what's the commonly used metrics for each operator, and add them to `summary` analyze level, finally set the `summary` analyze level to default. 2. Add a `MetricType` field associated with `Metric` for detail level or potentially category in the future. For different configurations, a certain `MetricType` set will be shown accordingly. #### Demo ``` -- continuing the above example > set datafusion.explain.analyze_level = summary; 0 row(s) fetched. Elapsed 0.000 seconds. > explain analyze select * from lineitem where l_orderkey = 3000000; +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | plan_type | plan | +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Plan with Metrics | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=5, elapsed_compute=25.339µs] | | | FilterExec: l_orderkey@0 = 3000000, metrics=[output_rows=5, elapsed_compute=81.221µs] | | | DataSourceExec: file_groups={14 groups: [[Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-0.parquet:0..11525426], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-0.parquet:11525426..20311205, Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-1.parquet:0..2739647], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-1.parquet:2739647..14265073], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-1.parquet:14265073..20193593, Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-2.parquet:0..5596906], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-2.parquet:5596906..17122332], ...]}, projection=[l_orderkey, l_partkey, l_suppkey, l_linenumber, l_quantity, l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate, l_commitdate, l_receiptdate, l_shipinstruct, l_shipmode, l_comment], file_type=parquet, predicate=l_orderkey@0 = 3000000, pruning_predicate=l_orderkey_null_count@2 != row_count@3 AND l_orderkey_min@0 <= 3000000 AND 3000000 <= l_orderkey_max@1, required_guarantees=[l_orderkey in (3000000)], metrics=[output_rows=19813, elapsed_compute=14ns] | | | | +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row(s) fetched. Elapsed 0.025 seconds. ``` Only `BaselineMetrics` are shown. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 4. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> UT ## Are there any user-facing changes? No <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Adrian Garcia Badaracco
|
c84e3cf5a5 |
feat: Add percentile_cont aggregate function (#17988)
## Summary Adds exact `percentile_cont` aggregate function as the counterpart to the existing `approx_percentile_cont` function. ## What changes were made? ### New Implementation - Created `percentile_cont.rs` with full implementation - `PercentileCont` struct implementing `AggregateUDFImpl` - `PercentileContAccumulator` for standard aggregation - `DistinctPercentileContAccumulator` for DISTINCT mode - `PercentileContGroupsAccumulator` for efficient grouped aggregation - `calculate_percentile` function with linear interpolation ### Features - **Exact calculation**: Stores all values in memory for precise results - **WITHIN GROUP syntax**: Supports `WITHIN GROUP (ORDER BY ...)` - **Interpolation**: Uses linear interpolation between values - **All numeric types**: Works with integers, floats, and decimals - **Ordered-set aggregate**: Properly marked as `is_ordered_set_aggregate()` - **GROUP BY support**: Efficient grouped aggregation via GroupsAccumulator ### Tests Added comprehensive tests in `aggregate.slt`: - Error conditions validation - Basic percentile calculations (0.0, 0.25, 0.5, 0.75, 1.0) - Comparison with `median` function - Ascending and descending order - GROUP BY aggregation - NULL handling - Edge cases (empty sets, single values) - Float interpolation - Various numeric data types ## Example Usage ```sql -- Basic usage with WITHIN GROUP syntax SELECT percentile_cont(0.75) WITHIN GROUP (ORDER BY column_name) FROM table_name; -- With GROUP BY SELECT category, percentile_cont(0.95) WITHIN GROUP (ORDER BY value) FROM sales GROUP BY category; -- Compare with median (percentile_cont(0.5) == median) SELECT percentile_cont(0.5) WITHIN GROUP (ORDER BY price) FROM products; ``` ## Performance Considerations Like `median`, this function stores all values in memory before computing results. For large datasets or when approximation is acceptable, use `approx_percentile_cont` instead. ## Related Issues Closes #6714 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Pepijn Van Eeckhoudt
|
ea83c2644e |
#17982 Make nvl a thin wrapper for coalesce (#17991)
## Which issue does this PR close? - Closes #17982 ## Rationale for this change By making `NVLFunc` a wrapper for `CoalesceFunc` with a more restrictive signature the implementation automatically benefits from any optimisation work related to `coalesce`. ## What changes are included in this PR? - Make `NVLFunc` a thin wrapper of `CoalesceFunc`. This seemed like the simplest way to reuse the coalesce logic, but keep the stricter signature of `nvl`. - Add `ScalarUDF::conditional_arguments` as a more precise complement to `ScalarUDF::short_circuits`. By letting each function expose which arguments are eager and which are lazy, we provide more precise information to the optimizer which may enable better optimisation. ## Are these changes tested? Assumed to be covered by sql logic tests. Unit tests for the custom implementation were removed since those are no longer relevant. ## Are there any user-facing changes? The rewriting of `nvl` to `case when ... then ... else ... end` is visible in the physical query plan. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Sriram Sundar
|
77ec31981a |
Feat: Make current_date aware of execution timezone. (#18034)
* Feat: Make current_date aware of execution timezone. * CI Fixes: Rectify assertion error of slt and update scalar_functions.md * CI Fixes: Comment out flaky test involving now(). * CI Fixes: Resolve slt error. * Feat: Add helper function to calculate current_date and fix tests. * Chore: Refactor timezone conversion logic. |
||
|
Simon Vandel Sillesen
|
6479e43f86 |
Support JOIN pipe operator (#17969)
* support WHERE pipe operator * support order by * support limit * select pipe * extend support * document supported pipe operators in user guide * fmt * fix where pipe before extend * support AS * support union * support intersection * support except * support aggregate * support join operator * support pivot * remove unused * revert parquet-testing * remove simon * move docs to select.md * remove unnecessary comments * back out pivot operator * remove prompt marker |
||
|
Simon Vandel Sillesen
|
a7b113c455 |
Support AS, UNION, INTERSECTION, EXCEPT, AGGREGATE pipe operators (#17312)
* support WHERE pipe operator * support order by * support limit * select pipe * extend support * document supported pipe operators in user guide * fmt * fix where pipe before extend * support AS * support union * support intersection * support except * support aggregate * revert diff from main * simplify using mut * remove useless comments * remove dummy data * move docs to select.md * simplify using alias_if_changed * deduplicate fn * add aggregate toc * revert parquet testing |
||
|
aditya singh rathore
|
84b327c564 |
doc: add missing examples for multiple math functions (#17018)
* Update Scalar_functions.md * pretier fix * Updated files * Updated Scalar functions * Update datafusion/functions/src/math/log.rs Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> * Update datafusion/functions/src/math/monotonicity.rs Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> * Update datafusion/functions/src/math/monotonicity.rs Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> * Update datafusion/functions/src/math/nans.rs Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> * Update datafusion/functions/src/math/nanvl.rs Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> * Fix tanh example to be tanh not trunc * Run update_function_docs.sh --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |
||
|
Simon Vandel Sillesen
|
23d91c59bc |
Support WHERE, ORDER BY, LIMIT, SELECT, EXTEND pipe operators (#17278)
* support WHERE pipe operator * support order by * support limit * select pipe * extend support * document supported pipe operators in user guide * fmt * fix where pipe before extend * don't rebind * remove clone * move docs into select.md * avoid confusion by removing `>` in examples --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> |