mirror of
https://github.com/langchain-ai/datafusion.git
synced 2026-08-28 04:39:59 -04:00
kosiew
0c82adec71
## Which issue does this PR close? * Closes #17285. ## Rationale for this change DataFusion’s struct casting and some coercion paths were effectively positional: when two structs had the same field types but different field *orders*, casting could silently swap values. This is surprising to users and can lead to silent data corruption (e.g. `{b: 3, a: 4}::STRUCT(a INT, b INT)` yielding `{a: 3, b: 4}`). The goal of this PR is to make struct casting behavior match user expectations by matching fields by **name** (case-sensitive) and recursively applying the same logic to nested structs, while keeping a compatible fallback for structs with **no** shared field names. ## What changes are included in this PR? * **Name-based struct casting implementation** in `datafusion_common::nested_struct`: * Match struct fields by **name**, reorder to match target schema, recursively cast nested structs. * Fill **missing target fields** with null arrays. * Ignore **extra source fields**. * **Positional mapping fallback** when there is *no name overlap* **and** field counts match (avoids breaking `struct(1, 'x')::STRUCT(a INT, b VARCHAR)` style casts). * Improved handling for **NULL / all-null struct inputs** by producing a correctly typed null struct array. * Centralized validation via `validate_field_compatibility` and helper `fields_have_name_overlap`. * **Ensure struct casting paths use the name-based logic**: * `ScalarValue::cast_to_with_options`: route `Struct` casts through `nested_struct::cast_column`. * `ColumnarValue::cast_to`: for `Struct` targets, cast via `nested_struct::cast_column`; non-struct casts still use Arrow’s standard casting. * **Type coercion improvements for structs in binary operators / CASE**: * When two structs have at least one overlapping name, coerce **by name**. * Otherwise, preserve prior behavior by coercing **positionally**. * **Planning-time cast validation for struct-to-struct**: * `physical-expr` CAST planning now validates struct compatibility using the same rules as runtime (`validate_struct_compatibility`) to fail fast. * `ExprSchemable` allows struct-to-struct casts to pass type checking; detailed compatibility is enforced by the runtime / planning-time validator. * **Optimizer safety**: * Avoid const-folding struct casts when field counts differ. * Avoid const-folding casts of **0-row** struct literals due to evaluation batch dimension mismatches. * **Tests and SQL logic tests**: * New unit tests covering: * name-based reordering * missing fields (nullable vs non-nullable) * null struct fields and nested nulls * positional fallback with no overlap * coercion behavior and simplifier behavior * Updated/added `.slt` cases to reflect the new semantics and to add coverage for struct casts and nested struct reordering. * **Minor docs/maintenance**: * Adjusted doc comment referencing `ParquetWriterOptions` so it doesn’t break when the `parquet` feature is disabled. ## Are these changes tested? Yes. * Added/updated Rust unit tests in: * `datafusion/common/src/nested_struct.rs` * `datafusion/expr-common/src/columnar_value.rs` * `datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs` * Added/updated SQL logic tests in: * `datafusion/sqllogictest/test_files/case.slt` * `datafusion/sqllogictest/test_files/struct.slt` These tests cover: * correct value mapping when struct field order differs * nested struct reordering * insertion of nulls for missing nullable fields * erroring on missing non-nullable target fields * positional mapping fallback when there is no name overlap * planning-time validation vs runtime behavior alignment ## Are there any user-facing changes? Yes. * **Struct casts are now name-based** (case-sensitive): fields are matched by name, reordered to the target schema, missing fields are null-filled (if nullable), and extra fields are ignored. * **Fallback behavior**: if there is *no* name overlap and field counts match, casting proceeds **positionally**. * **Potential behavior change** in queries relying on the prior positional behavior when structs shared names but were out of order (previously could yield swapped values). This PR changes that to the safer, expected behavior. No public API changes are introduced, but this is a semantic change in struct casting. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested.
39 lines
1.1 KiB
ReStructuredText
39 lines
1.1 KiB
ReStructuredText
.. Licensed to the Apache Software Foundation (ASF) under one
|
|
.. or more contributor license agreements. See the NOTICE file
|
|
.. distributed with this work for additional information
|
|
.. regarding copyright ownership. The ASF licenses this file
|
|
.. to you under the Apache License, Version 2.0 (the
|
|
.. "License"); you may not use this file except in compliance
|
|
.. with the License. You may obtain a copy of the License at
|
|
|
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
.. Unless required by applicable law or agreed to in writing,
|
|
.. software distributed under the License is distributed on an
|
|
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
.. KIND, either express or implied. See the License for the
|
|
.. specific language governing permissions and limitations
|
|
.. under the License.
|
|
|
|
SQL Reference
|
|
=============
|
|
|
|
.. toctree::
|
|
:maxdepth: 2
|
|
|
|
data_types
|
|
struct_coercion
|
|
select
|
|
subqueries
|
|
ddl
|
|
dml
|
|
explain
|
|
information_schema
|
|
operators
|
|
aggregate_functions
|
|
window_functions
|
|
scalar_functions
|
|
special_functions
|
|
format_options
|
|
prepared_statements
|