langchain-ai/datafusion
1
0
Fork 0
mirror of https://github.com/langchain-ai/datafusion.git synced 2026-07-18 21:24:40 -04:00
Code Issues 3 Packages Projects Releases Wiki Activity
12,745 Commits 4 Branches 0 Tags
e937cadbcceff6a42bee2c5fc8d03068fa0eb30c
Commit Graph

1036 Commits

Author SHA1 Message Date
Neil Conway 585bbf35d3 perf: Optimize array_has_any() with scalar arg (#20385) 
## Which issue does this PR close?

- Closes #20384.
- See #18181 for related context.

## Rationale for this change

When `array_has_any` is passed a scalar for either of its arguments, we
can use a much faster algorithm: rather than doing O(N*M) comparisons
for each row of the columnar arg, we can build a hash table on the
scalar argument and probe it instead.

## What changes are included in this PR?

* Add benchmark to cover the one-scalar-arg case
* Implement optimization as described above

Note that we fallback to a linear scan when the scalar arg is smaller
than a threshold (<= 8 elements), because benchmarks suggested probing a
HashSet is not profitable for very small arrays.

## Are these changes tested?

Yes. Tests pass and benchmarked.

## Are there any user-facing changes?

No.

---------

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
 2026-02-24 20:59:08 +00:00
Albert Skalt 34dad2ccee Cache PlanProperties, add fast-path for with_new_children (#19792) 
- closes https://github.com/apache/datafusion/issues/19796

This patch aims to implement a fast-path for the
ExecutionPlan::with_new_children function for some plans, moving closer
to a physical plan re-use implementation and improving planning
performance. If the passed children properties are the same as in self,
we do not actually recompute self's properties (which could be costly if
projection mapping is required). Instead, we just replace the children
and re-use self's properties as-is.

To be able to compare two different properties --
ExecutionPlan::properties(...) signature is modified and now returns
`&Arc<PlanProperties>`. If `children` properties are the same in
`with_new_children` -- we clone our properties arc and then a parent
plan will consider our properties as unchanged, doing the same.

- Return `&Arc<PlanProperties>` from `ExecutionPlan::properties(...)`
instead of a reference.
- Implement `with_new_children` fast-path if there is no children
properties changes for all
  major plans.

Note: currently, `reset_plan_states` does not allow to re-use plan in
general: it is not
supported for dynamic filters and recursive queries features, as in this
case state reset
should update pointers in the children plans.

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
 2026-02-24 20:58:06 +00:00
Oleks V ed0323a2bb feat: support arrays_zip function (#20440) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->


- Closes #.

## Rationale for this change

Summary
- Adds a new arrays_zip scalar function that combines multiple arrays
into a single array of structs, where each struct field corresponds to
an input array
- Shorter arrays within a row are padded with NULLs to match the longest
array's length
- Compatible with Spark's arrays_zip behavior

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?

```

  arrays_zip takes N list arrays and produces a List<Struct<c0, c1, ..., cN>> where each struct contains the elements at the same index from each input array.

  > SELECT arrays_zip([1, 2, 3], ['a', 'b', 'c']);
  [{c0: 1, c1: a}, {c0: 2, c1: b}, {c0: 3, c1: c}]

  > SELECT arrays_zip([1, 2], [3, 4, 5]);
  [{c0: 1, c1: 3}, {c0: 2, c1: 4}, {c0: NULL, c1: 5}]

  Implementation details:
  - Implemented in set_ops.rs following existing array function patterns
  - Uses MutableArrayData builders per column with row-by-row processing for efficient memory handling
  - For each row, computes the max array length, copies values from each input array, and pads shorter arrays with NULLs
  - Supports variadic arguments (2 or more arrays)
  - Handles NULL list entries, NULL elements, empty arrays, mixed types, and Null-typed arguments
  - Registered as arrays_zip with alias list_zip
  - Uses Signature::variadic_any with Volatility::Immutable
```

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
 2026-02-23 16:08:36 +00:00
Dmitrii Blaginin 60457d0b0a Runs-on for more actions (#20274) 
Follow up on https://github.com/apache/datafusion/pull/20107: switch
more actions to the new flow

| Job | OLD | NEW | Delta |
|---|---|---|---|
| **linux build test** (from #20107) | 3m 55s | 1m 46s | -2m 09s (55%
faster) |
| **cargo test (amd64)** (from #20107) | 11m 34s | 3m 13s | -8m 21s (72%
faster) |
| **cargo check datafusion features** | 11m 18s | 6m 21s | -4m 57s (44%
faster) |
| **cargo examples (amd64)** | 9m 13s | 4m 35s | -4m 38s (50% faster) |
| **verify benchmark results (amd64)** | 11m 48s | 4m 22s | -7m 26s (63%
faster) |
| **cargo check datafusion-substrait features** | 10m 20s | 3m 56s | -6m
24s (62% faster) |
| **cargo check datafusion-proto features** | 4m 48s | 2m 25s | -2m 23s
(50% faster) |
| **cargo test datafusion-cli (amd64)** | 5m 42s | 1m 58s | -3m 44s (65%
faster) |
| **cargo test doc (amd64)** | 8m 07s | 3m 16s | -4m 51s (60% faster) |
| **cargo doc** | 5m 10s | 1m 56s | -3m 14s (63% faster) |
| **Run sqllogictest with Postgres runner** | 6m 06s | 2m 46s | -3m 20s
(55% faster) |
| **Run sqllogictest in Substrait round-trip mode** | 6m 42s | 2m 38s |
-4m 04s (61% faster) |
| **clippy** | 6m 01s | 2m 10s | -3m 51s (64% faster) |
| **check configs.md and \*\*\*_functions.md is up-to-date** | 6m 54s |
2m 12s | -4m 42s (68% faster) |
 2026-02-23 10:04:37 +00:00
Eren Avsarogullari a936d0de95 test: Extend Spark Array functions: array_repeat , shuffle and slice test coverage (#20420) 
## Which issue does this PR close?
- Closes #20419.

## Rationale for this change
This PR adds new positive test cases for `datafusion-spark` array
functions: `array_repeat `, `shuffle`, `slice` for the following
use-cases:
```
- nested function execution,
- different datatypes such as timestamp,
- casting before function execution
```
Also, being updated contributor-guide testing documentation with minor
addition.

## What changes are included in this PR?
Being added new positive test cases to `datafusion-spark` array
functions: `array_repeat `, `shuffle`, `slice`.

## Are these changes tested?
Yes, adding new positive test cases.

## Are there any user-facing changes?
No
 2026-02-20 18:39:37 +00:00
Adrian Garcia Badaracco 1ee782f783 Migrate Python usage to uv workspace (#20414) 
I was having trouble getting benchmarks to gen data.

## Summary
- Replace three independent `requirements.txt` files with a uv workspace
(`benchmarks`, `dev`, `docs` projects)
- Single `uv.lock` lockfile for reproducible dependency resolution
- Simplify `bench.sh` by removing all ad-hoc venv/pip logic in favor of
`uv run`

## Test plan
- [ ] `uv sync` resolves all deps from repo root
- [ ] `uv run --project benchmarks python3 benchmarks/compare.py` works
- [ ] `uv run --project docs sphinx-build docs/source docs/build` builds
docs
- [ ] Run a benchmark from `bench.sh` that uses Python (e.g., h2o data
gen or compare flow)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-20 16:29:56 +00:00
Neil Conway ace9cd44b7 perf: Optimize trim UDFs for single-character trims (#20328) 
## Which issue does this PR close?

- Closes #20327 

## Rationale for this change

By default, btrim(), ltrim(), and rtrim() trim space characters; it is
also reasonably common for queries to specify a non-default trim pattern
that is still a single ASCII character.

We can optimize for this case by doing a byte-level scan, rather than
invoking the more heavyweight std::string machinery used for more
complex trim scenarios.

## What changes are included in this PR?

Add a benchmark for trimming spaces, and implement the optimization
described above. Also fixed an error in the documentation.

## Are these changes tested?

Yes, and benchmarked.

## Are there any user-facing changes?

No.

---------

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
 2026-02-20 09:28:53 +00:00
Neil Conway b7f6090874 perf: Optimize lpad, rpad for ASCII strings (#20278) 
The previous implementation incurred the overhead of Unicode machinery,
even for the common case that both the input string and the fill string
consistent only of ASCII characters. For the ASCII-only case, we can
assume that the length in bytes equals the length in characters, and
avoid expensive graphene-based segmentation. This follows similar
optimizations applied elsewhere in the codebase.

Benchmarks indicate this is a significant performance win for ASCII-only
input (4x-10x faster) but only a mild regression for Unicode input (2-5%
slower).

Along the way:

* Combine: a few instances of `write_str(str)? + append_value("")` with
`append_value(str)`, which saves a few cycles
* Add a missing test case for truncating the input string
* Add benchmarks for Unicode input

## Which issue does this PR close?

- Closes #20277.

## Are these changes tested?

Covered by existing tests. Added new benchmarks for Unicode inputs.

## Are there any user-facing changes?

No.

---------

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
 2026-02-20 02:32:42 +00:00
Neil Conway c3f080774c perf: Optimize translate() UDF for scalar inputs (#20305) 
## Which issue does this PR close?

- Closes #20302.

## Rationale for this change

`translate()` is commonly invoked with constant values for its second
and third arguments. We can take advantage of that to significantly
optimize its performance by precomputing the translation lookup table,
rather than recomputing it for every row. For ASCII-only inputs, we can
further replace the hashmap lookup table with a fixed-size array that
maps ASCII byte values directly.

For scalar ASCII inputs, this yields roughly a 10x performance
improvement. For scalar UTF8 inputs, the performance improvement is more
like 50%, although less so for long strings.

Along the way, add support for `translate()` on `LargeUtf8` input, along
with an SLT test, and improve the docs.

## What changes are included in this PR?

* Add a benchmark for scalar/constant input to translate
* Add a missing test case
* Improve translate() docs
* Support translate() on LargeUtf8 input
* Optimize translate() for scalar inputs by precomputing lookup hashmap
* Optimize translate() for ASCII inputs by precomputing ASCII byte-wise
lookup table

## Are these changes tested?

Yes. Added an extra test case and did a bunch of benchmarking.

## Are there any user-facing changes?

No.

---------

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
 2026-02-19 07:03:19 +00:00
dependabot[bot] 4e43ac5a06 chore(deps): bump maturin from 1.11.5 to 1.12.2 in /docs (#20400) 
Bumps [maturin](https://github.com/pyo3/maturin) from 1.11.5 to 1.12.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/pyo3/maturin/releases">maturin's
releases</a>.</em></p>
<blockquote>
<h2>v1.12.2</h2>
<h2>What's Changed</h2>
<ul>
<li>fix: allow absolute paths for <code>--sbom-include</code> by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/3004">PyO3/maturin#3004</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/PyO3/maturin/compare/v1.12.1...v1.12.2">https://github.com/PyO3/maturin/compare/v1.12.1...v1.12.2</a></p>
<h2>v1.12.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Add <code>--sbom-include</code> CLI argument for additional SBOM
files by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2999">PyO3/maturin#2999</a></li>
<li>fix: resolve include patterns relative to python-source for sdist
and wheel by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/3000">PyO3/maturin#3000</a></li>
<li>feat: support including <code>OUT_DIR</code> assets in wheel builds
by <a href="https://github.com/messense"><code>@​messense</code></a> in
<a
href="https://redirect.github.com/PyO3/maturin/pull/3001">PyO3/maturin#3001</a></li>
<li>add test case for uniffi with multiple crates by <a
href="https://github.com/davidhewitt"><code>@​davidhewitt</code></a> in
<a
href="https://redirect.github.com/PyO3/maturin/pull/2839">PyO3/maturin#2839</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/PyO3/maturin/compare/v1.12.0...v1.12.1">https://github.com/PyO3/maturin/compare/v1.12.0...v1.12.1</a></p>
<h2>v1.12.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Use pypi compatibility validation for own CI by <a
href="https://github.com/konstin"><code>@​konstin</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2929">PyO3/maturin#2929</a></li>
<li>Update toml crates for toml 1.1 support by <a
href="https://github.com/konstin"><code>@​konstin</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2934">PyO3/maturin#2934</a></li>
<li>Use a single location for MSRV by <a
href="https://github.com/konstin"><code>@​konstin</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2936">PyO3/maturin#2936</a></li>
<li>Fix editable install for binary projects with Python modules by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2938">PyO3/maturin#2938</a></li>
<li>Release to crates.io only after the builds passed by <a
href="https://github.com/konstin"><code>@​konstin</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2939">PyO3/maturin#2939</a></li>
<li>Use <code>mymindstorm/setup-emsdk@v14</code> in generated GitHub
Actions workflow by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2941">PyO3/maturin#2941</a></li>
<li>Use trusted publishing for crates.io by <a
href="https://github.com/konstin"><code>@​konstin</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2943">PyO3/maturin#2943</a></li>
<li>Filter linked_paths by KIND and linked_libs by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2949">PyO3/maturin#2949</a></li>
<li>Update bytes to 1.11.1 by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2960">PyO3/maturin#2960</a></li>
<li>Normalize wheel distribution names to match the PyPA spec by <a
href="https://github.com/artob"><code>@​artob</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2954">PyO3/maturin#2954</a></li>
<li>Allow build loongarch64 and riscv64 for musllinux by <a
href="https://github.com/wojiushixiaobai"><code>@​wojiushixiaobai</code></a>
in <a
href="https://redirect.github.com/PyO3/maturin/pull/2963">PyO3/maturin#2963</a></li>
<li>Strip excluded cargo targets in sdist by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2964">PyO3/maturin#2964</a></li>
<li>Normalize wheel <code>RECORD</code> paths (on Windows) by <a
href="https://github.com/texodus"><code>@​texodus</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2965">PyO3/maturin#2965</a></li>
<li>Bump MSRV to 1.88.0 by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2966">PyO3/maturin#2966</a></li>
<li>Support MATURIN_STRIP env var and --strip true/false to override
pyproject.toml by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2968">PyO3/maturin#2968</a></li>
<li>fix: copy bin artifacts before auditwheel repair to avoid rerun
failures by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2969">PyO3/maturin#2969</a></li>
<li>fix: rewrite python-source in pyproject.toml when building sdist by
<a href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2972">PyO3/maturin#2972</a></li>
<li>fix: resolve wheel include patterns relative to project root by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2973">PyO3/maturin#2973</a></li>
<li>fix: always include workspace Cargo.toml in sdist by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2974">PyO3/maturin#2974</a></li>
<li>refactor: simplify source_distribution.rs by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2976">PyO3/maturin#2976</a></li>
<li>feat: support PEP 735 dependency groups in develop command by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2978">PyO3/maturin#2978</a></li>
<li>Fix license file handling for workspace-level license files by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2970">PyO3/maturin#2970</a></li>
<li>Support PEP 739 build-details.json when cross compiling by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2979">PyO3/maturin#2979</a></li>
<li>Fix .libs directory name for namespace packages by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2981">PyO3/maturin#2981</a></li>
<li>fix: exclude duplicate python source files from sdist for workspace
members by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2982">PyO3/maturin#2982</a></li>
<li>fix: remove default-members from workspace Cargo.toml in sdist by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2983">PyO3/maturin#2983</a></li>
<li>fix: correctly filter workspace members in sdist by directory path
by <a href="https://github.com/messense"><code>@​messense</code></a> in
<a
href="https://redirect.github.com/PyO3/maturin/pull/2984">PyO3/maturin#2984</a></li>
<li>feat: Add PEP 770 SBOM support by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2980">PyO3/maturin#2980</a></li>
<li>Error when python-source is set but Python module is missing by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2986">PyO3/maturin#2986</a></li>
<li>feat: add auditwheel SBOM for grafted shared libraries by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2985">PyO3/maturin#2985</a></li>
<li>Fix sdist duplicate README error when readme is in both Cargo.toml
and pyproject.toml by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2987">PyO3/maturin#2987</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/PyO3/maturin/blob/main/Changelog.md">maturin's
changelog</a>.</em></p>
<blockquote>
<h2>1.12.2</h2>
<ul>
<li>Fix: allow absolute paths for <code>--sbom-include</code> (<a
href="https://redirect.github.com/pyo3/maturin/issues/3004">#3004</a>)</li>
</ul>
<h2>1.12.1</h2>
<ul>
<li>Replace addnab/docker-run-action with direct docker run command</li>
<li>Add <code>--sbom-include</code> CLI argument for additional SBOM
files (<a
href="https://redirect.github.com/pyo3/maturin/issues/2999">#2999</a>)</li>
<li>Fix: resolve include patterns relative to python-source for sdist
and wheel (<a
href="https://redirect.github.com/pyo3/maturin/issues/3000">#3000</a>)</li>
<li>Feat: log external shared libraries and their dependents before
patchelf</li>
<li>Feat: support including <code>OUT_DIR</code> assets in wheel builds
(<a
href="https://redirect.github.com/pyo3/maturin/issues/3001">#3001</a>)</li>
<li>Add test case for uniffi with multiple crates (<a
href="https://redirect.github.com/pyo3/maturin/issues/2839">#2839</a>)</li>
</ul>
<h2>1.12.0</h2>
<ul>
<li>Use pypi compatibility validation for own CI (<a
href="https://redirect.github.com/pyo3/maturin/issues/2929">#2929</a>)</li>
<li>Update toml crates for toml 1.1 support (<a
href="https://redirect.github.com/pyo3/maturin/issues/2934">#2934</a>)</li>
<li>Use a single location for MSRV (<a
href="https://redirect.github.com/pyo3/maturin/issues/2936">#2936</a>)</li>
<li>Fix editable install for binary projects with Python modules (<a
href="https://redirect.github.com/pyo3/maturin/issues/2938">#2938</a>)</li>
<li>Release to crates.io only after the builds passed (<a
href="https://redirect.github.com/pyo3/maturin/issues/2939">#2939</a>)</li>
<li>Use <code>mymindstorm/setup-emsdk@v14</code> in generated GitHub
Actions workflow (<a
href="https://redirect.github.com/pyo3/maturin/issues/2941">#2941</a>)</li>
<li>Use trusted publishing for crates.io (<a
href="https://redirect.github.com/pyo3/maturin/issues/2943">#2943</a>)</li>
<li>Filter linked_paths by KIND and linked_libs (<a
href="https://redirect.github.com/pyo3/maturin/issues/2949">#2949</a>)</li>
<li>Update bytes to 1.11.1 (<a
href="https://redirect.github.com/pyo3/maturin/issues/2960">#2960</a>)</li>
<li>Normalize wheel distribution names to match the PyPA spec (<a
href="https://redirect.github.com/pyo3/maturin/issues/2954">#2954</a>)</li>
<li>Allow build loongarch64 and riscv64 for musllinux (<a
href="https://redirect.github.com/pyo3/maturin/issues/2963">#2963</a>)</li>
<li>Strip excluded cargo targets in sdist (<a
href="https://redirect.github.com/pyo3/maturin/issues/2964">#2964</a>)</li>
<li>Normalize wheel <code>RECORD</code> paths (on Windows) (<a
href="https://redirect.github.com/pyo3/maturin/issues/2965">#2965</a>)</li>
<li>Bump MSRV to 1.88.0 (<a
href="https://redirect.github.com/pyo3/maturin/issues/2966">#2966</a>)</li>
<li>Support MATURIN_STRIP env var and --strip true/false to override
pyproject.toml (<a
href="https://redirect.github.com/pyo3/maturin/issues/2968">#2968</a>)</li>
<li>Fix: copy bin artifacts before auditwheel repair to avoid rerun
failures (<a
href="https://redirect.github.com/pyo3/maturin/issues/2969">#2969</a>)</li>
<li>Fix: rewrite python-source in pyproject.toml when building sdist (<a
href="https://redirect.github.com/pyo3/maturin/issues/2972">#2972</a>)</li>
<li>Fix: resolve wheel include patterns relative to project root (<a
href="https://redirect.github.com/pyo3/maturin/issues/2973">#2973</a>)</li>
<li>Fix: always include workspace Cargo.toml in sdist (<a
href="https://redirect.github.com/pyo3/maturin/issues/2974">#2974</a>)</li>
<li>Refactor: simplify source_distribution.rs (<a
href="https://redirect.github.com/pyo3/maturin/issues/2976">#2976</a>)</li>
<li>Feat: support PEP 735 dependency groups in develop command (<a
href="https://redirect.github.com/pyo3/maturin/issues/2978">#2978</a>)</li>
<li>Fix license file handling for workspace-level license files (<a
href="https://redirect.github.com/pyo3/maturin/issues/2970">#2970</a>)</li>
<li>Support PEP 739 build-details.json when cross compiling (<a
href="https://redirect.github.com/pyo3/maturin/issues/2979">#2979</a>)</li>
<li>Fix .libs directory name for namespace packages (<a
href="https://redirect.github.com/pyo3/maturin/issues/2981">#2981</a>)</li>
<li>Fix: exclude duplicate python source files from sdist for workspace
members (<a
href="https://redirect.github.com/pyo3/maturin/issues/2982">#2982</a>)</li>
<li>Fix: remove default-members from workspace Cargo.toml in sdist (<a
href="https://redirect.github.com/pyo3/maturin/issues/2983">#2983</a>)</li>
<li>Fix: correctly filter workspace members in sdist by directory path
(<a
href="https://redirect.github.com/pyo3/maturin/issues/2984">#2984</a>)</li>
<li>Feat: Add PEP 770 SBOM support (<a
href="https://redirect.github.com/pyo3/maturin/issues/2980">#2980</a>)</li>
<li>Error when python-source is set but Python module is missing (<a
href="https://redirect.github.com/pyo3/maturin/issues/2986">#2986</a>)</li>
<li>Feat: add auditwheel SBOM for grafted shared libraries (<a
href="https://redirect.github.com/pyo3/maturin/issues/2985">#2985</a>)</li>
<li>Fix sdist duplicate README error when readme is in both Cargo.toml
and pyproject.toml (<a
href="https://redirect.github.com/pyo3/maturin/issues/2987">#2987</a>)</li>
<li>Fix: support python-source pointing outside Rust source directory
(<a
href="https://redirect.github.com/pyo3/maturin/issues/2988">#2988</a>)</li>
<li>Relax ziglang dependency version requirement (<a
href="https://redirect.github.com/pyo3/maturin/issues/2990">#2990</a>)</li>
<li>Stop adding link-native-libraries flag by default in Emscripten
platform in latest Rust (<a
href="https://redirect.github.com/pyo3/maturin/issues/2991">#2991</a>)</li>
<li>Fix docker build github workflow</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/PyO3/maturin/commit/adb2906f90bb6b70722d0546c907b03e545c02f9"><code>adb2906</code></a>
Release v1.12.2</li>
<li><a
href="https://github.com/PyO3/maturin/commit/647356824548faf76cfb30421b21eb415323fe1d"><code>6473568</code></a>
fix: allow absolute paths for <code>--sbom-include</code> (<a
href="https://redirect.github.com/pyo3/maturin/issues/3004">#3004</a>)</li>
<li><a
href="https://github.com/PyO3/maturin/commit/e772489789a7052506d1884ac863b4ac59ffe5da"><code>e772489</code></a>
Release v1.12.1</li>
<li><a
href="https://github.com/PyO3/maturin/commit/0672082797e44ad3e0fd3570cc0b6f406f3680ef"><code>0672082</code></a>
add test case for uniffi with multiple crates (<a
href="https://redirect.github.com/pyo3/maturin/issues/2839">#2839</a>)</li>
<li><a
href="https://github.com/PyO3/maturin/commit/f201566f2a67d18bd4e85baa05e5d249f2be59a7"><code>f201566</code></a>
feat: support including <code>OUT_DIR</code> assets in wheel builds (<a
href="https://redirect.github.com/pyo3/maturin/issues/3001">#3001</a>)</li>
<li><a
href="https://github.com/PyO3/maturin/commit/66b4869413c9e174c3a5a17528f132a1426c7878"><code>66b4869</code></a>
feat: log external shared libraries and their dependents before
patchelf</li>
<li><a
href="https://github.com/PyO3/maturin/commit/8f8f9f7cbd035305328e4e3d7ef9b58cea665a3b"><code>8f8f9f7</code></a>
fix: resolve include patterns relative to python-source for sdist and
wheel (...</li>
<li><a
href="https://github.com/PyO3/maturin/commit/1688a73eec327f3e2fabdeb37e829da2bc2ceff0"><code>1688a73</code></a>
netlify: update mdbook to 0.5.2</li>
<li><a
href="https://github.com/PyO3/maturin/commit/aefe09d14376d88773d3dcf4c4c6a7693bf69e8b"><code>aefe09d</code></a>
Add <code>--sbom-include</code> CLI argument for additional SBOM files
(<a
href="https://redirect.github.com/pyo3/maturin/issues/2999">#2999</a>)</li>
<li><a
href="https://github.com/PyO3/maturin/commit/8a688caf69782bbab1370a0bce03ddbf3ada900e"><code>8a688ca</code></a>
Document SBOM support in user guide</li>
<li>Additional commits viewable in <a
href="https://github.com/pyo3/maturin/compare/v1.11.5...v1.12.2">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=maturin&package-manager=pip&previous-version=1.11.5&new-version=1.12.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-02-17 18:13:48 +00:00
Adrian Garcia Badaracco 7d217b1ec3 Reduce ExtractLeafExpressions optimizer overhead with fast pre-scan (#20341) 
## Summary

Follow-up to #20117 which added the `ExtractLeafExpressions` and
`PushDownLeafProjections` optimizer rules for get_field pushdown.

Benchmarking revealed that these rules added 5-31% overhead on *all*
queries (including those with no struct/get_field expressions) because
they unconditionally allocated column HashSets, extractors, and walked
every expression tree for every Filter/Sort/Limit/Aggregate/Join node.

This PR adds:
- **`has_extractable_expr()` pre-scan**: A lightweight check using
`Expr::exists()` that short-circuits before any expensive allocations
when no `MoveTowardsLeafNodes` expressions are present
- **Config option**
`datafusion.optimizer.enable_leaf_expression_pushdown` to disable the
rules entirely

### Benchmark Results (vs no-rules baseline)

| Benchmark | Before Fix | After Fix |
|---|---|---|
| physical_select_aggregates_from_200 | +31.1% | +3.7% |
| physical_many_self_joins | +12.9% | +2.2% |
| physical_join_consider_sort | +12.9% | +1.0% |
| physical_unnest_to_join | +12.5% | +1.4% |
| physical_select_one_from_700 | +12.2% | +2.6% |
| physical_theta_join_consider_sort | +8.7% | +0.2% |
| physical_plan_tpch_q18 | +9.3% | +1.4% |
| physical_plan_tpch_all | +4.8% | +2.1% |
| physical_plan_tpcds_all | +5.6% | +2.2% |

## Test plan

- [x] All 47 `extract_leaf_expressions` unit tests pass
- [x] Benchmarked with `cargo bench -p datafusion --bench sql_planner`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-13 22:54:44 +00:00
dependabot[bot] ff15500676 chore(deps): bump setuptools from 80.10.2 to 82.0.0 in /docs (#20255) 
Bumps [setuptools](https://github.com/pypa/setuptools) from 80.10.2 to
82.0.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/pypa/setuptools/blob/main/NEWS.rst">setuptools's
changelog</a>.</em></p>
<blockquote>
<h1>v82.0.0</h1>
<h2>Deprecations and Removals</h2>
<ul>
<li><code>pkg_resources</code> has been removed from Setuptools. Most
common uses of <code>pkg_resources</code> have been superseded by the
<code>importlib.resources
&lt;https://docs.python.org/3/library/importlib.resources.html&gt;</code>_
and <code>importlib.metadata
&lt;https://docs.python.org/3/library/importlib.metadata.html&gt;</code>_
projects. Projects and environments relying on
<code>pkg_resources</code> for namespace packages or other behavior
should depend on older versions of <code>setuptools</code>. (<a
href="https://redirect.github.com/pypa/setuptools/issues/3085">#3085</a>)</li>
</ul>
<h1>v81.0.0</h1>
<h2>Deprecations and Removals</h2>
<ul>
<li>Removed support for the --dry-run parameter to setup.py. This one
feature by its nature threads through lots of core and ancillary
functionality, adding complexity and friction. Removal of this parameter
will help decouple the compiler functionality from distutils and thus
the eventual full integration of distutils. These changes do affect some
class and function signatures, so any derivative functionality may
require some compatibility shims to support their expected interface.
Please report any issues to the Setuptools project for investigation.
(<a
href="https://redirect.github.com/pypa/setuptools/issues/4872">#4872</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/pypa/setuptools/commit/03f3615362c4eb19c770b71be5bd58e38f235528"><code>03f3615</code></a>
Bump version: 81.0.0 → 82.0.0</li>
<li><a
href="https://github.com/pypa/setuptools/commit/530d11498af526c4210d8eeaa1ed6c63f44a390f"><code>530d114</code></a>
Merge pull request <a
href="https://redirect.github.com/pypa/setuptools/issues/5007">#5007</a>
from pypa/feature/remove-more-pkg_resources</li>
<li><a
href="https://github.com/pypa/setuptools/commit/11efe9f552290bf536515d458aa85752a0606aa8"><code>11efe9f</code></a>
Merge branch 'maint/75.3'</li>
<li><a
href="https://github.com/pypa/setuptools/commit/118f129dd0fb319058bd05f382c50188fd60a60e"><code>118f129</code></a>
Bump version: 75.3.3 → 75.3.4</li>
<li><a
href="https://github.com/pypa/setuptools/commit/90561ffde1220a590b7644745f48b5837b1a130d"><code>90561ff</code></a>
Merge pull request <a
href="https://redirect.github.com/pypa/setuptools/issues/5150">#5150</a>
from UladzimirTrehubenka/backport_cve_47273</li>
<li><a
href="https://github.com/pypa/setuptools/commit/4595034db8aab4ea33035a47a068b04fd8aa00cc"><code>4595034</code></a>
Add news fragment.</li>
<li><a
href="https://github.com/pypa/setuptools/commit/fc008006fc072af02eb7e0b601172c67eba395e3"><code>fc00800</code></a>
Merge pull request <a
href="https://redirect.github.com/pypa/setuptools/issues/5171">#5171</a>
from cclauss/ruff-v0.15.0</li>
<li><a
href="https://github.com/pypa/setuptools/commit/127e561362a2b4e560faabe9e979ed848106b62d"><code>127e561</code></a>
Remove tests reliant on pkg_resources, rather than xfailing them.</li>
<li><a
href="https://github.com/pypa/setuptools/commit/64bc21e10b5d749b1b75fa334caedb67cc7414c4"><code>64bc21e</code></a>
Reference the superseding libraries.</li>
<li><a
href="https://github.com/pypa/setuptools/commit/cf1ff459ea997b615a75d99304f6c9aa1fc94c06"><code>cf1ff45</code></a>
Merge branch 'main' into debt/pbr-without-pkg_resources</li>
<li>Additional commits viewable in <a
href="https://github.com/pypa/setuptools/compare/v80.10.2...v82.0.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=setuptools&package-manager=pip&previous-version=80.10.2&new-version=82.0.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-02-10 11:37:46 +00:00
Michael Kleen 8fc9681b74 feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations (#19930) 
## Which issue does this PR close?

Cleaned-up version of https://github.com/apache/datafusion/pull/18806
with:

- Removed `outer_queries_schema` from PlannerContext
- Planning logic only (optimizer modifications removed)
- sql logic tests moved to sql_integration.rs

- Closes https://github.com/apache/datafusion/issues/19816


## Rationale for this change

See https://github.com/apache/datafusion/pull/18806

## What changes are included in this PR?

See https://github.com/apache/datafusion/pull/18806

## Are these changes tested?

Yes

## Are there any user-facing changes?

`outer_queries_schema` is removed from PlannerContext.

---------

Co-authored-by: Duong Cong Toai <duongcongtoai.fjc@gmail.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
 2026-02-09 13:51:59 +00:00
mishop-15 15eea084fb Break upgrade guides into separate pages (#20183) 
Fixes #20155

## Which issue does this PR close?

Closes #20155

## Rationale for this change

The upgrade guide was one giant page with all versions. When you search
for something like "Aggregate", you get results from multiple versions
and can't tell if you're looking at v46 or v44 changes.

## What changes are included in this PR?

Split upgrading.md into separate files - one per version. Created an
upgrading/ directory with an index similar to how user-guide/sql/ is
organized.

## Are these changes tested?

Built the docs locally with `make html`. Each version now gets its own
page and ctrl-F works within that version only.

## Are there any user-facing changes?

Yes - upgrade guides are now separate pages instead of one long page.
Makes it easier to find version-specific info.
 2026-02-07 19:26:02 +00:00
Neil Conway 75428f1e0a fix: Avoid integer overflow in split_part() (#20198) 
Along the way, improve the docs slightly.

## Rationale for this change

Evaluating `SELECT SPLIT_PART('', '', -9223372036854775808);` yields (in
a debug build):

```
thread 'main' (41405991) panicked at datafusion/functions/src/string/split_part.rs:236:47:
attempt to negate with overflow
```

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## Are these changes tested?

Yes, added unit test.

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
 2026-02-07 12:46:31 +00:00
kosiew 8c478e9452 Disallow positional struct casting when field names don’t overlap (#19955) 
## Which issue does this PR close?

* Closes #19841.

## Rationale for this change

Struct-to-struct casting previously fell back to **positional mapping**
when there was **no field-name overlap** and the number of fields
matched. That behavior is ambiguous and can silently produce incorrect
results when source/target schemas have different field naming
conventions or ordering.

This PR makes struct casting **strictly name-based**: when there is no
overlap in field names between the source and target structs, the cast
is rejected with a clear planning error. This prevents accidental,
hard-to-detect data corruption and forces callers to provide explicit
field names (or align schemas) when casting.

## What changes are included in this PR?

* Removed the positional fallback logic for struct casting in
`cast_struct_column`; child fields are now resolved **only by name**.
* Updated `validate_struct_compatibility` to **error out** when there is
**no field name overlap**, instead of allowing positional compatibility
checks.
* Updated unit tests to reflect the new behavior (no-overlap casts now
fail with an appropriate error).
* Updated SQLLogicTest files to construct structs using **explicit field
names** (e.g. `{id: 1}` / `{a: 1, b: 'x'}` or `struct(… AS field)`),
avoiding reliance on positional behavior.
* Improved error messaging to explicitly mention the lack of field name
overlap.

## Are these changes tested?

Yes.

* Updated existing Rust unit tests in `nested_struct.rs` to assert the
new failure mode and error message.
* Updated SQLLogicTest coverage (`struct.slt`, `joins.slt`) to use named
struct literals so tests continue to validate struct behavior without
positional casting.

## Are there any user-facing changes?

Yes — behavioral change / potential breaking change.

* Casting between two `STRUCT`s with **no overlapping field names** now
fails (previously it could succeed via positional mapping if field
counts matched).
* Users must provide explicit field names (e.g. `{a: 1, b: 'x'}` or
`struct(expr AS a, expr AS b)`) or ensure schemas share field names.
* Error messages are more explicit: casts are rejected when there is “no
field name overlap”.

## LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed and tested.

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
 2026-02-03 02:22:13 +00:00
Dmitrii Blaginin 96a6bd71ed Runs-on for linux-build-lib and linux-test (2X faster CI) (#20107) 
## Which issue does this PR close?

Related to https://github.com/apache/datafusion/issues/13813

Thanks to the infra team and @gmcdonald specifically, we now have the
ability to use more powerful AWS-provided runners in our CI 🥳

DataFusion has one of the largest runtimes across Apache projects -
that's why we're bringing those runners here first. Since we're first to
test this, I think it's reasonable to do a gradual transition, so I
updated the two most frequently failing actions to be hosted in AWS. The
plan is to test that everything works fine and then transition the
remaining actions.

## What changes are included in this PR?

If the org is `apache`, we'll now use ASF-provisioned runners in the ASF
infra AWS account. Forks will not have access to those runners, so they
will fall back to GitHub-provisioned ones.

## Are these changes tested?

Yes.

- Forks in non-apache orgs:
https://github.com/blaginin/datafusion/pull/10
- Apache org: https://github.com/apache/datafusion-sandbox/pull/163

## Are there any user-facing changes?

No
 2026-02-03 02:07:31 +00:00
karuppuchamysuresh 9de192af29 docs: update data_types.md to reflect current Arrow type mappings (#20072) 
## Which issue does this PR close?
- Closes #18314
## Rationale for this change
The documentation in `data_types.md` was outdated and showed `Utf8` as
the default mapping for character types (CHAR, VARCHAR, TEXT, STRING),
but the current implementation defaults to `Utf8View`. This caused
confusion for users reading the
documentation as it didn't match the actual behavior.
Additionally, the "Supported Arrow Types" section at the end was
redundant since `arrow_typeof` now supports all Arrow types, making the
comprehensive list unnecessary.
## What changes are included in this PR?
1. **Updated Character Types table**: Changed the Arrow DataType column
from `Utf8` to `Utf8View` for CHAR, VARCHAR, TEXT, and STRING types
2. **Added configuration note**: Documented the
`datafusion.sql_parser.map_string_types_to_utf8view` setting that allows
users to switch back to `Utf8` if needed
3. **Removed outdated section**: Deleted the "Supported Arrow Types"
section (39 lines) as it's no longer necessary
## Are these changes tested?
This is a documentation-only change. The documentation accurately
reflects the current behavior of DataFusion:
- The default mapping to `Utf8View` is the current implementation
behavior
- The `datafusion.sql_parser.map_string_types_to_utf8view` configuration
option exists and works as documented
## Are there any user-facing changes?
Yes, documentation changes only. Users will now see accurate information
about:
- The correct default Arrow type mappings for character types
- How to configure the string type mapping behavior if they need the old
`Utf8` behavior

---------

Co-authored-by: Claude (claude-sonnet-4.5) <noreply@anthropic.com>
 2026-01-31 17:50:23 +00:00
Nuno Faria 92f60ad513 docs: Automatically update DataFusion version in docs (#20001) 
## Which issue does this PR close?

- N/A.

## Rationale for this change

I was looking at the download page
(https://datafusion.apache.org/download.html) when I noticed the version
there was not updated in a while. Likewise, there are other places in
the docs where the version is not updated.

This PR changes the `update_datafusion_versions.py` script to
automatically update those files, in addition to the Cargo files.

## What changes are included in this PR?

- Updated `update_datafusion_versions.py` to check some doc files.
- Updated the doc files to the current major version.

## Are these changes tested?

Yes, script runs correctly.

## Are there any user-facing changes?

No.

---------

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
 2026-01-29 03:02:37 +00:00
Kumar Ujjawal 36c0cda206 fix: respect DataFrameWriteOptions::with_single_file_output for paths without extensions (#19931) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes #13323.

## Rationale for this change

When using `DataFrameWriteOptions::with_single_file_output(true)`, the
setting was being ignored if the output path didn't have a file
extension. For example:

```rust
df.write_parquet("/path/to/output", 
    DataFrameWriteOptions::new().with_single_file_output(true), 
    None).await?;
```
Would create a directory /path/to/output/ with files inside instead of a
single file at /path/to/output.

This happened because the demuxer used a heuristic based solely on file
extension, ignoring the explicit user setting.

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?

- New FileOutputMode enum: Uses explicit modes (Automatic, SingleFile,
Directory) in FileSinkConfig for clearer output path handling.
- The demuxer now uses the user's explicit setting instead of always
relying on extension-based heuristics.

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?

- New unit test test_single_file_output_without_extension tests the
fixed behavior
- All sqllogictest pass

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

Breaking for direct FileSinkConfig construction: The struct now requires
file_output_mode: FileOutputMode field. Use FileOutputMode::Automatic to
preserve existing behavior.

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
 2026-01-28 20:57:37 +00:00
Andrew Lamb 0a68b81ade [main] Update version to 52.1.0 (#19878) (#20028) 
## Which issue does this PR close?

- part of https://github.com/apache/datafusion/issues/19784 

## Rationale for this change

Forward port changes from branch-52 to main
## What changes are included in this PR?

Update release version to 52.1.0 and add changelo by cherry-picking
- 9f3ddcecd6033a9d55161175d5dbe29697a9a922

## Are these changes tested?

By CI
## Are there any user-facing changes?

New version
 2026-01-27 21:51:24 +00:00
Tim Saucer 41d48b3f6c feat: implement protobuf converter trait to allow control over serialization and deserialization processes (#19437) 
## Which issue does this PR close?

- Closes https://github.com/apache/datafusion/issues/18477

## Rationale for this change

This PR adds a new trait for converting to and from Protobuf objects and
Physical expressions and plans.

## What changes are included in this PR?

- Add `PhysicalExtensionProtoCodec` and default implementation.
- Update all methods in the physical encoding/decoding methods to use
this trait.
- Added two examples
- Added unit test

## Are these changes tested?

Two examples and round trip unit test are added.

## Are there any user-facing changes?

If users are going through the recommended interfaces in the
documentation, `logical_plan_to_bytes` and `logical_plan_from_bytes`
they will have no user facing change. If they are instead calling into
the inner methods `PhysicalPlanNode::try_from_physical_plan` and so on,
then they will need to provide a proto converter. A default
implementation is provided.

---------

Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
 2026-01-27 18:09:59 +00:00
dependabot[bot] 52deee513f chore(deps): bump setuptools from 80.10.1 to 80.10.2 in /docs (#20022) 
Bumps [setuptools](https://github.com/pypa/setuptools) from 80.10.1 to
80.10.2.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/pypa/setuptools/blob/main/NEWS.rst">setuptools's
changelog</a>.</em></p>
<blockquote>
<h1>v80.10.2</h1>
<h2>Bugfixes</h2>
<ul>
<li>Update vendored dependencies. (<a
href="https://redirect.github.com/pypa/setuptools/issues/5159">#5159</a>)</li>
</ul>
<h2>Misc</h2>
<ul>
<li><a
href="https://redirect.github.com/pypa/setuptools/issues/5115">#5115</a>,
<a
href="https://redirect.github.com/pypa/setuptools/issues/5128">#5128</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/pypa/setuptools/commit/5cf2d085186f2c8053940076db99045b826ec22a"><code>5cf2d08</code></a>
Bump version: 80.10.1 → 80.10.2</li>
<li><a
href="https://github.com/pypa/setuptools/commit/852cd5e9fa507c91f5f6683425f157649715d268"><code>852cd5e</code></a>
Merge pull request <a
href="https://redirect.github.com/pypa/setuptools/issues/5166">#5166</a>
from pypa/bugfix/5159-vendor-bin-free</li>
<li><a
href="https://github.com/pypa/setuptools/commit/11115ee8e5b533c2cd948272b02f339f23b6d20a"><code>11115ee</code></a>
Suppress deprecation warning.</li>
<li><a
href="https://github.com/pypa/setuptools/commit/5cf9185dc8f2b3fbf140ebf6558798ccc0ce1077"><code>5cf9185</code></a>
Update vendored dependencies.</li>
<li><a
href="https://github.com/pypa/setuptools/commit/cf59f41400c75326d381f2d1989027b229b59a59"><code>cf59f41</code></a>
Delete all binaries generated by vendored package install.</li>
<li><a
href="https://github.com/pypa/setuptools/commit/89a598167c614ebaf7da441389bce35534b7cd7f"><code>89a5981</code></a>
Add missing newsfragments</li>
<li><a
href="https://github.com/pypa/setuptools/commit/c0114af5484625c25e48cd85429445f9d6a1cfc0"><code>c0114af</code></a>
Postpone deprecation warnings related to PEP 639 to 2027-Feb-18 (<a
href="https://redirect.github.com/pypa/setuptools/issues/5115">#5115</a>)</li>
<li><a
href="https://github.com/pypa/setuptools/commit/de076038f164a3629c91e3f2bc88a7b9c4f5312d"><code>de07603</code></a>
Revert &quot;[CI] Constraint transient test dependency on pyobjc&quot;
(<a
href="https://redirect.github.com/pypa/setuptools/issues/5128">#5128</a>)</li>
<li><a
href="https://github.com/pypa/setuptools/commit/3afd5d66606c092131052e982266b322f0a0dd4b"><code>3afd5d6</code></a>
Revert &quot;[CI] Constraint transient test dependency on
pyobjc&quot;</li>
<li>See full diff in <a
href="https://github.com/pypa/setuptools/compare/v80.10.1...v80.10.2">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=setuptools&package-manager=pip&previous-version=80.10.1&new-version=80.10.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-01-27 08:44:24 +00:00
Nuno Faria 073174b034 feat: Show the number of matched Parquet pages in DataSourceExec (#19977) 
## Which issue does this PR close?

- Closes #19875.

## Rationale for this change

Show the number of matched (and pruned) pages in the explain analyze
plan to help make decisions about file optimization.

Example:

```sql
DataSourceExec: ..., metrics=[
  ...
  page_index_rows_pruned=1.00 K total → 100 matched,
  page_index_pages_pruned=100 total → 10 matched,
  ...
]
```

## What changes are included in this PR?

- Added `page_index_pages_pruned` metric to DataSourceExec.
- Updated and extended existing tests.

## Are these changes tested?

Yes.

## Are there any user-facing changes?

New metric in the explain plans.
 2026-01-27 06:03:41 +00:00
Nuno Faria adddd4c32b fix: Make generate_series return an empty set with invalid ranges (#19999) 
## Which issue does this PR close?

- Closes #19998.

## Rationale for this change

Make the `generate_series` table function follow the Postgres's
convention.

## What changes are included in this PR?

- Removed the error when the range is invalid.
- Updated existing tests.

## Are these changes tested?

Yes.

## Are there any user-facing changes?

`generate_series`/`range` return an empty set with invalid ranges.
 2026-01-27 05:21:36 +00:00
cht42 8efc2b605e feat(spark): add base64 and unbase64 functions (#19968) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes #19967
- Part of #15914 

## Rationale for this change

Add spark compatible base64/unbase64 functions

## What changes are included in this PR?

- new encoding mode in DF encoding UDF for padded base64
- spark udfs for base64/unbase64

## Are these changes tested?

yes in SLT

## Are there any user-facing changes?

yes
 2026-01-26 16:36:42 +00:00
Andrew Lamb f8a22a51bf docs: Fix some broken / missing links in the DataFusion documentation (#19958) 
## Which issue does this PR close?



## Rationale for this change

While reading the DataFusion documentation, I noticed a broken link
here:



<img width="787" height="885" alt="Screenshot 2026-01-23 at 9 34 43 AM"
src="https://github.com/user-attachments/assets/9013151d-599b-4973-968e-0033c6614643"
/>


## What changes are included in this PR?

Fix several broken and missing links (this was done by codex and I
reviewed the PR)

## Are these changes tested?

Yes, by CI
## Are there any user-facing changes?
Better docs. No functional changes intended
 2026-01-26 12:37:27 +00:00
Nuno Faria 4d63f8c927 minor: Add favicon (#20000) 
## Which issue does this PR close?

- N/A.

## Rationale for this change

There is no favicon currently set for the DataFusion site.

For example, when searching on Google:
<img width="228" height="139" alt="image"
src="https://github.com/user-attachments/assets/c0d2cd43-26fb-4715-a10c-0e60b7853a0a"
/>

When a tab is opened:
<img width="257" height="42" alt="image"
src="https://github.com/user-attachments/assets/4deb74b1-f29f-4bb5-a18b-d634e8ac0b16"
/>

Adding the logo makes it look nicer.


## What changes are included in this PR?

Added favicon.

## Are these changes tested?

Manual checks.

## Are there any user-facing changes?

No.
 2026-01-26 05:19:56 +00:00
Andrew Lamb d1eea0755a Fix broken links in the documentation (#19964) 
## Which issue does this PR close?
- Similarly to https://github.com/apache/datafusion/pull/19958

## Rationale for this change
I found some broken links in our docs using codex
]
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?
Update broken links (will comment inline) 

## Are these changes tested?

By CI

## Are there any user-facing changes?

Better docs / fewer broken links

---------

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
 2026-01-25 22:02:46 +00:00
Nuno Faria e062ae5053 minor: Fix doc about write_batch_size (#19979) 
## Which issue does this PR close?

- N/A.

## Rationale for this change

In the docs, `write_batch_size` was defined as representing in bytes,
while it actually represents number of rows.

## What changes are included in this PR?

- Updates documentation about `write_batch_size`.

## Are these changes tested?

Yes.

## Are there any user-facing changes?

No.
 2026-01-25 22:01:27 +00:00
dependabot[bot] 0b60c588be chore(deps): bump setuptools from 80.9.0 to 80.10.1 in /docs (#19988) 
Bumps [setuptools](https://github.com/pypa/setuptools) from 80.9.0 to
80.10.1.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/pypa/setuptools/blob/main/NEWS.rst">setuptools's
changelog</a>.</em></p>
<blockquote>
<h1>v80.10.1</h1>
<h2>Misc</h2>
<ul>
<li><a
href="https://redirect.github.com/pypa/setuptools/issues/5152">#5152</a></li>
</ul>
<h1>v80.10.0</h1>
<h2>Features</h2>
<ul>
<li>Remove post-release tags on setuptools' own build. (<a
href="https://redirect.github.com/pypa/setuptools/issues/4530">#4530</a>)</li>
<li>Refreshed vendored dependencies. (<a
href="https://redirect.github.com/pypa/setuptools/issues/5139">#5139</a>)</li>
</ul>
<h2>Misc</h2>
<ul>
<li><a
href="https://redirect.github.com/pypa/setuptools/issues/5033">#5033</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/pypa/setuptools/commit/adfb0c9e3d1789587d609228d9ea1d79272e4107"><code>adfb0c9</code></a>
Bump version: 80.10.0 → 80.10.1</li>
<li><a
href="https://github.com/pypa/setuptools/commit/8535d107c2ff20e8e4a0aca2d780461918f54180"><code>8535d10</code></a>
docs: Link pyproject.toml to ext_modules (<a
href="https://redirect.github.com/pypa/setuptools/issues/5125">#5125</a>)</li>
<li><a
href="https://github.com/pypa/setuptools/commit/fafbe2c6566a9562300046b088ceb71efa9eb07f"><code>fafbe2c</code></a>
[CI] Workaround for GHA handling of 'skipped' in job dependency chain
(<a
href="https://redirect.github.com/pypa/setuptools/issues/5152">#5152</a>)</li>
<li><a
href="https://github.com/pypa/setuptools/commit/d171023e5b023bbe2ce8e29e7ae3314c01925783"><code>d171023</code></a>
Add news fragment</li>
<li><a
href="https://github.com/pypa/setuptools/commit/3dbba0672ad44d1b985ef47ebd098d10bee8e1d0"><code>3dbba06</code></a>
Refine comment to reference issue</li>
<li><a
href="https://github.com/pypa/setuptools/commit/e4922c88a5ebe7d7ca40a0abfaa59e1377372bf2"><code>e4922c8</code></a>
Apply suggestion from <a
href="https://github.com/webknjaz"><code>@​webknjaz</code></a></li>
<li><a
href="https://github.com/pypa/setuptools/commit/218c146ba37dabb9513f53510985dd6c3758dd23"><code>218c146</code></a>
[CI] Workaround for GHA handling of 'skipped' in job dependency
chain</li>
<li><a
href="https://github.com/pypa/setuptools/commit/29031718a55e5c7d5bbfc572b84d35d1f1f52aff"><code>2903171</code></a>
Bump version: 80.9.0 → 80.10.0</li>
<li><a
href="https://github.com/pypa/setuptools/commit/23a2b180ef81e6cda7fe55c14cdfca6385e8903e"><code>23a2b18</code></a>
[CI] Allow the action <code>check-changed-folders</code> to be skipped
in the <code>check</code> ac...</li>
<li><a
href="https://github.com/pypa/setuptools/commit/660e5817c2b7631494adb2e044e17fcf59f683fc"><code>660e581</code></a>
[CI] Allow the action <code>check-changed-folders</code> to be skipped
in the <code>check</code> ac...</li>
<li>Additional commits viewable in <a
href="https://github.com/pypa/setuptools/compare/v80.9.0...v80.10.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=setuptools&package-manager=pip&previous-version=80.9.0&new-version=80.10.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-01-25 09:29:33 +00:00
kosiew 0c82adec71 Fix struct casts to align fields by name (prevent positional mis-casts) (#19674) 
## Which issue does this PR close?

* Closes #17285.

## Rationale for this change

DataFusion’s struct casting and some coercion paths were effectively
positional: when two structs had the same field types but different
field *orders*, casting could silently swap values. This is surprising
to users and can lead to silent data corruption (e.g. `{b: 3, a:
4}::STRUCT(a INT, b INT)` yielding `{a: 3, b: 4}`).

The goal of this PR is to make struct casting behavior match user
expectations by matching fields by **name** (case-sensitive) and
recursively applying the same logic to nested structs, while keeping a
compatible fallback for structs with **no** shared field names.

## What changes are included in this PR?

* **Name-based struct casting implementation** in
`datafusion_common::nested_struct`:

* Match struct fields by **name**, reorder to match target schema,
recursively cast nested structs.
  * Fill **missing target fields** with null arrays.
  * Ignore **extra source fields**.
* **Positional mapping fallback** when there is *no name overlap*
**and** field counts match (avoids breaking `struct(1, 'x')::STRUCT(a
INT, b VARCHAR)` style casts).
* Improved handling for **NULL / all-null struct inputs** by producing a
correctly typed null struct array.
* Centralized validation via `validate_field_compatibility` and helper
`fields_have_name_overlap`.

* **Ensure struct casting paths use the name-based logic**:

* `ScalarValue::cast_to_with_options`: route `Struct` casts through
`nested_struct::cast_column`.
* `ColumnarValue::cast_to`: for `Struct` targets, cast via
`nested_struct::cast_column`; non-struct casts still use Arrow’s
standard casting.

* **Type coercion improvements for structs in binary operators / CASE**:

* When two structs have at least one overlapping name, coerce **by
name**.
  * Otherwise, preserve prior behavior by coercing **positionally**.

* **Planning-time cast validation for struct-to-struct**:

* `physical-expr` CAST planning now validates struct compatibility using
the same rules as runtime (`validate_struct_compatibility`) to fail
fast.
* `ExprSchemable` allows struct-to-struct casts to pass type checking;
detailed compatibility is enforced by the runtime / planning-time
validator.

* **Optimizer safety**:

  * Avoid const-folding struct casts when field counts differ.
* Avoid const-folding casts of **0-row** struct literals due to
evaluation batch dimension mismatches.

* **Tests and SQL logic tests**:

  * New unit tests covering:

    * name-based reordering
    * missing fields (nullable vs non-nullable)
    * null struct fields and nested nulls
    * positional fallback with no overlap
    * coercion behavior and simplifier behavior
* Updated/added `.slt` cases to reflect the new semantics and to add
coverage for struct casts and nested struct reordering.

* **Minor docs/maintenance**:

* Adjusted doc comment referencing `ParquetWriterOptions` so it doesn’t
break when the `parquet` feature is disabled.

## Are these changes tested?

Yes.

* Added/updated Rust unit tests in:

  * `datafusion/common/src/nested_struct.rs`
  * `datafusion/expr-common/src/columnar_value.rs`
  * `datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs`
* Added/updated SQL logic tests in:

  * `datafusion/sqllogictest/test_files/case.slt`
  * `datafusion/sqllogictest/test_files/struct.slt`

These tests cover:

* correct value mapping when struct field order differs
* nested struct reordering
* insertion of nulls for missing nullable fields
* erroring on missing non-nullable target fields
* positional mapping fallback when there is no name overlap
* planning-time validation vs runtime behavior alignment

## Are there any user-facing changes?

Yes.

* **Struct casts are now name-based** (case-sensitive): fields are
matched by name, reordered to the target schema, missing fields are
null-filled (if nullable), and extra fields are ignored.
* **Fallback behavior**: if there is *no* name overlap and field counts
match, casting proceeds **positionally**.
* **Potential behavior change** in queries relying on the prior
positional behavior when structs shared names but were out of order
(previously could yield swapped values). This PR changes that to the
safer, expected behavior.

No public API changes are introduced, but this is a semantic change in
struct casting.

## LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed and tested.
 2026-01-23 04:05:34 +00:00
dependabot[bot] 84e1ccc0d3 chore(deps): bump sphinx from 8.2.3 to 9.1.0 in /docs (#19647) 
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 8.2.3 to
9.1.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/sphinx-doc/sphinx/releases">sphinx's
releases</a>.</em></p>
<blockquote>
<h2>Sphinx 9.1.0</h2>
<p>Changelog: <a
href="https://www.sphinx-doc.org/en/master/changes.html">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Dependencies</h2>
<ul>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14153">#14153</a>:
Drop Python 3.11 support.</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/12555">#12555</a>:
Drop Docutils 0.20 support.
Patch by Adam Turner</li>
</ul>
<h2>Features added</h2>
<ul>
<li>Add <code>add_static_dir()</code> for copying static
assets from extensions to the build output.
Patch by Jared Dillard</li>
</ul>
<h2>Bugs fixed</h2>
<ul>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14189">#14189</a>:
autodoc: Fix duplicate <code>:no-index-entry:</code> for modules.
Patch by Adam Turner</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/13713">#13713</a>:
Fix compatibility with MyST-Parser.
Patch by Adam Turner</li>
<li>Fix tests for Python 3.15.
Patch by Adam Turner</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14089">#14089</a>:
autodoc: Fix default option parsing.
Patch by Adam Turner</li>
<li>Remove incorrect static typing assertions.
Patch by Adam Turner</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14050">#14050</a>:
LaTeXTranslator fails to build documents using the &quot;acronym&quot;
standard role.
Patch by Günter Milde</li>
<li>LaTeX: Fix rendering for grid filled merged vertical cell.
Patch by Tim Nordell</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14228">#14228</a>:
LaTeX: Fix overrun footer for cases of merged vertical table cells.
Patch by Tim Nordell</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14207">#14207</a>:
Fix creating <code>HTMLThemeFactory</code> objects in third-party
extensions.
Patch by Adam Turner</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/3099">#3099</a>:
LaTeX: PDF build crashes if a code-block contains more than
circa 1350 codelines (about 27 a4-sized pages at default pointsize).
Patch by Jean-François B.</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14064">#14064</a>:
LaTeX: TABs ending up in sphinxVerbatim fail to obey tab stops.
Patch by Jean-François B.</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14089">#14089</a>:
autodoc: Improve support for non-weakreferencable objects.
Patch by Adam Turner</li>
<li>LaTeX: Fix accidental removal at <code>3.5.0</code> (<a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/8854">#8854</a>)
of the documentation of
<code>literalblockcappos</code> key of  sphinxsetup.
Patch by Jean-François B.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/sphinx-doc/sphinx/blob/master/CHANGES.rst">sphinx's
changelog</a>.</em></p>
<blockquote>
<h1>Release 9.1.0 (released Dec 31, 2025)</h1>
<h2>Dependencies</h2>
<ul>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14153">#14153</a>:
Drop Python 3.11 support.</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/12555">#12555</a>:
Drop Docutils 0.20 support.
Patch by Adam Turner</li>
</ul>
<h2>Features added</h2>
<ul>
<li>Add :meth:<code>~sphinx.application.Sphinx.add_static_dir</code> for
copying static
assets from extensions to the build output.
Patch by Jared Dillard</li>
</ul>
<h2>Bugs fixed</h2>
<ul>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14189">#14189</a>:
autodoc: Fix duplicate <code>:no-index-entry:</code> for modules.
Patch by Adam Turner</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/13713">#13713</a>:
Fix compatibility with MyST-Parser.
Patch by Adam Turner</li>
<li>Fix tests for Python 3.15.
Patch by Adam Turner</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14089">#14089</a>:
autodoc: Fix default option parsing.
Patch by Adam Turner</li>
<li>Remove incorrect static typing assertions.
Patch by Adam Turner</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14050">#14050</a>:
LaTeXTranslator fails to build documents using the &quot;acronym&quot;
standard role.
Patch by Günter Milde</li>
<li>LaTeX: Fix rendering for grid filled merged vertical cell.
Patch by Tim Nordell</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14228">#14228</a>:
LaTeX: Fix overrun footer for cases of merged vertical table cells.
Patch by Tim Nordell</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14207">#14207</a>:
Fix creating <code>HTMLThemeFactory</code> objects in third-party
extensions.
Patch by Adam Turner</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/3099">#3099</a>:
LaTeX: PDF build crashes if a code-block contains more than
circa 1350 codelines (about 27 a4-sized pages at default pointsize).
Patch by Jean-François B.</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14064">#14064</a>:
LaTeX: TABs ending up in sphinxVerbatim fail to obey tab stops.
Patch by Jean-François B.</li>
<li><a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14089">#14089</a>:
autodoc: Improve support for non-weakreferencable objects.
Patch by Adam Turner</li>
<li>LaTeX: Fix accidental removal at <code>3.5.0</code> (<a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/8854">#8854</a>)
of the documentation of
<code>literalblockcappos</code> key of :ref:<code>'sphinxsetup'
&lt;latexsphinxsetup&gt;</code>.
Patch by Jean-François B.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/sphinx-doc/sphinx/commit/cc7c6f435ad37bb12264f8118c8461b230e6830c"><code>cc7c6f4</code></a>
Bump to 9.1.0 final</li>
<li><a
href="https://github.com/sphinx-doc/sphinx/commit/b127b9478aa4654a82eeadf1a1e89715d3927608"><code>b127b94</code></a>
Add <code>app.add_static_dir()</code> for copying extension static files
(<a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14219">#14219</a>)</li>
<li><a
href="https://github.com/sphinx-doc/sphinx/commit/20f1c46790d370b50fa7396cca3e1cc658ce9f89"><code>20f1c46</code></a>
LaTeX: Inhibit breaks for rows with merged vertical cells (<a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14227">#14227</a>)</li>
<li><a
href="https://github.com/sphinx-doc/sphinx/commit/3c85411fd06a1c3026f7991818312e5358ef52e5"><code>3c85411</code></a>
Polish CHANGES.rst (<a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14225">#14225</a>)</li>
<li><a
href="https://github.com/sphinx-doc/sphinx/commit/9ee5446c6cfb997a6b92f5cfb84d045ec947417a"><code>9ee5446</code></a>
LaTeX: restore 1.7 documentation of literalblockcappos (<a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14224">#14224</a>)</li>
<li><a
href="https://github.com/sphinx-doc/sphinx/commit/d75d602005be7567abc7741ee777f6f8c302d827"><code>d75d602</code></a>
LaTeX: improve (again...) some code comments in time for 9.1.0 (<a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14222">#14222</a>)</li>
<li><a
href="https://github.com/sphinx-doc/sphinx/commit/8dca61d69a4a05c56702980e4f6cbe6451dd9ebc"><code>8dca61d</code></a>
Improve some LaTeX code comments (<a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14220">#14220</a>)</li>
<li><a
href="https://github.com/sphinx-doc/sphinx/commit/8ab960098a8e12b8893bd3ceafc394759118346b"><code>8ab9600</code></a>
Bump to 9.1.0 candidate 2</li>
<li><a
href="https://github.com/sphinx-doc/sphinx/commit/d59b2375945ed04414a11b0adb3e5655525f3e9e"><code>d59b237</code></a>
autodoc: Improve support for non-weakreferencable objects</li>
<li><a
href="https://github.com/sphinx-doc/sphinx/commit/964424b3dbc92ed1718272fd45123878f3eec14d"><code>964424b</code></a>
Use the correct reference for using existing extensions (<a
href="https://redirect.github.com/sphinx-doc/sphinx/issues/14157">#14157</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/sphinx-doc/sphinx/compare/v8.2.3...v9.1.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=sphinx&package-manager=pip&previous-version=8.2.3&new-version=9.1.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
 2026-01-20 13:33:21 +00:00
Adrian Garcia Badaracco 936f959ddf Add FilterExecBuilder to avoid recomputing properties multiple times (#19854) 
- Closes #19608, 
- replaces #19619

---------

Co-authored-by: Ganesh Patil <7030871503ganeshpatil@gmail.com>
 2026-01-18 17:09:43 +00:00
Jeffrey Vo ac67ae4af5 Ensure null inputs to array setop functions return null output (#19683) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes #19682

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

Explained in issue.

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

Change array_except, array_intersect and array_union UDFs to return null
if either input is null.

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

Added & fixed tests.

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

Behaviour change to a function output.

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
 2026-01-16 12:28:20 +00:00
Andrew Lamb c91dcfdb6a Docs: add additional links to blog posts (#19833) 
## Which issue does this PR close?

- Part of #7013 

## Rationale for this change

We have written some good blogs recently that provide additional context
and backstory. Let's make sure they are available for others to read

## What changes are included in this PR?

Add links to select doc pages
## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
 2026-01-16 12:26:02 +00:00
cht42 f3f6dec7c9 feat: Add support for 'isoyear' in date_part function (#19821) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes #19820.

## Rationale for this change

`isoyear` part is available in both PG and Spark EXTRACT functions. 


https://www.postgresql.org/docs/current/functions-datetime.html#:~:text=the%20week%20numbering.-,isoyear,-The%20ISO%208601

https://github.com/apache/spark/blob/a03bedb6c1281c5263a42bfd20608d2ee005ab05/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L3360

## What changes are included in this PR?

Support for part `isoyear` in date_part function.

## Are these changes tested?
yes in SLT

## Are there any user-facing changes?

yes
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
 2026-01-16 12:22:58 +00:00
xudong.w ca904b3086 Row group limit pruning for row groups that entirely match predicates (#18868) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Part of https://github.com/apache/datafusion/issues/18860

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

See
https://github.com/apache/datafusion/issues/18860#issuecomment-3563442093

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

1. How to decide if we can do limit pruning without messing up the sql
semantics.
2. Add logic to decide if a row group is fully matched, all rows in the
row group are matched the predicated.
3. Use the fully matched row groups to return limit rows.

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
3. Serve as another way to document the expected behavior of the code


If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

Yes

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

No, no new configs, or API change
 2026-01-16 06:51:06 +00:00
Miao 5edda9b309 fix: calculate total seconds from interval fields for extract(epoch) (#19807) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes #19799.

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
The epoch function incorrectly used `date_part(array, DatePart::Second)`
for intervals which only extracts the seconds component of the interval
structure (0 for "15 minutes"), not the total seconds from all
components.

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

- Modified epoch() function in date_part.rs to properly handle interval
types by extracting struct fields and calculating total seconds
  - Added regression tests in date_part.slt

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

Yes. Added tests verifying correct conversion for all interval types and
precision levels.

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
Yes. `extract(epoch from interval)` now returns correct total seconds
instead of 0.

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
 2026-01-16 04:00:42 +00:00
Andrew Lamb de40f0c9f5 Docs: Fix some links in docs (#19834) 
Small doc tweaks
 2026-01-15 22:50:28 +00:00
Yongting You 567ba75840 doc: Add an auto-generated dependency graph for internal crates (#19280) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes #.

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
A dependency graph for workspace member crates are often needed when
doing refactors, I want it to be included in the doc, and have a script
to update it automatically.

Here is the preview:
<img width="1203" height="951" alt="image"
src="https://github.com/user-attachments/assets/527c18fc-258e-465f-a150-f2aafe3e6db9"
/>


## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
- adds a script to generate the dependency graph `deps.svg`, and verify
if the existing one is up to date.
- adds a documentation page in `Contributor Guide` to show this graph
- adds a CI job to check if the generated dependency graph is up to date
with the code.

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
I tested the dependency graph display locally, see above.
Is it possible to see the preview from this PR's change online?

I also included a dummy crate in the initial commit, to test if the CI
can catch it and throw understandable error message.

## Are there any user-facing changes?
No
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

---------

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
 2026-01-14 02:33:41 +00:00
xudong.w 2fb9fb3fb9 Update the upgrading.md (#19769) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- part of 18566 

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
 2026-01-12 15:48:58 +00:00
xudong.w 383e673ac3 Update 52.0.0 release version number and changelog (#19767) 
The 52.0.0 has released, the PR updates version number and changlog on
main branch
 2026-01-12 15:48:46 +00:00
Kumar Ujjawal d103d8886f chore: remove LZO Parquet compression (#19726) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes #19720.

## Rationale for this change

- Choosing LZO compression errors, I think it might never get supported
so the best option moving forward is to remove it algother and update
the docs.

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?

- Removed LZO from parse_compression_string() function
- Removed docs
- Updated exptected test output

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?

Yes

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

User choosing LZO as compression will get a clear error message:

```
Unknown or unsupported parquet compression: lzo. Valid values are: uncompressed, snappy, gzip(level), brotli(level), lz4, zstd(level), and lz4_raw.
```

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
 2026-01-12 02:31:20 +00:00
dependabot[bot] 84ea07029f chore(deps): bump maturin from 1.10.2 to 1.11.5 in /docs (#19740) 
Bumps [maturin](https://github.com/pyo3/maturin) from 1.10.2 to 1.11.5.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/pyo3/maturin/releases">maturin's
releases</a>.</em></p>
<blockquote>
<h2>v1.11.5</h2>
<h2>1.11.5</h2>
<ul>
<li>Allow combining <code>--compatibility pypi</code> with other
<code>--compatibility</code> values (<a
href="https://redirect.github.com/pyo3/maturin/pull/2928">#2928</a>)</li>
</ul>
<h2>v1.11.4</h2>
<h2>1.11.4</h2>
<ul>
<li>Support armv6l and armv7l in pypi compatibility (<a
href="https://redirect.github.com/pyo3/maturin/pull/2926">#2926</a>)</li>
<li>Improve the reliability of maturin's own CI</li>
</ul>
<h2>v1.11.2</h2>
<h2>What's Changed</h2>
<p><strong>Important</strong> <code>maturin upload</code> is deprecated
and will be removed in maturin 2.0 (<a
href="https://redirect.github.com/PyO3/maturin/issues/2334">PyO3/maturin#2334</a>)</p>
<ul>
<li>Release 1.11.2 by <a
href="https://github.com/konstin"><code>@​konstin</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2921">PyO3/maturin#2921</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/PyO3/maturin/compare/v1.11.1...v1.11.2">https://github.com/PyO3/maturin/compare/v1.11.1...v1.11.2</a></p>
<h2>v1.11.1</h2>
<h1>Release Notes</h1>
<p><strong>Important</strong> <code>maturin upload</code> is deprecated
and will be removed in maturin 2.0 (<a
href="https://redirect.github.com/PyO3/maturin/issues/2334">PyO3/maturin#2334</a>)</p>
<ul>
<li>Fix compiled artifacts being excluded by source path matching (<a
href="https://redirect.github.com/pyo3/maturin/pull/2910">#2910</a>)</li>
<li>Better error reporting for missing interpreters (<a
href="https://redirect.github.com/pyo3/maturin/pull/2918">#2918</a>)</li>
<li>Ignore unreadable excluded directories (<a
href="https://redirect.github.com/pyo3/maturin/pull/2916">#2916</a>)</li>
</ul>
<h2>v1.11.0</h2>
<p><strong>Important</strong> <code>maturin upload</code> is deprecated
and will be removed in maturin 2.0 (<a
href="https://redirect.github.com/PyO3/maturin/issues/2334">PyO3/maturin#2334</a>)</p>
<p>This release was yanked from PyPI.</p>
<h2>What's Changed</h2>
<ul>
<li>Remove unused code by <a
href="https://github.com/e-nomem"><code>@​e-nomem</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2848">PyO3/maturin#2848</a></li>
<li>Correct tagging for x86_64 iOS simulator wheels. by <a
href="https://github.com/freakboy3742"><code>@​freakboy3742</code></a>
in <a
href="https://redirect.github.com/PyO3/maturin/pull/2851">PyO3/maturin#2851</a></li>
<li>Bump MSRV to 1.85.0 and use Rust 2024 edition by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2850">PyO3/maturin#2850</a></li>
<li>Upgrade goblin to 0.10 by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2853">PyO3/maturin#2853</a></li>
<li>Set entry type when adding to the tar file by <a
href="https://github.com/e-nomem"><code>@​e-nomem</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2859">PyO3/maturin#2859</a></li>
<li>Split up module_writer.rs code for code organization by <a
href="https://github.com/e-nomem"><code>@​e-nomem</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2857">PyO3/maturin#2857</a></li>
<li>Update environment variables for Android cross-compilation support
by <a href="https://github.com/ririv"><code>@​ririv</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2825">PyO3/maturin#2825</a></li>
<li>Upgrade some Rust dependencies by <a
href="https://github.com/messense"><code>@​messense</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2860">PyO3/maturin#2860</a></li>
<li>Swap outer and inner loops in write_python_part() by <a
href="https://github.com/e-nomem"><code>@​e-nomem</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2861">PyO3/maturin#2861</a></li>
<li>Split out convenience methods from ModuleWriter trait by <a
href="https://github.com/e-nomem"><code>@​e-nomem</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2842">PyO3/maturin#2842</a></li>
<li>Update cargo_metadata to 0.20.0 by <a
href="https://github.com/e-nomem"><code>@​e-nomem</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2864">PyO3/maturin#2864</a></li>
<li>Calculate file options for WheelWriter once and cache the result by
<a href="https://github.com/e-nomem"><code>@​e-nomem</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2865">PyO3/maturin#2865</a></li>
<li>fix link to pyo3 config file documentation by <a
href="https://github.com/DetachHead"><code>@​DetachHead</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2869">PyO3/maturin#2869</a></li>
<li>Clean up internal fields of WheelWriter by <a
href="https://github.com/e-nomem"><code>@​e-nomem</code></a> in <a
href="https://redirect.github.com/PyO3/maturin/pull/2870">PyO3/maturin#2870</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/PyO3/maturin/blob/main/Changelog.md">maturin's
changelog</a>.</em></p>
<blockquote>
<h2>1.11.5</h2>
<ul>
<li>Allow combining <code>--compatibility pypi</code> with other
<code>--compatibility</code> values (<a
href="https://redirect.github.com/pyo3/maturin/pull/2928">#2928</a>)</li>
</ul>
<h2>1.11.4</h2>
<ul>
<li>Support armv6l and armv7l in pypi compatibility (<a
href="https://redirect.github.com/pyo3/maturin/pull/2926">#2926</a>)</li>
<li>Improve the reliability of maturin's own CI</li>
</ul>
<h2>1.11.3</h2>
<ul>
<li>Fix manylinux2014 compliance check (<a
href="https://redirect.github.com/pyo3/maturin/pull/2922">#2922</a>)</li>
</ul>
<h2>1.11.2</h2>
<ul>
<li>Fix failed release</li>
</ul>
<h2>1.11.1</h2>
<ul>
<li>Fix compiled artifacts being excluded by source path matching (<a
href="https://redirect.github.com/pyo3/maturin/pull/2910">#2910</a>)</li>
<li>Better error reporting for missing interpreters (<a
href="https://redirect.github.com/pyo3/maturin/pull/2918">#2918</a>)</li>
<li>Ignore unreadable excluded directories (<a
href="https://redirect.github.com/pyo3/maturin/pull/2916">#2916</a>)</li>
</ul>
<h2>[1.11.0] - Yanked</h2>
<p>Note: This release was yanked to a regression: <a
href="https://redirect.github.com/PyO3/maturin/issues/2909">PyO3/maturin#2909</a></p>
<ul>
<li>Refactor <code>ModuleWriter</code> to be easier to implement and
use</li>
<li>Add Android cross compilation support, fix wheel tags for
Android</li>
<li>Update generate-ci to macos-15-intel and add windows arm
support</li>
<li>Deprecate 'upload' and 'publish' CLI commands</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/PyO3/maturin/commit/5d50051c9928a0aabe7d52cf868abf17f7e8f407"><code>5d50051</code></a>
Release 1.11.5 (<a
href="https://redirect.github.com/pyo3/maturin/issues/2930">#2930</a>)</li>
<li><a
href="https://github.com/PyO3/maturin/commit/5b54ce4549fcbef1898c4903ae3a0268609848d5"><code>5b54ce4</code></a>
Allow combining <code>--compatibility pypi</code> with other
<code>--compatibility</code> values (#...</li>
<li><a
href="https://github.com/PyO3/maturin/commit/b80ab1f95443db3d9614cdfa3f147b610e136890"><code>b80ab1f</code></a>
Release 1.11.4 (<a
href="https://redirect.github.com/pyo3/maturin/issues/2927">#2927</a>)</li>
<li><a
href="https://github.com/PyO3/maturin/commit/e5fa2779a13ce814314f0eb4e8900b45a2905051"><code>e5fa277</code></a>
Support armv6l and armv7l in pypi compatibility (<a
href="https://redirect.github.com/pyo3/maturin/issues/2926">#2926</a>)</li>
<li><a
href="https://github.com/PyO3/maturin/commit/ff7f3efa71e9e04bbab8ba8f72cfeb74904926bd"><code>ff7f3ef</code></a>
Use uv instead of sudo pip and pin release bootstrapping version (<a
href="https://redirect.github.com/pyo3/maturin/issues/2925">#2925</a>)</li>
<li><a
href="https://github.com/PyO3/maturin/commit/a4f232de88a7691182fc25f03d98c1cac9bc7142"><code>a4f232d</code></a>
Automate release dry runs (<a
href="https://redirect.github.com/pyo3/maturin/issues/2924">#2924</a>)</li>
<li><a
href="https://github.com/PyO3/maturin/commit/795565daaddd42021149223c773ca9d1e7c4ed99"><code>795565d</code></a>
Release 1.11.3 (<a
href="https://redirect.github.com/pyo3/maturin/issues/2923">#2923</a>)</li>
<li><a
href="https://github.com/PyO3/maturin/commit/67a59dec1545dd917a2728ba9a75f88909602441"><code>67a59de</code></a>
Fix manylinux2014 compliance check (<a
href="https://redirect.github.com/pyo3/maturin/issues/2922">#2922</a>)</li>
<li><a
href="https://github.com/PyO3/maturin/commit/3034dfc13d07402f4e57317706efc960ffab58ad"><code>3034dfc</code></a>
Release 1.11.2 (<a
href="https://redirect.github.com/pyo3/maturin/issues/2921">#2921</a>)</li>
<li><a
href="https://github.com/PyO3/maturin/commit/9a0942aa33fbe8eefcc0744d856e5f1a74b77f62"><code>9a0942a</code></a>
Release 0.11.1 (<a
href="https://redirect.github.com/pyo3/maturin/issues/2920">#2920</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/pyo3/maturin/compare/v1.10.2...v1.11.5">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=maturin&package-manager=pip&previous-version=1.10.2&new-version=1.11.5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-01-11 11:30:42 +00:00
Andrew Lamb 013efb4fe1 docs: Refine Communication documentation to highlight Discord (#19714) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Part of #7013 

## Rationale for this change

I was speaking with someone who wanted to get more involved with
DataFusion recently, and it was not clear from the website that discord
is far more active than Slack. I think it would help new new users to
make this clearer in the documentation to lower the barrier to joining
the communitu


## What changes are included in this PR?


Improve the communication documentation here:
https://datafusion.apache.org/contributor-guide/communication.html and
mention that discord is more active than slack

<img width="1191" height="856" alt="Screenshot 2026-01-09 at 8 54 39 AM"
src="https://github.com/user-attachments/assets/9bbef5d4-645d-45f0-85aa-6b12a2a7eea5"
/>


While I was reviewing the communication documentation, I noticed some
other minor issues that could be improved which I will call out inline

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
 2026-01-11 03:21:26 +00:00
Adam Gutglick 8ba46466d3 docs: Fix two small issues in introduction.md (#19712) 
This is a small maintanence PR to fix a couple of small issues:
1. The `datafusion-tui` isn't used anywhere, and as far as I can tell
its now basically a part of `datafusion-dft` (which is already
referenced).
2. The call for action wasn't rendered into the HTML page. I was there
up until #4903, which included this small mistake which every subsequent
change just maintained.
 2026-01-09 13:39:02 +00:00
UBarney c98fa5616e perfect hash join (#19411) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Closes #17635.

## Rationale for this change

This PR introduces a Perfect Hash Join optimization by using an
array-based direct mapping(`ArrayMap`) instead of a HashMap.
The array-based approach outperforms the standard Hash Join when the
build-side keys are **_dense_** (i.e., the ratio of `count / (max -
min+1)` is high) or when the key range `(max - min)` is sufficiently
**small**.

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

The following results from the hj.rs benchmark suite. The benchmark was
executed with the optimization enabled by setting
`DATAFUSION_EXECUTION_PERFECT_HASH_JOIN_MIN_KEY_DENSITY=0.1`
```

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query                                                  ┃   base_hj ┃ density=0.1 ┃        Change ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1_density=1_prob_hit=1_25*1.5M                  │   5.50 ms │     4.54 ms │ +1.21x faster │
│ QQuery 2_density=0.026_prob_hit=1_25*1.5M              │   6.13 ms │     5.43 ms │ +1.13x faster │
│ QQuery 3_density=1_prob_hit=1_100K*60M                 │ 132.59 ms │    97.42 ms │ +1.36x faster │
│ QQuery 4_density=1_prob_hit=0.1_100K*60M               │ 146.66 ms │    97.75 ms │ +1.50x faster │
│ QQuery 5_density=0.75_prob_hit=1_100K*60M              │ 139.85 ms │   103.82 ms │ +1.35x faster │
│ QQuery 6_density=0.75_prob_hit=0.1_100K*60M            │ 256.62 ms │   192.15 ms │ +1.34x faster │
│ QQuery 7_density=0.5_prob_hit=1_100K*60M               │ 136.27 ms │    91.64 ms │ +1.49x faster │
│ QQuery 8_density=0.5_prob_hit=0.1_100K*60M             │ 234.89 ms │   185.35 ms │ +1.27x faster │
│ QQuery 9_density=0.2_prob_hit=1_100K*60M               │ 132.76 ms │    98.44 ms │ +1.35x faster │
│ QQuery 10_density=0.2_prob_hit=0.1_100K*60M            │ 240.04 ms │   184.93 ms │ +1.30x faster │
│ QQuery 11_density=0.1_prob_hit=1_100K*60M              │ 133.02 ms │   108.11 ms │ +1.23x faster │
│ QQuery 12_density=0.1_prob_hit=0.1_100K*60M            │ 235.44 ms │   209.10 ms │ +1.13x faster │
│ QQuery 13_density=0.01_prob_hit=1_100K*60M             │ 135.64 ms │   132.52 ms │     no change │
│ QQuery 14_density=0.01_prob_hit=0.1_100K*60M           │ 235.88 ms │   234.62 ms │     no change │
│ QQuery 15_density=0.2_prob_hit=0.1_100K_(20%_dups)*60M │ 178.49 ms │   147.55 ms │ +1.21x faster │
└────────────────────────────────────────────────────────┴───────────┴─────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary          ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (base_hj)       │ 2349.79ms │
│ Total Time (density=0.1)   │ 1893.37ms │
│ Average Time (base_hj)     │  156.65ms │
│ Average Time (density=0.1) │  126.22ms │
│ Queries Faster             │        13 │
│ Queries Slower             │         0 │
│ Queries with No Change     │         2 │
│ Queries with Failure       │         0 │
└────────────────────────────┴───────────┘

```


The following results from tpch-sf10
```
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃       base ┃ perfect_hj ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │  739.66 ms │  743.84 ms │     no change │
│ QQuery 2     │  315.94 ms │  317.53 ms │     no change │
│ QQuery 3     │  655.79 ms │  669.24 ms │     no change │
│ QQuery 4     │  215.48 ms │  218.79 ms │     no change │
│ QQuery 5     │ 1131.42 ms │ 1146.03 ms │     no change │
│ QQuery 6     │  202.32 ms │  190.83 ms │ +1.06x faster │
│ QQuery 7     │ 1734.06 ms │ 1710.50 ms │     no change │
│ QQuery 8     │ 1185.05 ms │ 1173.90 ms │     no change │
│ QQuery 9     │ 2036.76 ms │ 1994.30 ms │     no change │
│ QQuery 10    │  907.32 ms │  893.20 ms │     no change │
│ QQuery 11    │  306.63 ms │  275.46 ms │ +1.11x faster │
│ QQuery 12    │  404.00 ms │  381.95 ms │ +1.06x faster │
│ QQuery 13    │  531.67 ms │  498.58 ms │ +1.07x faster │
│ QQuery 14    │  317.63 ms │  303.04 ms │     no change │
│ QQuery 15    │  602.24 ms │  572.18 ms │     no change │
│ QQuery 16    │  200.00 ms │  201.68 ms │     no change │
│ QQuery 17    │ 1848.67 ms │ 1790.60 ms │     no change │
│ QQuery 18    │ 2130.63 ms │ 2179.84 ms │     no change │
│ QQuery 19    │  501.81 ms │  529.85 ms │  1.06x slower │
│ QQuery 20    │  637.91 ms │  661.72 ms │     no change │
│ QQuery 21    │ 1882.43 ms │ 1917.10 ms │     no change │
│ QQuery 22    │  130.68 ms │  141.76 ms │  1.08x slower │
└──────────────┴────────────┴────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary         ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (base)         │ 18618.10ms │
│ Total Time (perfect_hj)   │ 18511.93ms │
│ Average Time (base)       │   846.28ms │
│ Average Time (perfect_hj) │   841.45ms │
│ Queries Faster            │          4 │
│ Queries Slower            │          2 │
│ Queries with No Change    │         16 │
│ Queries with Failure      │          0 │
└───────────────────────────┴────────────┘
```

## What changes are included in this PR?

- During the `collect_left_input` (build) phase, we now conditionally
use an `ArrayMap` instead of a standard `JoinHashMapType`. This
optimization is triggered only when the following conditions are met:
  - There is exactly one join key.
- The join key can be any integer type convertible to u64 (excluding
i128 and u128).
- The key distribution is sufficiently dense or the key range (max -
min) is small enough to justify an array-based allocation.
  - build_side.num_rows() < `u32::MAX`
- The `ArrayMap` works by storing the minimum key as an offset and using
a Vec to directly map a key `k` to its build-side index via `data[k-
offset]`.
- Rewrite Hash Join micro-benchmarks in benchmarks/src/hj.rs to evaluate
ArrayMap and
  HashMap performance across varying key densities and probe hit rates



<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?
Yes
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

Yes, this PR introduces two new session configuration parameters to
control the behavior of the Perfect Hash Join optimization:

- `perfect_hash_join_small_build_threshold`: This parameter defines the
maximum key range (max_key - min_key) for the build side to be
considered "small." If the key range is below this threshold, the
array-based join will be triggered regardless of key density.
- `perfect_hash_join_min_key_density`: This parameter sets the minimum
density (row_count / key_range) required to enable the perfect hash join
optimization for larger key ranges
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
 2026-01-09 10:08:53 +00:00
jizezhang e6049de5a7 Make default ListingFilesCache table scoped (#19616) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Builds on https://github.com/apache/datafusion/pull/19388
- Closes https://github.com/apache/datafusion/issues/19573

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

This PR explores one way to make `ListFilesCache` table scoped. A
session level cache is still used, but the cache key is made a
"table-scoped" path, for which a new struct
```
pub struct TableScopedPath(pub Option<TableReference>, pub Path);
```
is defined. `TableReference` comes from `CreateExternalTable` passed to
`ListingTableFactory::create`.

Additionally, when a table is dropped, all entries related to a table is
dropped by modifying `SessionContext::find_and_deregister` method.

Testing (change on adding `list_files_cache()` for cli is included for
easier testing).
- Testing cache reuse on a single table.
```
> \object_store_profiling summary
ObjectStore Profile mode set to Summary
> create external table test
stored as parquet
location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/';
0 row(s) fetched. 
Elapsed 14.878 seconds.

Object Store Profiling
Instrumented Object Store: instrument_mode: Summary, inner: AmazonS3(overturemaps-us-west-2)
Summaries:
+-----------+----------+-----------+-----------+-------------+-------------+-------+
| Operation | Metric   | min       | max       | avg         | sum         | count |
+-----------+----------+-----------+-----------+-------------+-------------+-------+
| Get       | duration | 0.030597s | 0.209235s | 0.082396s   | 36.254189s  | 440   |
| Get       | size     | 204782 B  | 857230 B  | 497304.88 B | 218814144 B | 440   |
| List      | duration | 0.192037s | 0.192037s | 0.192037s   | 0.192037s   | 1     |
| List      | size     |           |           |             |             | 1     |
+-----------+----------+-----------+-----------+-------------+-------------+-------+
> select table, path, unnest(metadata_list) from list_files_cache() limit 1;
+-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| table | path                            | UNNEST(list_files_cache().metadata_list)                                                                                                                                                                                                                  |
+-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| test  | release/2025-12-17.0/theme=base | {file_path: release/2025-12-17.0/theme=base/type=bathymetry/part-00000-dd0f2f50-b436-4710-996f-f1b06181a3a1-c000.zstd.parquet, file_modified: 2025-12-17T22:32:50, file_size_bytes: 40280159, e_tag: "15090401f8f936c3f83bb498cb99a41d-3", version: NULL} |
+-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched. 
Elapsed 0.058 seconds.

Object Store Profiling
> select count(*) from test where type = 'infrastructure';
+-----------+
| count(*)  |
+-----------+
| 142969564 |
+-----------+
1 row(s) fetched. 
Elapsed 0.028 seconds.

Object Store Profiling
```
- Test separate cache entries are created for two tables with same path
```
> create external table test2
stored as parquet
location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/';
0 row(s) fetched. 
Elapsed 14.798 seconds.

Object Store Profiling
Instrumented Object Store: instrument_mode: Summary, inner: AmazonS3(overturemaps-us-west-2)
Summaries:
+-----------+----------+-----------+-----------+-------------+-------------+-------+
| Operation | Metric   | min       | max       | avg         | sum         | count |
+-----------+----------+-----------+-----------+-------------+-------------+-------+
| Get       | duration | 0.030238s | 0.350465s | 0.073670s   | 32.414654s  | 440   |
| Get       | size     | 204782 B  | 857230 B  | 497304.88 B | 218814144 B | 440   |
| List      | duration | 0.133334s | 0.133334s | 0.133334s   | 0.133334s   | 1     |
| List      | size     |           |           |             |             | 1     |
+-----------+----------+-----------+-----------+-------------+-------------+-------+
> select count(*) from test2 where type = 'bathymetry';
+----------+
| count(*) |
+----------+
| 59963    |
+----------+
1 row(s) fetched. 
Elapsed 0.009 seconds.

Object Store Profiling
> select table, path from list_files_cache();
+-------+---------------------------------+
| table | path                            |
+-------+---------------------------------+
| test  | release/2025-12-17.0/theme=base |
| test2 | release/2025-12-17.0/theme=base |
+-------+---------------------------------+
2 row(s) fetched. 
Elapsed 0.004 seconds.
```
- Test cache associated with a table is dropped when table is dropped,
and the other table with same path is unaffected.
```
> drop table test;
0 row(s) fetched. 
Elapsed 0.015 seconds.

Object Store Profiling
> select table, path from list_files_cache();
+-------+---------------------------------+
| table | path                            |
+-------+---------------------------------+
| test2 | release/2025-12-17.0/theme=base |
+-------+---------------------------------+
1 row(s) fetched. 
Elapsed 0.005 seconds.

Object Store Profiling
> select count(*) from list_files_cache() where table = 'test';
+----------+
| count(*) |
+----------+
| 0        |
+----------+
1 row(s) fetched. 
Elapsed 0.014 seconds.
> select count(*) from test2 where type = 'infrastructure';
+-----------+
| count(*)  |
+-----------+
| 142969564 |
+-----------+
1 row(s) fetched. 
Elapsed 0.013 seconds.

Object Store Profiling
```
- Test that dropping a view does not remove cache
```
> create view test2_view as (select * from test2 where type = 'infrastructure');
0 row(s) fetched. 
Elapsed 0.103 seconds.

Object Store Profiling
> select count(*) from test2_view;
+-----------+
| count(*)  |
+-----------+
| 142969564 |
+-----------+
1 row(s) fetched. 
Elapsed 0.094 seconds.

Object Store Profiling
> drop view test2_view;
0 row(s) fetched. 
Elapsed 0.002 seconds.

Object Store Profiling
> select table, path from list_files_cache();
+-------+---------------------------------+
| table | path                            |
+-------+---------------------------------+
| test2 | release/2025-12-17.0/theme=base |
+-------+---------------------------------+
1 row(s) fetched. 
Elapsed 0.007 seconds.
```
## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
 2026-01-08 14:34:10 +00:00
Kumar Ujjawal 646213ec7c feat: add Time type support to date_trunc function (#19640) 
## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->

- Part of #19025.

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?

- Added Time64/Time32 signatures to date_trunc
- Added time truncation logic (hour, minute, second, millisecond,
microsecond)
- Error for invalid granularities (day, week, month, quarter, year)

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?

Yes

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

---------

Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
 2026-01-08 01:42:17 +00:00