I was having trouble getting benchmarks to gen data.
## Summary
- Replace three independent `requirements.txt` files with a uv workspace
(`benchmarks`, `dev`, `docs` projects)
- Single `uv.lock` lockfile for reproducible dependency resolution
- Simplify `bench.sh` by removing all ad-hoc venv/pip logic in favor of
`uv run`
## Test plan
- [ ] `uv sync` resolves all deps from repo root
- [ ] `uv run --project benchmarks python3 benchmarks/compare.py` works
- [ ] `uv run --project docs sphinx-build docs/source docs/build` builds
docs
- [ ] Run a benchmark from `bench.sh` that uses Python (e.g., h2o data
gen or compare flow)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #https://github.com/apache/datafusion/issues/20024.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
- Closes #.
## Rationale for this change
Add confirmation dialogue to confirm tarball release
## What changes are included in this PR?
Update `tarball-release.sh` to ask `y/N` confirmation before it proceeds
to release upload
## Are these changes tested?
## Are there any user-facing changes?
No
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- part of #https://github.com/apache/datafusion/issues/20024.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close?
- N/A.
## Rationale for this change
I was looking at the download page
(https://datafusion.apache.org/download.html) when I noticed the version
there was not updated in a while. Likewise, there are other places in
the docs where the version is not updated.
This PR changes the `update_datafusion_versions.py` script to
automatically update those files, in addition to the Cargo files.
## What changes are included in this PR?
- Updated `update_datafusion_versions.py` to check some doc files.
- Updated the doc files to the current major version.
## Are these changes tested?
Yes, script runs correctly.
## Are there any user-facing changes?
No.
---------
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
## Which issue does this PR close?
- part of https://github.com/apache/datafusion/issues/19784
## Rationale for this change
Forward port changes from branch-52 to main
## What changes are included in this PR?
Update release version to 52.1.0 and add changelo by cherry-picking
- 9f3ddcecd6033a9d55161175d5dbe29697a9a922
## Are these changes tested?
By CI
## Are there any user-facing changes?
New version
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
Part of https://github.com/apache/datafusion/issues/19227
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
See issue for details.
The existing script `./dev/rust_lint.sh` do checks for all
non-functional tests include formater/clippy checks.
Some check tools support auto fix options, so this PR add an option to
the lint scripts to perform auto-fixes.
Now `./dev/rust_lint.sh --write --allow-dirty` can perform auto-fixes
for all linter etc. violations
```
yongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (auto-fix)> ./dev/rust_lint.sh --help
Usage: ./dev/rust_lint.sh [--write] [--allow-dirty]
Runs the local Rust lint suite similar to CI.
--write Run formatters, clippy and other non-functional checks in best-effort write/fix mode (requires a clean git worktree, no uncommitted changes; some checks are test-only and ignore this flag).
--allow-dirty Allow `--write` to run even when the git worktree has uncommitted changes.
```
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
- Adds `[--write] [--allow-dirty]` flag to `rust_lint.sh` to support
auto fixes
- `rust_lint.sh` consists of several sub-scripts like `rust_fmt.sh`,
they're all extended with auto-fix feature through `--write` flag, and
the `rust_lint.sh` is optionally calling them with an additional flag
for auto fixes.
- Clean up `rust_lint.sh`
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
Yes, commit 8c99417929fcf133423dd3392f1939ab13a0bc93 intentionally
introduced one violation for each available lint check, and the auto-fix
command is able to fix all of them.
The test may not be comprehensive, but it provides a reasonable starting
point. We can begin using this script now and iterate on it if we
discover cases where the auto-fix does not behave correctly.
## Are there any user-facing changes?
No
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
A small step towards https://github.com/apache/datafusion/issues/19227
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
- Move doc prettier check command to a local script
- Add this script to the lint-check runner `dev/rust_lint.sh` to make
local development easier
(Technically it's neither rust nor lint, but keeping all non functional
tests here seem to be the most convenient.)
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
There is a typos check CI job introduced recently. I always end up with
typos in my PRs, so I want an easier way to run this typo check locally.
`dev/rust_lint.rs` is used to perform all lint checks (fmt, clippy)
locally, the typos check should also be suitable to get included.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
1. Added `ci/scripts/typos_check.sh` for local typos check, and include
it into `dev/rust_lint.sh`
2. Update CI to also use `ci/scripts/typos_check.sh` for consistency
3. Moved typo check CI job from `Rust workflow` to `Dev workflow`
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
4. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
Yes, since CI job is changed, I included one typo in this PR, and it
should trigger CI failure.
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close?
This addresses part of https://github.com/apache/datafusion/issues/15804
but does not close it.
## Rationale for this change
Now that we are on MSRV 1.88 we can use rust edition 2024, which brings
let chains and other nice features. It also improves `unsafe` checking.
In order to introduce these changes in slower way instead of one massive
PR that is too difficult to manage we are updating a few crates at a
time.
## What changes are included in this PR?
Updates 6 crates to 2024.
- common
- common-runtime
- doc
- expr-common
- macros
- proto-common
## Are these changes tested?
Existing unit tests. There are no functional code changes.
## Are there any user-facing changes?
None.
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes https://github.com/apache/datafusion/issues/17134
## Rationale for this change
ASF Infra works in tricky way, to apply a protection for the new branch
its needed to get:
- branch created in repo
- add rules to `.asf.yaml` in `main`
- tricky part ASF reapply policies ONLY if `.asf.yaml` changed in PR,
means that "future" branches won't be covered until the file changed and
pushed.
Adding a script to add automatically a block with new RC branch
protection rules and updated documentation
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---------
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
## Which issue does this PR close?
- related to https://github.com/apache/datafusion/issues/17558
## Rationale for this change
I have hit rate limit issues in the past when I didn't set my github
token correctly, and @xudong963 reported something similar here:
https://github.com/apache/datafusion/issues/17558#issuecomment-3531704481
## What changes are included in this PR?
Add a note in the release docs about how to solve the rate limiting
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- part of
https://github.com/apache/datafusion/issues/17558#issuecomment-3531704481
-
## Rationale for this change
Forward port changes from branch-51 to main:
- https://github.com/apache/datafusion/pull/18705
## What changes are included in this PR?
Bring changes from this PR to main
- https://github.com/apache/datafusion/pull/18705
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
Closes#18109.
## Rationale for this change
Previously, the SQL planner accepted `WITHIN GROUP` clauses for all
aggregate UDAFs, even those that did not explicitly support ordered-set
semantics. This behavior was too permissive and inconsistent with
PostgreSQL. For example, queries such as `SUM(x) WITHIN GROUP (ORDER BY
x)` were allowed, even though `SUM` is not an ordered-set aggregate.
This PR enforces stricter validation so that only UDAFs that explicitly
return `true` from `supports_within_group_clause()` may use `WITHIN
GROUP`. All other aggregates now produce a clear planner error when this
syntax is used.
## What changes are included in this PR?
* Added type alias `WithinGroupExtraction` to simplify complex tuple
return types used by helper functions.
* Introduced a new helper method `extract_and_prepend_within_group_args`
to centralize logic for handling `WITHIN GROUP` argument rewriting.
* Updated the planner to:
* Validate that only UDAFs with `supports_within_group_clause()` can
accept `WITHIN GROUP`.
* Prepend `WITHIN GROUP` ordering expressions to function arguments only
for supported ordered-set aggregates.
* Produce clear error messages when `WITHIN GROUP` is used incorrectly.
* Added comprehensive unit tests verifying correct behavior and failure
cases:
* `WITHIN GROUP` rejected for non-ordered-set aggregates (`MIN`, `SUM`,
etc.).
* `WITHIN GROUP` accepted for ordered-set aggregates such as
`percentile_cont`.
* Validation for named arguments, multiple ordering expressions, and
semantic conflicts with `OVER` clauses.
* Updated SQL logic tests (`aggregate.slt`) to reflect new rejection
behavior.
* Updated documentation:
* `aggregate_functions.md` and developer docs to clarify when and how
`WITHIN GROUP` can be used.
* `upgrading.md` to inform users of this stricter enforcement and
migration guidance.
## Are these changes tested?
✅ Yes.
* New tests in `sql_integration.rs` validate acceptance, rejection, and
argument behavior of `WITHIN GROUP` for both valid and invalid cases.
* SQL logic tests (`aggregate.slt`) include negative test cases
confirming planner rejections.
## Are there any user-facing changes?
✅ Yes.
* Users attempting to use `WITHIN GROUP` with regular aggregates (e.g.
`SUM`, `AVG`, `MIN`, `MAX`) will now see a planner error:
> `WITHIN GROUP is only supported for ordered-set aggregate functions`
* Documentation has been updated to clearly describe `WITHIN GROUP`
semantics and provide examples of valid and invalid usage.
No API-breaking changes were introduced; only stricter planner
validation and improved error messaging.
## Which issue does this PR close?
- part of https://github.com/apache/datafusion/issues/17558
## Rationale for this change
- Bring some small changes to changelog to main
## What changes are included in this PR?
Cherry-pick this PR to main:
- https://github.com/apache/datafusion/pull/18592
## Are these changes tested?
By CI
## Are there any user-facing changes?
New changelog content
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
- part of https://github.com/apache/datafusion/issues/17558
## Rationale for this change
As our releases have become more complex, it is important that our
release documentation reflects the current reality.
While working on the 51 release in
https://github.com/apache/datafusion/issues/17558 I found
it was somewhat out of date (didn't refer to the branches, for example)
## What changes are included in this PR?
Update readme
## Are these changes tested?
N/A
## Are there any user-facing changes?
No -- this is internal developer documentation
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Matt Butrovich <mbutrovich@users.noreply.github.com>
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## Rationale for this change
Allow users to understand some decisions for when to change certain
joins configurations.
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
Add readme to joins physical plan
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
- Related to https://github.com/apache/datafusion/issues/18072
## Rationale for this change
Bring 50.3.0 changes to main
## What changes are included in this PR?
- cherry-pick https://github.com/apache/datafusion/pull/18173 into main:
- CHANGELOG
- Version number change
## Are these changes tested?
Yes by CI
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
- This addresses part of
https://github.com/apache/datafusion/issues/17713 but it does not close
it.
## Rationale for this change
In order to remove `core` from `proto` crate, we need `ArrowFormat` to
be available. Similar to the other datasource types (csv, avro, json,
parquet) this splits the Arrow IPC file format into its own crate.
## What changes are included in this PR?
This is a straight refactor. Code is merely moved around.
The size of the diff is the additional files that are required
(cargo.toml, readme.md, etc)
## Are these changes tested?
Existing unit tests.
## Are there any user-facing changes?
Users that include `ArrowSource` may need to update their include paths.
For most, the reexports will cover this need.
* feat: Support `FILTER` clause in aggregate window functions
* fix: Box `WindowFunction` in `ExprFuncKind` enum to reduce enum total size
As suggested by `clippy`:
```
warning: large size difference between variants
--> datafusion/expr/src/expr_fn.rs:772:1
|
772 | / pub enum ExprFuncKind {
773 | | Aggregate(AggregateFunction),
| | ---------------------------- the second-largest variant contains at least 72 bytes
774 | | Window(WindowFunction),
| | ---------------------- the largest variant contains at least 288 bytes
775 | | }
| |_^ the entire enum is at least 288 bytes
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#large_enum_variant
= note: `#[warn(clippy::large_enum_variant)]` on by default
help: consider boxing the large fields to reduce the total size of the enum
|
774 - Window(WindowFunction),
774 + Window(Box<WindowFunction>),
|
```
* test: Add DataFrame API test for FILTER clause on aggregate window functions
* docs: Update aggregate and window function documentation with FILTER support
* docs: Link missing proto fields to github issue in TODO comment
* [branch-49] Update version to `49.0.0`, add changelog (#16822)
* Update version to 49.0.0
* Add 49.0.0 changelog
* prettier
* Update changelog and config
* chore: use `equals_datatype` for `BinaryExpr` (#16813) (#16847)
* chore: use `equals_datatype` instead of direct type comparison for `BinaryExpr`
* chore: use `equals_datatype` instead of direct type comparison for `BinaryExpr`
(cherry picked from commit acff1b6bdd)
* [branch-49] Final Changelog Tweaks (#16852)
---------
Co-authored-by: Oleks V <comphead@users.noreply.github.com>
* Set DataFusion runtime configurations through SQL interface
* fix clippy warnings
* use spill count based tests for checking applied memory limit
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* [branch-46] Update changelog for backports to 46.0.0 (#14977)
* Add note about upgrade guide into the release notes (#14979)
* Add new crates
---------
Co-authored-by: xudong.w <wxd963996380@gmail.com>
Adrian Garcia Badaracco