datafusion

mirror of https://github.com/langchain-ai/datafusion.git synced 2026-08-26 18:06:36 -04:00

Files

T

History

Neil Conway d6fb3608b0 perf: Optimize array_position for scalar needle (#20532 )

## Which issue does this PR close?

- Closes #20530 

## Rationale for this change

The previous implementation of `array_position` used
`compare_element_to_list` for every input row. When the needle is a
scalar (quite common), we can do much better by searching over the
entire flat haystack values array with a single call to
`arrow_ord::cmp::not_distinct`. We can then iterate over the resulting
set bits to determine per-row results.

This is ~5-10x faster than the previous implementation for typical
inputs.

## What changes are included in this PR?

* Implement new fast path for `array_position` with scalar needle
* Improve docs for `array_position`
* Don't use `internal_err` to report a user-visible error

## Are these changes tested?

Yes, and benchmarked. Additional tests added in a separate PR (#20531)

## Are there any user-facing changes?

No.

2026-02-26 18:40:55 +00:00

cli

Make default ListingFilesCache table scoped (#19616 )

2026-01-08 14:34:10 +00:00

sql

perf: Optimize array_position for scalar needle (#20532 )

2026-02-26 18:40:55 +00:00

arrow-introduction.md

docs: Fix some broken / missing links in the DataFusion documentation (#19958 )

2026-01-26 12:37:27 +00:00

concepts-readings-events.md

Fix broken links in the documentation (#19964 )