mirror of
https://github.com/langchain-ai/datafusion.git
synced 2026-07-21 03:05:28 -04:00
Neil Conway
c3f080774c
## Which issue does this PR close? - Closes #20302. ## Rationale for this change `translate()` is commonly invoked with constant values for its second and third arguments. We can take advantage of that to significantly optimize its performance by precomputing the translation lookup table, rather than recomputing it for every row. For ASCII-only inputs, we can further replace the hashmap lookup table with a fixed-size array that maps ASCII byte values directly. For scalar ASCII inputs, this yields roughly a 10x performance improvement. For scalar UTF8 inputs, the performance improvement is more like 50%, although less so for long strings. Along the way, add support for `translate()` on `LargeUtf8` input, along with an SLT test, and improve the docs. ## What changes are included in this PR? * Add a benchmark for scalar/constant input to translate * Add a missing test case * Improve translate() docs * Support translate() on LargeUtf8 input * Optimize translate() for scalar inputs by precomputing lookup hashmap * Optimize translate() for ASCII inputs by precomputing ASCII byte-wise lookup table ## Are these changes tested? Yes. Added an extra test case and did a bunch of benchmarking. ## Are there any user-facing changes? No. --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>