mirror of
https://github.com/langchain-ai/datafusion.git
synced 2026-07-19 23:35:20 -04:00
jpeeter
a774371961
The only code change required, AFAICT, is the calculation of num_dicts, which is no longer simply the number of fields, but rather the unique number of id's they point to. I'm calculating this on-demand, as it's quite cheap and not frequently called, but could also (p)re-compute this on every addField. For now, I've added tests that read materialised data generated from Java, as we don't support writing IPC with shared dictionaries in C++ either yet (and out of scope here). Down the line, I would like full read & write support for shared dictionaries across at least C++, Python, Java and Julia, so I'll be coming back to this with follow-up PR's where needed. As part of that, I'll also change the tests to no longer rely on materialised files, but use the round-trip mechanism. Closes #9629 from jmgpeeters/ARROW-11838 Lead-authored-by: jpeeter <jpeeter@loswjpeeter1.intra.gsacapital.com> Co-authored-by: jmgpeeters <joris.mg.peeters@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
37 lines
1.2 KiB
Bash
Executable File
37 lines
1.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
set -ex
|
|
|
|
arrow_dir=${1}
|
|
source_dir=${1}/cpp
|
|
build_dir=${2}/cpp
|
|
|
|
gold_dir=$arrow_dir/testing/data/arrow-ipc-stream/integration
|
|
|
|
pip install -e $arrow_dir/dev/archery
|
|
|
|
archery integration --with-all --run-flight \
|
|
--gold-dirs=$gold_dir/0.14.1 \
|
|
--gold-dirs=$gold_dir/0.17.1 \
|
|
--gold-dirs=$gold_dir/1.0.0-bigendian \
|
|
--gold-dirs=$gold_dir/1.0.0-littleendian \
|
|
--gold-dirs=$gold_dir/2.0.0-compression \
|
|
--gold-dirs=$gold_dir/4.0.0-shareddict \
|