This seems to save about 2GB in the image's root directory.
Closes#9891 from pitrou/ARROW-12191-conda-integration
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
# Rationale
The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations.
The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after https://github.com/apache/arrow/pull/9879 (see @pitrou 's comment on https://github.com/apache/arrow/pull/9884#issuecomment-813037756).
# Changes
1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them
2. Remove incremental build artifacts
3. Moves `cargo check` to ensure we can build arrow without default features to a github action and out of the integration test
This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build).
# Results
According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D) after this PR's changes, the rust artifacts are much smaller.
Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time:
```
2021-04-05T11:30:22.6164120Z 244M /opt/go/pkg
2021-04-05T11:30:22.6164599Z 393M /opt/go
2021-04-05T11:30:22.7201596Z 110M /opt/conda/envs/arrow/share
2021-04-05T11:30:22.7220002Z 171M /opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0
2021-04-05T11:30:22.7220911Z 171M /opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu
2021-04-05T11:30:22.7221489Z 171M /opt/conda/envs/arrow/libexec/gcc
2021-04-05T11:30:22.7221945Z 186M /opt/conda/envs/arrow/libexec
2021-04-05T11:30:22.7314210Z 186M /opt/conda/envs/arrow/lib/valgrind
2021-04-05T11:30:22.8109246Z 129M /opt/conda/envs/arrow/lib/python3.8/site-packages
2021-04-05T11:30:22.8290900Z 177M /opt/conda/envs/arrow/lib/python3.8
2021-04-05T11:30:22.8636030Z 1.8G /opt/conda/envs/arrow/lib
2021-04-05T11:30:22.9861104Z 171M /opt/conda/envs/arrow/include/boost
2021-04-05T11:30:23.0128341Z 260M /opt/conda/envs/arrow/include
2021-04-05T11:30:23.0163805Z 417M /opt/conda/envs/arrow/bin
2021-04-05T11:30:23.0449297Z 102M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64
2021-04-05T11:30:23.0525177Z 133M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr
2021-04-05T11:30:23.0526023Z 139M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot
2021-04-05T11:30:23.0527418Z 189M /opt/conda/envs/arrow/x86_64-conda-linux-gnu
2021-04-05T11:30:23.0612702Z 108M /opt/conda/envs/arrow/jre/lib
2021-04-05T11:30:23.0613140Z 109M /opt/conda/envs/arrow/jre
2021-04-05T11:30:23.0618402Z 3.2G /opt/conda/envs/arrow
2021-04-05T11:30:23.0618775Z 3.2G /opt/conda/envs
2021-04-05T11:30:23.1055113Z 159M /opt/conda/lib
2021-04-05T11:30:23.5619580Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind
2021-04-05T11:30:23.5625488Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib
2021-04-05T11:30:23.5626989Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0
2021-04-05T11:30:23.5793611Z 349M /opt/conda/pkgs
2021-04-05T11:30:23.5845265Z 3.7G /opt/conda
2021-04-05T11:30:23.5846086Z 4.1G /opt
2021-04-05T11:30:23.6810817Z 131M /root/.m2/repository/org
2021-04-05T11:30:23.6813841Z 260M /root/.m2/repository
2021-04-05T11:30:23.6814253Z 260M /root/.m2
2021-04-05T11:30:23.7360947Z 124M /root/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack
2021-04-05T11:30:23.7362421Z 124M /root/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects
2021-04-05T11:30:23.7376583Z 125M /root/.cargo/registry/index/github.com-1ecc6299db9ec823/.git
2021-04-05T11:30:23.7447433Z 133M /root/.cargo/registry/index/github.com-1ecc6299db9ec823
2021-04-05T11:30:23.7448341Z 133M /root/.cargo/registry/index
2021-04-05T11:30:23.7448796Z 232M /root/.cargo/registry
2021-04-05T11:30:23.7449202Z 245M /root/.cargo
2021-04-05T11:30:23.8658627Z 126M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core
2021-04-05T11:30:23.8663924Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html
2021-04-05T11:30:23.8668130Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust
2021-04-05T11:30:23.8669570Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc
2021-04-05T11:30:23.8670941Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share
2021-04-05T11:30:23.8672364Z 149M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8673637Z 155M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8674758Z 157M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
2021-04-05T11:30:23.8675721Z 376M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8676657Z 865M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8677417Z 865M /root/.rustup/toolchains
2021-04-05T11:30:23.8677825Z 865M /root/.rustup
2021-04-05T11:30:23.8678171Z 1.5G /root
2021-04-05T11:30:23.9219564Z 113M /go/pkg/mod/google.golang.org/genproto@v0.0.0-20200911024640-645f7a48b24f/googleapis
2021-04-05T11:30:23.9220667Z 113M /go/pkg/mod/google.golang.org/genproto@v0.0.0-20200911024640-645f7a48b24f
2021-04-05T11:30:23.9306825Z 129M /go/pkg/mod/google.golang.org
2021-04-05T11:30:23.9346436Z 264M /go/pkg/mod
2021-04-05T11:30:23.9346938Z 264M /go/pkg
2021-04-05T11:30:23.9347281Z 264M /go
2021-04-05T11:30:23.9383575Z 129M /build/cpp/src/arrow
2021-04-05T11:30:23.9384025Z 131M /build/cpp/src
2021-04-05T11:30:23.9387580Z 223M /build/cpp
2021-04-05T11:30:23.9387915Z 223M /build
2021-04-05T11:30:23.9388188Z 6.1G /
2021-04-05T11:30:24.0044084Z 846M /arrow/
```
Closes#9889 from alamb/alamb/reduce_integration_size
Authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>
# Rationale
Rust debug symbols are quite verbose, taking up memory during the final link time as well as significant disk space. Turning off the creation of symbols should save us compile / test time for CI as well as space on the integration test
# Change
Do not produce debug symbols on Rust CI (keep enough to have line numbers in `panic!` traceback, but not enough to interpret a core file, which no one does to my knowledge anyways)
Note that the integration test passed: https://github.com/apache/arrow/pull/9879/checks?check_run_id=2256148363Closes#9879 from alamb/less_symbols_in_integration
Authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>
This adds a one hour timeout to the R builds. It changes the R CI test script to use `reporter="location"` by default. It adds a dump test logs step to the end of the build that will always dump the test output regardless of success/failure.
These three changes combined will make it much easier to debug test failures in R tests.
Closes#9846 from westonpace/feature/arrow-12143
Lead-authored-by: Weston Pace <weston.pace@gmail.com>
Co-authored-by: Jonathan Keane <jkeane@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
The only code change required, AFAICT, is the calculation of num_dicts, which is no longer simply the number of fields, but rather the unique number of id's they point to. I'm calculating this on-demand, as it's quite cheap and not frequently called, but could also (p)re-compute this on every addField.
For now, I've added tests that read materialised data generated from Java, as we don't support writing IPC with shared dictionaries in C++ either yet (and out of scope here).
Down the line, I would like full read & write support for shared dictionaries across at least C++, Python, Java and Julia, so I'll be coming back to this with follow-up PR's where needed. As part of that, I'll also change the tests to no longer rely on materialised files, but use the round-trip mechanism.
Closes#9629 from jmgpeeters/ARROW-11838
Lead-authored-by: jpeeter <jpeeter@loswjpeeter1.intra.gsacapital.com>
Co-authored-by: jmgpeeters <joris.mg.peeters@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This resolves the sanitizer error (thank you @lidavidm for the c++ solution!). I also added a step to print `testthat.out` if `RDsan` fails which we weren't doing before. Hopefully that helps diagnose these a bit easier in the future.
Closes#9814 from jonkeane/ARROW-11914-r-sanitizer
Authored-by: Jonathan Keane <jkeane@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
Based on the c++ implementation but tuned and optimized for Go, I spent the first couple months this year creating a Go implementation for Parquet with the goal of native/easy integration with the Arrow library while still being highly performant and at minimum reaching feature parity with the C++ implementation.
Based on the conversations on the JIRA card, rather than dumping a huge code bomb (there's a ton), I've chunked it up. This is the initial chunk of code comprising of an internal utils directory that is analogous to the cpp/arrow/utils/ bit readers/writers/bit run readers/etc. which were ultimately used by the go implementation, while using c2goasm to reach the performance necessary for certain areas.
This is part 1 of the implementation as I chunk it up and push it out. I'll wait for each chunk to get merged before pushing the next PR in order to make sure that everything stays in sync.
CC: @emkornfield @wesm @sbinet @nickpoorman
Closes#9671 from zeroshade/arrow-7905-p1
Authored-by: Matthew Topol <mtopol@factset.com>
Signed-off-by: Micah Kornfield <emkornfield@gmail.com>
This should work in theory, I'm building the image to see how well does vcpkg support arm packages.
```bash
ARCH=arm64v8 archery docker build python-wheel-manylinux-2014
```
Closes#9285 from kszucs/manylinux-arm64
Lead-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
This builds upon ARROW-11340 and makes it simpler to use vcpkg for dependencies.
Previously, to use vcpkg it was necessary to specify:
- `-DARROW_DEPENDENCY_SOURCE=SYSTEM`
- `-DCMAKE_TOOLCHAIN_FILE=/path/to/vcpkg.cmake`
- `-DVCPKG_TARGET_TRIPLET=xxx-yyy-zzz`
- Various`*_MSVC_*` variables on Windows
- Various `*_ROOT`, `*_LIB`, and `*_INCLUDE` variables in some cases
- `-D_VCPKG_INSTALLED_DIR=/path/to/vcpkg/installed/dir` in some cases
Now all that is necessary is:
- `-DARROW_DEPENDENCY_SOURCE=VCPKG`
Other variables are now optional and can be specified to override default behaviors.
After this is merged, I will make additional updates to the docs, focusing initially on Windows (ARROW-11336).
Closes#9553 from ianmcook/ARROW-11580-take2
Authored-by: Ian Cook <ianmcook@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@kszucs could you review this please? My main purpose in adding this is to improve the experience for Arrow C++ devs using Windows, but I noticed it also relates to your [TODO in #9096](https://github.com/apache/arrow/pull/9096/files#diff-990134cce6657dbbcf95457cf1a56810a7efa1f6cd58ecc27557c7d6ff45b533R67-R68). vcpkg does not have any `requirements.txt`-style package enumeration mechanism, but it supports this JSON manifest as a mechanism of defining dependencies.
In the `vcpkg install` command, you can specify the path to the directory containing this manifest file with `--x-manifest-root` which later will change to `--manifest-root`. See details at https://vcpkg.readthedocs.io/en/stable/specifications/manifests/.
There are some differences between the packages listed in this manifest versus the packages you listed in the `vcpkg install` commands in #9096
- This installs `gtest` and `benchmark`
- This installs `boost` instead of separate `boost-filesystem`, `boost-regex`, etc.
- This does not explicitly include the `core` feature of `aws-sdk-cpp` because explicitly including it causes an error, and it gets installed anyway
Closes#9287 from ianmcook/ARROW-11340
Lead-authored-by: Ian Cook <ianmcook@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
This PR creates a test to receive RecordBatch for different endian (e.g. receive RecordBatch with big-endian schema on little-endian platform).
This PR changes
1. Introduce Endianness enum class to represent endianness
2. Add a new flag `endianness` to `arrow::schema` to represent endianness of Array in `RecordBatch`.
3. Eagerly convert non-nativeendian data to nativeendian data in a batch if `IpcReadOption.use_native_endian = true` (`true` by default).
4. Add golden arrow files for integration test in both endians and test script
Regarding 3., other possible choices are as follows:
- Lazily convert non-nativeendian data to nativeendian data for a column in each RecordBatch.
Pros: Avoid conversion for columns that will not be read
Cons: Complex management of endianness of each column. Inconsistency of endianness between schema and column data.
- Convert non-nativeendian data to nativeendian data when each element is read
Pros: Can keep the original schema without batch conversion
Cons: 1) Each RecordBatch may need an additional field to show whether the endian conversion is necessary or not. 2) Need to update test cases to accept different endianness between expected and actual schemas.
Now, this PR uses the simplest approach (always see native endianness in schemas) that eagerly converts all of the columns in a batch.
TODO
- [x] Support to convert endian of each element for primitive types
- [x] Support to convert endian of each element for complex types
- [x] Support to convert endian of each element for all types for stream
- [x] Add golden arrow files in both endians
For creating this PR, @kou helps me greatly (prototype of arrow::schema and teaching me about RecordBatch).
Closes#7507 from kiszk/ARROW-8797
Lead-authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Includes dependency toolchain updates (utf8proc, re2) and bindings for
* "nchar", "str_length" -> "binary_length"
* "tolower", "str_to_lower" -> "utf8_lower"
* "toupper", "str_to_upper" -> "utf8_upper"
* "str_trim" is handled a little differently since it's one function in `stringr` but three functions in arrow (utf8_ltrim_whitespace, utf8_rtrim_whitespace, utf8_trim_whitespace)
Closes#9423 from nealrichardson/r-strings
Authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
The tests use "gold" files generated by PyArrow 2.0.0 and are only enabled for C++.
Closes#9408 from pitrou/ARROW-11417-compression-integration
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Add following Datasets APIs to Java:
- DatasetFactory
- Dataset
- Scanner
- ScanTask
- ScanTask.BatchIterator
Add a native dataset path to bridge c++ Datasets components to Java:
- NativeDatasetFactory (c++ class : DatasetFactory)
- NativeDataset (c++ class: Dataset)
- NativeScanner (c++ class: DisposableScannerAdaptor)
Following c++ components are not JNI-mapped to keep the initial implementation simple:
- Fragment
- ScanTask
- (arrow::)RecordBatchIterator
Add following API to `FileSystemDatasetFactory` to avoid passing file system objects via JNI bridge:
- `FileSystemDatasetFactory::Make( std::string uri, std::shared_ptr<FileFormat> format, FileSystemFactoryOptions options)`
Unit tests are based on `FileSystemDatasetFactory`.
Closes#7030 from zhztheplayer/ARROW-7808
Authored-by: Hongze Zhang <hongze.zhang@intel.com>
Signed-off-by: Micah Kornfield <emkornfield@gmail.com>
This adds an additional docker-compose task that will test a Spark branch with the latest PyArrow, instead of the default which also builds Spark with the latest Arrow Java. This is so existing Spark releases with fixed Arrow Java dependencies can still be tested with current PyArrow changes to ensure there is still compatibility. The nightly test against Spark branch-3.0 is changed to use this new task.
Closes#9210 from BryanCutler/ci-spark-branch-3.0-ARROW-10457
Authored-by: Bryan Cutler <cutlerb@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Where newer versions were available but we could not upgrade, I left comments in the versions.txt file for future reference.
This PR turns off re2 in the R package builds since we don't use it (yet).
Closes#9296 from nealrichardson/bump-mimalloc
Lead-authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Also resolves:
- ARROW-11213: [Packaging][Python] Dockerize wheel building on windows
- ARROW-11215: [CI] Use named volumes by default for caching in docker-compose
- ARROW-11231: [Python][Packaging] Enable mimalloc in wheels
Main features:
- dockerize windows builds
- use vcpkg as the dependency source where we can explicitly pin a working version without worrying about the drag of build environment over time
Potential follow-up:
- use vcpkg on macos as well
cc @kou @xhochy
## Manylinux testing
Should be straightforward (I'm going to shared pre-built images for quicker testing).
## Windows testing
Only windows host is able to run windows containers. I'm virtualizing windows on macOS (should work on linux as well) using virtualbox:
```bash
git clone https://github.com/StefanScherer/windows-docker-machine
cd windows-docker-machine
# grant more resources in the vagrantfile under the virtualbox section (I use 8 cores and 16GiB of ram)
vagrant up --provider virtualbox 2019-box
docker context ls
docker context use 2019-box
docker image ls
```
Now docker should use the windows docker daemon.
```bash
pip install -e dev/archery[docker]
# the volumes defined in the docker compose file should be commented out and pass arrow's source explicitly
# this is required because compose is being executed on a unix host and the directory is mounted through a virtual machine
PYTHON=3.6 archery docker run --no-pull --using-docker-cli -v C:$(pwd):C:/arrow python-wheel-windows-vs2017 cmd.exe
# then within the container execute a build or do other interactive things
arrow\ci\scripts\python_wheel_windown_build.bat
```
Closes#9096 from kszucs/wheel-vcpkg
Lead-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Includes fixes for compiling on R 3.4 and R 3.3, plus a crossbow job to build on old R versions
Closes#8833 from nealrichardson/old-r-versions
Lead-authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Co-authored-by: Romain Francois <romain@rstudio.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
ARROW_DEFAULT_MEMORY_POOL can take the name of the desired memory pool backend
('jemalloc', 'mimalloc', 'system').
Closes#9105 from pitrou/ARROW-11009-memory-pool-env-var
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
Ensure that the environment variables are properly passed to cmake.
Closes#8932 from jonkeane/centos-ARROW-10856
Authored-by: Jonathan Keane <jkeane@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
Because upstream uses "re2". If we don't use different package name
such as "RE2", RE2 package is conflicted with other package that uses
RE2 such as gRPC.
This change also includes resolve_dependency() cleanup. There were
some custom resolve_dependency() like codes for packages that need our
custom "Alt" version such as Boost and gRPC. We can use
resolved_dependency() for these packages with the new HAVE_ALT option.
This change also renamed our Boost dependency name to "Boost" from
"BOOST". It means that users need to use -DBoost_SOURCE not
-DBOOST_SOURCE. To keep backward compatibility, -DBOOST_SOURCE is
still accepted when -DBoost_SOURCE isn't specified.
Users also need to use -Dre2_SOURCE not -DRE2_SOURCE. To keep backward
compatibility, -DRE2_SOURCE is still accepted when -Dre2_SOURCE isn't
specified.
Closes#8908 from kou/cpp-re2
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
This PR refactors the CI with the purpose of making it faster, easier to extend, and more robust. The main consequences of this PR are:
1. build time of non-integration jobs reduced by half (10m vs 20m)
2. coverage is now part of all builds
3. integration with the c data interface is now part of all builds
4. significantly reduced the risk of cache misses, and the size of the caches
5. reduced complexity by removing the need to build a docker image
6. macos now have caching in place, like other builds
7. significantly easier to extend to other architectures, os, etc.
The overall design of this PR is that most steps of the CI process should be independent of all others as much as possible, so that
* they can run in parallel
* it is clearer where the failure is
* it is easier to extend the build to other variations (arch, os, flags, etc.)
Intrinsically, the CI pipeline is just a DAG where certain jobs benefit from shared compute and thus depend on common nodes, while others run independently.
This PR re-writes most of our CI as simple github jobs that share caches whenever makes sense, thereby describing the aforementioned DAG. Consequently, many of the sequential steps are now executed in parallel, causing the items describe above.
There are two tasks depending on rust that are not in the `rust.yaml`:
* lint, that happens via archery
* integration tests, that happens via archery
The bottleneck of a green pipeline are the integration tests, that currently re-compile a significant number of components (as I already flagged on the mailing list), but at least we get faster failures for compilation (2.5m) and clippy (4.5m).
Closes#8821 from jorgecarleitao/build_tests
Authored-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>
This PR is a proposal to add support to the [C data interface](https://arrow.apache.org/docs/format/CDataInterface.html) by implementing the necessary functionality to both consume and produce structs with its ABI and lifetime rules.
This is for now limited to primitive types and strings (utf8), but it is easily generalized for all types whose data is encapsulated in `ArrayData` (things with buffers and child data).
Some design choices:
* import and export does not care about the type of the data that is in memory (previously `BufferData`, now `Bytes`) - it only cares about how they should be converted from and to `ArrayData` to the C data interface.
* import wraps incoming pointers on a struct behind an `Arc`, so that we thread-safely refcount them and can share them between buffers, arrays, etc.
* `export` places `Buffer`s in `private_data` for bookkeeping and release them when the consumer releases it via `release`.
I do not expect this PR to be easy to review, as it is touching sensitive (aka `unsafe`) code. However, based on the tests I did so far, I am sufficiently happy to PR it.
This PR has three main parts:
1. Addition of an `ffi` module that contains the import and export functionality
2. Add some helpers to import and export an Array from C Data Interface
3. A crate to test this against Python/C++'s API
It also does a small refactor of `BufferData`, renaming it to `Bytes` (motivated by the popular `bytes` crate), and moving it to a separate file.
What is tested:
* round-trip `Python -> Rust -> Python` (new separate crate, `arrow-c-integration`)
* round-trip `Rust -> Python -> Rust` (new separate crate, `arrow-c-integration`)
* round-trip `Rust -> Rust -> Rust`
* memory allocation counts
Finally, this PR has a large contribution of @pitrou , that took _a lot_ of his time to explain to me how the C++ was doing it and the main things that I had to worry about here.
Closes#8401 from jorgecarleitao/arrow-c-inte
Authored-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
Signed-off-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
Fixes Ci failures by separating running tests, and discarding artefacts after testing
Also adds a separate test for `arrow` on stable.
Changes made:
* Reorder the Rust tests to test other modules without `arrow`, reducing test duplication
* Add stable toolchain test for the `arrow` crate, test it next to `feature "simd"`
* Clear the test artefacts before testing `arrow`, to reclaim some disk space
* Update nightly to `2020-11-24` so that the tests would get triggered
Closes#8793 from nevi-me/ARROW-10757
Authored-by: Neville Dipale <nevilledips@gmail.com>
Signed-off-by: Neville Dipale <nevilledips@gmail.com>
This PR copies the required shared libraries for Java into the directory (`${ARROW_HOME}/lib`) where the test will refer to thru LD_LIBRARY_PATH.
Closes#8683 from kiszk/ARROW-10621
Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
# Rationale
We do currently run clippy (aka rust linter) on all PRs. However, the way we invoke clippy means that any clippy warnings introduced in PRs do not cause a CI failure
I believe the intended behavior is that we keep the arrow project "Clippy clean"
# Changes
* Upgrade the rust_lint.sh script to test all targets and treat warnings as fatal
* Updates the code so that the project passes all clippy lints 🎉
# Example of this not working today
For example, this PR has clippy flag a warning and yet the test still passes (note the PR doesn't introduce the warning).
https://github.com/apache/arrow/runs/1408182529?check_suite_focus=true
```
Checking arrow v3.0.0-SNAPSHOT (/Users/runner/work/arrow/arrow/rust/arrow)
warning: unneeded unit return type
--> arrow/src/buffer.rs:694:45
|
694 | pub fn resize(&mut self, new_len: usize) -> () {
| ^^^^^^ help: remove the `-> ()`
|
= note: `#[warn(clippy::unused_unit)]` on by default
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unused_unit
warning: 1 warning emitted
Finished dev [unoptimized + debuginfo] target(s) in 1m 11s
~/work/arrow/arrow
```
# Testing of this PR:
* The initial commit of this PR f50ca5a only includes the updated clippy command and it fails CI (see https://github.com/apache/arrow/pull/8687/checks?check_run_id=1408834360)
* When the second commit to fix the clippy warning is introduced, then CI passes: https://github.com/apache/arrow/pull/8687/checks?check_run_id=1409016544Closes#8687 from alamb/ARROW-10626-clippy-clean
Authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Signed-off-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
This takes us from 2020-04-22 to 2020-11-14.
There have been a lot of clippy lints introduced since April, and an incompleteness feature warning about specialization (which causes CI to fail).
I have allowed almost all clippy lints in datafusion, as we have >150 warnings there.
I have made changes to fix arrow and parquet lints where I perceive them not to affect open PRs.
Closes#8666 from nevi-me/ARROW-10269
Authored-by: Neville Dipale <nevilledips@gmail.com>
Signed-off-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
Dominik Moritz