## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #https://github.com/apache/datafusion/issues/20024.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- part of #https://github.com/apache/datafusion/issues/20024.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #https://github.com/apache/datafusion/issues/19294.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---------
Co-authored-by: Sergey Zhukov <szhukov@aligntech.com>
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
Part of https://github.com/apache/datafusion/issues/19227
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
See issue for details.
The existing script `./dev/rust_lint.sh` do checks for all
non-functional tests include formater/clippy checks.
Some check tools support auto fix options, so this PR add an option to
the lint scripts to perform auto-fixes.
Now `./dev/rust_lint.sh --write --allow-dirty` can perform auto-fixes
for all linter etc. violations
```
yongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (auto-fix)> ./dev/rust_lint.sh --help
Usage: ./dev/rust_lint.sh [--write] [--allow-dirty]
Runs the local Rust lint suite similar to CI.
--write Run formatters, clippy and other non-functional checks in best-effort write/fix mode (requires a clean git worktree, no uncommitted changes; some checks are test-only and ignore this flag).
--allow-dirty Allow `--write` to run even when the git worktree has uncommitted changes.
```
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
- Adds `[--write] [--allow-dirty]` flag to `rust_lint.sh` to support
auto fixes
- `rust_lint.sh` consists of several sub-scripts like `rust_fmt.sh`,
they're all extended with auto-fix feature through `--write` flag, and
the `rust_lint.sh` is optionally calling them with an additional flag
for auto fixes.
- Clean up `rust_lint.sh`
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
Yes, commit 8c99417929fcf133423dd3392f1939ab13a0bc93 intentionally
introduced one violation for each available lint check, and the auto-fix
command is able to fix all of them.
The test may not be comprehensive, but it provides a reasonable starting
point. We can begin using this script now and iterate on it if we
discover cases where the auto-fix does not behave correctly.
## Are there any user-facing changes?
No
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
…#19294)
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- part of #https://github.com/apache/datafusion/issues/19294.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---------
Co-authored-by: Sergey Zhukov <szhukov@aligntech.com>
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
A small step towards https://github.com/apache/datafusion/issues/19227
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
- Move doc prettier check command to a local script
- Add this script to the lint-check runner `dev/rust_lint.sh` to make
local development easier
(Technically it's neither rust nor lint, but keeping all non functional
tests here seem to be the most convenient.)
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #https://github.com/apache/datafusion/issues/18512.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---------
Co-authored-by: Sergey Zhukov <szhukov@aligntech.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
There is a typos check CI job introduced recently. I always end up with
typos in my PRs, so I want an easier way to run this typo check locally.
`dev/rust_lint.rs` is used to perform all lint checks (fmt, clippy)
locally, the typos check should also be suitable to get included.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
1. Added `ci/scripts/typos_check.sh` for local typos check, and include
it into `dev/rust_lint.sh`
2. Update CI to also use `ci/scripts/typos_check.sh` for consistency
3. Moved typo check CI job from `Rust workflow` to `Dev workflow`
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
4. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
Yes, since CI job is changed, I included one typo in this PR, and it
should trigger CI failure.
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close?
- Closes#14197
## Rationale for this change
Our python downstream library no longer uses the feature `pyarrow` in
this repository
[#1000](https://github.com/apache/datafusion-python/pull/1000). As such,
this feature is no longer needed.
## What changes are included in this PR?
Code removal
## Are these changes tested?
Tested against datafusion-python repo. No build issues.
## Are there any user-facing changes?
If any user needs this code, they will need to migrate to the
`datafusion-python` repository as their dependency.
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
An initial attempt towards
https://github.com/apache/datafusion/issues/18467
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
### Rationale for the additional lint rule
`clippy::needless_pass_by_value`
There is a clippy lint rule that is not turned on by the current
strictness level in CI:
https://rust-lang.github.io/rust-clippy/master/index.html#needless_pass_by_value
Note it has the `Clippy` category `pedantic`, and its description is
`lints which are rather strict or have occasional false positives` from
https://doc.rust-lang.org/nightly/clippy
It seems we have been suffering from the excessive copying issue for
quite some time, and @alamb is on the front line now
https://github.com/apache/datafusion/issues/18413. I think this extra
lint rule is able to help.
### Implementation plan
This PR only enables this rule in `datafusion-common` package, and apply
`#[allow(clippy::needless_pass_by_value)]` for all violations.
If this PR makes sense, we can open a tracking issue and roll out this
check to the remaining workspace packages. At least this can help
prevent new inefficient patterns and identify existing issues that we
can fix gradually.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
* Add `Cargo.lock`
* Style fix
* Update `Cargo.lock`
* Remove unused path
* Remove cli-specific ci job and dependabot config
* Remove home pin
* Make cli test work with backtrace feature
* More changes resulting from moving the cli crate in the workspace
* Exclude `depcheck` `Cargo.lock`
* Remove `--locked` from `depcheck` run
* Refer to guidance instead of updated guidance
* Remove `--locked` from benchmark script
* Only run with `--locked` in `linux-build-lib` job of `Rust` workflow
* Remove unrelated formatting changes
* Add a note about using `--locked`
* Update cargo.lock
* Update .github/workflows/rust.yml
* Update Cargo.lock
* fix yaml
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* Add optional reference to tokio runtime for table providers
* Add function to return the library version over FFI
* Resolve clippy warnings
* Function does not need to be defined as unsafe
* Add integration test for FFI table provider
* Add version call on FFI integration test
* Making use explicit on crate to try to get CI to ensure it builds first
* Add license text
* Fix unit test to find deps in ci profile
* Remove ffitest crate and put test lib behind a feature flag
* Add integation-tests feature to ci tests
* Add integration-tests feature to CI run
* Add clarifying text
* Update CI to only run integration tests for certain checks
* When the feature integtation-tests is enabled, we get conflicting library entries for the example table provider and integration test, so disable the example during CI run
* Remove typo
* Specify each excluded crate separately
* Doc tests did not need the exclusion
* Integration tests shouldn't need doc test
* Fix md5 return_type to only return Utf8 as per current code impl.
* ci improvements
* Lock taiki-e/install-action to a githash for apache action policy - Release 2.46.19 in the case of this hash.
* Lock taiki-e/install-action to a githash for apache action policy - Release 2.46.19 in the case of this hash.
* Revert nextest change until action is approved.
* Exclude requires workspace
* Fixing minor typo to verify ci caching of builds is working as expected.
* Updates from PR review.
* Adding issue link for disabling intel mac build
* improve performance of running examples
* remove cargo check
Since a4ac0829ec commit there was a
discrepancy between rust.yml GitHub workflow and the `dev/rust_lint.sh`
script behavior. Sync the behaviors. Reuse common script to prevent
future discrepancies.
* Fix running examples readme
Some examples are runnable from any place (e.g. `csv_sql`), but some
expect a specific working directory (e.g. `regexp`). Running from
`datafusion-examples/examples` is tested on CI so guaranteed to work,
let's put this path in the README.
As a follow-up, we should look what it would take to make examples
runnable directly from an IDE such as RustRover.
* Remove doubled fmt check from CI
Examples' format is checked in `check-fmt` job, so can be skipped in
`rust_example.sh`.
* feat: add example for copy to
* better docs plus tempdir
* build: clean examples if over 10GB
* only 1GB
* build: try clearing some disk space before running
* build: remove sudo
* build: try clean
* build: run clean
* build: only clean examples
* docs: better output for example
* fix: downgrade tonic for arrow compatibility
Tonic 0.10 and 0.11 are not API compatible.
Arrow 50 depends on tonic 0.10, and datafusion must match that dependency for compatibility reasons.
* feat: make nested examples runnable
cargo run --example doesn't support nested examples. Nested examples need an explicit block to be runnable.
* fix: fix custom catalog typo and formatting
* docs: add note about upgrading tonic with arrow
* ci: add cargo check for all examples
* Add datafusion-cli tests to the CI Job
* Fix clippy error of datafusion-cli
* update tests of datafusion-cli
* fix tests error of datafusion-cli
* Splits out "cargo test datafusion-cli (amd64)" into separate CI job, fix "no space left on device" error
This PR is a follow-up of #10058. #10058 avoids executing flatc during the Java build process by statically generating Java files from the schema. Now, flatc is not necessary.
Since flatc for s390x is explicitly downloaded in a script, we can drop this download.
Closes#10067 from kiszk/ARROW-12419
Authored-by: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Also use Ubuntu 20.04 by default, instead of 18.04.
Closes#9933 from pitrou/ARROW-12262-ubuntu-docs
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Added a bash script to the workflow that checks that all .Rd files in /r/man/ that aren't internal reference ones are listed as sections in the _pkgdown.yml reference sections
Closes#9946 from thisisnic/ARROW-12236-pkgdown-ci
Authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
The installation looks like this now in the default case if the build script errors:
```
* installing *source* package ‘arrow’ ...
** using staged installation
*** Found local C++ source
*** Building C++ libraries
**** cmake
**** arrow
**** Error building Arrow C++. Re-run with ARROW_R_DEV=true for debug information.
------------------------- NOTE ---------------------------
See https://arrow.apache.org/docs/r/articles/install.html
for help installing Arrow C++ libraries
---------------------------------------------------------
```
This PR also (1) restores the arrow-without-arrow wrapping (from #9689) and (2) adds an .onAttach message for the arrow-without-arrow case to hopefully alert users earlier that they have an incomplete/useless build. If you do get a without-arrow build, this is what the loading message looks like:
```
> library(arrow)
The Arrow C++ library is not available. To retry installation with debug output, run:
install_arrow(verbose = TRUE)
See https://arrow.apache.org/docs/r/articles/install.html for more guidance and troubleshooting.
Attaching package: ‘arrow’
The following object is masked from ‘package:utils’:
timestamp
```
It *also* adds an .onAttach message if you have a build with optional features disabled (e.g. S3, lz4, etc.):
```
> library(arrow)
See arrow_info() for available features
Attaching package: ‘arrow’
The following object is masked from ‘package:utils’:
timestamp
```
`arrow_info()` will then (on Linux only) also print a message pointing you to the installation vignette if there are missing features:
```
> arrow_info()
Arrow package version: 3.0.0.9000
Capabilities:
dataset TRUE
parquet TRUE
s3 TRUE
utf8proc TRUE
re2 TRUE
snappy TRUE
gzip TRUE
brotli TRUE
zstd TRUE
lz4 TRUE
lz4_frame TRUE
lzo FALSE
bz2 TRUE
jemalloc TRUE
mimalloc FALSE
To reinstall with more features enabled, see
https://arrow.apache.org/docs/r/articles/install.html
...
```
Certain compression libraries (like lzo) are on a blocklist that excludes them from this extra messaging. The purpose of all of this is to give more hints to users when they have limited builds and give them guidance on how to enhance them, while at the same time not overly broadcasting this (which would promote FUD) and trying to be clear that you don't *always* have to `install_arrow()` after `install.packages()`.
Closes#9896 from nealrichardson/nix-install-debug
Authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
This seems to save about 2GB in the image's root directory.
Closes#9891 from pitrou/ARROW-12191-conda-integration
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
# Rationale
The [integration test](https://github.com/apache/arrow/pull/9884/checks?check_run_id=2263730460) has a fixed size builder docker image and has builds from several Arrow implementations.
The Rust build artifacts (compiled binaries) in the integration tests still consume ~ 1GB of space even after https://github.com/apache/arrow/pull/9879 (see @pitrou 's comment on https://github.com/apache/arrow/pull/9884#issuecomment-813037756).
# Changes
1. Only build the `arrow-integration-testing` crate (which has the binaries needed for integration testing) rather than all of them
2. Remove incremental build artifacts
3. Moves `cargo check` to ensure we can build arrow without default features to a github action and out of the integration test
This should both speed up the rust build as part of integration testing as well as reduce the amount of disk space required in the builder (both peak and after the rust build).
# Results
According to this [log](https://pipelines.actions.githubusercontent.com/CKhEcbzaHKPSiCoVWyWCFardoFMA9UN2zDnXYiBlRYO31o0IIv/_apis/pipelines/1/runs/179726/signedlogcontent/3?urlExpires=2021-04-05T12%3A37%3A13.1888704Z&urlSigningMethod=HMACV1&urlSignature=zROZZZham0BGH5f2vbe65n3DL13xXLgD9RJXybRqBAA%3D) after this PR's changes, the rust artifacts are much smaller.
Note we could still save space by removing the rust toolchain and `.cargo` files but I think that can be an optimization for another time:
```
2021-04-05T11:30:22.6164120Z 244M /opt/go/pkg
2021-04-05T11:30:22.6164599Z 393M /opt/go
2021-04-05T11:30:22.7201596Z 110M /opt/conda/envs/arrow/share
2021-04-05T11:30:22.7220002Z 171M /opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu/9.3.0
2021-04-05T11:30:22.7220911Z 171M /opt/conda/envs/arrow/libexec/gcc/x86_64-conda-linux-gnu
2021-04-05T11:30:22.7221489Z 171M /opt/conda/envs/arrow/libexec/gcc
2021-04-05T11:30:22.7221945Z 186M /opt/conda/envs/arrow/libexec
2021-04-05T11:30:22.7314210Z 186M /opt/conda/envs/arrow/lib/valgrind
2021-04-05T11:30:22.8109246Z 129M /opt/conda/envs/arrow/lib/python3.8/site-packages
2021-04-05T11:30:22.8290900Z 177M /opt/conda/envs/arrow/lib/python3.8
2021-04-05T11:30:22.8636030Z 1.8G /opt/conda/envs/arrow/lib
2021-04-05T11:30:22.9861104Z 171M /opt/conda/envs/arrow/include/boost
2021-04-05T11:30:23.0128341Z 260M /opt/conda/envs/arrow/include
2021-04-05T11:30:23.0163805Z 417M /opt/conda/envs/arrow/bin
2021-04-05T11:30:23.0449297Z 102M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr/lib64
2021-04-05T11:30:23.0525177Z 133M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot/usr
2021-04-05T11:30:23.0526023Z 139M /opt/conda/envs/arrow/x86_64-conda-linux-gnu/sysroot
2021-04-05T11:30:23.0527418Z 189M /opt/conda/envs/arrow/x86_64-conda-linux-gnu
2021-04-05T11:30:23.0612702Z 108M /opt/conda/envs/arrow/jre/lib
2021-04-05T11:30:23.0613140Z 109M /opt/conda/envs/arrow/jre
2021-04-05T11:30:23.0618402Z 3.2G /opt/conda/envs/arrow
2021-04-05T11:30:23.0618775Z 3.2G /opt/conda/envs
2021-04-05T11:30:23.1055113Z 159M /opt/conda/lib
2021-04-05T11:30:23.5619580Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib/valgrind
2021-04-05T11:30:23.5625488Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0/lib
2021-04-05T11:30:23.5626989Z 146M /opt/conda/pkgs/valgrind-3.15.0-he513fc3_0
2021-04-05T11:30:23.5793611Z 349M /opt/conda/pkgs
2021-04-05T11:30:23.5845265Z 3.7G /opt/conda
2021-04-05T11:30:23.5846086Z 4.1G /opt
2021-04-05T11:30:23.6810817Z 131M /root/.m2/repository/org
2021-04-05T11:30:23.6813841Z 260M /root/.m2/repository
2021-04-05T11:30:23.6814253Z 260M /root/.m2
2021-04-05T11:30:23.7360947Z 124M /root/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack
2021-04-05T11:30:23.7362421Z 124M /root/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects
2021-04-05T11:30:23.7376583Z 125M /root/.cargo/registry/index/github.com-1ecc6299db9ec823/.git
2021-04-05T11:30:23.7447433Z 133M /root/.cargo/registry/index/github.com-1ecc6299db9ec823
2021-04-05T11:30:23.7448341Z 133M /root/.cargo/registry/index
2021-04-05T11:30:23.7448796Z 232M /root/.cargo/registry
2021-04-05T11:30:23.7449202Z 245M /root/.cargo
2021-04-05T11:30:23.8658627Z 126M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/core
2021-04-05T11:30:23.8663924Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html
2021-04-05T11:30:23.8668130Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust
2021-04-05T11:30:23.8669570Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc
2021-04-05T11:30:23.8670941Z 423M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share
2021-04-05T11:30:23.8672364Z 149M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8673637Z 155M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8674758Z 157M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib
2021-04-05T11:30:23.8675721Z 376M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib
2021-04-05T11:30:23.8676657Z 865M /root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu
2021-04-05T11:30:23.8677417Z 865M /root/.rustup/toolchains
2021-04-05T11:30:23.8677825Z 865M /root/.rustup
2021-04-05T11:30:23.8678171Z 1.5G /root
2021-04-05T11:30:23.9219564Z 113M /go/pkg/mod/google.golang.org/genproto@v0.0.0-20200911024640-645f7a48b24f/googleapis
2021-04-05T11:30:23.9220667Z 113M /go/pkg/mod/google.golang.org/genproto@v0.0.0-20200911024640-645f7a48b24f
2021-04-05T11:30:23.9306825Z 129M /go/pkg/mod/google.golang.org
2021-04-05T11:30:23.9346436Z 264M /go/pkg/mod
2021-04-05T11:30:23.9346938Z 264M /go/pkg
2021-04-05T11:30:23.9347281Z 264M /go
2021-04-05T11:30:23.9383575Z 129M /build/cpp/src/arrow
2021-04-05T11:30:23.9384025Z 131M /build/cpp/src
2021-04-05T11:30:23.9387580Z 223M /build/cpp
2021-04-05T11:30:23.9387915Z 223M /build
2021-04-05T11:30:23.9388188Z 6.1G /
2021-04-05T11:30:24.0044084Z 846M /arrow/
```
Closes#9889 from alamb/alamb/reduce_integration_size
Authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>
# Rationale
Rust debug symbols are quite verbose, taking up memory during the final link time as well as significant disk space. Turning off the creation of symbols should save us compile / test time for CI as well as space on the integration test
# Change
Do not produce debug symbols on Rust CI (keep enough to have line numbers in `panic!` traceback, but not enough to interpret a core file, which no one does to my knowledge anyways)
Note that the integration test passed: https://github.com/apache/arrow/pull/9879/checks?check_run_id=2256148363Closes#9879 from alamb/less_symbols_in_integration
Authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>
This adds a one hour timeout to the R builds. It changes the R CI test script to use `reporter="location"` by default. It adds a dump test logs step to the end of the build that will always dump the test output regardless of success/failure.
These three changes combined will make it much easier to debug test failures in R tests.
Closes#9846 from westonpace/feature/arrow-12143
Lead-authored-by: Weston Pace <weston.pace@gmail.com>
Co-authored-by: Jonathan Keane <jkeane@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
The only code change required, AFAICT, is the calculation of num_dicts, which is no longer simply the number of fields, but rather the unique number of id's they point to. I'm calculating this on-demand, as it's quite cheap and not frequently called, but could also (p)re-compute this on every addField.
For now, I've added tests that read materialised data generated from Java, as we don't support writing IPC with shared dictionaries in C++ either yet (and out of scope here).
Down the line, I would like full read & write support for shared dictionaries across at least C++, Python, Java and Julia, so I'll be coming back to this with follow-up PR's where needed. As part of that, I'll also change the tests to no longer rely on materialised files, but use the round-trip mechanism.
Closes#9629 from jmgpeeters/ARROW-11838
Lead-authored-by: jpeeter <jpeeter@loswjpeeter1.intra.gsacapital.com>
Co-authored-by: jmgpeeters <joris.mg.peeters@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This resolves the sanitizer error (thank you @lidavidm for the c++ solution!). I also added a step to print `testthat.out` if `RDsan` fails which we weren't doing before. Hopefully that helps diagnose these a bit easier in the future.
Closes#9814 from jonkeane/ARROW-11914-r-sanitizer
Authored-by: Jonathan Keane <jkeane@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
Based on the c++ implementation but tuned and optimized for Go, I spent the first couple months this year creating a Go implementation for Parquet with the goal of native/easy integration with the Arrow library while still being highly performant and at minimum reaching feature parity with the C++ implementation.
Based on the conversations on the JIRA card, rather than dumping a huge code bomb (there's a ton), I've chunked it up. This is the initial chunk of code comprising of an internal utils directory that is analogous to the cpp/arrow/utils/ bit readers/writers/bit run readers/etc. which were ultimately used by the go implementation, while using c2goasm to reach the performance necessary for certain areas.
This is part 1 of the implementation as I chunk it up and push it out. I'll wait for each chunk to get merged before pushing the next PR in order to make sure that everything stays in sync.
CC: @emkornfield @wesm @sbinet @nickpoorman
Closes#9671 from zeroshade/arrow-7905-p1
Authored-by: Matthew Topol <mtopol@factset.com>
Signed-off-by: Micah Kornfield <emkornfield@gmail.com>
Sergey Zhukov