Neal Richardson
e433ca40db
The installation looks like this now in the default case if the build script errors: ``` * installing *source* package ‘arrow’ ... ** using staged installation *** Found local C++ source *** Building C++ libraries **** cmake **** arrow **** Error building Arrow C++. Re-run with ARROW_R_DEV=true for debug information. ------------------------- NOTE --------------------------- See https://arrow.apache.org/docs/r/articles/install.html for help installing Arrow C++ libraries --------------------------------------------------------- ``` This PR also (1) restores the arrow-without-arrow wrapping (from #9689) and (2) adds an .onAttach message for the arrow-without-arrow case to hopefully alert users earlier that they have an incomplete/useless build. If you do get a without-arrow build, this is what the loading message looks like: ``` > library(arrow) The Arrow C++ library is not available. To retry installation with debug output, run: install_arrow(verbose = TRUE) See https://arrow.apache.org/docs/r/articles/install.html for more guidance and troubleshooting. Attaching package: ‘arrow’ The following object is masked from ‘package:utils’: timestamp ``` It *also* adds an .onAttach message if you have a build with optional features disabled (e.g. S3, lz4, etc.): ``` > library(arrow) See arrow_info() for available features Attaching package: ‘arrow’ The following object is masked from ‘package:utils’: timestamp ``` `arrow_info()` will then (on Linux only) also print a message pointing you to the installation vignette if there are missing features: ``` > arrow_info() Arrow package version: 3.0.0.9000 Capabilities: dataset TRUE parquet TRUE s3 TRUE utf8proc TRUE re2 TRUE snappy TRUE gzip TRUE brotli TRUE zstd TRUE lz4 TRUE lz4_frame TRUE lzo FALSE bz2 TRUE jemalloc TRUE mimalloc FALSE To reinstall with more features enabled, see https://arrow.apache.org/docs/r/articles/install.html ... ``` Certain compression libraries (like lzo) are on a blocklist that excludes them from this extra messaging. The purpose of all of this is to give more hints to users when they have limited builds and give them guidance on how to enhance them, while at the same time not overly broadcasting this (which would promote FUD) and trying to be clear that you don't *always* have to `install_arrow()` after `install.packages()`. Closes #9896 from nealrichardson/nix-install-debug Authored-by: Neal Richardson <neal.p.richardson@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
Arrow Developer Scripts
This directory contains scripts useful to developers when packaging, testing, or committing to Arrow.
Merging a pull request requires being a committer on the project. In addition you need to have linked your GitHub and ASF accounts on https://gitbox.apache.org/setup/ to be able to push to GitHub as the main remote.
NOTE: It may take some time (a few hours) between when you complete the setup at GitBox, and when your GitHub account will be added as a committer.
How to merge a Pull request
Please don't merge PRs using the Github Web interface. Instead, set up
your git clone such as to have a remote named apache pointing to the
official Arrow repository:
git remote add apache git@github.com:apache/arrow.git
and then run the following command:
./dev/merge_arrow_pr.sh
This creates a new Python virtual environment under dev/.venv[PY_VERSION]
and installs all the necessary dependencies to run the Arrow merge script.
After installed, it runs the merge script.
(we don't provide a wrapper script for Windows yet, so under Windows you'll
have to install Python dependencies yourself and then run dev/merge_arrow_pr.py
directly)
The merge script uses the GitHub REST API; if you encounter rate limit issues,
you may set a ARROW_GITHUB_API_TOKEN environment variable to use a Personal
Access Token.
You can specify the username and the password of your JIRA account in
APACHE_JIRA_USERNAME and APACHE_JIRA_PASSWORD environment variables.
If these aren't supplied, the script will ask you the values of them.
Note that the directory name of your Arrow git clone must be called arrow.
example output:
Which pull request would you like to merge? (e.g. 34):
Type the pull request number (from https://github.com/apache/arrow/pulls) and hit enter.
=== Pull Request #X ===
title Blah Blah Blah
source repo/branch
target master
url https://api.github.com/repos/apache/arrow/pulls/X
Proceed with merging pull request #3? (y/n):
If this looks good, type y and hit enter.
From git-wip-us.apache.org:/repos/asf/arrow.git
* [new branch] master -> PR_TOOL_MERGE_PR_3_MASTER
Switched to branch 'PR_TOOL_MERGE_PR_3_MASTER'
Merge complete (local ref PR_TOOL_MERGE_PR_3_MASTER). Push to apache? (y/n):
A local branch with the merge has been created. type y and hit enter to push it to apache master
Counting objects: 67, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (26/26), done.
Writing objects: 100% (36/36), 5.32 KiB, done.
Total 36 (delta 17), reused 0 (delta 0)
To git-wip-us.apache.org:/repos/arrow-mr.git
b767ac4..485658a PR_TOOL_MERGE_PR_X_MASTER -> master
Restoring head pointer to b767ac4e
Note: checking out 'b767ac4e'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b new_branch_name
HEAD is now at b767ac4... Update README.md
Deleting local branch PR_TOOL_MERGE_PR_X
Deleting local branch PR_TOOL_MERGE_PR_X_MASTER
Pull request #X merged!
Merge hash: 485658a5
Would you like to pick 485658a5 into another branch? (y/n):
For now just say n as we have 1 branch
Verifying Release Candidates
We have provided a script to assist with verifying release candidates:
bash dev/release/verify-release-candidate.sh 0.7.0 0
Currently this only works on Linux (patches to expand to macOS welcome!). Read the script for information about system dependencies.
On Windows, we have a script that verifies C++ and Python (requires Visual Studio 2015):
dev/release/verify-release-candidate.bat apache-arrow-0.7.0.tar.gz
Verifying the JavaScript release
For JavaScript-specific releases, use a different verification script:
bash dev/release/js-verify-release-candidate.sh 0.7.0 0
Integration testing
Build the following base image used by multiple tests:
docker build -t arrow_integration_xenial_base -f docker_common/Dockerfile.xenial.base .
HDFS C++ / Python support
docker-compose build conda-cpp
docker-compose build conda-python
docker-compose build conda-python-hdfs
docker-compose run --rm conda-python-hdfs
Apache Spark Integration Tests
Tests can be run to ensure that the current snapshot of Java and Python Arrow works with Spark. This will run a docker image to build Arrow C++ and Python in a Conda environment, build and install Arrow Java to the local Maven repository, build Spark with the new Arrow artifact, and run Arrow related unit tests in Spark for Java and Python. Any errors will exit with a non-zero value. To run, use the following command:
docker-compose build conda-cpp
docker-compose build conda-python
docker-compose build conda-python-spark
docker-compose run --rm conda-python-spark
If you already are building Spark, these commands will map your local Maven repo to the image and save time by not having to download all dependencies. Be aware, that docker write files as root, which can cause problems for maven on the host.
docker-compose run --rm -v $HOME/.m2:/root/.m2 conda-python-spark
NOTE: If the Java API has breaking changes, a patched version of Spark might need to be used to successfully build.