langchain-ai/datafusion
1
0
Fork 0
mirror of https://github.com/langchain-ai/datafusion.git synced 2026-07-19 15:24:21 -04:00
Code Issues 3 Packages Projects Releases Wiki Activity
2,764 Commits 4 Branches 0 Tags
bec1082756394d3fbb9aacc400a90c35b78e5dde
Commit Graph

7 Commits

Author SHA1 Message Date
Zhuo Peng fcf51041ae ARROW-7362: [Python][C++] Added ListArray.Flatten() that properly flattens a ListArray 
Currently ListArray.flatten() simply returns the child array. If a ListArray is a slice of another ListArray, they will share the same child array, however the expected behavior (I think) of flatten() should be returning an Array that's a concatenation of all the sub-lists in the ListArray, so the slicing offset should be taken into account.

For example:
```python
a = pa.array([[1], [2], [3]])

assert a.flatten().equals(pa.array([1,2,3]))

# expected:
a.slice(1).flatten().equals(pa.array([2, 3]))
```

Closes #6006 from brills/flatten and squashes the following commits:

4702f59da <Antoine Pitrou> Improve implementation characteristics
d14210dc4 <Antoine Pitrou> Address review comments, add a test for non-canonical list arrays
c789812d0 <Antoine Pitrou> Fix git merge error
7d0b864bf <Antoine Pitrou> Fix typo + print out conda env in conda docker builds
5f3650f90 <Zhuo Peng> comments
6aa3181b8 <Zhuo Peng> Take care of nulls correctly.
3c8746241 <Zhuo Peng> Also changed the python wrapper.
a3d4d2fbb <Zhuo Peng> Added a C++ method Flatten() to ListArray.

Lead-authored-by: Zhuo Peng <1835738+brills@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
 2019-12-18 17:57:32 +01:00
Krisztián Szűcs 17d67eb5a2 ARROW-6341: [Python] Implement low-level bindings for Dataset 
Closes #5237 from kszucs/ARROW-6341 and squashes the following commits:

45121f77a <Krisztián Szűcs> Fix tests for MockFs
069d8f55e <Krisztián Szűcs> Don't expose SimpleDataSource
48b5556ef <Krisztián Szűcs> Test projected partitions
27dbe56ab <Krisztián Szűcs> Don't deprecate RandomFileAccess just remove in favor of CRandomFileAccess
64ca71245 <Krisztián Szűcs> Execute instead of scan
bd0d1d2e7 <Krisztián Szűcs> more type_name
f2ab5ebfe <Krisztián Szűcs> type_name
3988865ca <Krisztián Szűcs> Rebase again
b8949a3ce <Krisztián Szűcs> Result iterator api
7553caa96 <Krisztián Szűcs> Clang format
a1d82546f <Krisztián Szűcs> Remove ScanContext
6260734a3 <Krisztián Szűcs> Fix api changes
e6a562356 <Krisztián Szűcs> Expose root_partition setter; resolve a couple of review issues
c9ba0fb93 <Krisztián Szűcs> Fix review comments
f589ecb23 <Krisztián Szűcs> Removed todo notes
9b38a40cf <Krisztián Szűcs> Docstring additions
3210e9160 <Krisztián Szűcs> Fixing review issues
4384b74cf <Krisztián Szűcs> Enable PYARROW_BUILD_DATASET
f52e735a0 <Krisztián Szűcs> Remove DataFragment and ScanOptions
13eaf46a0 <Krisztián Szűcs> Remove move workaround
620ba6ffa <Krisztián Szűcs> schema as property
e9f77bd6b <Krisztián Szűcs> Expose root_partition
01510bcf8 <Krisztián Szűcs> Some docstrings
5beb0d26c <Krisztián Szűcs> Pxd definition fixes
f89dc4913 <Krisztián Szűcs> Data fragments
c9881c858 <Krisztián Szűcs> Downcast data fragments
032a4358c <Krisztián Szűcs> More expressions and testing
1a4a8544a <Krisztián Szűcs> Fix import errors if dataset is not enabled
2da1b5c76 <Krisztián Szűcs> Please the linters
d1bc74efe <Krisztián Szűcs> HivePartitionScheme
bf5dd17f4 <Krisztián Szűcs> Release the gil for std::move
a76dc6c3c <Krisztián Szűcs> Remove the move headers from flight
8cdfe1054 <Krisztián Szűcs> Expose more methods
53b64910e <Krisztián Szűcs> Expose Scalar/Comparison/Boolean expressions
444ae58a0 <Krisztián Szűcs> Expressions and scalar wrapping
01029a666 <Krisztián Szűcs> Test parquet data discovery
2e416ea8f <Krisztián Szűcs> Expressions
d14cf502b <Krisztián Szűcs> PartitionScheme
bd6e1d656 <Krisztián Szűcs> FileSystemDataSourceDiscovery
0c0e3752f <Krisztián Szűcs> Working scanner
18cfd949b <Krisztián Szűcs> Resolve issues with handling iterator results
43c3d2bb2 <Krisztián Szűcs> MockFileSystem and use Results
3bbc51bd0 <Krisztián Szűcs> MockFileSystem
5e279ead2 <Krisztián Szűcs> Basic scanning
c0d6715aa <Krisztián Szűcs> Rebase datasets
132e2f553 <Krisztián Szűcs> license
d51b560d7 <Krisztián Szűcs> data fragments and data sources
a0b35c7a0 <Krisztián Szűcs> skeleton and headers

Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
 2019-12-13 12:26:07 -05:00
Krisztián Szűcs cdece26054 ARROW-7310: [Python] Expose HDFS implementation for pyarrow.fs 
Closes #5990 from kszucs/ARROW-7310 and squashes the following commits:

1521226f2 <Krisztián Szűcs> Typo
206492bae <Krisztián Szűcs> Fix docstring
65796f8fc <Krisztián Szűcs> Use hdfs options by default in the test suite
ce5c77b1b <Krisztián Szűcs> Docstrings
f38fdfd16 <Krisztián Szűcs> Add user to HDFS uri
8b2f44e7e <Krisztián Szűcs> Remove the unreleased s3fs module in favor of pyarrow.fs
8cbae0d60 <Krisztián Szűcs> Fix hdfs input/output stream closing; fix append to file tests; test hdfs options
9eb10adb0 <Krisztián Szűcs> Enable hdfs tests
808e22725 <Krisztián Szűcs> Fix import in conftest
73d916842 <Krisztián Szűcs> Python bindingds for HDFS filesystem

Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
 2019-12-12 11:57:53 +01:00
Krisztián Szűcs 24fbf74691 ARROW-7346: [CI] Explicit usage of ccache across the builds 
- ccache configuration is centralised in the docker-compose.yml
- stricter build directory paths
- java jni tests were not running, since the vendoring of flatbuffers there was a hidden linking error

Closes #5982 from kszucs/explicit_use_ccache and squashes the following commits:

90fcd0c05 <Krisztián Szűcs> Sort env definitions alphabetically
30a64dbb5 <Krisztián Szűcs> Lint CMake
d76282bde <Krisztián Szűcs> enable plasma java client; fix flatbuffers link error because it is bundled now
b6d47db8c <Krisztián Szűcs> Fix PLASMA_STORE binary path
c4648afa4 <Krisztián Szűcs> Explicit ccache; centralized config in docker-compose

Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
 2019-12-08 13:07:08 +01:00
Sutou Kouhei a1c7a67ab2 ARROW-7334: [CI][Python] Use Python 3 on macOS 
Currently, Python 2 is used.

Closes #5979 from kou/ci-python-macos-use-python3 and squashes the following commits:

519c5f1d3 <Sutou Kouhei> Use sys.executable
240ab7743 <Sutou Kouhei>  Use Python 3 on macOS

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
 2019-12-06 14:15:21 +01:00
Antoine Pitrou 918080c4e3 ARROW-6954: [Python] [CI] Add Python 3.8 to CI matrix 
Also use a separate conda environment rather than installing everything
in the base conda environment, as conda can complain about conflicts.

Closes #5877 from pitrou/ARROW-6954-py38-ci and squashes the following commits:

0f411a482 <Krisztián Szűcs> Remove pandas master build for python 2.7 because pandas has already dropped support
9924be5bd <Krisztián Szűcs> Fix argument validation for install scripts
0ef0ae453 <Krisztián Szűcs> Fix conda-r image
5737cddd5 <Krisztián Szűcs> Missing CONDA_PREFIX
7b6bd7c14 <Krisztián Szűcs> Use a default conda environment other than base
e620b3b3f <Antoine Pitrou> ARROW-6954:   Add Python 3.8 to CI matrix

Lead-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
 2019-11-26 15:37:47 +01:00
Krisztián Szűcs 6102db6b1f ARROW-7101: [CI] Refactor docker-compose setup and use it with GitHub Actions 
## Projecting ideas from ursabot

### Parametric docker images

The images are better parameterized now, meaning that we can build more variant of the same service. Couple of examples:

```console
UBUNTU=16.04 docker-compose build ubuntu-cpp
ARCH=arm64v8 UBUNTU=18.04 docker-compose build ubuntu-cpp
PYTHON=3.6 docker-compose build conda-python
ARCH=arm32v7 PYTHON=3.6 PANDAS=0.25 docker-compose build conda-python-pandas
```

Each variant has it's own docker image following a string naming schema:
`{org}/{arch}-{platform}-{platform-version}[[-{variant}-{variant-version}]..]:latest`

### Use *_build.sh and *_test.sh for each job

The docker images provide the environment, and each language backend usually should implement  two scripts, a `build.sh` and a `test.sh`. This way dependent build like the docker python, r or c glib are able to reuse the build script of the ancestor without running its tests.

With small enough scripts, if the environment is properly set up even the non-docker builds should be reproducible locally. GitHub Actions support bash scripts across all three platforms, so we can reuse the same `*_build.sh` and `*_test.sh` scripts to execute the builds either in docker, on the CI or locally.

## Using GitHub Actions for running the builds

Regardless of the CI we're going to choose, the isolation constraint of different platforms requires some sort of virtualisation. Currently linux (and windows, but I have not tried it yet) has lightweight containerisation, so we should keep the linux builds isolated in docker containers. The rest of the platforms (windows and macOS) should be executed on the CI system.

GitHub Actions support all three major platforms, linux, windows and macOS. I've added cross platform builds for a couple of languages, like Rust, and Go, the rest are work in progress.

### Workflow

A workflow should define all builds of a language, mostly because the path filters can be defined on workflow level. For example the python builds should be triggered if either a cpp/** or a python/** file changes which can be covered in the same workflow file.

## Feature parity with the current builds

Reaching feature parity with all of the builds below is not a goal for this PR, the difficult ones should at least have a tracking JIRA ticket.

### Travis-CI

- [x] **Lint, Release tests**:
  - `Lint / C++, Python, R, Rust, Docker, RAT`
  - `Dev / Source Release`
- [x] **C++ unit tests w/ conda-forge toolchain, coverage**: without coverage
  - `C++ / AMD64 Conda C++`
- [x] **Python 3.6 unit tests, conda-forge toolchain, coverage**: without coverage
  - `Python / AMD64 Conda Python 3.6`
- [x] **[OS X] C++ w/ Xcode 9.3**:
  - `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3
- [x] **[OS X] Python w/ Xcode 9.3**:
  - `Python / AMD64 MacOS 10.14 Python 3`: with Xcode 10.3
- [x] **Java OpenJDK8 and OpenJDK11**:
  - `Java / AMD64 Debian Java JDK 8 Maven 3.5.2`
  - `Java / AMD64 Debian Java JDK 11 Maven 3.6.2`
- [x] **Protocol / Flight Integration Tests**:
  - `Dev / Protocol Test`
- [x] **NodeJS**: without running lint and coverage
  - `NodeJS / AMD64 Debian NodeJS 11`
- [x] **C++ & GLib & Ruby w/ gcc 5.4**:
  - `C++ / AMD64 Debian 10 C++`: with GCC 8.3
  - `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4
  - `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4
  - `C GLib / AMD64 Ubuntu 18.04 C GLib`
  - `Ruby / AMD64 Ubuntu 18.04 Ruby`
- [x] **[OS X] C++ & GLib & Ruby w/ XCode 10.2 & Homebrew**
  - `C++ / AMD64 MacOS 10.14 C++`: with Xcode 10.3
  - `C GLib / AMD64 MacOS 10.14 C Glib`: with Xcode 10.3
  - `Ruby / AMD64 MacOS 10.14 Ruby`: with Xcode 10.3
- [x] **Go**: without coverage
  - `Go / AMD64 Debian Go 1.12`
- [x] **R (with and without libarrow)**:
  - `R / AMD64 Conda R 3.6`: with libarrow
  - `R / AMD64 Ubuntu 18.04 R 3.6` with libarrow

### Appveyor

- ~JOB=Build, GENERATOR=Ninja, CONFIGURATION=Release, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017~
- ~JOB=Toolchain, GENERATOR=Ninja, CONFIGURATION=Release, ARROW_S3=ON, ARROW_BUILD_FLIGHT=ON, ARROW_BUILD_GANDIVA=ON~
- ~JOB=Build_Debug, GENERATOR=Ninja, CONFIGURATION=Debug~
- ~JOB=MinGW32, MINGW_ARCH=i686, MINGW_PACKAGE_PREFIX=mingw-w64-i686, MINGW_PREFIX=c:\msys64\mingw32, MSYSTEM=MINGW32, USE_CLCACHE=false~
- ~JOB=MinGW64, MINGW_ARCH=x86_64, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64, MINGW_PREFIX=c:\msys64\mingw64, MSYSTEM=MINGW64, USE_CLCACHE=false~
- [x] **JOB=Rust, TARGET=x86_64-pc-windows-msvc, USE_CLCACHE=false**:
  - `Rust / AMD64 Windows 2019 Rust nightly-2019-09-25`
- [x] **JOB=C#, APPVEYOR_BUILD_WORKER_IMAGE=Visual Studio 2017, USE_CLCACHE=false**
  - `C# / AMD64 Windows 2019 C# 2.2.103`
- [x] **JOB=Go, MINGW_PACKAGE_PREFIX=mingw-w64-x86_64 ...**:
  - `Go / AMD64 Windows 2019 Go 1.12`
- ~JOB=R with libarrow, USE_CLCACHE=false, TEST_R_WITH_ARROW=TRUE, RWINLIB_LOCAL=%APPVEYOR_BUILD_FOLDER%\libarrow.zip~

### Github Actions

- [x] **Windows MSVC C++ / Build (Visual Studio 16 2019)**:
  - `C++ / AMD64 Windows 2019 C++`: without tests
- [x] **Windows MSVC C++ / Build (Visual Studio 15 2017)**:
  - `C++ / AMD64 Windows 2016 C++`: without tests
- [x] **Linux docker-compose / Test (C++ w/ clang-7 & system packages)**: all have llvm for gandiva but the compiler is set to gcc
  - `C++ / AMD64 Debian 10 C++`: with GCC 8.3
  - `C++ / AMD64 Ubuntu 16.04 C++`: with GCC 5.4
  - `C++ / AMD64 Ubuntu 18.04 C++`: with GCC 7.4
- [x] **Linux docker-compose / Test (Rust)**: without rustfmt
  - `Rust / AMD64 Debian Rust nightly-2019-09-25`
- [x] **Linux docker-compose / Test (Lint, Release tests)**:
  - `Lint / C++, Python, R, Rust, Docker, RAT`
  - `Dev / Source Release`

### Nightly Crossbow tests

The packaging builds are out of the scope if this PR, but the nightly **dockerized test** task are in.

Nightly tests:
- [x] docker-r
- [x] docker-r-conda
- [x] docker-r-sanitizer
- [x] docker-rust
- [x] docker-cpp
- [x] docker-cpp-cmake32
- [x] docker-cpp-release
- [x] docker-cpp-static-only
- [x] docker-c_glib
- [x] docker-go
- [x] docker-python-2.7
- [x] docker-python-3.6
- [x] docker-python-3.7
- [x] docker-python-2.7-nopandas
- [x] docker-python-3.6-nopandas
- [x] docker-java
- [x] docker-js
- [x] docker-docs
- [x] docker-lint
- [x] docker-iwyu: included in the lint
- [x] docker-clang-format: included in the lint
- [x] docker-pandas-master
- [x] docker-dask-integration
- [x] docker-hdfs-integration
- [x] docker-spark-integration
- [x] docker-turbodbc-integration

# TODOs left:

- [x] Fix the Apidoc generation for c_glib
- [x] Fix the JNI test for Gandiva and ORC
- [x] Test that crossbow tests are passing
- ~Optionally restore the travis configuration to incrementally decommission old builds~

## Follow-up JIRAs:

- [Archery] Consider porting the docker tool of ursabot to archery
- [Archery] Consider to use archery with or instead of the pre-commit hooks
- [Archery] Create a wrapper script in archery for docker compose in order to run the containers with the host's user and group
- [C++] GCC 5.4.0 has a compile errors, reproduce with UBUNTU=16.04 docker-compose run ubuntu-cpp
- [C++][CI] Test the ported fuzzit integration image
- [C++][CI] Turn off unnecessary features in the integration tests (spark/turbodbc/dask/hdfs)
- [C++][CI] Revisit ASAN UBSAN settings in every C++ based image
- [CI] Consider re-adding the removed debian testing image is removed
- [Go][CI] Pre-install the go dependencies in the dockerfile using go get
- [JS][CI] Pre-install the JS dependencies in the dockerfile
- [Rust][CI] Pre-install the rust dependencies in the dockerfile
- [Java][CI] Pre-install the java dependencies in the dockerfile
- [Ruby][CI] Pre-install the ruby dependencies in the dockerfile and remove it from the test script
- [C#][CI] Pre-install the C# dependencies in the dockerfile
- [R][CI] Fix the r-sanitizer build https://issues.apache.org/jira/browse/ARROW-6957
- [GLIB][MacOS] Fail to execute lua examples (fails to load 'lgi.corelgilua51' despite that lgi is installed)
- [C++][CMake] Automatically set ARROW_GANDIVA_PC_CXX_FLAGS for conda and OSX sdk (see cpp_build.sh)
- [C++][CI] Hiveserver2 instegration test fails to connect to impala container
- [CI][Spark] Support specific Spark version in the integration tet including latest
- [JS][CI] Move nodejs linting from js_build.sh to archery
- [Python][CI] create a docker image for python ASV benchmarks and fix the script
- [CI] Find a short but related prefix for the env vars used for the docker-compose file to prevent collisions
- [C#] the docker container fails to run because of the ubuntu host versions, see https://github.com/dotnet/core/issues/3509
- [C++][Windows] Enable more features on the windows GHA build
- [Doc] document docker-compose usage in the developer sphinx guide
- [CI][C++] Add .ccache to the docker-compose mounts
- [Archery][CI] Refactor the ci/scripts to a sourceable bash functions or to archery directly
- [C++][CI] Use scripts/util_coredump.sh to show automatic backtraces
- [C++] Fix the hanging C++ tests in Windows 2019
- [CI] Ask INFRA to set up the DOCKERHUB_* secrets for GitHub actions
- [C++][CI] Running Gandiva tests fails on Fedora:
    Reproduce with: `docker-compose run -e ARROW_GANDIVA=ON fedora-cpp`
    ```
    Running gandiva-internals-test, redirecting output into /build/cpp/build/test-logs/gandiva-internals-test.txt (attempt 1/1)
    1364
    : CommandLine Error: Option 'x86-experimental-vector-widening-legalization' registered more than once!
    1365
    LLVM ERROR: inconsistency in registered CommandLine options
    1366
    /build/cpp/src/gandiva
    ```
- [JS][CI] NodeJS build fails on Github Actions Windows node
    ```
    > NODE_NO_WARNINGS=1 gulp build
    # 'NODE_NO_WARNINGS' is not recognized as an internal or external command,
    # operable program or batch file.
    # npm ERR! code ELIFECYCLE
    # npm ERR! errno 1
    # npm ERR! apache-arrow@1.0.0-SNAPSHOT build: `NODE_NO_WARNINGS=1 gulp build`
    # npm ERR! Exit status 1
    # npm ERR!
    # npm ERR! Failed at the apache-arrow@1.0.0-SNAPSHOT build script.
    # npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
    ```

Closes #5589 from kszucs/docker-refactor and squashes the following commits:

5105d12e6 <Krisztián Szűcs> Rename pull-request folder to dev_cron
e9e9a7eec <Krisztián Szűcs> Use underscores for naming the workflow files
a92c99d03 <Krisztián Szűcs> Disable hanging C++ tests on windows
f158c89b5 <Krisztián Szűcs> Attempt to push from apache/arrow master; Don't push from crossbow tasks
0e1d470a1 <Krisztián Szűcs> Turn off ORC on macOS C++ test due to link error
258db5cff <Krisztián Szűcs> Only push docker images from apache/arrow repository
acdfcf086 <Krisztián Szűcs> Remove ORC from the brewfile
5102b85b1 <Krisztián Szűcs> Fix nodeJS workflow
032d6a388 <Krisztián Szűcs> Turn off 2 python builds
7f15b97a8 <Krisztián Szűcs> Filter branches
48b8d128a <Krisztián Szűcs> Fix workflows
36ad9d297 <Krisztián Szűcs> Disable builds
0f603af0c <Krisztián Szűcs> master only and cron workflows
28cc2d78d <Krisztián Szűcs> Rename Java JNI workflow
bcd8af7b7 <Krisztián Szűcs> Port the remaining travis utility scripts
ed5688154 <Krisztián Szűcs> Usage comments; recommend installing pandas from the docs because of its removal from conda_env_python
3c8c023ce <Krisztián Szűcs> Use Arch in volumes; some comments; remove conda version 'latest' from the images
771b023a8 <Krisztián Szűcs> Cleanup files; separate JNI builds
97ff8a122 <Krisztián Szűcs> Push docker images only from master
dc00b4297 <Krisztián Szűcs> Enable path filters
e0e2e1f46 <Krisztián Szűcs> Fix pandas master build
3814e0828 <Krisztián Szűcs> Fix manylinux volumes
c18edda70 <Krisztián Szűcs> Add CentOS version to the manylinux image names
c8b9dd6b1 <Krisztián Szűcs> Missing --pyargs argument for the python test command
33e646981 <Krisztián Szűcs> Turn off gandiva and flight for the HDFS test
b9c547889 <Krisztián Szűcs> Refactor docker-compose file and use it with github actions.

Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
 2019-11-12 11:07:48 +01:00