mirror of
https://github.com/langchain-ai/datafusion.git
synced 2026-07-19 15:24:21 -04:00
Krisztián Szűcs
b26b2cf17c
Also resolves: - ARROW-11213: [Packaging][Python] Dockerize wheel building on windows - ARROW-11215: [CI] Use named volumes by default for caching in docker-compose - ARROW-11231: [Python][Packaging] Enable mimalloc in wheels Main features: - dockerize windows builds - use vcpkg as the dependency source where we can explicitly pin a working version without worrying about the drag of build environment over time Potential follow-up: - use vcpkg on macos as well cc @kou @xhochy ## Manylinux testing Should be straightforward (I'm going to shared pre-built images for quicker testing). ## Windows testing Only windows host is able to run windows containers. I'm virtualizing windows on macOS (should work on linux as well) using virtualbox: ```bash git clone https://github.com/StefanScherer/windows-docker-machine cd windows-docker-machine # grant more resources in the vagrantfile under the virtualbox section (I use 8 cores and 16GiB of ram) vagrant up --provider virtualbox 2019-box docker context ls docker context use 2019-box docker image ls ``` Now docker should use the windows docker daemon. ```bash pip install -e dev/archery[docker] # the volumes defined in the docker compose file should be commented out and pass arrow's source explicitly # this is required because compose is being executed on a unix host and the directory is mounted through a virtual machine PYTHON=3.6 archery docker run --no-pull --using-docker-cli -v C:$(pwd):C:/arrow python-wheel-windows-vs2017 cmd.exe # then within the container execute a build or do other interactive things arrow\ci\scripts\python_wheel_windown_build.bat ``` Closes #9096 from kszucs/wheel-vcpkg Lead-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Co-authored-by: Sutou Kouhei <kou@clear-code.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
55 lines
1.8 KiB
Batchfile
Executable File
55 lines
1.8 KiB
Batchfile
Executable File
@rem Licensed to the Apache Software Foundation (ASF) under one
|
|
@rem or more contributor license agreements. See the NOTICE file
|
|
@rem distributed with this work for additional information
|
|
@rem regarding copyright ownership. The ASF licenses this file
|
|
@rem to you under the Apache License, Version 2.0 (the
|
|
@rem "License"); you may not use this file except in compliance
|
|
@rem with the License. You may obtain a copy of the License at
|
|
@rem
|
|
@rem http://www.apache.org/licenses/LICENSE-2.0
|
|
@rem
|
|
@rem Unless required by applicable law or agreed to in writing,
|
|
@rem software distributed under the License is distributed on an
|
|
@rem "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
@rem KIND, either express or implied. See the License for the
|
|
@rem specific language governing permissions and limitations
|
|
@rem under the License.
|
|
|
|
@echo on
|
|
|
|
set PYARROW_TEST_CYTHON=OFF
|
|
set PYARROW_TEST_DATASET=ON
|
|
set PYARROW_TEST_GANDIVA=OFF
|
|
set PYARROW_TEST_HDFS=ON
|
|
set PYARROW_TEST_ORC=OFF
|
|
set PYARROW_TEST_PANDAS=ON
|
|
set PYARROW_TEST_PARQUET=ON
|
|
set PYARROW_TEST_PLASMA=OFF
|
|
set PYARROW_TEST_S3=OFF
|
|
set PYARROW_TEST_TENSORFLOW=ON
|
|
set PYARROW_TEST_FLIGHT=ON
|
|
|
|
set ARROW_TEST_DATA=C:\arrow\testing\data
|
|
set PARQUET_TEST_DATA=C:\arrow\submodules\parquet-testing\data
|
|
|
|
@REM Install the built wheels
|
|
python -m pip install numpy
|
|
python -m pip install --no-index --find-links=C:\arrow\python\dist\ pyarrow || exit /B
|
|
|
|
@REM Test that the modules are importable
|
|
python -c "import pyarrow"
|
|
python -c "import pyarrow._hdfs"
|
|
python -c "import pyarrow._s3fs"
|
|
python -c "import pyarrow.csv"
|
|
python -c "import pyarrow.dataset"
|
|
python -c "import pyarrow.flight"
|
|
python -c "import pyarrow.fs"
|
|
python -c "import pyarrow.json"
|
|
python -c "import pyarrow.parquet"
|
|
|
|
@REM Install testing dependencies
|
|
pip install -r C:\arrow\python\requirements-wheel-test.txt
|
|
|
|
@REM Execute unittest
|
|
pytest -r s --pyargs pyarrow
|