# Description
Exposes min_commit_interval in the Python API to `optimize.compact` and
`optimize.z_order`. Added one test-case to verify the
min_commit_interval.
# Related Issue(s)
closes#1640
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
# Description
Updated the API to:
```python
DeltaTable.optimize.compact()
DeltaTable.optimize.z_order()
```
The old API of `DeltaTable.optimize()` still works, but now issues a
deprecation warning.
# Related Issue(s)
- closes#1442
# Documentation
<!---
Share links to useful documentation
--->
# Description
The description of the main changes of your pull request
# Related Issue(s)
- closes#715
- closes#373
# Documentation
<!---
Share links to useful documentation
--->
---------
Co-authored-by: Robert Pack <42610831+roeap@users.noreply.github.com>
# Description
This PR sets up ruff for python linting. It also enables isort from
within ruff and removes the standalone isort.
# Related Issue(s)
closes#1149
---------
Signed-off-by: Marijn Valk <marijncv@hotmail.com>
# Description
Adds a more specific ValueError if a wrong `table_or_uri` is provided in
the `write_delta_lake` function. It also adds support for a pathlib
`Path` object.
# Related Issue(s)
- closes#1123
# Documentation
<!---
Share links to useful documentation
--->
---------
Signed-off-by: Marijn Valk <marijncv@hotmail.com>
# Description
Recently we moved some of our storage configuration via a property bag
upstream to the object_store crate. This allows us to simplify our
configuration handling here and make S3 configuration consistent with
azure and gcp.
I think as a follow up it would be great to migrate dynamodb_lock to
using the official SDKs as well, and then see what we still need form
the s3 storage options.
# Related Issue(s)
closes#999
# Documentation
<!---
Share links to useful documentation
--->
Co-authored-by: Will Jones <willjones127@gmail.com>
* feat: start moving toward Rust implementation of Python Schema
* feat: implemention schema
* feat: Move implementation into schema.rs
* fix: get tests passing
* fix: cargo fmt
* fix: use default names so we can roundtrip maptype and listtype
* feat: complete docs for Schema
* fix: import literal from typing_extensions
* fix: add back original warnings
* fix: Add pyarrow type stubs
* Expose read and write options in public API
* Add test for reading
* Remove small date as there seems to be another issue with it
* Test writer options
* Make linter happy
* Add proper test with writer options and make mypy happy
* Remove unused imports from test_writer
* PR comments
* Lint
* Format using direct command
* Apply suggestions from reviewers
* Initial writer implementation
* Add basic partitioning support
* Update docs and link to other projects
* Add Pandas support
* Test writer stats and partitioning
* Test statistics
* Enforce protocol version
* Add experimental to docstring
* Need tying extensions for checking now
* Add nipick ignore for typing_extensions
* Update ADLSGen2-HOWTO.md
I had a hard time figuring out how to connect to a delta table that is stored in ADLS Gen2 and only found a way by digging into the source code. I would like to save other people the same trouble by adding this to the docs.
* altered prefix for azure
* Refresh Python usage documentation
* Reflowed the usage documentation to start with loading, then look at
log introspection, and finally querying tables.
* Added info about supported backends and data catalogs.
* Added more examples and guidance on how to query Delta tables.
* Finish drafting docs
* Clarify history and version retention
* Don't link to dask
* mention how to auth to Glue
* only release new doc after successful pypi release
* document steps for creating new python release
* update docs to reference the new `to_pandas` method
Ion Koutsouris