This adds a static method `is_deltatable(path, opts)` to the
`DeltaTable` Python class, which returns `True` if able to
locate a delta table at specified `path` and `False` otherwise.
It does so by reusing the Rust internal `is_delta_table_location()`
via the `DeltaTableBuilder`.
Additionally, this also adds documentation of the usage with
examples for the `DeltaTable.is_deltatable()` method.
This replaces the old "overwrite_schema" parameter with a
schema_write_mode parameter that basically allows to distinguish between
overwrite/merge/none
Fixes#1386
# Description
As requested by @ion-elgreco in #2229 , we should fix the formatter
versions
# Related Issue(s)
<!---
For example:
- closes#106
--->
# Documentation
<!---
Share links to useful documentation
--->
# Description
First/naive implementation of `replaceWhere` for `write`. Code compiles
and there is a test to verify the outcome. I would appreciate any
feedback on improving the structure/implementation. For example, I
copied the part of code from `delete` operation because there is no way
to call that code in `delete` directly from `write` - should I look into
extracting that code from `delete` to somewhere central?
Seems to also works with partitions columns.
# Related Issue(s)
https://github.com/delta-io/delta-rs/issues/1957
# Documentation
Added a section in docs
---------
Signed-off-by: Nikolay Ulmasov <ulmasov@hotmail.com>
Co-authored-by: Ion Koutsouris <15728914+ion-elgreco@users.noreply.github.com>
# Description
I have added documentation in the API and also on the Python usage page
regarding this configuration. Please let me know if it is satisfactory,
and if not, I am more than happy to address any issues or make any
necessary adjustments.
# Related Issue(s)
- closes#1674
# Documentation
# Description
Completed the outstanding tasks in #1708
Also changed theme from readthedocs to mkdocs - both are built-in but
latter looks sleeker
# Related Issue(s)
closes#1708
---------
Co-authored-by: Robert Pack <42610831+roeap@users.noreply.github.com>
Co-authored-by: R. Tyler Croy <rtyler@brokenco.de>
# Description
- Exposes UPDATE operation to Python.
- Added two test cases, with predicate and without
- Took some learnings in simplifying the code (will apply it in MERGE PR
as well)
# Related Issue(s)
<!---
For example:
- closes#106
--->
Closes#1505
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
# Description
Exposes min_commit_interval in the Python API to `optimize.compact` and
`optimize.z_order`. Added one test-case to verify the
min_commit_interval.
# Related Issue(s)
closes#1640
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
# Description
Updated the API to:
```python
DeltaTable.optimize.compact()
DeltaTable.optimize.z_order()
```
The old API of `DeltaTable.optimize()` still works, but now issues a
deprecation warning.
# Related Issue(s)
- closes#1442
# Documentation
<!---
Share links to useful documentation
--->
# Description
The description of the main changes of your pull request
# Related Issue(s)
- closes#715
- closes#373
# Documentation
<!---
Share links to useful documentation
--->
---------
Co-authored-by: Robert Pack <42610831+roeap@users.noreply.github.com>
# Description
This PR sets up ruff for python linting. It also enables isort from
within ruff and removes the standalone isort.
# Related Issue(s)
closes#1149
---------
Signed-off-by: Marijn Valk <marijncv@hotmail.com>
# Description
Adds a more specific ValueError if a wrong `table_or_uri` is provided in
the `write_delta_lake` function. It also adds support for a pathlib
`Path` object.
# Related Issue(s)
- closes#1123
# Documentation
<!---
Share links to useful documentation
--->
---------
Signed-off-by: Marijn Valk <marijncv@hotmail.com>
# Description
Recently we moved some of our storage configuration via a property bag
upstream to the object_store crate. This allows us to simplify our
configuration handling here and make S3 configuration consistent with
azure and gcp.
I think as a follow up it would be great to migrate dynamodb_lock to
using the official SDKs as well, and then see what we still need form
the s3 storage options.
# Related Issue(s)
closes#999
# Documentation
<!---
Share links to useful documentation
--->
Co-authored-by: Will Jones <willjones127@gmail.com>
* feat: start moving toward Rust implementation of Python Schema
* feat: implemention schema
* feat: Move implementation into schema.rs
* fix: get tests passing
* fix: cargo fmt
* fix: use default names so we can roundtrip maptype and listtype
* feat: complete docs for Schema
* fix: import literal from typing_extensions
* fix: add back original warnings
* fix: Add pyarrow type stubs
* Expose read and write options in public API
* Add test for reading
* Remove small date as there seems to be another issue with it
* Test writer options
* Make linter happy
* Add proper test with writer options and make mypy happy
* Remove unused imports from test_writer
* PR comments
* Lint
* Format using direct command
* Apply suggestions from reviewers
* Initial writer implementation
* Add basic partitioning support
* Update docs and link to other projects
* Add Pandas support
* Test writer stats and partitioning
* Test statistics
* Enforce protocol version
* Add experimental to docstring
* Need tying extensions for checking now
* Add nipick ignore for typing_extensions
* Update ADLSGen2-HOWTO.md
I had a hard time figuring out how to connect to a delta table that is stored in ADLS Gen2 and only found a way by digging into the source code. I would like to save other people the same trouble by adding this to the docs.
* altered prefix for azure
* Refresh Python usage documentation
* Reflowed the usage documentation to start with loading, then look at
log introspection, and finally querying tables.
* Added info about supported backends and data catalogs.
* Added more examples and guidance on how to query Delta tables.
* Finish drafting docs
* Clarify history and version retention
* Don't link to dask
* mention how to auth to Glue
* only release new doc after successful pypi release
* document steps for creating new python release
* update docs to reference the new `to_pandas` method
R. Tyler Croy