mirror of
https://github.com/langchain-ai/datafusion.git
synced 2026-08-27 20:30:06 -04:00
main
14 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Gene Bordegaray
|
552dbe4f19 |
fix: Eliminate consecutive repartitions (#18521)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #18341. - Closes https://github.com/apache/datafusion/issues/9370 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Cases where two RepartitionExec operators appear consecutively in the plan. This is unneeded overhead that eliminating provides speed ups. Full Report: [The Physical Optimizer and Fixing Consecutive Repartitions In the Enforce Distribution Rule.pdf](https://github.com/user-attachments/files/23420831/The.Physical.Optimizer.and.Fixing.Consecutive.Repartitions.In.the.Enforce.Distribution.Rule.pdf) Issue Report: [Fixing Consecutive Repartitions In the Enforce Distribution Rule.pdf](https://github.com/user-attachments/files/23420880/Fixing.Consecutive.Repartitions.In.the.Enforce.Distribution.Rule.pdf) ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Change to repartition adding logic in `enforce_distribution.rs` A ton of test and bench updates to mirror new behavior ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes benchmarked and tested, check report for benchmarks ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Yongting You
|
ec2402aee9 |
feat: Support configurable EXPLAIN ANALYZE detail level (#18098)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> `EXPLAIN ANALYZE` can be used for profiling and displays the results alongside the EXPLAIN plan. The issue is that it currently shows too many low-level details. It would provide a better user experience if only the most commonly used metrics were shown by default, with more detailed metrics available through specific configuration options. ### Example In `datafusion-cli`: ``` > CREATE EXTERNAL TABLE IF NOT EXISTS lineitem STORED AS parquet LOCATION '/Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem'; 0 row(s) fetched. Elapsed 0.000 seconds. explain analyze select * from lineitem where l_orderkey = 3000000; ``` The parquet reader includes a large number of low-level details: ``` metrics=[output_rows=19813, elapsed_compute=14ns, batches_split=0, bytes_scanned=2147308, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=18, num_predicate_creation_errors=0, page_index_rows_matched=19813, page_index_rows_pruned=729088, predicate_cache_inner_records=0, predicate_cache_records=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=1, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=21.997µs, metadata_load_time=273.83µs, page_index_eval_time=29.915µs, row_pushdown_eval_time=42ns, statistics_eval_time=76.248µs, time_elapsed_opening=4.02146ms, time_elapsed_processing=24.787461ms, time_elapsed_scanning_total=24.17671ms, time_elapsed_scanning_until_data=23.103665ms] ``` I believe only a subset of it is commonly used, for example `output_rows`, `metadata_load_time`, and how many file/row-group/pages are pruned, and it would better to only display the most common ones by default. ### Existing `VERBOSE` keyword There is a existing verbose keyword in `EXPLAIN ANALYZE VERBOSE`, however it's turning on per-partition metrics instead of controlling detail level. I think it would be hard to mix this partition control and the detail level introduced in this PR, so they're separated: the following config will be used for detail level and the semantics of `EXPLAIN ANALYZE VERBOSE` keep unchanged. ### This PR: configurable explain analyze level 1. Introduced a new config option `datafusion.explain.analyze_level`. When set to `dev` (default value), all existing metrics will be shown. If set to `summary`, only `BaselineMetrics` will be displayed (i.e. `output_rows` and `elapsed_compute`). Note now we only include `BaselineMetrics` for simplicity, in the follow-up PRs we can figure out what's the commonly used metrics for each operator, and add them to `summary` analyze level, finally set the `summary` analyze level to default. 2. Add a `MetricType` field associated with `Metric` for detail level or potentially category in the future. For different configurations, a certain `MetricType` set will be shown accordingly. #### Demo ``` -- continuing the above example > set datafusion.explain.analyze_level = summary; 0 row(s) fetched. Elapsed 0.000 seconds. > explain analyze select * from lineitem where l_orderkey = 3000000; +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | plan_type | plan | +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Plan with Metrics | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=5, elapsed_compute=25.339µs] | | | FilterExec: l_orderkey@0 = 3000000, metrics=[output_rows=5, elapsed_compute=81.221µs] | | | DataSourceExec: file_groups={14 groups: [[Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-0.parquet:0..11525426], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-0.parquet:11525426..20311205, Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-1.parquet:0..2739647], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-1.parquet:2739647..14265073], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-1.parquet:14265073..20193593, Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-2.parquet:0..5596906], [Users/yongting/Code/datafusion/benchmarks/data/tpch_sf1/lineitem/part-2.parquet:5596906..17122332], ...]}, projection=[l_orderkey, l_partkey, l_suppkey, l_linenumber, l_quantity, l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate, l_commitdate, l_receiptdate, l_shipinstruct, l_shipmode, l_comment], file_type=parquet, predicate=l_orderkey@0 = 3000000, pruning_predicate=l_orderkey_null_count@2 != row_count@3 AND l_orderkey_min@0 <= 3000000 AND 3000000 <= l_orderkey_max@1, required_guarantees=[l_orderkey in (3000000)], metrics=[output_rows=19813, elapsed_compute=14ns] | | | | +-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row(s) fetched. Elapsed 0.025 seconds. ``` Only `BaselineMetrics` are shown. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 4. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> UT ## Are there any user-facing changes? No <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Chen Chongchen
|
4417d5c022 |
doc: fix indent format explain (#16085)
* doc: fix indent format explain * update |
||
|
Namgung Chan
|
0af945f7ec | update userguide (#15710) | ||
|
kosiew
|
63f37a3440 | docs: reorder - tree format (default) (#15706) | ||
|
Andrew Lamb
|
41e7aed3a9 |
Support EXPLAIN ... FORMAT <indent | tree | json | graphviz > ... (#15166)
* Support EXPLAIN FORMAT <format> * Update datafusion/sql/src/parser.rs Co-authored-by: Ruihang Xia <waynestxia@gmail.com> * Improve documentation * Remove to_stringified and simplify code * mark deprecated rather than remove * Add a note about explain format configuration --------- Co-authored-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
Dawei H.
|
3f900ac5e1 |
Test all examples from library-user-guide & user-guide docs (#14544)
* add mut annotation * fix rust examples * fix rust examples * update * fix first doctest * fix first doctest * fix more doctest * fix more doctest * fix more doctest * adopt rustdoc syntax * adopt rustdoc syntax * adopt rustdoc syntax * fix more doctest * add missing imports * final udtf * reenable * remove dep * run prettier * api-health * update doc * update doc * temp fix * fix doc * fix async schema provider * fix async schema provider * fix doc * fix doc * reorder * refactor * s * finish * minor update * add missing docs * add deps (#3) * fix doctest * update doc * fix doctest * fix doctest * tweak showkeys * fix doctest * fix doctest * fix doctest * fix doctest * update to use user_doc * add rustdoc preprocessing * fix dir * revert to original doc * add allocator * mark type * update * fix doctest * add doctest * add doctest * fix doctest * fix doctest * fix doctest * fix doctest * fix doctest * fix doctest * fix doctest * fix doctest * fix doctest * prettier format * revert change to datafusion-testing * add apache header * install cmake in setup-builder for ci workflow dependency * taplo + fix snmalloc * Update function docs * preprocess user-guide * Render examples as sql * fix intro * fix docs via script --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
mertak-synnada
|
5e1e693b6e |
Introduce unified DataSourceExec for provided datasources, remove ParquetExec, CsvExec, etc (#14224)
* unify ParquetExec, AvroExec, ArrowExec, NDJsonExec, MemoryExec into one DataSourceExec plan * fix license headers * fix compile errors on documents * separate non-parquet code * format code * fix typo * fix imports * fix clippy fix csv_json example * add comment to the example * fix cargo docs * change MemoryExec with MemorySourceConfig * merge fixes * change MemoryExec to DataSourceExec * fix merge conflicts * apply some syntactic sugars * fix imports and comment line * simplify some lines * rename source_config as file_source * format code * format code * make memory metrics default behavior * remove unnecessary cfg check * format code * remove ParquetExec strings * fix documents and imports * fix imports * add constraints and fix tests * delete redundant file * make metrics and statistics a part of File type specific configurations make cache a part of DataSourceExec * format code * fix tests * format code * split repartitioning into DataSourceExec and FileSourceConfig parts * move properties into DataSourceExec and split eq_properties and output_partitioning in DataSource trait * clone source with Arc * return file type as enum and do not downcast if not necessary create fmt_extra method * format code * re-add deprecated plans in order to support backward compatibility * reduce diff * fix doc * merge fixes * remove unnecessary files * rename config structs to source * remove empty files fix tests * removed FileSourceConfig projected_statistics must be solved! * fix base_config formatting * format code * fix repartition logic * fix merge conflicts * fix csv projection error * clippy fix * use new() on initialization * use DataSourceExec on deprecated file operators as well * move ParquetSource into source.rs fix doc errors * use ParquetSource only if parquet feature is enabled * fix slt tests * add with_fetch API to MemorySourceConfig and re-add deprecated MemoryExec * fix merge conflicts fix memory source fetch error * format code * change FileType enum into a dyn Trait so that it can be extensible * remove metadata_size_hint from required ParquetSource parameters * remove FileType trait and split with_predicate logic for ParquetSource * remove predicate from initialization of ParquetSource * remove unnecessary imports * deprecate ParquetExecBuilder and add doc hints * fix slt * fix clippy * fix fmt * return reference of the Arc in source() * re-add deprecated exec files * fix doc error |
||
|
WeblWabl
|
7d8bb0b346 |
Add documentation on EXPLAIN and EXPLAIN ANALYZE (#12122)
* feat(wip): working on adding explain docs Signed-off-by: Devan <devandbenz@gmail.com> * working on it Signed-off-by: Devan <devandbenz@gmail.com> * working on it -- adding plan descriptions Signed-off-by: Devan <devandbenz@gmail.com> * adds descriptions for phys plan and note on parallel Signed-off-by: Devan <devandbenz@gmail.com> * adds information about the logical plan Signed-off-by: Devan <devandbenz@gmail.com> * Add page to index * Update example, add information on how to get the datafile * Add example of reading explain analyze * Add section on partitioning * prettier * adding aggregate plan explain docs Signed-off-by: Devan <devandbenz@gmail.com> * field -> column Signed-off-by: Devan <devandbenz@gmail.com> * repartition update Signed-off-by: Devan <devandbenz@gmail.com> * prettier Signed-off-by: Devan <devandbenz@gmail.com> * clarify some points --------- Signed-off-by: Devan <devandbenz@gmail.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Chih Wang
|
ae88235617 |
Fix sphinx warnings (#9142)
* Fix 'Configuration Settings' link in write_options.md * Fix multiple reference definition in scalar_functions.md * Fix typo: 'contcat_ws' to 'concat_ws' in scalar_functions.md * Add missing section 'array_contains' in scalar_functions.md * Fix reference warnings in operators.md * Remove unsupported syntax 'csv' from code blocks of adding-udfs.md * Fix syntax tag in example-usage.md * Change syntax tag of code blocks that contain multiple syntaxes of cli.md * Remove 'sql' syntax tags from code blocks * Remove leading ❯ in sql code blocks |
||
|
Tzu-Chiao Yeh
|
6be75ff2dc |
Supply consistent format output for FileScanConfig params (#6202)
* Supply consistent format output for FileScanConfig params * Compact display output and optimize output ordering display * Appease clippy * update sqllogictest --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> |
||
|
Jeffrey
|
3ad7734a7e |
Update sql doc (#6025)
* Update/touch-up user-guide sql doc pages * Update ddl doc |
||
|
Martin Grigorov
|
7310385e85 |
Minor: Fix typos in the documentation (#4376)
* Rename 'function_table' to 'function' in the User Guide cli docs 1. function_table sounds weird 2. The cli help uses just 'function' - https://github.com/apache/arrow-datafusion/blob/e1204a5bf72c119123404463befb716adbdcff25/datafusion-cli/src/command.rs#L140 Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org> * Improve wording Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org> * Fix typos Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org> Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org> |
||
|
unvalley
|
e395e30cd9 |
User Guide: Add EXPLAIN to SQL reference (#3767)
* docs: add explain.md * docs: add explain to sql index * docs: describe explain at the beginning * docs: mention VERBOSE |