Add Parquet export support for trace data #15

New Issue

2026-02-16T03:17:20-05:00

yindo commented

2026-02-16 03:17:20 -05:00

Originally created by @nishanthp on GitHub (Feb 4, 2026).

Originally assigned to: @AstraBert on GitHub.

Is your feature request related to a problem? Please describe.

When working with large volumes of trace data from OtelTracesSqlEngine, I need to export and archive traces for offline analysis, sharing with team members, or integration with data analytics pipelines. Currently, the only export option is to_pandas() which keeps data in memory, or to_sql_database() which requires a database connection. For long-term storage and efficient querying of historical traces, there's no built-in way to export to a performant file format.

Describe the solution you'd like

Add a to_parquet() method to export trace data in Apache Parquet format. This would enable:
-> Fast, efficient storage of trace data with 50-80% compression
-> Easy sharing of trace datasets without database dependencies
-> Integration with modern data analytics tools (DuckDB, Spark, Polars)
-> Partitioning support for time-series trace data (by service, date, etc.)

Describe alternatives you've considered

-> Manual export via pandas: Users can call engine.to_pandas().to_parquet(), but this requires extra steps and doesn't provide built-in partitioning or date extraction logic
-> CSV export: Simpler but much larger file sizes and slower query performance
-> JSON export: Good for interoperability but inefficient for large datasets
-> Keep everything in SQL: Works but requires maintaining database infrastructure for historical data

Additional context
Parquet is the industry standard for analytical workloads and offers 10-50x faster queries compared to CSV while using significantly less storage. It's already widely supported in the Python data ecosystem (pandas, pyarrow, polars, duckdb) making it a natural fit for trace data analysis workflows.
Dependencies: Requires pyarrow or fastparquet (commonly already installed with pandas).

Originally created by @nishanthp on GitHub (Feb 4, 2026). Originally assigned to: @AstraBert on GitHub. **Is your feature request related to a problem? Please describe.** When working with large volumes of trace data from OtelTracesSqlEngine, I need to export and archive traces for offline analysis, sharing with team members, or integration with data analytics pipelines. Currently, the only export option is to_pandas() which keeps data in memory, or to_sql_database() which requires a database connection. For long-term storage and efficient querying of historical traces, there's no built-in way to export to a performant file format. **Describe the solution you'd like** Add a to_parquet() method to export trace data in Apache Parquet format. This would enable: -> Fast, efficient storage of trace data with 50-80% compression -> Easy sharing of trace datasets without database dependencies -> Integration with modern data analytics tools (DuckDB, Spark, Polars) -> Partitioning support for time-series trace data (by service, date, etc.) **Describe alternatives you've considered** -> Manual export via pandas: Users can call engine.to_pandas().to_parquet(), but this requires extra steps and doesn't provide built-in partitioning or date extraction logic -> CSV export: Simpler but much larger file sizes and slower query performance -> JSON export: Good for interoperability but inefficient for large datasets -> Keep everything in SQL: Works but requires maintaining database infrastructure for historical data **Additional context** Parquet is the industry standard for analytical workloads and offers 10-50x faster queries compared to CSV while using significantly less storage. It's already widely supported in the Python data ecosystem (pandas, pyarrow, polars, duckdb) making it a natural fit for trace data analysis workflows. Dependencies: Requires **pyarrow** or **fastparquet** (commonly already installed with pandas).

yindo added the enhancement triage labels 2026-02-16 03:17:20 -05:00

yindo commented

2026-02-16 03:17:20 -05:00