StarRocks Community 2025 Roadmap for Paimon

用户6616

2025年2月20日修改

Backgroud

•
Paimon 1.0 official release: this marks a milestone version of Paimon, a production-ready official release.​

•
Alibaba Group's OpenLake project with StarRocks as the primary Lakehouse engine: many Paimon-specific features have been stably validated in internal versions and will be fully contributed to the community.​

•
By 2025, we will achieve full feature and performance parity with Apache Iceberg, empowering open-source users to unlock the full potential of Streaming Lakehouse.​

DLA Query Optimization

•
Metadata super optimization: ​
◦
Paimon metadata distributed plan​
◦
Paimon manifest cache & background refresh​
◦
Enhanced statistics collection​
◦
End-to-end observability for metadata​

•
Advanced query optimization​
◦
Index optimization (e.g., Bitmap file index support)​
◦
Improve complex predicate pushdown capabilities​
◦
Optimizations for specific query patterns (e.g., count(*), select limit, topn)​
◦
Point lookup optimization: Paimon supports record-level optimization​
◦
Support primary key (PK)/foreign key (FK) constraints in Paimon tables​

•
Asynchronous scan range distribution: support for All-At-Once and Phased scheduler​

Fully Featured LakeHouse (Achieve parity with Iceberg/Polaris)

•
Performant and cost-effective writes: collaborate with compute engines to build end-to-end data processing pipelines purely based on the data lake​

•
Full operation support: time-travel, management for tagging & branching, enable DDL, DML, procedures, and seamless table migration​

•

Support Paimon rest catalog

◦

Integrate policy management for rest catalog (see discussion here) https://docs.google.com/document/d/1kIiVkFFg9tPa5SH70b9WwzbmclrzH3qWHKfCKXw5lbs/edit?tab=t.0#heading=h.nly223xz13km

◦
Compaction and layout optimization: introduce compaction services and automatic layout arrangement.​

Paimon Specific Optimization

•
Support Paimon variant type​

•
Suppot Paimon view, materialized view/table：use Flink/Spark/StarRocks for Paimon materialized view processing, with StarRocks performing SQL rewrite.​

•
Incremental computation framework for Paimon (TBD; initially supports Paimon append tables): for now "ETL on lake" use cases categorized as:​
◦
Scenario1：Paimon (ODS，DWD) -> StarRocks OlapTable (DWS，ADS) ​
◦
Scenario2：Paimon (ODS，DWD) -> StarRocks (full computation)-> Paimon（DWS，DWD）​
◦
Scenario3：Paimon (ODS，DWD) -> StarRocks (changelog-based incremental) -> Paimon (DWS，DWD)​

StarRocks Community 2025 Roadmap for Paimon​

StarRocks Community 2025 Roadmap for Paimon