- Compaction: Merges small data files and delete files into larger, optimized files to improve read performance.
- Snapshot Expiration: Removes old, unneeded snapshots and their associated data files to reclaim storage space.
Automatic maintenance for Iceberg was added in RisingWave v2.5.0.
Automatic maintenance
You can enable automatic maintenance to run periodically in the background for your Iceberg sinks and internal tables.Dedicated Compactor RequiredAutomatic Iceberg maintenance requires a dedicated compactor service. Please contact us via the RisingWave Slack workspace to have the necessary resources allocated for your cluster.
Parameters
Configure automatic maintenance by specifying the following parameters in theWITH
clause of a CREATE SINK
or CREATE TABLE ... ENGINE = iceberg
statement.
Parameter | Description |
---|---|
enable_compaction | Required. Set to true to enable automatic compaction and snapshot expiration. |
compaction_interval_sec | Optional. The interval in seconds between maintenance runs. Default: 3600 . |
enable_snapshot_expiration | Optional. Set to true to enable snapshot expiration. By default, it removes snapshots older than 5 days. |
snapshot_expiration_max_age_millis | Optional. The maximum age (in milliseconds) for a snapshot to be retained. To keep only the latest snapshot, set this to 0 . |
snapshot_expiration_retain_last | Optional. The minimum number of snapshots to retain, regardless of their age. |
max_snapshots_num_before_compaction | Optional. The maximum number of snapshots allowed since the last rewrite operation. If set, the sink will pause if this number is exceeded until compaction completes. |
Examples
Internal Iceberg table
Iceberg Sink
Manual maintenance
In addition to automatic background maintenance, you can trigger compaction and snapshot expiration manually at any time using theVACUUM
command.
This gives you on-demand control over table optimization and storage cleanup.