Deliver data to Iceberg tables

Need help generating SQL? Use Claude Code or Cursor with the RisingWave MCP server to generate and run SQL interactively. This guide explains how to deliver processed data from RisingWave into existing Iceberg tables. Use this when you have Iceberg tables managed by external systems and want RisingWave to deliver processed results into them.

Prerequisites

An upstream source, table, or materialized view in RisingWave to output data from.
Existing Iceberg tables that you can deliver to, or the ability to create them via external systems.
Appropriate permissions to deliver to the target Iceberg catalog and storage.
Access credentials for the underlying object storage (e.g., S3 access key and secret key).

Create an Iceberg sink

To write data to an external Iceberg table, create a SINK. This statement defines how data from an upstream object should be formatted and delivered to the target Iceberg table.

CREATE SINK my_iceberg_sink FROM processed_events
WITH (
    connector = 'iceberg',
    type = 'append-only',
    warehouse.path = 's3://my-data-lake/warehouse',
    database.name = 'analytics',
    table.name = 'processed_user_events',
    create_table_if_not_exists = 'true',
    catalog.type = 'glue',
    catalog.name = 'my_glue_catalog',
    s3.access.key = 'your-access-key',
    s3.secret.key = 'your-secret-key',
    s3.region = 'us-west-2',
    partition_by = 'partition_by_column_name'
);

Configuration parameters

Parameter	Required	Description
`connector`	Yes	Must be `'iceberg'`.
`type`	Yes	Sink mode. `'append-only'` for new records only; `'upsert'` to handle updates and deletes.
`database.name`	Yes	The name of the target Iceberg database.
`table.name`	Yes	The name of the target Iceberg table.
`primary_key`	Yes, if `type` is `upsert`	A comma-separated list of columns that form the primary key.
`write_mode`	No	Write mode for the sink. Options: `'merge-on-read'` (default) or `'copy-on-write'`. Important: Copy-on-write is only supported for upsert sinks. Append-only sinks must use merge-on-read. See Write modes for details.
`force_append_only`	No	If `true`, converts an `upsert` stream to `append-only`. Updates become inserts and deletes are ignored. Default: `false`.
`is_exactly_once`	No	Set to `true` to enable exactly-once delivery semantics. This provides stronger consistency but may impact performance. Default: `true`. Exactly-once delivery requires sink decoupling to be enabled (the default behavior). If you `SET sink_decouple = false;`, exactly-once semantics will be automatically disabled for the sink.
`commit_checkpoint_interval`	No	Controls how often RisingWave commits to Iceberg. Default: `60` (about every 60 seconds in the default configuration).
`commit_retry_num`	No	The number of times to retry a failed commit. Default: `8`.
`create_table_if_not_exists`	No	If `true`, RisingWave creates the external Iceberg table automatically when it does not exist. Default: `false`.
`partition_by`	No	Specify partitioning using column names or transformations. Supported transformations include `identity`, `truncate(n)`, `bucket(n)`, `year`, `month`, `day`, `hour`, and `void`. Multiple columns can be separated by commas. Example: `partition_by = 'truncate(4,v2),bucket(5,v1)'`. For more details on Iceberg partitioning, see Partition transforms.
`auto.schema.change`	No	Set to `true` to enable automatic schema evolution. When enabled, RisingWave automatically propagates schema changes (for example, adding columns) from the upstream source to the Iceberg table. Automatic schema evolution is only supported when exactly-once semantics are active (`is_exactly_once = 'true'` and sink decoupling is enabled); if exactly-once is disabled for the sink, this option has no effect and schema changes are not propagated. Currently supports `ADD COLUMN` operations. Default: `false`.

For detailed storage and catalog configuration:

Object storage: Object storage configuration
Catalogs: Catalog configuration

You can configure commit_checkpoint_interval and commit_retry_num to manage commit frequency and retry behavior.

Schema evolution

RisingWave supports automatic schema evolution for Iceberg sinks, allowing schema changes from upstream sources to be automatically propagated to the target Iceberg table. This feature works with exactly-once semantics. To enable schema evolution, set auto.schema.change = 'true' when creating the sink:

CREATE SINK my_iceberg_sink FROM upstream_source
WITH (
    connector = 'iceberg',
    type = 'append-only',
    warehouse.path = 's3://my-bucket/warehouse',
    database.name = 'my_database',
    table.name = 'my_table',
    catalog.type = 'glue',
    s3.access.key = 'your-access-key',
    s3.secret.key = 'your-secret-key',
    auto.schema.change = 'true',
    is_exactly_once = 'true'
);

Supported operations

Currently, RisingWave supports the following schema change operations:

ADD COLUMN: Automatically adds new columns to the Iceberg table when they are added to the upstream source.

How it works

When schema evolution is enabled:

RisingWave detects schema changes in the upstream source
Schema changes are applied to the Iceberg table as atomic schema updates in the catalog
The sink coordinates schema commits with data commits to ensure consistency
In exactly-once mode, RisingWave ensures that schema changes are applied exactly once, even in the presence of failures

Schema changes use separate commit operations from data writes, but the sink coordinates them so that the required schema updates are durably committed and not duplicated during recovery. The system checks whether schema changes have already been applied to avoid duplicate schema updates during recovery.

Table maintenance

When you continuously sink data to an Iceberg table, it is important to perform periodic maintenance, including compaction and snapshot expiration, to maintain good query performance and manage storage costs. RisingWave provides both automatic and manual maintenance options. For complete details, see the Iceberg table maintenance guide.

Monitoring

-- Check sink status
SHOW SINKS;

-- View sink details
DESCRIBE SINK my_iceberg_sink;

​Prerequisites

​Create an Iceberg sink

​Configuration parameters

​Schema evolution

​Supported operations

​How it works

​Table maintenance

​Monitoring