Catalog configuration for Apache Iceberg

These configuration parameters are used when creating Iceberg sources, sinks, or connections to specify the Iceberg catalog. The catalog is responsible for managing table metadata (schema, partitioning, location). RisingWave supports several catalog types. These parameters are used in the WITH clause of CREATE SOURCE, CREATE SINK, and CREATE CONNECTION statements.

Catalog parameters

Parameter	Description	Required (Conditional)
`catalog.name`	The name of the Iceberg catalog. This is a user-defined identifier. Optional for the `storage` catalog, but required for all other catalog types.	Conditional
`catalog.type`	The type of Iceberg catalog to use. Supported values: `'storage'`: Uses the underlying file system (e.g., S3) directly for metadata. `'rest'`: Uses the Iceberg REST catalog. `'hive'`: Uses a Hive Metastore. `'jdbc'`: Uses a JDBC catalog. `'glue'`: Uses AWS Glue Data Catalog `'snowflake'`: Uses Snowflake catalog If not specified, defaults to `'storage'`.	No (defaults to `storage`)
`catalog.uri`	The URI of the catalog. The required format depends on the `catalog.type`: `rest`: The base URL of the REST catalog server (e.g., `'http://rest-catalog:8181'`). `hive`: The Hive Metastore URI (e.g., `'thrift://hive-metastore:9083'`). `jdbc`: The JDBC connection string (e.g., `'jdbc:postgresql://postgres:5432/iceberg'`). `snowflake`: The Snowflake JDBC connection string (e.g., `'jdbc:snowflake://<account_identifier>.snowflakecomputing.com/'`).	Conditional
`catalog.credential`	Credential for accessing the Iceberg catalog, used to exchange for a token in the OAuth2 client credentials flow. Applicable only in the `rest` catalog.	No
`catalog.token`	A Bearer token for accessing the Iceberg catalog, used for interaction with the server. Applicable only in the `rest` catalog.	No
`catalog.oauth2_server_uri`	The `oauth2_server_uri` for accessing the Iceberg catalog, serving as the token endpoint URI to fetch a token if the `rest` catalog is not the authorization server. Applicable only in the `rest` catalog.	No
`catalog.scope`	Scope for accessing the Iceberg catalog, providing additional scope for OAuth2. Applicable only in the `rest` catalog.	No
`catalog.jdbc.user`	Username for JDBC catalog.	No
`catalog.jdbc.password`	Password for JDBC catalog.	No

Catalog types

Iceberg supports these types of catalogs:

Storage catalog

The Storage catalog stores all metadata in the underlying file system, such as Hadoop or S3. Currently, we only support S3 as the underlying file system.

Example

CREATE SINK sink_demo_storage FROM t
WITH (
    connector = 'iceberg',
    type = 'append-only',
    force_append_only = true,
    s3.endpoint = 'http://minio-0:9301',
    s3.access.key = 'xxxxxxxxxx',
    s3.secret.key = 'xxxxxxxxxx',
    s3.region = 'ap-southeast-1',
    catalog.type = 'storage',
    catalog.name = 'demo',
    warehouse.path = 's3://icebergdata/demo',
    database.name = 's1',
    table.name = 't1'
);

Example

CREATE SOURCE source_demo_storage
WITH (
    connector = 'iceberg',
    s3.endpoint = 'http://minio-0:9301',
    s3.access.key = 'xxxxxxxxxx',
    s3.secret.key = 'xxxxxxxxxx',
    s3.region = 'ap-southeast-1',
    catalog.type = 'storage',
    catalog.name = 'demo',
    warehouse.path = 's3://icebergdata/demo',
    database.name = 's1',
    table.name = 't1'
);

REST catalog

The REST catalog uses a RESTful interface to manage table metadata.

Example

CREATE SINK sink_demo_rest FROM t
WITH (
    connector = 'iceberg',
    type = 'append-only',
    force_append_only = true,
    s3.endpoint = 'https://s3.ap-southeast-2.amazonaws.com',
    s3.region = 'ap-southeast-2',
    s3.access.key = 'xxxx',
    s3.secret.key = 'xxxx',
    s3.path.style.access = 'true',
    catalog.type = 'rest',
    catalog.uri = 'http://localhost:8181/api/catalog',
    warehouse.path = 'quickstart_catalog',
    database.name = 'ns',
    table.name = 't1',
    catalog.credential='123456:123456',
    catalog.scope='PRINCIPAL_ROLE:ALL',
    catalog.oauth2_server_uri='xxx'
    catalog.scope='xxx',
);

JDBC catalog

The JDBC catalog stores metadata in a relational database like PostgreSQL.

Example

CREATE SINK sink_demo_jdbc FROM t
WITH (
    connector = 'iceberg',
    type = 'append-only',
    force_append_only = true,
    s3.endpoint = 'http://minio-0:9301',
    s3.access.key = 'xxxxxxxxxx',
    s3.secret.key = 'xxxxxxxxxx',
    s3.region = 'ap-southeast-1',
    catalog.type = 'jdbc',
    catalog.uri = 'jdbc:postgresql://127.0.0.1:8432/metadata',
    catalog.jdbc.user = 'postgres',
    catalog.jdbc.password = '123',
    catalog.name = 'dev',
    warehouse.path = 's3://icebergdata/demo',
    database.name = 's1',
    table.name = 't1'
);

Glue catalog

AWS Glue Data Catalog provides a centralized metadata store for table definitions.

Example

CREATE SINK sink_demo_glue FROM t
WITH (
    connector = 'iceberg',
    type = 'append-only',
    force_append_only = true,
    s3.endpoint = 'https://s3.ap-southeast-2.amazonaws.com',
    s3.region = 'ap-southeast-2',
    s3.access.key = 'xxxx',
    s3.secret.key = 'xxxx',
    catalog.type = 'glue',
    catalog.name = 'demo',
    warehouse.path = 's3://icebergdata/demo',
    database.name = 's1',
    table.name = 't1'
);

Snowflake catalog

The Snowflake catalog uses Snowflake’s built-in Iceberg catalog functionality.

Example

CREATE SINK sink_demo_snowflake FROM t
WITH (
    connector = 'iceberg',
    type = 'append-only',
    force_append_only = true,
    catalog.type = 'snowflake',
    catalog.name = 'SNOWFLAKE',
    catalog.uri = 'jdbc:snowflake://<account_identifier>.snowflakecomputing.com/',
    catalog.jdbc.user = '<your_username>',
    catalog.jdbc.password = '<your_password>',
    warehouse.path = '<warehouse_path>',
    database.name = '<your_database>',
    table.name = '<your_table>'
);

Hive Metastore catalog

The Hive Metastore catalog stores metadata using a Hive Metastore backend.

Example

CREATE SINK sink_demo_hive FROM t
WITH (
    connector = 'iceberg',
    type = 'append-only',
    force_append_only = true,
    s3.endpoint = 'http://minio-0:9301',
    s3.access.key = 'xxxxxxxxxx',
    s3.secret.key = 'xxxxxxxxxx',
    s3.region = 'ap-southeast-1',
    catalog.type = 'hive',
    catalog.uri = 'thrift://hive-metastore:9083',
    warehouse.path = 's3://icebergdata/demo',
    database.name = 's1',
    table.name = 't1'
);

Use with RisingWave Managed Iceberg

While these external catalogs are primarily used with the Bring Your Own Iceberg approach, they can also be used with RisingWave Managed Iceberg when you need to integrate with existing catalog infrastructure:

-- Use external JDBC catalog with Iceberg table engine
CREATE CONNECTION managed_with_jdbc WITH (
    type = 'iceberg',
    warehouse.path = 's3://my-bucket/warehouse/',
    s3.access.key = 'your-access-key',
    s3.secret.key = 'your-secret-key',
    s3.region = 'us-west-2',
    catalog.type = 'jdbc',
    catalog.uri = 'jdbc:postgresql://postgres:5432/catalog',
    catalog.jdbc.user = 'catalog_user',
    catalog.jdbc.password = 'catalog_password',
    catalog.name = 'production'
);

-- Create managed table using external catalog
CREATE TABLE managed_events (
    event_id BIGINT PRIMARY KEY,
    user_id INT,
    event_type VARCHAR,
    timestamp TIMESTAMPTZ
) ENGINE = iceberg
WITH (connection = 'managed_with_jdbc');

For the simplest managed setup, use the Hosted Iceberg Catalog instead.

Choosing the right catalog

For Bring Your Own Iceberg:

AWS Glue: If you’re on AWS and have existing Glue Data Catalog infrastructure
JDBC: If you have a PostgreSQL/MySQL database for metadata storage
REST: For custom catalog services or AWS S3 Tables
Storage: For simple setups without external catalog dependencies
Hive: For integration with existing Hadoop ecosystem
Snowflake: If using Snowflake as your data warehouse

For RisingWave Managed Iceberg:

Hosted catalog: Recommended for new projects and simple setups
External catalogs: When you need integration with existing enterprise catalog infrastructure

Best practices

Choose based on your ecosystem: Use catalogs that integrate with your existing infrastructure.
Consider security: Some catalogs offer better authentication and authorization features.
Plan for scale: Some catalogs handle large numbers of tables better than others.
Network connectivity: Ensure RisingWave can reach your catalog service.
Backup considerations: Understand how your catalog metadata is backed up and recovered.

Next steps

Object storage setup: Configure your storage backend with Object storage configuration.
Read from tables: Set up sources with Read from Iceberg.
Write to tables: Set up sinks with Write to Iceberg.
Managed alternative: Consider RisingWave Managed Iceberg for simpler setups.

Iceberg

​Catalog parameters

​Catalog types

​Storage catalog

​REST catalog

​JDBC catalog

​Glue catalog

​Snowflake catalog

​Hive Metastore catalog

​Use with RisingWave Managed Iceberg

​Choosing the right catalog

​For Bring Your Own Iceberg:

​For RisingWave Managed Iceberg:

​Best practices

​Next steps

Catalog parameters

Catalog types

Storage catalog

REST catalog

JDBC catalog

Glue catalog

Snowflake catalog

Hive Metastore catalog

Use with RisingWave Managed Iceberg

Choosing the right catalog

For Bring Your Own Iceberg:

For RisingWave Managed Iceberg:

Best practices

Next steps