These configuration parameters are used when creating Iceberg sources, sinks, or connections to specify the Iceberg catalog. The catalog is responsible for managing table metadata (schema, partitioning, location). RisingWave supports several catalog types. These parameters are used in the WITH clause of CREATE SOURCE, CREATE SINK, and CREATE CONNECTION statements.
Catalog parameters
| Parameter | Description | Required (Conditional) |
|---|
catalog.name | The name of the Iceberg catalog. This is a user-defined identifier. Optional for the storage catalog, but required for all other catalog types. | Conditional |
catalog.type | The type of Iceberg catalog to use. Supported values: 'storage': Uses the underlying file system (e.g., S3) directly for metadata.'rest': Uses the Iceberg REST catalog.'hive': Uses a Hive Metastore.'jdbc': Uses a JDBC catalog.-
'glue': Uses AWS Glue Data Catalog -
'snowflake': Uses Snowflake catalog
If not specified, defaults to 'storage'. | No (defaults to storage) |
catalog.uri | The URI of the catalog. The required format depends on the catalog.type: rest: The base URL of the REST catalog server (e.g., 'http://rest-catalog:8181').hive: The Hive Metastore URI (e.g., 'thrift://hive-metastore:9083').-
jdbc: The JDBC connection string (e.g., 'jdbc:postgresql://postgres:5432/iceberg'). -
snowflake: The Snowflake JDBC connection string (e.g., 'jdbc:snowflake://<account_identifier>.snowflakecomputing.com/').
| Conditional |
catalog.credential | Credential for accessing the Iceberg catalog, used to exchange for a token in the OAuth2 client credentials flow. Applicable only in the rest catalog. | No |
catalog.token | A Bearer token for accessing the Iceberg catalog, used for interaction with the server. Applicable only in the rest catalog. | No |
catalog.oauth2_server_uri | The oauth2_server_uri for accessing the Iceberg catalog, serving as the token endpoint URI to fetch a token if the rest catalog is not the authorization server. Applicable only in the rest catalog. | No |
catalog.scope | Scope for accessing the Iceberg catalog, providing additional scope for OAuth2. Applicable only in the rest catalog. | No |
catalog.jdbc.user | Username for JDBC catalog. | No |
catalog.jdbc.password | Password for JDBC catalog. | No |
Catalog types
Iceberg supports these types of catalogs:
Storage catalog
The Storage catalog stores all metadata in the underlying file system, such as Hadoop or S3. Currently, we only support S3 as the underlying file system.
CREATE SINK sink_demo_storage FROM t
WITH (
connector = 'iceberg',
type = 'append-only',
force_append_only = true,
s3.endpoint = 'http://minio-0:9301',
s3.access.key = 'xxxxxxxxxx',
s3.secret.key = 'xxxxxxxxxx',
s3.region = 'ap-southeast-1',
catalog.type = 'storage',
catalog.name = 'demo',
warehouse.path = 's3://icebergdata/demo',
database.name = 's1',
table.name = 't1'
);
CREATE SOURCE source_demo_storage
WITH (
connector = 'iceberg',
catalog.type = 'storage',
warehouse.path = 's3://icebergdata/demo',
s3.endpoint = 'http://minio-0:9301',
s3.access.key = 'xxxxxxxxxx',
s3.secret.key = 'xxxxxxxxxx',
s3.region = 'ap-southeast-1',
database.name = 's1',
table.name = 't1'
);
REST catalog
RisingWave supports the REST catalog, which acts as a proxy to other catalogs like Hive, JDBC, and Nessie catalog. This is the recommended approach to use RisingWave with Iceberg tables.
CREATE SINK sink_demo_rest FROM t
WITH (
connector = 'iceberg',
type = 'append-only',
force_append_only = true,
s3.endpoint = 'https://s3.ap-southeast-2.amazonaws.com',
s3.region = 'ap-southeast-2',
s3.access.key = 'xxxx',
s3.secret.key = 'xxxx',
s3.path.style.access = 'true',
catalog.type = 'rest',
catalog.uri = 'http://localhost:8181/api/catalog',
warehouse.path = 'quickstart_catalog',
database.name = 'ns',
table.name = 't1',
catalog.credential='123456:123456',
catalog.scope='PRINCIPAL_ROLE:ALL',
catalog.oauth2_server_uri='xxx'
catalog.scope='xxx',
);
CREATE SOURCE source_demo_rest
WITH (
connector = 'iceberg',
s3.endpoint = 'https://s3.ap-southeast-2.amazonaws.com',
s3.region = 'ap-southeast-2',
s3.access.key = 'xxxx',
s3.secret.key = 'xxxx',
s3.path.style.access = 'true',
catalog.type = 'rest',
catalog.uri = 'http://localhost:8181/api/catalog',
warehouse.path = 'quickstart_catalog',
database.name = 'ns',
table.name = 't1',
catalog.credential='123456:123456',
catalog.scope='PRINCIPAL_ROLE:ALL',
catalog.oauth2_server_uri='xxx'
catalog.scope='xxx',
);
Hive catalog
RisingWave supports the Hive catalog. You need to set catalog.type to hive to use it.
CREATE SINK sink_demo_hive FROM t
WITH (
connector = 'iceberg',
type = 'append-only',
force_append_only = true,
catalog.type = 'hive',
catalog.uri = 'thrift://metastore:9083',
warehouse.path = 's3://icebergdata/demo',
s3.endpoint = 'http://minio-0:9301',
s3.access.key = 'xxxxxxxxxx',
s3.secret.key = 'xxxxxxxxxx',
s3.region = 'ap-southeast-1',
catalog.name = 'demo',
database.name = 's1',
table.name = 't1'
);
CREATE SOURCE source_demo_hive
WITH (
connector = 'iceberg',
catalog.type = 'hive',
catalog.uri = 'thrift://metastore:9083',
warehouse.path = 's3://icebergdata/demo',
s3.endpoint = 'http://minio-0:9301',
s3.access.key = 'xxxxxxxxxx',
s3.secret.key = 'xxxxxxxxxx',
s3.region = 'ap-southeast-1',
catalog.name = 'demo',
database.name = 's1',
table.name = 't1'
);
JDBC catalog
RisingWave supports the JDBC catalog.
CREATE SINK sink_demo_jdbc FROM t
WITH (
connector = 'iceberg',
type = 'append-only',
force_append_only = true,
warehouse.path = 's3://icebergdata/demo',
s3.endpoint = 'http://minio-0:9301',
s3.access.key = 'xxxxxxxxxx',
s3.secret.key = 'xxxxxxxxxx',
s3.region = 'ap-southeast-1',
catalog.name = 'demo',
catalog.type = 'jdbc',
catalog.uri = 'jdbc:postgresql://postgres:5432/iceberg',
catalog.jdbc.user = 'admin',
catalog.jdbc.password = '123456',
database.name = 's1',
table.name = 't1'
);
CREATE SOURCE source_demo_jdbc
WITH (
connector = 'iceberg',
warehouse.path = 's3://icebergdata/demo',
s3.endpoint = 'http://minio-0:9301',
s3.access.key = 'xxxxxxxxxx',
s3.secret.key = 'xxxxxxxxxx',
s3.region = 'ap-southeast-1',
catalog.name = 'demo',
catalog.type = 'jdbc',
catalog.uri = 'jdbc:postgresql://postgres:5432/iceberg',
catalog.jdbc.user = 'admin',
catalog.jdbc.password = '123456',
database.name = 's1',
table.name = 't1'
);
Glue catalog
PREMIUM EDITION FEATURE
This is a Premium Edition feature. All Premium Edition features are available out of the box without additional cost on RisingWave Cloud. For self-hosted deployments, users need to purchase a license key to access this feature. To purchase a license key, please contact sales team at sales@risingwave-labs.com.
For a full list of Premium Edition features, see RisingWave Premium Edition.
RisingWave supports the Glue catalog. You should use AWS S3 if you use the Glue catalog. Below are example codes for using this catalog:
CREATE SINK sink_test FROM t
WITH (
type='upsert',
primary_key='col',
connector = 'iceberg',
catalog.type = 'glue',
catalog.name = 'test',
warehouse.path = 's3://my-iceberg-bucket/test',
s3.access.key = 'xxxxxxxxxx',
s3.secret.key = 'xxxxxxxxxx',
s3.region = 'ap-southeast-2',
database.name='test_db',
table.name='test_table'
);
CREATE SOURCE source_test
WITH (
connector = 'iceberg',
catalog.type = 'glue',
warehouse.path = 's3://my-iceberg-bucket/test',
s3.access.key = 'xxxxxxxxxx',
s3.secret.key = 'xxxxxxxxxx',
s3.region = 'ap-southeast-2',
database.name='test_db',
table.name='test_table'
);
Snowflake catalog
RisingWave supports the Snowflake catalog for Iceberg sources.
CREATE SOURCE iceberg_t1_source
WITH (
connector = 'iceberg',
s3.region = 'ap-northeast-2',
s3.access.key = 'xxx',
s3.secret.key = 'xxx',
catalog.name = 'demo1',
catalog.type = 'snowflake',
catalog.uri = 'jdbc:snowflake://<account_identifier>.snowflakecomputing.com/',
catalog.jdbc.user = 'xxx',
catalog.jdbc.password = 'xxx',
warehouse.path = 's3://xxx',
database.name = 'xxx',
table.name = 'xxx'
);
