Object storage configuration for Apache Iceberg
Configuration parameters for connecting RisingWave to object storage systems used by Apache Iceberg.
These configuration parameters are used when creating Iceberg sources, sinks, or connections to specify how RisingWave connects to object storage systems where Apache Iceberg data files are stored. These parameters are used in the WITH
clause of CREATE SOURCE
, CREATE SINK
, and CREATE CONNECTION
statements.
S3-compatible storage
To integrate RisingWave with Iceberg tables stored in S3-compatible object storage, you must provide the necessary connection details. These configurations tell RisingWave exactly where your data warehouse resides and how to authenticate and access it.
These parameters configure the connection to the underlying S3-compatible storage system where the Iceberg data files are stored. This includes AWS S3, MinIO, and other compatible services.
Parameter | Description | Required (Conditional) |
---|---|---|
warehouse.path | The base path to your Iceberg warehouse in the S3-compatible storage. Example: 's3://my-bucket/iceberg-warehouse' | Conditional |
s3.endpoint | Optional. Endpoint of the S3.
| Conditional |
s3.region | Optional. The region where the S3 bucket is hosted. Either s3.endpoint or s3.region must be specified. | Conditional |
s3.access.key | Required. Access key of the S3 compatible object store. | Conditional |
s3.secret.key | Required. Secret key of the S3 compatible object store. | Conditional |
s3.path.style.access | Optional. Determines the access style for S3. If true , use path-style; if false , use virtual-hosted–style. | No |
enable_config_load | Optional. Controls whether configuration is loaded from environment variables. Set to true will load warehouse credentials from environment variables. Only supported in self-hosted environments. | No |
Google Cloud Storage (GCS)
Added in v2.3.0.
RisingWave supports creating Iceberg sinks on GCS with catalog types storage
or rest
. To enable it, configure the following specific parameters:
These parameters configure the connection to the underlying GCS storage system where the Iceberg data files are stored.
Parameter name | Description |
---|---|
warehouse.path | Specifies the Google Cloud Storage path. |
gcs.credential | Base64-encoded credential key obtained from the GCS service account key JSON file. To get this JSON file, refer to the guides of GCS documentation.
|
For more details on sinking data to GCS, see Sink data to Google Cloud Storage.
Azure Blob Storage
Added in v2.4.0.
RisingWave supports creating Iceberg sinks on Azure Blob with catalog types storage
or rest
. To enable it, configure the following specific parameters:
These parameters configure the connection to Azure Blob storage where the Iceberg data files are stored.
Parameter name | Description |
---|---|
warehouse.path | Specifies the Azure Blob Storage path. |
azblob.account_name | The Azure Storage account name used to authenticate access. |
azblob.account_key | The Azure Storage account key associated with the account name, used for authentication. |
azblob.endpoint_url | The full endpoint URL of the Azure Blob service. |
For more details on sinking data to Azure Blob, see Sink data to Azure Blob.
Amazon S3 Tables
You can configure RisingWave Iceberg connectors to use Amazon S3 Tables as their catalog. This setup allows RisingWave to sink data into or read data from Iceberg tables managed by the AWS native S3 Tables catalog service.
To achieve this, specify the rest
catalog type within your CREATE SINK
or CREATE SOURCE
statement and include the necessary parameters for SigV4 authentication against the S3 Tables REST API.
Required REST Catalog Parameters for S3 Tables:
While these parameters might be optional in general Iceberg configurations, they are required when catalog.type = 'rest'
is used to connect to Amazon S3 Tables:
Parameter name | Description | Value for S3 Tables |
---|---|---|
catalog.rest.signing_region | The AWS region for signing REST catalog requests. | e.g., us-east-1 |
catalog.rest.signing_name | The service name for signing REST catalog requests. | s3tables |
catalog.rest.sigv4_enabled | Enables SigV4 signing for REST catalog requests. Set to true . | true |
Example for Iceberg Sink
Example for Iceberg Source
Once created, this sink will write data from source_table
into the specified Iceberg table (<your-table-name>
) within the <your-database-name>
database, using Amazon S3 Tables to manage the table’s metadata.
Once created, RisingWave will use this source to read data from the specified Iceberg table, leveraging Amazon S3 Tables for metadata discovery.