Syntax
To use Snowflake v2, specifyconnector = 'snowflake_v2'
in your CREATE SINK statement.
Parameters
Parameter | Description |
---|---|
type | Sink type (e.g., append-only or upsert) |
intermediate.table.name | Name of the intermediate table used for upsert mode. No need to fill this out in append-only mode |
table.name | Name of the target table |
database | Snowflake database name |
schema | Snowflake schema name |
write.target.interval.seconds | Interval in seconds for scheduled writes (default: 3600) |
warehouse | Snowflake warehouse name |
jdbc.url | JDBC URL to connect to Snowflake |
username | Snowflake username |
password | Snowflake password |
commit_checkpoint_interval | Commit every n checkpoints (default: 10) |
auto_schema_change | Enable automatic schema change for upsert sink; adds new columns to target table if needed |
create_table_if_not_exists | Create target table if it does not exist |
with_s3 = true
:
Parameter | Description |
---|---|
with_s3 | Enable writing via S3 (default: true) |
s3.region_name | AWS region for S3 |
s3.bucket_name | S3 bucket name |
s3.path | S3 folder path for sink files |
enable_config_load | Load S3 credentials from environment (self-hosted only) |
s3.credentials.access | AWS access key ID |
s3.credentials.secret | AWS secret access key |
s3.endpoint_url | Custom S3 endpoint URL (for self-hosted setups) |
s3.assume_role | IAM role ARN to assume for S3 access |
Auto schema change
Snowflake v2 supports sink auto schema change whensink_decoupling
is false
. When enabled, RisingWave will automatically alter the target table to add new columns from the upstream data source.
To enable it:
Configure RisingWave to write to S3
You need to configure how RisingWave authenticates with AWS S3. There are two primary methods:- Access Key / Secret Key (AK/SK) authentication
CREATE SINK
statement.
- Assume role authentication
- Obtaining the RisingWave service ARN from our support team.
- Creating an IAM policy with the necessary S3 read/write permissions for your bucket and prefix.
- Configuring the IAM Role’s trust policy to allow the RisingWave service ARN to assume it.
Append-only and upsert modes
Snowflake v2 sink connector supports both modes for flexible data handling. Inupsert
mode, performance is optimized through the use of an intermediate table:
-
An intermediate table is created to stage data before merging it into the target table. If
create_table_if_not_exists
is set to true, the table is automatically namedrw_<target_table_name>_<uuid>
. -
Data is periodically merged from the intermediate table into the target table according to the
write.target.interval.seconds
setting. - By default, an S3 bucket is required to achieve optimal ingestion performance into the intermediate table.
-
Alternatively, you can use
INSERT
SQL statements to load data directly into the intermediate table, though this approach is not recommended due to performance drawbacks.
- Snowflake v2 sink with S3 writer (Append-only mode)
- Snowflake v2 sink with S3 writer (Upsert mode):
Set up Snowflake
Before connecting RisingWave to Snowflake, you need to prepare a dedicated user, role, schema, and warehouse in Snowflake. If you already have a Snowflake user with the necessary privileges, you can skip this step. Follow the steps below to create and configure the required resources.- Set
database_name
,warehouse_name
, andrw_schema
accordingly to use an existing database, warehouse, and/or schema.
Example
- Create role and schema for RisingWave.
- Create Snowflake database.
- Create a user for RisingWave.
- Create a warehouse for RisingWave.
- Grant RisingWave role access to warehouse.
- Grant RisingWave access to database.
Set up Snowflake S3 integration
To enable RisingWave to write data into Snowflake through S3, you need to set up a Snowflake Storage Integration and a Stage in Snowflake. Follow these steps:- Use
ACCOUNTADMIN
role inRW_DB.RW_SCHEMA
created above.
- Create a Snowflake storage integration that connects to your S3 bucket. Replace the ARN and S3 path with your own.
- Configure IAM trust policy
STORAGE_AWS_IAM_USER_ARN
(e.g. arn:aws:iam::398211280607:user/....
) and STORAGE_AWS_EXTERNAL_ID
(e.g. ....c9aPS5npkOTkSO8=
) to configure AWS IAM trusted policy.
- Validate that Snowflake can access your S3 bucket.
- Create an external stage.
public
or other roles, you need to fully configure and grant these privileges:
Set up S3 IAM and role
RisingWave supports direct access to S3 using Access Key / Secret Key (AK/SK) by default. If you prefer to set up S3 IAM and role, follow the official Snowflake guide. Below are additional notes specific to RisingWave.- Create an IAM policy for S3 access.
Example
- Configure the following additional trusted entities for your IAM Role to allow RisingWave arn to assume the role. Contact our support team to obtain the correct role to use. Please don’t click required external id to trust RisingWave arn here.
Example