CREATE CONNECTION
Use the CREATE CONNECTION
command to create a reusable catalog for connector parameters.
The CREATE CONNECTION
command creates a reusable connection configuration that can be referenced when creating sources, sinks, or tables. Currently supported connection types are Kafka, Schema Registry, and Iceberg.
Added in v2.3.0: Support iceberg
connection.
Syntax
Parameter
Parameter or clause | Description |
---|---|
type | Required. The type of connection. Supported values: kafka , schema_registry , iceberg . |
secret_name | Use the SECRET keyword to reference secrets, allowing sensitive information to be stored securely and referenced in the connection configuration. Changes to the secret are automatically applied, so there’s no need to alter the connection. |
Click to see all supported properties for Kafka connection.
Click to see all supported properties for Kafka connection.
Required
properties.bootstrap.server
: The Kafka bootstrap server addresses.
Optional
For SSL/SASL authentication:
properties.security.protocol
properties.ssl.endpoint.identification.algorithm
properties.ssl.ca.location
properties.ssl.ca.pem
properties.ssl.certificate.location
properties.ssl.certificate.pem
properties.ssl.key.location
properties.ssl.key.pem
properties.ssl.key.password
properties.sasl.mechanism
properties.sasl.username
properties.sasl.password
properties.sasl.kerberos.service.name
properties.sasl.kerberos.keytab
properties.sasl.kerberos.principal
properties.sasl.kerberos.kinit.cmd
properties.sasl.kerberos.min.time.before.relogin
properties.sasl.oauthbearer.config
For PrivateLink connection:
privatelink.targets
privatelink.endpoint
For AWS authentication:
aws.region
endpoint
aws.credentials.access_key_id
aws.credentials.secret_access_key
aws.credentials.session_token
aws.credentials.role.arn
aws.credentials.role.external_id
Click to see all supported properties for Schema Registry.
Click to see all supported properties for Schema Registry.
Property name | Description |
---|---|
schema.registry | The base URL of the Schema Registry service. |
schema.registry.username | The username for authenticating to the Schema Registry service. |
schema.registry.password | The password for authenticating to the Schema Registry service. |
Click to see all supported properties for Iceberg connection.
Click to see all supported properties for Iceberg connection.
Property name | Description |
---|---|
catalog.type | Type of the Iceberg catalog. |
s3.region | AWS S3 region. |
s3.endpoint | AWS S3 endpoint. |
s3.access.key | AWS S3 access key. |
s3.secret.key | AWS S3 secret key. |
gcs.credential | Google Cloud Storage credential. |
azblob.account_name | The Azure Storage account name. |
azblob.account_key | The Azure Storage account key associated with the account name |
azblob.endpoint_url | The full endpoint URL of the Azure Blob service. |
warehouse.path | Path of the Iceberg warehouse, applicable only in storage catalog. |
catalog.name | Name of the catalog, optional for storage catalog, required for others. |
catalog.uri | URI of the Iceberg catalog, applicable only in rest catalog. |
catalog.credential | Credential for accessing Iceberg catalog, applicable only in rest catalog. |
catalog.token | Token for interacting with rest catalog server. |
catalog.oauth2_server_uri | OAuth2 token endpoint URI, applicable only in rest catalog. |
catalog.scope | Additional OAuth2 scope for accessing the Iceberg catalog, applicable only in rest catalog. |
catalog.rest.signing_region | The signing region when signing requests to the rest catalog. |
catalog.rest.signing_name | The signing name when signing requests to the rest catalog. |
catalog.rest.signing_region | Specify whether to use SigV4 for signing requests to the rest catalog. |
s3.path.style.access | Enables path-style access for AWS S3. |
catalog.jdbc.user | JDBC user for catalog access. |
catalog.jdbc.password | JDBC password for catalog access. |
enable_config_load | Controls whether configuration is loaded from environment variables. Set to true will load warehouse credentials from environment variables. Only supported in self-hosted environments. |
Example
To connect to a schema registry:
To create a Kafka connection that securely integrates secrets:
To create an Iceberg connection:
To create a source, table or sink from the connection, the name of connector and connection must match those specified above. Also, the attributes defined in the connection and the source/table/sink cannot overlap:
Create an AWS PrivateLink connection
If you are using a cloud-hosted source or sink, such as AWS MSK, there might be connectivity issues when your service is located in a different VPC from where you have deployed RisingWave. To establish a secure, direct connection between these two different VPCs and allow RisingWave to read consumer messages from the broker or send messages to the broker, use the AWS PrivateLink service.
Follow the steps below to create an AWS PrivateLink connection.
- Create a target group for each broker. Set the target type as IP addresses and the protocol as TCP. Ensure that the VPC of the target group is the same as your cloud-hosted source.
- Create a Network Load Balancer. Ensure that it is enabled in the same subnets your broker sources are in and the Cross-zone load balancing is also enabled.
- Create a TCP listener for each MSK broker that corresponds to the target groups created. Ensure the ports are unique.
- Complete the health check for each target group.
- Create a VPC endpoint service associated with the Network Load Balancer created. Be sure to add the AWS principal of the account that will access the endpoint service to allow the service consumer to connect. See Manage permissions for more details.
- Use the
CREATE CONNECTION
command in RisingWave to create an AWS PrivateLink connection referencing the endpoint service created. Here is an example of creating an AWS PrivateLink connection.
- Create a source or sink with AWS PrivateLink connection.
- Use the
CREATE SOURCE/TABLE
command to create a Kafka source with PrivateLink connection. For more details, see Create source with AWS PrivateLink connection. - Use the
CREATE SINK
command to create a Kafka sink with PrivateLink connection. For more details, see Create sink with AWS PrivateLink connection.
- Use the