Sink data from RisingWave to Amazon DynamoDB

Amazon DynamoDB is a fully managed, serverless, key-value NoSQL database designed to run high-performance applications at any scale. It provides consistent, single-digit millisecond response times and offers built-in security, backup and restore, and in-memory caching.

PREMIUM FEATUREThis is a premium feature. For a comprehensive overview of all premium features and their usage, please see RisingWave premium features.

Syntax

CREATE SINK [ IF NOT EXISTS ] sink_name
[FROM sink_from | AS select_query]
WITH (
   connector = 'dynamodb',
   connector_parameter = 'value', ...
)
FORMAT data_format ENCODE data_encode [ (
    format_parameter = 'value'
) ];

Parameters

Field	Note
table	Required. Name of the DynamoDB table where you want to write the data.
primary_key	Required. A pair of columns representing the partition key and sort key of DynamoDB, e.g., key1,key2, separated by comma.
aws.region	Required. AWS region where your DynamoDB table is hosted.
aws.endpoint_url	Optional. Custom endpoint URL for connecting to DynamoDB. Useful for testing with DynamoDB Local.
aws.credentials.access_key_id	Conditional. AWS access key ID for accessing DynamoDB. Must be specified when using access key ID and secret key.
aws.credentials.secret_access_key	Conditional. AWS secret access key for accessing DynamoDB. Must be specified when using access key ID and secret key.
aws.credentials.role.arn	Conditional. ARN of the IAM role to assume for accessing DynamoDB. Must be specified when using AssumeRole.
aws.credentials.role.external_id	Conditional. External ID for assuming the IAM role specified in aws.credentials.role.arn.
aws.profile	Optional. The name of the AWS CLI profile to use for accessing DynamoDB. If specified, it overrides the default profile.
dynamodb.max_batch_item_nums	Optional. The maximum number of items in the `BatchWriteItem` operation. It must be larger than 1, and less than or equal to 25. The default is 25.
dynamodb.max_future_send_nums	Optional. The maximum number of concurrent write futures in DynamoDB. It must be less than 360, and the default is 256. This is derived from user-defined `max_parallelism_units` (40000 by default). If the write throughput of RisingWave exceeds the `max_parallelism_units` set in DynamoDB, an error would be reported.
dynamodb.batch_write_retry_times	Optional. The maximum number of times to retry items returned by DynamoDB as unprocessed. The default is 3.
dynamodb.batch_write_retry_backoff_ms	Optional. The initial retry backoff in milliseconds for unprocessed items. The backoff increases exponentially, includes jitter, and is capped at 2 seconds. The default is 100.

When a BatchWriteItem request succeeds but returns unprocessed items, RisingWave retries only those items. If the configured retries are exhausted, the sink reports an error so that its normal recovery process can take over.

Partition key and sort key mapping

For the DynamoDB sink to function correctly, it is required that the source table in RisingWave has a composite primary key comprising two columns, corresponding to the partition key and sort key defined in the target DynamoDB table. For example, if you are sinking data into a DynamoDB table named movies that is configured with a partition key title and a sort key year, you must ensure that the RisingWave table schema also defines a composite primary key with the same columns. Below is an example schema definition for the RisingWave table:

CREATE TABLE IF NOT EXISTS movies (
    year integer,
    title varchar,
    description varchar,
    primary key (year, title)
);

This makes sure that the data structure in RisingWave aligns with the key definition in DynamoDB, thus preventing any further potential errors.

Data type mapping

RisingWave Data Type	DynamoDB Data Type
boolean	Bool
smallint	number (N)
integer	number (N)
bigint	number (N)
numeric	number (N)
real	number (N)
double precision	number (N)
serial	number (N)
character varying (varchar)	string (S)
bytea	binary
date	string (S)
time without time zone	string (S)
timestamp without time zone	string (S)
timestamp with time zone	string (S)
interval	string (S)
struct	map (M)
array	list (L)
JSONB	string (S)

The struct datatype in RisingWave will map to map (M) in DynamoDB in a recursive way. Refer to source code for details.

Get started

Work with data

Install & Operate

Performance

Troubleshooting

Reference

Cloud

Sink data from RisingWave to Amazon DynamoDB

Syntax

Parameters

Partition key and sort key mapping

Data type mapping

​Syntax

​Parameters

​Partition key and sort key mapping

​Data type mapping

Syntax

Parameters

Partition key and sort key mapping

Data type mapping