Skip to main content

Overview of data ingestion

You can ingest data into RisingWave in two ways:

  • Connect to and ingest data from external sources such as databases and message brokers.
  • Use SQL statements to insert, update or delete data in tables directly.

Ingest data from external sources

To ingest data from external sources into RisingWave, you need to create a source (CREATE SOURCE) or a table with connector settings (CREATE TABLE) in RisingWave.

The syntax for creating a source is similar to creating a table with connector settings:

CREATE {TABLE | SOURCE} source_or_table_name 
[optional_schema_definition]
WITH (
connector='kafka',
connector_parameter='value', ...
)
...

When connector settings are specified for a table in RisingWave, it is able to store streaming data. However, a table with connector settings is different from a source in RisingWave.

  • A source does not persist all data that flows in. It persists only results from materialized views. It accepts only append-only data, such as application events or log messages.
  • A table with connector settings persists all data that flows in. It accepts both append-only data and updatable data. To accept updatable data, you need to specify a primary key when creating the table. Data in upsert, debezium, canal, or maxwell format are examples of updatable data.

Regardless of whether data is persisted in RisingWave, you can create materialized views to transform or analyze them.

Insert data into tables (without connectors)

You can also load data to RisingWave by creating tables (CREATE TABLE) and inserting data into tables (INSERT).

Supported sources and formats

For the complete list of supported sources and formats, see Supported sources and formats.

Help us make this doc better!