@font-face {
    font-family: 'Monument Grotesk';
    src: url('https://fonts.gstatic.com/s/inter/v18/UcCo3FwrK3iLTcviYwYZ8UA3.woff2');
    font-weight: normal;
    font-style: normal;
    font-display: swap;
  }
  
  h1,
  h2,
  h3 {
    font-family: 'monument grotesk';
    letter-spacing: normal !important;
  }

This topic lists the source connectors in RisingWave.

Sources

Supported sources

RisingWave

Join our Slack

GitHub

Website

Guides

SQL/UDFs

Python SDK

Client Libraries

Changelog

FAQs

Sign up free

Blog

v2.0 and earlier

RisingWave is an event stream processing platform for developers. It offers an unified experience for real-time data ingestion, stream processing, and low-latency serving.

What is RisingWave?

RisingWave excels in a variety of real-time data processing scenarios, making it an ideal choice for several categories of use cases, including:

RisingWave use cases

This guide aims to provide a quick and easy way to get started with RisingWave.

Quick start

Discover the wide range of real-world use cases where RisingWave delivers concrete value with the list of runnable demos.

Overview

Example repository

This topic introduces the RisingWave Premium Edition and offers a complete list of all premium features.

Premium edition

RisingWave Premium Edition

This topic describes the pluggable disk cache feature of RisingWave.

Pluggable disk cache

This topic provides an overview of RisingWave's stream processing performance characteristics based on rigorous testing using the industry-standard Nexmark benchmark and an extended set of real-world SQL queries.

Stream processing benchmarks

Stream processing benchmarks: Nexmark results

This topic provides an overview of RisingWave's ad-hoc query serving performance characteristics (latency and throughput) based on the industry-standard sysbench benchmark.

Query serving benchmarks

Query serving benchmarks: sysbench results

Describes how to use the purpose-built Python SDK by RisingWave to build event-driven applications.

You can use third-party PostgreSQL drivers to seamlessly interact with RisingWave from your applications.

Use RisingWave in your Node.js application

Node.js

Describes how to use third-party Python drivers to interact with RisingWave.

Python

Ruby

Java

Change Data Capture (CDC) refers to the process of identifying and capturing data changes in a database, and then delivering the changes to a downstream service in real time.

MySQL CDC

Ingest CDC data from MySQL

You can ingest data from NATS JetStream into RisingWave by using the NATS source connector in RisingWave.

NATS JetStream

Ingest data from NATS JetStream

Data processing is a crucial step in converting raw data into valuable insights.

Overview of data processing

This guide provides instructions for using dbt to manage real-time data transformations in RisingWave.

Use dbt to manage pipelines

Use dbt for data transformations

Similar to traditional databases, RisingWave supports data updates and deletions within tables. Additionally, when data in a table is modified or deleted, any materialized views built on that table will be updated accordingly.

Deletes and updates

Transactions

This guide introduces how to maintain a wide table whose columns come from different sources. Traditional data warehouses or ETL use a join query for this purpose. However, streaming join brings issues such as low efficiency and high memory consumption.

Maintain wide table with table sinks

In stream processing, watermarks are integral when using event time processing logic with event time based operations. Watermarks are like markers or signals that track the progress of event time, allowing you to process events within their corresponding time windows. A watermark is an estimate of the maximum event time observed so far, or a threshold indicating that events received so far have a timestamp later than or equal to the current watermark. Events that arrive with a timestamp earlier than the current watermark are considered late and are not processed within its time window.

Watermarks

Emit on window close

Indexes in a database are designed to increase the speed at which the database management system (DBMS) can locate and retrieve the desired data from a table or a materialized view.

Indexes

This guide describes how to leverage the time travel feature. This feature helps access historical data at a specific time.

Time travel queries

RisingWave allows you to access and use insights from your streaming data immediately. It also functions like any other database, allowing you to query batch or raw data that you've inserted.

Access and use your processed data

Query with SELECT statements

A foreign data wrapper in PostgreSQL allows you to directly virtualize data stored in an external database as a local external table, also known as a foreign table. This tutorial will demonstrate how to interact between PostgreSQL and RisingWave. In this example, RisingWave will use CDC (Change Data Capture) to extract data from PostgreSQL, and analyze it using a materialized view. Then PostgreSQL will directly retrieve the computation results stored in RisingWave.

RisingWave as Postgres FDW

RisingWave as a PostgreSQL foreign data wrapper

Subscription is used to pull data change records for a specific table or materialized view (MV).

Subscription

RisingWave supports delivering data to downstream systems via its sink connectors.

Overview of data delivery

This guide describes how to sink data from RisingWave to Apache Iceberg using the Iceberg sink connector in RisingWave. Apache Iceberg is a table format designed to support huge tables. For more information, see [Apache Iceberg](https://iceberg.apache.org).

Apache Iceberg

Sink data from RisingWave to Apache Iceberg

This topic describes how to sink data from RisingWave to a Kafka broker and how to specify security (encryption and authentication) settings.

Apache Kafka

Sink to Kafka

 This guide describes how to sink data from RisingWave to ClickHouse using the ClickHouse sink connector in RisingWave.

ClickHouse

Sink data from RisingWave to ClickHouse

This guide describes how to sink data from RisingWave to Elasticsearch using the Elasticsearch sink connector in RisingWave.

Elasticsearch

Sink data from RisingWave to Elasticsearch

PostgreSQL

Sink data from RisingWave to PostgreSQL

This guide will introduce how to sink data from RisingWave to JDBC-available databases using the JDBC sink connector.

MySQL

Sink data from RisingWave to MySQL with the JDBC connector

SQL references

Overview of the SQL commands supported by RisingWave.

SQL commands

The `ALTER CONNECTION` command modifies the definition of a connection. To use this command, you must own the connection.

ALTER CONNECTION

The `ALTER DATABASE` command modifies the definition of a database.

ALTER DATABASE

The `ALTER FUNCTION` command modifies the definition of a function. To use this command, you must own the function.

ALTER FUNCTION

The `ALTER INDEX` command modifies the definition of an index.

ALTER INDEX

The `ALTER MATERIALIZED VIEW` command modifies the metadata of a materialized view. To use this command, you must own the materialized view.

ALTER MATERIALIZED VIEW

The `ALTER SCHEMA` command modifies the definition of a schema. To use this command, you must own the schema.

ALTER SCHEMA

Use the `ALTER SECRET` command to alter the secrets that store sensitive credentials.

ALTER SECRET

The `ALTER SINK` command modifies the metadata of a sink.

ALTER SINK

The `ALTER SOURCE` command modifies the definition of a source.

ALTER SOURCE

The `ALTER SUBSCRIPTION` command modifies the definition of a subscription.

ALTER SUBSCRIPTION

ALTER SWAP

The `ALTER SYSTEM` command modifies the value of a server configuration parameter.

ALTER SYSTEM

The `ALTER TABLE` command modifies the definition of a table.

ALTER TABLE

Use the `ALTER USER` command to modify the name, password, system permissions, and other properties of an existing user.

ALTER USER

The `ALTER VIEW` command modifies the metadata of a view. To use this command, you must own the view.

ALTER VIEW

Use the `AS CHANGELOG` clause to convert a changelog operation in a stream into a column.

AS CHANGELOG

RisingWave supports read-only transactions. You can use the `BEGIN READ ONLY` command to start a read-only transaction.

BEGIN

Use `CANCEL JOBS/JOB` to cancel specific streaming jobs that are in progress.

CANCEL JOBS

You can use the `COMMENT ON` command to add comments on tables or columns in RisingWave. Comments are stored as metadata of tables or columns.

COMMENT ON

RisingWave supports read-only transactions. You can use the `COMMIT` command to commit the current transaction.

COMMIT

The `CREATE AGGREGATE` command can be used to create [user-defined aggregate functions](/sql/udfs/user-defined-functions) (UDAFs). Currently, UDAFs are only supported in Python and JavaScript as embedded UDFs.

CREATE AGGREGATE

Use the `CREATE CONNECTION` command to create a reusable catalog for connector parameters.

CREATE CONNECTION

Use the `CREATE DATABASE` command to create a new database.

CREATE DATABASE

The `CREATE FUNCTION` command can be used to create [user-defined functions](/sql/udfs/user-defined-functions) (UDFs).

CREATE FUNCTION

Use the `CREATE INDEX` command to construct an [index](/processing/indexes) on a table or a materialized view.

CREATE INDEX

Use the `CREATE MATERIALIZED VIEW` command to create a materialized view. A materialized view can be created based on sources, tables, materialized views, or indexes.

CREATE MATERIALIZED VIEW

Use the `CREATE SCHEMA` command to create a new schema.

CREATE SCHEMA

Use the `CREATE SECRET` command to create secrets to store sensitive credentials securely.

CREATE SECRET

Use the `CREATE SINK INTO` command to create a sink into RisingWave's table.

CREATE SINK INTO

Use the `CREATE SINK` command to create a sink. A sink is an external target where you can send data processed in RisingWave. You can create a sink from a table or a materialized view.

CREATE SINK

A source is a resource that RisingWave can read data from. You can create a source in RisingWave using the `CREATE SOURCE` command.

CREATE SOURCE

Use the `CREATE TABLE` command to create a new table. Tables consist of fixed columns and insertable rows. 

CREATE TABLE

Use the `CREATE USER` command to create a new user account in RisingWave.

CREATE USER

Use the `CREATE VIEW` command to create a non-materialized view, which runs every time the view is referenced in a query. A non-materialized view can be created based on sources, tables, views, or indexes.

CREATE VIEW

Use the `DELETE` command to permanently remove rows from a table.

DELETE

Use the `DESCRIBE` command to view columns in the specified table, source, sink, view, or materialized view.

DESCRIBE

Use the `DISCARD` command to reset the state of current session

DISCARD

Use the `DROP AGGREGATE` command to remove an existing [user-defined aggregate function (UDAF)](/sql/udfs/user-defined-functions). The usage is similar to `DROP FUNCTION`, except that it's for aggregate functions.

DROP AGGREGATE

Use the `DROP CONNECTION` command to remove a connection.

DROP CONNECTION

Use the `DROP DATABASE` command to remove a database from your RisingWave instance.

DROP DATABASE

DROP FUNCTION

Use the `DROP INDEX` command to remove an index from a table or a materialized view.

DROP INDEX

Use the `DROP MATERIALIZED VIEW` command to remove a materialized view from the database.

DROP MATERIALIZED VIEW

Use the `DROP SCHEMA` command to remove a schema from a database.

DROP SCHEMA

Use the `DROP SECRET` command to drop the secrets that store sensitive credentials.

DROP SECRET

Use the `DROP SINK` command to remove a [sink](/sql/commands/sql-create-sink) if you no longer need to deliver data to the sink.

DROP SINK

Use the `DROP SOURCE` command to remove a [source](/sql/commands/sql-create-source) if you no longer need the data inflow from the source.

DROP SOURCE

Use the `DROP TABLE` command to remove a table from the database.

DROP TABLE

Use the `DROP USER` command to remove a user from RisingWave.

DROP USER

Use the `DROP VIEW` command to remove an existing view from a particular schema.

DROP VIEW

Use the `EXPLAIN` command to show the execution plan of a statement.

EXPLAIN

The `FLUSH` command commits any pending data changes and forces RisingWave to persist updated data to storage, which guarantees subsequent reads can access the latest data.

FLUSH

Use the `GRANT` command to grant specific privileges to a user.

Connector	Version
Kafka	3.1.0 or later
Redpanda	Latest
Pulsar	2.8.0 or later
Kinesis	Latest
PostgreSQL CDC	10, 11, 12, 13, 14
MySQL CDC	5.7, 8.0
SQL Server CDC	2019, 2022
MongoDB CDC
Citus CDC	10.2
CDC via Kafka
Google Pub/Sub
Amazon S3	Latest
Google Cloud Storage	Latest
Azure Blob	Latest
NATS JetStream
MQTT
Apache Iceberg
Load generator	Built-in

Get started

Working with data

Deploy & operate

Optimize performance

Troubleshooting

Reference

Cloud

Supported sources