Ingest data from Instaclustr Kafka
You can ingest data from Kafka deployed in Instaclustr, a fully managed and integrated platform with popular open-source tools such as Kafka, PostgreSQL, Cassandra, and Redis. It facilitates easy Kafka Connect integration and includes a dedicated ZooKeeper along with Kraft that delivers a seamless Kafka journey with a 100% open-source solution.
Set up Kafka on Instaclustr Cloud
This guide goes through the steps to create a Kafka cluster in Instaclustr and to connect it to RisingWave for data ingestion. For more information regarding the data ingestion from Instaclustr Cloud, please refer to Instaclustr Documentation.
Sign up for an Upstash Cloud account
Start by signing up for a free Instaclustr account, which will grant you access to Kafka services. To create your account, visit Instaclustr Cloud.
Create a Kafka cluster
Once you are logged in, follow the instructions in Getting Started with Apache Kafka to create a Kafka cluster in the Instaclustr Cloud.
After successfully creating a Kafka cluster and then, a Kafka topic, add the IP address of the connecting computer to the cluster Firewall Rules to produce and consume data.
Connect and interact with your Kafka cluster
You can connect with the Kafka cluster using various clients and languages such as Java, C#, Python, and Ruby to produce and consume Kafka messages using the Connection Info page that contains a list of your node addresses, authentication credentials to connect to your cluster, and a few connection examples for popular clients Kafka supports. Therefore, you can extract real-time flight data from the AviationStck API and feed it into a Kafka topic in the Kafka cluster deployed in the Instaclustr Cloud.
After these steps, you are on your way to build stream processing applications and pipelines both using Kafka deployed in Instaclustr cloud and RisingWave!
Ingest data from Instaclustr Kafka
Create a RisingWave project
You can create a RisingWave project and connect to it by following the steps in the Quick Start in the RisingWave documentation.
Create a source
Once you have successfully deployed the RisingWave project and connected to it, proceed to create a source in RisingWave to ingest data from Instaclustr Kafka.
Create a materialized view
Let’s create a materialized view named aviation_mv
based on the source aviation_source
, to transform and cast certain columns from the source data, effectively modifying their data types.
Query the materialized view
The materialized view can be queried to retrieve the latest data from the source:
The retrieved result should look like this:
You have successfully consumed data from a Kafka topic into the RisingWave, created a source and a materialized view, and then queried it.
Was this page helpful?