Sink data from RisingWave to WebHDFS - RisingWave

As a workaround for HDFS, WebHDFS allows external clients to execute Hadoop file system operations without necessarily running on the Hadoop cluster itself. Therefore, it reduces the dependency on the Hadoop environment when using HDFS.

Syntax

CREATE SINK [ IF NOT EXISTS ] sink_name
[FROM sink_from | AS select_query]
WITH (
   connector='webhdfs',
   connector_parameter = 'value', ...
);

Parameters

Parameter names	Description
connector	Required. Support the WebHDFS connector only.
webhdfs.endpoint	Required. The endpoint for the WebHDFS service.
webhdfs.path	Required. The directory where the sink file is located.
type	Required. Defines the type of the sink. The WebHDFS sink only supports `append-only`.

Example

CREATE SINK webhdfs_sink AS SELECT v1
FROM t1
WITH (
    connector='webhdfs',
    webhdfs.path = '<test_path>',
    webhdfs.endpoint = '<test_endpoint>',
    type = 'append-only',
)FORMAT PLAIN ENCODE PARQUET(force_append_only=true);

Advanced topics

For more information about encode Parquet or JSON, see Sink data in parquet or json format. For more information about batching strategy, see Batching strategy for file sink.