This guide describes how to sink data from RisingWave to Google BigQuery.
big-query-sink
demo in the integration_test directory of the RisingWave repository.
Parameter Names | Description |
---|---|
sink_name | Name of the sink to be created. |
sink_from | A clause that specifies the direct source from which data will be output. sink_from can be a materialized view or a table. Either this clause or select_query query must be specified. |
AS select_query | A SELECT query that specifies the data to be output to the sink. Either this query or a sink_from clause must be specified. See SELECT for the syntax and examples of the SELECT command. |
type | Required. Data format. Allowed formats:
|
force_append_only | Optional. If true, forces the sink to be append-only, even if it cannot be. |
bigquery.local.path | Optional. The file path leading to the JSON key file located in your local server. Details can be found in Service Accounts under your Google Cloud account. Either bigquery.local.path or bigquery.s3.path must be specified. |
bigquery.s3.path | Optional. The file path leading to the JSON key file located in S3. Details can be found in Service Accounts under your Google Cloud account. At least one of bigquery.local.path or bigquery.s3.path must be specified. |
bigquery.project | Required. The BigQuery project ID. |
bigquery.dataset | Required. The BigQuery dataset ID. |
bigquery.table | Required. The BigQuery table you want to sink to. |
bigquery.credentials | Optional. Enter the JSON key credentials as a string. At least one of bigquery.credentials or bigquery.local.path or bigquery.s3.path must be specified. |
bigquery.retry_times | Optional. The number of times the system should retry a BigQuery insert operation before ultimately returning an error. Defaults to 5. |
auto_create | Optional. Defaults to false. If true, a new table will be automatically created in BigQuery when the specified table is not found. |
aws.credentials.access_key_id | Optional. The access key of the S3 file. This must be specified if sinking to an S3 file. |
aws.credentials.secret_access_key | Optional. The secret access key of the S3 file. This must be specified if sinking to an S3 file. |
region | Optional. The service region of the S3 file. This must be specified if sinking to an S3 file. |
RisingWave Data Type | BigQuery Data Type |
---|---|
boolean | bool |
smallint | int64 |
integer | int64 |
bigint | int64 |
real | unsupported |
double precision | float64 |
numeric | numeric |
date | date |
character varying (varchar) | string |
time without time zone | time |
timestamp without time zone | datetime |
timestamp with time zone | timestamp |
interval | interval |
struct | struct |
array | array |
bytea | bytes |
JSONB | JSON |
serial | int64 |