Supported protobuf types

Conversion

RisingWave converts well-known types from the protobuf library to specific types in RisingWave. The conversion is as follows:

Protobuf type	RisingWave type
any	`jsonb`
double	`double precision`
float	`real`
int32	`int`
int64	`bigint`
uint32	`bigint`
uint64	`decimal`
sint32	`int`
sint64	`bigint`
fixed32	`bigint`
fixed64	`decimal`
sfixed32	`int`
sfixed64	`bigint`
bool	`boolean`
string	`varchar`
bytes	`bytea`
enum	`varchar`
message	`struct`. See details in Nested messages.
messages_as_jsonb	`jsonb`. See details in Handle recursive definitions.
repeated	`array`
map	`map`. See details in Map.
google.protobuf.Struct	Not supported
google.protobuf.Timestamp	`struct<seconds bigint, nanos int>`
google.protobuf.Duration	`struct<seconds bigint, nanos int>`
google.protobuf.Any	`struct<type_url varchar, value bytea>`
google.protobuf.Int32Value	`struct<value int>`
google.protobuf.StringValue	`struct<value varchar>`

Nested messages

The nested fields are transformed into columns within a struct type. For example, a Protobuf message with the following structure:

Example

message NestedMessage {
  int32 id = 1;
  string name = 2;
}

Will be converted to struct<id int, name varchar> in RisingWave.

Handle recursive definitions

When detecting a recursive definition in the protobuf, RisingWave will reject the statement and show the circular dependency. Adding dependency items to messages_as_jsonb with full type name separated by comma can solve the case. For example:

CREATE TABLE opentelemetry_test 
WITH ( 
    ${RISEDEV_KAFKA_WITH_OPTIONS_COMMON}, 
    topic = 'opentelemetry_test' 
) 
FORMAT PLAIN 
ENCODE PROTOBUF ( 
    schema.registry = '${RISEDEV_SCHEMA_REGISTRY_URL}', 
    message = 'opentelemetry_test.OTLPTestMessage', 
    messages_as_jsonb = 'opentelemetry.proto.common.v1.ArrayValue,opentelemetry.proto.common.v1.KeyValueList,opentelemetry.proto.common.v1.AnyValue'
);

Field presence

RisingWave correctly handles protobuf field presence to distinguish between missing fields and fields explicitly set to their default values. The handling depends on the field type:

Regular primitive fields

Regular primitive fields (without the optional keyword) receive their default values when absent from the protobuf message:

Example

syntax = "proto3";
message Example {
  int32 regular_int = 1;      // Gets 0 if missing
  string regular_string = 2;  // Gets "" if missing
  bool regular_bool = 3;      // Gets false if missing
}

Optional fields

Fields marked with the optional keyword are set to NULL when absent:

Example

syntax = "proto3";
message Example {
  optional int32 optional_int = 1;      // Gets NULL if missing
  optional string optional_string = 2;  // Gets NULL if missing
}

Message fields

Message fields (nested messages) are set to NULL when absent:

Example

syntax = "proto3";
message InnerMessage {
  string content = 1;
}
message Example {
  InnerMessage inner = 1;  // Gets NULL if missing
}

Oneof fields

Fields within a oneof group are set to NULL when the oneof is not set or a different field in the group is set:

Example

syntax = "proto3";
message Example {
  oneof test_oneof {
    string oneof_string = 1;  // NULL if oneof_int is set or if oneof is not set
    int32 oneof_int = 2;      // NULL if oneof_string is set or if oneof is not set
  }
}

Only one field within a oneof can be non-NULL at a time.

CREATE SOURCE

SQL reference

Python SDK

User-defined functions

Supported protobuf types

Conversion

Nested messages

Handle recursive definitions

Field presence

Regular primitive fields

Optional fields

Message fields

Oneof fields

​Conversion

​Nested messages

​Handle recursive definitions

​Field presence

​Regular primitive fields

​Optional fields

​Message fields

​Oneof fields

​Related topics

Conversion

Nested messages

Handle recursive definitions

Field presence

Regular primitive fields

Optional fields

Message fields

Oneof fields

Related topics