Skip to main content
RisingWave exposes Prometheus metrics for post-checkpoint connector acknowledgments. These metrics apply to source connectors that require explicit message acknowledgment (for example, Pulsar, NATS JetStream, and Google Pub/Sub).

​
Metrics

​
source_connector_ack_success_count

Counts successful acknowledgments sent to the broker after a checkpoint. Labels:
  • connector_type: connector identifier. Current values include pulsar, nats_jetstream, and pubsub (note: this may differ from the SQL connector value, for example nats and google_pubsub).

​
source_connector_ack_failure_count

Counts failed acknowledgment attempts after a checkpoint. Labels:
  • source_name: name of the source.
  • connector_type: connector identifier (pulsar, nats_jetstream, or pubsub).
  • error_type: reason the acknowledgment failed. The set of values is bounded. Which values a source emits depends on the connector:
    • Emitted by pubsub and nats_jetstream:
      • error — the broker RPC returned an error.
      • timeout — the ack RPC exceeded the internal ack timeout.
    • Emitted by pulsar:
      • empty_message_id — the checkpoint ack batch contained no message ID.
      • channel_missing — the internal ack channel was not registered when the checkpoint completed.
      • channel_send_error — sending the message ID to the internal ack channel failed.
      • decode_error — the Pulsar reader failed to decode the message ID.
      • broker_error — the Pulsar broker rejected the acknowledgment.

​
Grafana panels

If you use the risingwave_dev_dashboard Grafana dashboard, the following panels display these metrics:
  • Source Connector Ack Successes — plots source_connector_ack_success_count over time.
  • Source Connector Ack Failures — plots source_connector_ack_failure_count by error_type.
For instructions on setting up the Grafana dashboard, see Monitor a RisingWave cluster.

​
Troubleshooting

Pub/Sub and NATS JetStream:
  • Rising error count: the broker returned an error on the ack RPC. Check broker logs, subscription/consumer configuration, and network connectivity from the RisingWave compute nodes.
  • Rising timeout count: the ack RPC exceeded RisingWave’s internal ack timeout. Investigate broker latency or backpressure; sustained timeouts can cause redelivery.
Pulsar:
  • Rising broker_error count: the Pulsar broker is rejecting acknowledgments. Check broker logs and connectivity from RisingWave compute nodes.
  • channel_missing or channel_send_error: the internal acknowledgment channel was not ready when the checkpoint completed. This may indicate a race condition during source restart; monitor whether the count stabilizes after the source reconnects.
  • empty_message_id: a message was consumed without a valid message ID. This is typically a connector-specific error; review the source configuration.
  • decode_error: the Pulsar reader failed to decode a message ID before acking. Review recent source or Pulsar client changes.