High latency

​

Symptoms

• High barrier latency: Examine the Barrier Latency panel in the Grafana dashboard (dev) > Streaming section. A consistently high latency curve indicates that barriers are getting stuck, signifying a slowdown in the streaming pipeline. See Monitoring and metrics - Barrier monitoring for details.

• Backpressure: Observe the Actor Output Blocking Time Ratio (Backpressure) panel in Grafana dashboard (dev) > Streaming Actors. High backpressure between fragments indicates that a downstream component is unable to keep up with the upstream data flow. See Workload analysis for more details.
• Low source throughput: Check Source throughput to see if the input is the cause. Low throughput from the source, without significant backpressure, often indicates a problem with the upstream system itself.
• Data lag: If RisingWave is falling behind in processing data, this indicates a slowdown. This might be observed as a growing delay between when data is generated in the upstream source and when it’s reflected in RisingWave’s materialized views.
• Increasing number of barriers: A steadily increasing number of uncollected barriers is a sign of processing delays.

​

Potential causes

​

Backpressure

​

Resource constraints (CPU, memory, disk I/O)

​

Slow upstream sources

​

Network issues

​

Inefficient queries

• Heavy aggregations: Functions like array_agg can be computationally expensive, especially with large datasets or high cardinality grouping.
• Joins with unreasonable amplification: Joining tables on low-cardinality columns (columns with few distinct values) can result in a massive increase in the number of rows processed, slowing down the join operation.

• Using UNION instead of UNION ALL: The UNION operator removes duplicate rows, which requires extra processing. If you don’t need to deduplicate rows, use UNION ALL for significantly better performance.
• UDF performance issues: Slow User-Defined Functions (UDFs) can become a bottleneck.
• Improper configuration: Incorrect configuration settings can negatively impact performance.

​

Diagnosis

​

General approach

1. Check barrier latency and number: Start by examining the “Barrier Latency” and “Barrier Number” panels in Grafana, as described in the “Symptoms” section.
2. Analyze backpressure: Go to Grafana dashboard (dev) > Streaming Actors > Actor Output Blocking Time Ratio (Backpressure) panel and analyze backpressure between fragments. High backpressure points to a bottleneck in the downstream fragment.
3. Examine await tree dump: Check the Await Tree Dump of all compute nodes in RisingWave Dashboard hosted on http://meta-node:5691. If the barrier is stuck, the Await Tree Dump will reveal which operation it’s waiting for, helping you pinpoint the problematic part of your streaming pipeline. To help identify bottlenecks in the await tree, you can use the Await-Tree Analyzer for analysis. This tool is also available for self-hosting, please consult the repository for setup instructions.
4. Examine resource utilization: Check CPU usage, memory usage, and cache miss ratios as described in Monitoring and metrics. This will help you identify potential resource constraints.
5. Simplify the query: If you suspect an inefficient query, try removing parts of the SQL query (e.g., individual JOIN clauses, WHERE conditions, or complex functions) to isolate the specific part causing the slowdown.
6. Check for errors: Examine RisingWave’s logs for any error messages.

​

Identifying specific causes

​

Identify back-pressured fragments

​

Resource constraints

​

Slow upstream sources

​

Network issues

​

Inefficient queries

hash_join_amplification: large rows matched for join key matched_rows_len=200000 update_table_id=31 match_table_id=33 join_key=OwnedRow([Some(Int32(1)), Some(Utf8("abc"))]) actor_id=119 fragment_id=30

​

Solutions

​

General strategies

• Optimize queries: The most effective way to improve performance is often to improve the efficiency of your SQL queries used to create materialized views and sources. See Best practices - Query optimization for general guidance.
• Increase resources: If resource constraints are the issue, consider scaling up (adding more resources to existing nodes) or scaling out (adding more nodes). See Best practices - Resource allocation.
• Address upstream bottlenecks: If the upstream source is slow, you’ll need to address the bottleneck in the upstream system itself.
• Address network issues: Troubleshoot network connectivity.

​

Specific solutions for each cause

​

Backpressure

• Rewrite queries: Simplify or optimize the queries used to create your materialized views and sources. Focus on reducing the load on the bottleneck fragments identified during diagnosis. This might involve restructuring joins, adding filters, or optimizing aggregations.
• Increase parallelism: If a specific fragment is a bottleneck due to computational limitations, increasing its parallelism (if possible) can help distribute the workload. See Best practices - Resource allocation.
• Address resource constraints: Underlying resource limitations can cause backpressure. See Specific bottlenecks for detailed guidance.

​

Resource constraints

​

Slow upstream sources

​

Network issues

​

Inefficient queries

• Reduce join amplification:
  • Rewrite the query: Use better join conditions or restructure your query to minimize the number of rows produced by joins. See Maintain wide table with table sinks for details.
  • Use the workaround: If you have identified specific join keys causing high amplification, you can split the original materialized view into multiple materialized views with filtering conditions to distribute the workload: For example, for this join:
    Copy

    Ask AI

    SELECT * FROM orders
    INNER JOIN product_description
    ON orders.product_id = product_description.product_id
    

    If product_id = 1 is a hot-selling product causing amplification, split the materialized view:
    1. Create one MV with orders.user_id % 7 = 0.
    2. Create another with orders.user_id % 7 = 1.
    3. Repeat up to orders.user_id % 7 = 6.
    If user_id is not directly usable with modulo, use a hash function: hash_value(orders.user_id). Account for negative hash values: hash_value(orders.user_id) % 7 = -1/-2/-3/-4/-5/-6. Finally, UNION ALL the results from all the materialized views.
• Optimize heavy aggregations: Consider alternative aggregation strategies, pre-aggregate data if possible, or use approximations if acceptable for your use case.
• Use UNION ALL instead of UNION: If you don’t need to deduplicate rows, use UNION ALL.
• Improve UDF performance: Optimize your UDF code or consider alternative approaches if UDFs are a bottleneck.