Troubleshoot and resolve high latency and slow stream processing issues in RisingWave.
array_agg
can be computationally expensive, especially with large datasets or high cardinality grouping.A join B on A.nation = B.nation
, any operation on a row in A or B will be amplified by the join, potentially leading to performance issues.UNION
instead of UNION ALL
: The UNION
operator removes duplicate rows, which requires extra processing. If you don’t need to deduplicate rows, use UNION ALL
for significantly better performance.http://meta-node:5691
. If the barrier is stuck, the Await Tree Dump will reveal which operation it’s waiting for, helping you pinpoint the problematic part of your streaming pipeline.
To help identify bottlenecks in the await tree, you can use the Await-Tree Analyzer for analysis. This tool is also available for self-hosting, please consult the repository for setup instructions.
JOIN
clauses, WHERE
conditions, or complex functions) to isolate the specific part causing the slowdown.
15002→15001
and 15003→15002
show high backpressure. Since backpressure propagates upstream, the root cause is likely fragment 15001
.
To correlate this with your SQL query, use the RisingWave Dashboard and access the “Fragment” panel. You can also run EXPLAIN CREATE MATERIALIZED VIEW …
to see the query plan and identify the part of the query corresponding to the problematic fragments.
high_join_amplification
with the problematic join keys will be printed, such as
product_id = 1
is a hot-selling product causing amplification, split the materialized view:
orders.user_id % 7 = 0
.orders.user_id % 7 = 1
.orders.user_id % 7 = 6
.user_id
is not directly usable with modulo, use a hash function: hash_value(orders.user_id)
. Account for negative hash values: hash_value(orders.user_id) % 7 = -1/-2/-3/-4/-5/-6
.
Finally, UNION ALL
the results from all the materialized views.
UNION ALL
instead of UNION
: If you don’t need to deduplicate rows, use UNION ALL
.