Out-of-Memory (OOM) is a common issue in data processing systems that can have multiple causes. This guide aims to help you identify the root causes of OOM errors and efficiently resolve the issues.
total_memory
set for the compute node. Memory settings can be found in the booting logs of the compute node. Search for the keyword “Memory outline” to locate the specific section.streaming_parallelism
to a smaller number:
4 * source_rate_limit
records/s.
Alternatively, you may use risectl
to alter the streaming rate limit of an existent materialized view, where the <id>
can be found either from the RisingWave Dashboard or rw_catalog
schema.
storage.prefetch_buffer_capacity_mb
parameter in the TOML file.
The storage.prefetch_buffer_capacity_mb
configuration defines the maximum memory allowed for prefetching. It aims to optimize streaming executor and batch query performance through pre-reading. This feature allows hummock to read larger chunks of data in a single I/O operation, but at a higher memory cost. When the memory usage during prefetch operations reaches this limit, hummock will revert to the original read method, processing data in 64 KB blocks. If you set the parameter to 0, this feature will be disabled. By default, it is set to 7% of the total machine memory.
MALLOC_CONF=prof:true
for the compute nodes.127.0.0.1:5691
.
By default, the heap profile data is automatically dumped when the memory usage reaches 90%. You also have the option to manually dump heap profile data. Once the data is dumped, you can click on Analyze within the dashboard to examine memory usage patterns and potential issues without leaving the dashboard.
#troubleshooting
channel. You may also file an issue in GitHub.
Please include the following details when you are reporting an issue: