This topic introduces the concept and the tuning method of reserved memory and cache eviction policy.
A simplified description of our memory control mechanism will help you understand these two concepts more straightforwardly. In simple words, RisingWave manages memory usage through a three-step process:
We allow you to tune both reserved memory and cache eviction policy.
Starting from version 1.10, RisingWave calculates the reserved memory based on the following gradient:
Details
Read an example.For example, let’s consider a compute node with 32GB of memory. The reserved memory would be calculated as follows:
This calculation method ensures that in scenarios with less memory, the system reserves more memory for critical tasks. On the other hand, in scenarios with more memory, it reserves less memory, thus achieving a better balance between system performance and memory utilization.
However, this may not be suitable for all workloads and machine setups. To address this, you can use the startup option --reserved-memory-bytes
or the environment variable RW_RESERVED_MEMORY_BYTES
to override the reserved memory configuration for compute nodes. But note that the memory reserved should be at least 512MB.
For instance, suppose you are deploying a compute node on a machine or pod with 64GB of memory. By default, the reserved memory would be calculated as follows:
However, if you find this excessive for your specific use case, you have the option to specify a different value. You can set either RW_RESERVED_MEMORY_BYTES=8589934592
as an environment variable or --reserved-memory-bytes=8589934592
as a command line parameter when starting up the compute node. This will allocate 8GB as the reserved memory instead.
Reserved memory setting before 1.10
Before version 1.9, RisingWave just allocated 30% of the total memory as reserved memory by default. However, through practical application, we realized that this default setting may not be suitable for all scenarios. Therefore, in version 1.9, we introduced the ability to customize the reserved memory.
To further optimize this feature, we changed the calculation method for reserved memory in version 1.10 and introduced the current gradient calculation method. These changes improve memory utilization and provide enhanced performance for our users.
By continuously improving the reserved memory feature, we strive to offer a more flexible and efficient memory management solution to meet the diverse needs of our users.
You can tune the cache enviction policy by configuring the memory_controller_eviction_factor_XXX
variables in the configuration toml file.
Variables in the configuration toml file
RisingWave uses a tiered eviction strategy to manage cache based on the current memory usage:
memory_controller_threshold_stable
, RisingWave begins to evict data from the operator cache with mild intensity.memory_controller_threshold_graceful
triggers a more aggressive eviction strategy.memory_controller_threshold_aggressive
, the eviction process becomes most aggressive to free up memory.Each threshold corresponds to a specific configuration variable in the toml
configuration file.
For more detailed information on how the policy adjusts the intensity of eviction, you can refer to the source code directly.
Suppose the reserved memory is set to 30%, and memory_controller_threshold_stable
is 0.8. RisingWave will aim to use 56% of the total memory stably during normal operations, calculated as:
Then memory usage typically stays below 56%. It only exceeds this limit during sudden throughput spikes or when creating a new materialized view that processes large amounts of historical data. In these cases, more aggressive memory eviction strategies are triggered.
The best configuration depends on your specific workload and data pattern. We recommend you tune the parameters based on the metrics on Grafana.