By default, RisingWave is configured to utilize the maximum available CPUs across compute nodes for streaming jobs. In RisingWave, a streaming job refers to the creation of a table, source, index, materialized view, or sink. This design aims to achieve high performance by fully utilizing compute resources.
risingwave/src/config/<your-config>.yaml
.
default_parallelism
setting determines the parallelism for newly created streaming jobs. Change the streaming_parallelism
before creating streaming jobs can achieve the same effect. For guidance on how to set this value, refer to How to adjust the resources allocated to each streaming query?.
max_concurrent_creating_streaming_jobs
is helpful. It controls the maximum number of streaming jobs created concurrently. However, please do not set it too high, as it may introduce excessive pressure on the cluster.
default
. The parallelism will then fall back to the global streaming_parallelism
setting.
rw_fragment_parallelism
, and you can alter the streaming jobs’s parallelism with the ALTER
statement. For more information, refer to Cluster scaling.
Here is an example of how to adjust the parallelism.
/risingwave/bin/risingwave ctl scale horizon --include-workers all
is used to scale out all streaming jobs to avoid the skewed actor distribution. However, this approach may not be sufficient when dealing with a large number of streaming jobs, as it does not consider the default_parallelism
parameter.{"level":"ERROR","fields":{"message":"lease keeper failed","error":"grpc request error: status: Unavailable, message: \"etcdserver: request timed out, waiting for the applied index took too long\",}
, please try to scale up the nodes.