Operate
Cluster limit
This guide introduces the cluster limit designed to enhance user experience when cluster resources are limited. It sets a limit on the number of actors per parallelism (i.e., CPU core). Exceeding this limit when you create materialized view, table, or sink will trigger a warning message.
Limit types
- Soft limit:
meta_actor_cnt_per_worker_parallelism_soft_limit
, defaults to 100. If exceeding this limit, creating materialized view, table, or sink can still succeed but will generate a SQL notice message when the statement returns. - Hard limit:
meta_actor_cnt_per_worker_parallelism_hard_limit
, defaults to 400. If exceeding this limit, creating materialized view, table, or sink will fail and generate an error message when the statement returns.
Notes
- The limit applies only when you create new materialized view, table, or sink. Existing database objects are unaffected.
- To resolve issues related to these limits, consider scaling the cluster or reducing the streaming job parallelism.
- Default limits may be adjusted in future releases.
Override cluster limits
If it’s safe for the cluster to operate with limits exceeded, you can override the default behavior using one of the following methods:
- Bypass the limit check via session variable:
SET bypass_cluster_limits TO true
. - Increase the limit via meta developer config:
Please be aware that once you bypass the check or increase the limits, the cluster could become overloaded, leading to issues with stability, availability, or performance.
Examples
Create a materialized view exceeding the soft limit
Create a materialized view exceeding the hard limit
Was this page helpful?