Stream processing benchmarks: Nexmark results

​

Performance highlights

RisingWave demonstrates significant performance advantages in a majority of stream processing scenarios, particularly those involving complex state management. Here are some key takeaways from the latest benchmark results:

• High throughput: RisingWave consistently delivers high throughput across various Nexmark queries, reaching up to 893.2 thousand records per second (kr/s) in q1 and 770.0 kr/s in q7-rewrite.
• Efficient resource utilization: RisingWave showcases efficient resource utilization, achieving up to 127.36 kr/s per core in q2.
• Strong performance in complex queries: RisingWave maintains robust performance even in complex queries like q7, q9, q18, and q20, which involve significant state management.

​

Detailed benchmark results

​

Benchmark methodology

The performance tests were conducted using the Nexmark benchmark, a widely recognized standard in the stream processing field. In addition to the standard Nexmark queries, an extended set of 5 queries was included to cover a broader range of common SQL operators.

​

Environment

• Hardware: Cloud Isolation Environment using single-node instances (8 vCPUs, 16GB memory) with S3 storage. Each pod exclusively occupies one machine.

• RisingWave version: nightly-20230309

• RisingWave configuration:

  [storage]
  block_cache_capacity_mb = 2048
  meta_cache_capacity_mb = 512
  compactor_memory_limit_mb = 2560
  shared_buffer_capacity_mb = 2048
  

​

Test procedure

1. Data was populated into Kafka using the nexmark-bench tool.
2. Nexmark Kafka sources were created using these SQLs.
3. CREATE SINK (blackhole) was used to test the performance and cost of Nexmark queries.
4. The RisingWave cluster was deployed via kube-bench.

​

Performance comparison with Flink

RisingWave’s performance has been extensively benchmarked against Apache Flink, providing valuable insights into its relative strengths. The comparison reveals significant performance advantages in most streaming scenarios.

​

Key comparative findings

• RisingWave outperformed Flink in 22 out of 27 queries
• 12 queries showed performance improvements of ≥50% compared to Flink
• 10 queries demonstrated more than 2x performance improvement
• Exceptional improvements were observed in:
  • Dynamic filtering operations (q102: 520x faster)
  • Anti-join operations (q104: 660x faster)
  • Complex state management queries (q7: 62x faster)

​

Performance characteristics by query type

​

Strong performance advantage (>2x faster than Flink)

• Queries with large internal states (>20GB)
• Complex join operations
• Dynamic filtering operations
• Aggregations with large state management

​

Comparable performance (90-110% of Flink)

• Basic stateless computations
• Simple filtering operations
• Network I/O bound operations

​

Areas for optimization (performing below Flink)

• Window aggregations without EOWC support (q5)
• Distinct aggregations without split optimization (q16)

​

Key performance factors

RisingWave’s performance advantages can be attributed to several design and implementation choices:

1. Rust-based architecture: RisingWave is built from the ground up in Rust and minimizes reliance on third-party JVM components. This provides inherent performance benefits at the computation layer.
2. Direct SQL optimization: Unlike systems with multiple abstraction layers, RisingWave directly optimizes SQL queries, enabling highly customized performance tuning.
3. Computation-aware storage: RisingWave’s custom storage implementation is aware of the computation, allowing for intelligent state management and reduced storage costs by leveraging remote storage (e.g., S3, HDFS).

​

Areas of strength

RisingWave excels in scenarios with:

• Complex stateful computations: Queries requiring large and complex internal states benefit significantly from RisingWave’s efficient state management.
• Multi-stream joins: Initial tests indicate that RisingWave can efficiently handle joins of multiple data streams, making it suitable for scenarios with multiple data sources.

​

Considerations

• Stateless computations: Performance gains in stateless computations might be less pronounced when network I/O between RisingWave and the data source (e.g., Kafka) becomes the bottleneck.
• Performance degradation: Reducing compute node memory or introducing irregular access patterns can lead to a high cache miss rate, impacting performance due to increased access to remote storage.
• Emit On Window Close (EOWC): RisingWave will soon support EOWC semantics, which can optimize window aggregation functions.
• Split distinct aggregation: RisingWave will add the optimization for split distinct aggregation in the future, which will further improve the performance for queries like q16.

​

Future performance evaluations

Further testing is planned to evaluate RisingWave’s performance in areas such as:

• Complex stateful computations: Including operations like JSON parsing and string processing.
• Multi-stream joins: Releasing detailed results of multi-stream join experiments.
• Advanced features: Assessing the impact of UDFs, watermarks, and other advanced features on performance.