During initial data loading, AP analytical scenarios often require importing massive datasets.
If the traditional serial loading approach is used, loading tens or even hundreds of millions of rows from InnoDB into Rapid can take an excessively long time, severely delaying the start of analytical tasks. To address this, the Rapid engine introduces coroutine-based parallel parsing, accelerating column data parsing and assembly to significantly reduce query rendering time for wide tables.
During initial data loading, AP analytical scenarios often require importing massive datasets.
If the traditional serial loading approach is used, loading tens or even hundreds of millions of rows from InnoDB into Rapid can take an excessively long time, severely delaying the start of analytical tasks.
In addition, analytical workloads often operate on wide tables (tables with a large number of columns).
In a columnar storage layout, if a traditional serial parsing approach is used, the system must combine data from each column sequentially during query execution to produce the final results — a process that can be time-consuming.
To address this, the Rapid engine introduces coroutine-based parallel parsing, accelerating column data parsing and assembly to significantly reduce query rendering time for wide tables.
Feature Introduction
To address these bottlenecks, two key system variables were introduced:
During row scans, if the number of columns exceeds this value, the column parsing tasks are split into batches and executed concurrently in a coroutine pool.
Reduce per-row latency in wide-table scans and increase query throughput.
Performance Benefits
Bulk Loading
When the row count exceeds rapid_parallel_load_max, parallel loading reduces the loading time for tens of millions of rows from minutes to seconds, achieving a 3–5× speedup.
Wide Table Scans
When the column count exceeds rapid_async_column_threshold, batch-based coroutine column parsing reduces per-row scan latency by over 40%.
CPU Utilization
Parallel and coroutine-based execution keeps multi-core CPUs fully utilized during load and scan phases, reducing idle time.
Core Implementation
System Variable Control
Registered via MySQL plugin variable mechanism, adjustable at runtime (some variables can only be changed before data is loaded).
Validation checks ensure no active load/scan tasks are running before changes, preserving consistency.
Parallel Loading (**rapid_parallel_load_max**)
During secondary_load, when row count exceeds the threshold, data is partitioned and imported into Rapid’s in-memory columnar store in parallel.
