Accelerating Spark with the Java Vector API

Apache Spark has had a columnar execution direction for some time, and projects such as Apache DataFusion Comet have demonstrated how much performance can be gained by moving away from row-at-a-time execution. But there is another interesting question:

Can we bring SIMD-style execution directly into Spark’s JVM execution path, using Java itself?

That is the idea behind VecRuntime, a JVM-native vectorized execution engine for Spark SQL. The project uses the Java Vector API to execute Spark SQL operators over Arrow-layout columnar batches. Unsupported operations simply fall back to Spark like on Comet.

Consider a completely ordinary PySpark query:

df = (
    sales
    .where(...)
    .groupBy(...)
    .agg(...)
    .orderBy(...)
)

There is nothing VecRuntime specific here. The application still uses the standard DataFrame API. The difference happens later.

Spark generates a physical plan and VecRuntime applies a planning rule that replaces supported Spark operators with vectorized implementations.

Anything that cannot be converted remains a normal Spark operator. That fallback model is important because it means acceleration can be incremental rather than all-or-nothing.

Why the JVM?

When people hear “vectorized execution”, they often think of native code. That is a reasonable assumption. Modern analytical engines frequently use C++, SIMD intrinsics, native memory layouts, and specialized runtimes. VecRuntime takes a different route. Its kernels are implemented in Java using the Java Vector API. The Java Vector API allows code to operate on multiple values per instruction while still remaining inside the JVM.

On an ARM64 machine, the JVM can use the available NEON or SVE vector instructions. On machines with wider x86 SIMD capabilities, the Vector API can select wider species automatically like AVX512.

What does VecRuntime accelerate?

The project currently implements vectorized versions of several core Spark SQL operators covering TPC-H and TPC-DS queries (details here).

However, the important part is not simply the number of operators. It is the composability. A vectorized Filter can feed a vectorized Aggregate, which can feed another vectorized operator, without converting back to Spark rows between every step.

That is where the architecture starts becoming interesting.

The columnar representation

VecRuntime operates on columnar batches using an Arrow-compatible layout. That means an execution pipeline can look like:

The operators work directly on columns rather than repeatedly materializing UnsafeRow objects. This matters because a vectorized operator is only useful if the surrounding pipeline preserves the vectorized representation. And that leads to one of the biggest lessons from the project.

The hidden cost: shuffle boundaries

Suppose we have:

VectorAggregate -> Spark Shuffle -> VectorAggregate

Spark’s traditional shuffle machinery is fundamentally row-oriented. That means a columnar batch can end up going through a sequence conceptually similar to:

The vectorized operator may be faster, but the conversion overhead can erase much of the benefit. This is why the example repository contains an optional columnar shuffle.

A shuffle designed for columnar execution

The VecRuntime shuffle moves Arrow record batches rather than rows. The architecture is roughly:

Columnar Shuffle overview

For remote shuffle, the current implementation uses Apache Arrow Flight over gRPC. The important property is not the transport itself.

The data can remain columnar across the shuffle boundary.

That removes the row-to-columnar conversion that otherwise happens at every stage boundary.

The PySpark example

The example repository was built specifically to demonstrate this approach without requiring a large Spark deployment.

The repository uses uv to manage the Python environment ( you will need Java 25too). The quick start is simply:

The repository uses uv to manage the Python environment.

The quick start is simply:

git clone https://github.com/vecruntime/pyspark-example
cd pyspark-example
uv run example.py

The first execution generates a small star schema:

sales
  ~20 million rows
stores
  1,000 rows

The generated data is written as Parquet and reused across subsequent runs. This gives us a simple but realistic analytical workload without needing an external dataset.

The example contains three DataFrame queries.

1. A TPC-H Q1-style aggregation

The first query filters rows, computes derived expressions, performs a grouped aggregation and orders the result. Conceptually:

Parquet scan
     ↓
Filter
     ↓
HashAggregate
     ↓
Sort

This is a particularly good shape for vectorization because it contains arithmetic, predicates and aggregation over large numbers of rows.

2. A broadcast join and top-N query

The second query joins the large sales table with the small stores dimension:

sales
   │
 Filter
   │
   ├──────── Broadcast stores
   │
 BroadcastHashJoin
   │
 Aggregate
   │
 TakeOrdered / LIMIT

This tests a different part of the engine: joins plus aggregation plus top-N processing.

3. Ranking within each region

The third query performs an aggregation followed by a ranking window:

Join
  ↓
Aggregate
  ↓
Window / Rank
  ↓
Filter rank <= 3

This is useful because it combines several operators and exercises the ability to keep a larger section of the plan columnar.

The most useful part: same code, two execution modes

The example deliberately runs each query twice with just a config change. It also inspects the final physical plan to show which operators were accelerated.

This makes the experiment much more interesting than simply comparing two unrelated implementations.

Get Angel Conde’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

We are comparing:

Same PySpark code
Same Spark session
Same data
Same query
        │
        ├── Spark execution
        │
        └── VecRuntime execution

What does the example show?

On my Apple M3 Pro system, using Spark 4.1.3 and 20 million rows, the repository reports the following indicative results ( relative speed up):

• 1.19× for the aggregation query
• 1.24× for the join/top-N query
• 1.17× for the ranking query

These are intentionally presented as indicative rather than universal benchmarks. The example is running locally on one machine, the machine was doing other work, and the data size is relatively small.

The more interesting observation is what happens without the columnar shuffle. For this workload, the README notes that stage boundaries can spend enough time converting between columnar batches and rows that much of the operator-level gain disappears.

Faster operators alone are not enough. The execution pipeline has to remain columnar.

The kernels are where the fun starts

The repository also contains lower-level JMH benchmarks for the Java kernels. These are kernel-level numbers, not end-to-end Spark query speedups.

That distinction matters enormously. A kernel can be 8× faster while the overall query only becomes 1.2× faster because the application also spends time scanning Parquet, scheduling tasks, shuffling, materializing results, performing unsupported expressions and so on.

This is precisely why the columnar pipeline matters.

Aggregation is more subtle

Vectorization is not automatically beneficial everywhere. Grouped aggregation is a good example.

A naive implementation might perform a SIMD reduction separately for every group. That sounds attractive.

But imagine four groups and a batch of thousands of rows. You don’t want to scan the entire batch four times. The implementation therefore uses a scatter-oriented approach where rows update grouped accumulators.

SIMD alone doesn’t make an algorithm efficient.

The algorithm still has to respect memory access patterns, cache behavior and data dependencies.

Floating-point correctness is another trade-off

There is another interesting issue with vectorized aggregation: floating-point associativity. Spark’s normal aggregation order can produce a particular rounding result. A vectorized reduction may use multiple partial sums and therefore perform additions in a different order.

But floating-point arithmetic can produce slightly different results. VecRuntime therefore has a strict floating-point mode. With strict mode enabled, it can reproduce Spark’s rounding semantics rather than simply maximizing SIMD throughput.

This is a useful example of a principle that gets overlooked in vectorized execution projects:

Performance is only useful when the semantics are correct.

And this is still Spark

One of the architectural decisions I particularly like is the fallback behavior. VecRuntime does not try to implement every Spark expression and operator before users can try it. Instead, it can partially accelerate a plan. For example:

VectorFilter
     ↓
VectorProject
     ↓
SparkPythonUDF
     ↓
SparkWrite

The vectorized operators run where possible, and Spark handles the rest. The README lists unsupported or non-vectorized areas (yet). This makes the approach much closer to an accelerator layer than a replacement Spark engine.

There is also a Spark UI

One of the nicest features of the example is the Vector Acceleration tab in the Spark UI.

With:

uv run example.py --keep-alive

The example keeps the Spark session alive and exposes the Spark UI. The custom tab visualizes the execution plan and shows which operators were executed by VecRuntime and which remained on Spark.

Detail plan of a SQL query

That is especially useful for debugging partial acceleration. Instead of asking:

“Why didn’t this query speed up?”

you can ask:

“Which operator prevented the pipeline from remaining vectorized?”

That is a much more actionable question.

Looking beyond a local laptop

The local PySpark example is intentionally simple. The main project has also been tested on larger workloads, including TPC-H and TPC-DS.

The current project documentation reports a 1 TB TPC-DS comparison on EKS using eight 13-core executors.

In that configuration, the documented current as publication date totals are:

They demonstrate that a JVM-native execution path can be pushed far beyond a small local experiment and can execute a substantial analytical workload.

The project reports that accelerated 82 of the 103 TPC-DS queries in that campaign.

JVM-native does not mean anti-native

An interesting part of the design is that the project is designed to coexist with Apache DataFusion Comet.

A potential pipeline looks like:

Native Parquet decoding
        ↓
Arrow / columnar representation
        ↓
VecRuntime
        ↓
Comet or other native component

The common currency is the columnar representation. That opens up a more interesting architecture than the usual:

JVM engine vs native engine

Instead, we can think about:

Different execution components sharing the same columnar memory representation.

That is a much more composable model.

Final thoughts

There is a lot of attention on native execution engines for Spark. And for good reason. But the Java ecosystem has an interesting capability that is still relatively under-explored for analytical execution: the Java Vector API.

VecRuntime is an experiment around that idea. The goal is not to replace Spark. It is to ask whether Spark’s familiar execution model can be pushed further by:

Not necessarily need to leaving the JVM to get a vectorized engine.

Resources

• VecRuntime: https://github.com/vecruntime/vecruntime
• PySpark example: https://github.com/vecruntime/pyspark-example
• Java Vector API: https://openjdk.org/jeps/338
• Apache DataFusion Comet: https://github.com/apache/datafusion-comet
• Apache Arrow: https://arrow.apache.org/