Accelerating Apache Spark with Gluten & Velox

Apache Spark is widely adopted for large‑scale data processing, but many of its performance optimizations have plateaued. Spark’s Whole‑Stage Code Generation introduced in Spark 2.0 delivered ~2× speed‑ups ( against Spark 1.x), yet subsequent improvements have focused mostly on query‑plan optimizations; the per‑operator execution speed has stagnated.

Meanwhile, several native SQL engines (ClickHouse, Apache DataFusion, Databricks’ Photon, Sail and Velox) implement columnar, vectorized execution in C++ and can outperform Spark’s JVM‑based engine. To make Spark scale with the speed of native engines, the open‑source Gluten project “glues” them together.

Why Gluten?

Gluten is a middle layer that offloads compute‑intensive Spark SQL operators to native engines while preserving Spark’s distributed control flow. Its design focuses on the following:

• Plan conversion: Spark’s whole‑stage physical plan is translated to a Substrait plan and passed to the native engine.
• Offload execution: Performance‑critical data processing is executed in native code. Gluten exposes clear JNI interfaces so Spark can call the native backend and get data back as Arrow columnar batches.
• Memory & shuffle: Gluten manages its own off‑heap memory and implements a columnar shuffle manager; it falls back to vanilla Spark for unsupported operators.
• Backend flexibility: The plugin can switch native backends (currently Velox and ClickHouse) without changing Spark applications.

The goal is to let users keep their DataFrame/SQL API unchanged and simply enable the plugin via configuration to accelerate Spark’s core computation.

What is Velox?

Velox is an open‑source C++ execution engine that provides reusable, high‑performance building blocks for data systems. Developed by Meta and maintained with partners like IBM/Ahana, Intel, Voltron Data, Microsoft and ByteDance, Velox takes an optimized query plan (from a SQL engine or dataframe API) and executes it. It does not include its own SQL parser, dataframe API or optimizer; instead it offers a library of components that compute engines can integrate ( remember, Composable Data Stack):

• Type and vector system: An Arrow‑compatible columnar memory layout with encodings like flat, dictionary and run‑length.
• Vectorized expression evaluation: A fully vectorized evaluator for scalar, aggregate and window functions.
• Operators & functions: Built‑in relational operators (scan, projection, filter, join, group by, order by, unnest, etc.).
• I/O and connectors: Interfaces for multiple file formats (Parquet, ORC, Nimble) and storage systems (S3, HDFS, GCS, ABFS, local files).
• Resource management: Memory arenas, buffer management, CPU thread pools, spilling and caching

By using Velox as a backend, Gluten can offload Spark’s physical operators to a highly optimized native engine while maintaining Spark’s fault tolerance and distributed scheduling.

Demo project: spark‑gluten‑velox

To illustrate how Gluten with a Velox backend accelerates Spark workloads, the repository Neuw84/spark-gluten-velox provides a Docker‑Compose deployment and a test notebook. The repo is a self‑contained environment built for S3‑backed benchmarks:

• Components: It assembles a Spark 3.5.2 cluster with Hadoop 3.4.1, Gluten 1.4.0 using the Velox backend, JupyterLab for interactive notebooks, and S3A integration. The images are based on Ubuntu 22.04.
• Cluster layout: A Spark master (port 8080), two worker nodes (4 cores & 4 GB RAM on-heap each), a Spark history server (port 18080) and a JupyterLab interface (port 8888)GitHub.
• Requirements: You need Docker and Docker Compose on x86‑64 hardware; an AWS S3 bucket for reading/writing data; and either an IAM role (recommended on EC2) or AWS credentials on your workstation configured.

Quick start

• Clone and build the environment: docker Compose builds images for the master, workers, history server and JupyterLab, and starts the cluster.

git clone https://github.com/Neuw84/spark-gluten-velox.git
cd spark-gluten-velox
docker-compose up - build

• Open JupyterLab. Navigate to http://localhost:8888 in your browser. Authentication is disabled for convenience; upload the test.ipynb notebook provided in the repo.
• Configure S3 access. On EC2 you can attach an IAM role that grants s3:GetObject, s3:PutObject, s3:DeleteObject and s3:ListBucket permissions on your bucket. For local development configure your AWS credentials via aws configure or environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) and update the s3_base value in the notebook.
• Run the benchmark. Upload test.ipynb to the JupyterLab and execute its cells. It will first run queries with Velox enabled and then with vanilla Spark for comparison. Remember to configure the S3 bucket.

What the notebook does

The test notebook synthesizes a 50 million‑row fact table and small dimension tables to simulate a retail analytics workload. It defines constants such as number of products (N_PROD), countries, channels and days, and uses PySpark’s random functions to generate skewed data. After caching the fact and dimension tables, it defines a simple timing function that writes each query result to S3 and counts the rows back to ensure full execution. The notebook runs three representative SQL workloads

• Query A — Heavy multi‑aggregation: Groups the fact table by country_id and channel_id and computes counts, sums, averages, standard deviation and approximate percentiles. This pattern stresses hash aggregation, projection and filter operators.
• Query B — Rollup (cube) aggregation: Joins the fact table with a date dimension and uses rollup(date_key, country_id, product_id) to compute revenue, quantity and average discount across multiple grouping levels.
• Query C — Star schema join and top‑K sort: Joins the fact table with broadcast dimensions (countries and channels) and the date dimension, computes gross and net revenue, and orders by gross descending, taking the top 5,000 rows. Broadcast joins and top‑K sorts test Velox’s vectorized join and sort operators.

After executing the queries, the notebook prints wall‑clock times and resets spark.gluten.enabled to false to run the same workload with the vanilla Spark engineGitHub.

Configuration highlights

To enable Gluten/Velox acceleration, the notebook constructs a SparkSession with several critical settings:

• Enable the Gluten plugin: spark.plugins = org.apache.gluten.GlutenPlugin.
• Use the columnar shuffle manager: spark.shuffle.manager = org.apache.spark.shuffle.sort.ColumnarShuffleManager.
• Turn on off‑heap memory and allocate 8 GB: spark.memory.offHeap.enabled = true and spark.memory.offHeap.size = 8g.
• Configure the Velox bundle JAR on the driver and executors: the Docker image already includes gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar and adds it to the classpath in the notebook.

For S3 integration, the compose file and notebook use the Hadoop 3.4.1 S3A connector with “magic” committers, improving write performance. You can view progress and metrics in the Spark UIs (master at :8080, history server at :18080, and driver UI at :4040).

Customizing and scaling

The demo is designed to be flexible:

• Scaling workers: Adjust SPARK_WORKER_CORES and SPARK_WORKER_MEMORY in the docker-compose.yml to give executors more cores and memory.
• Adding workers: Duplicate the worker service and assign different ports.
• Changing data size: Modify N_ROWS, REPART or other parameters in the notebook to simulate different workload sizes.
• Custom queries: Feel free to add your own workloads or business logic to the notebook; the notebook’s timing function is generic.
• Enhancing the compose setup: The current compose file keeps event logs local; you can improve it to store logs on S3 so the Spark history server works across runs and workers.

Benchmarks

Let’s do a concrete benchmark. We will run each set 3 times and do the average with different test sizes. We will run them on a c5.4xlarge instance. Remember as we are running all containers on a single instance we are not touching real network. On a real production scenario this could impact the results ( shuffling data around the network is not cheap). Test parameters:

Get Angel Conde’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Default settings:

N_ROWS = 50_000_000 
N_PROD = 50_000
N_CTRY = 32
N_CH = 6
N_DAYS = 90
REPART = 256

Medium settings:

N_ROWS = 80_000_000   
N_PROD = 70_000
N_CTRY = 32
N_CH   = 6
N_DAYS = 120
REPART = 256         

High settings:

N_ROWS = 110_000_000
N_PROD = 80_000
N_CTRY = 32
N_CH   = 6
N_DAYS = 120
REPART = 256

Then, let’s see the results for each query:

Query A:

Query B

Query C

Velox reduces total wall‑time by roughly 23 % compared to Vanilla Spark. Expressed as an average of the relative improvements for each query‑size combination, the overall performance gain is about 25 % faster for Velox versus Vanilla Spark on this specific test.

Summary

Gluten with a Velox backend brings modern vectorized execution to Apache Spark without requiring users to change their DataFrame/SQL code. By translating Spark’s physical plan to a Substrait plan and offloading to Velox, it combines Spark’s robust distributed scheduling with Velox’s native performance and memory efficiency. The spark-gluten-velox repository demonstrates how to set up a local Spark cluster with Gluten/Velox, configure S3 integration, and benchmark performance across realistic analytics workloads. Once you run the notebook and measure the wall‑clock times, you’ll have concrete numbers to quantify how much faster your queries can run on Spark when backed by Velox.

Remember to test for your specific workload!