Python + DuckDB UDFs with PyArrow: Columnar Speedups for Feature Engineering

How to keep your features in SQL, your logic in Python, and still be fast.

Use Python UDFs and PyArrow inside DuckDB to build fast, columnar feature engineering pipelines without shipping data back and forth to Pandas.

You load a Parquet file into DuckDB.

You write a neat SQL query.

Then you do the “one little thing” data people always do:

“I’ll just add a tiny Python UDF for that custom feature… how bad could it be?”

Suddenly your sub-second query is taking 20 seconds and one CPU core is pegged at 100%. The DuckDB magic is gone, and you’re back in single-threaded Python land.

The trick is not to swear off Python UDFs. It’s to use them the way DuckDB’s engine wants to work: in columnar chunks, ideally with PyArrow doing the heavy lifting.

Let’s walk through what that looks like in practice.

Where naive Python UDFs go wrong

DuckDB is a columnar engine. It:

• Scans Parquet/CSV in vectorized batches.
• Operates on columns, not rows.