Spark series : Why col("status") == None Returns No Rows in PySpark

Photo by Chad Madden on Unsplash

If you work with PySpark long enough, you’ll eventually hit this bug:

#python
from pyspark.sql.functions import col
df.filter(col("status") == None)

You know the status column has nulls, but this filter returns zero rows. It feels wrong, but it’s actually how Spark is designed.

In this post, we’ll see why this happens and where it can silently break your logic in real projects.

The core reason: SQL null semantics, not Python semantics

Spark DataFrames use SQL semantics for expressions, not Python’s.

In SQL (and Spark SQL):

• NULL means “unknown / missing value”, not just “nothing”.
• Comparisons with NULL using =, !=, <, > do not return TRUE or FALSE.
• They return a third logical value: UNKNOWN.

Example in SQL world:

• NULL = NULL → UNKNOWN
• price = NULL → UNKNOWN
• price != NULL → UNKNOWN

And in a WHERE clause:

• Rows are kept only if the condition is TRUE.
• Rows where the condition is FALSE or UNKNOWN are dropped.

So in Spark, when you write:

#python
df.filter(col("status") == None)

for a row with status = null, Spark effectively evaluates:

#sql
status = NULL  -- result: UNKNOWN, not TRUE

Because the result is not TRUE, the row is discarded. That’s why you get zero rows back.

The correct way: use isNull / isNotNull

To work with nulls in Spark, you must use the dedicated null predicates.

#python
from pyspark.sql.functions import col
# rows where status IS NULL
df_nulls = df.filter(col("status").isNull())
# rows where status IS NOT NULL
df_not_nulls = df.filter(col("status").isNotNull())

Or with SQL strings:

#python
df_nulls = df.filter("status IS NULL")
df_not_nulls = df.filter("status IS NOT NULL")

Under the hood, Spark translates these to proper IS NULL / IS NOT NULL SQL checks, which do understand how to match null values.

A realistic example where this bug hurts you

Imagine you’re cleaning event data and want to drop rows where the status is missing. A common (but wrong) pattern is:

#python
clean = df.filter(col("status") != None)

You think you’re “removing null statuses”, but because status != None is also UNKNOWN for nulls, those rows are not kept — which sounds correct at first.

Get Amit Dass’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The real danger appears when you do the opposite: trying to select only rows where status is null.

#python
# ❌ BUGGY: returns zero rows
missing_status = df.filter(col("status") == None)

In many pipelines, that branch feeds:

• a “data quality” dashboard (you report 0 missing values, which is wrong), or
• special handling logic (e.g., default routing for missing status).

Because you get an empty DataFrame, you might conclude “we have no missing statuses”, while in reality they’re there — they just failed the == None comparison.

The correct version is:

#python
# ✅ correct null filter
missing_status = df.filter(col("status").isNull())

Now you’ll see the rows you were missing before.

Bonus: null or empty string in one condition

A very common real‑world requirement is “status is null or empty string”. In Spark, do it like this:

#python
from pyspark.sql.functions import col
df_bad_status = df.filter(
    col("status").isNull() | (col("status") == "")
)

Or in SQL syntax:

#python
df_bad_status = df.filter("status IS NULL OR status = ''")

This covers both genuinely missing values and “present but blank” values.

Key takeaway

When you’re writing Spark code, remember:

• Spark expressions are SQL-like, not pure Python.
• Never rely on == None or != None in DataFrame filters.
• Always use .isNull(), .isNotNull() (or IS NULL, IS NOT NULL in SQL) for null handling.

This small habit will save you from quietly wrong filters and misleading “no nulls” conclusions in production pipelines.

Want to do automatic ?

Automatic (just configure) — Databricks provide these data quality features

• ✅ Use expectations to explicitly track status IS NOT NULL as a data quality metric for your pipelines.
• ✅ Use anomaly detection to automatically flag unusual changes in null rate, row counts, or other table‑level metrics over time.