Processing JSON Log Files with PyArrow Variant Types

Modern systems generate massive volumes of log data, often in JSON format. While JSON is flexible, that flexibility becomes a challenge when schemas evolve or fields contain mixed types. Traditional data processing tools struggle with this variability.

This is where PyArrow’s variant (union-like) types and columnar processing shine.

Get Vaibhav Khanduja’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

In this blog, we’ll walk through:

• The problem with semi-structured JSON logs
• How PyArrow handles variant data
• A practical pipeline for processing logs efficiently

The Challenge: JSON Logs in the Wild

Consider a typical application log:

{"timestamp": "2026-04-20T10:00:00Z", "user_id": 123, "event": "login", "metadata": {"ip": "1.2.3.4"}}
{"timestamp": "2026-04-20T10:01:00Z", "user_id": "anonymous", "event": "view", "metadata": {"page": "home"}}
{"timestamp": "2026-04-20T10:02:00Z", "event": "error", "metadata": "timeout"}

Problems:

• user_id can be int or string
• metadata can be object or string
• Fields may be missing entirely

Why PyArrow?

PyArrow provides:

• Columnar memory format (Apache Arrow)
• Efficient JSON ingestion
• Support for nested and union-like data
• Zero-copy interoperability with analytics engines

Most importantly, it allows us to model inconsistent fields explicitly instead of flattening or losing information.

Step 1: Reading JSON Logs with PyArrow

import pyarrow as pa
import pyarrow.json as paj

table = paj.read_json("logs.json")
print(table.schema)

PyArrow automatically infers schema, but mixed types may default to large unions or fallback types.

Step 2: Handling Variant Fields

PyArrow doesn’t always infer clean union types automatically. For production pipelines, it’s better to define them.

Example: Defining a Variant (Union) Type

user_id_type = pa.union([
    pa.field("int", pa.int64()),
    pa.field("str", pa.string())
], mode="sparse")

For metadata:

metadata_type = pa.union([
    pa.field("json", pa.struct([
        pa.field("ip", pa.string()),
        pa.field("page", pa.string())
    ])),
    pa.field("text", pa.string())
], mode="sparse")

Step 3: Normalizing Incoming Data

Since raw JSON won’t automatically map into unions cleanly, we normalize before converting:

def normalize_record(record):
    # Normalize user_id
    if isinstance(record.get("user_id"), int):
        user_id = ("int", record["user_id"])
    else:
        user_id = ("str", str(record.get("user_id")))

    # Normalize metadata
    meta = record.get("metadata")
    if isinstance(meta, dict):
        metadata = ("json", meta)
    else:
        metadata = ("text", str(meta))
    return {
        "timestamp": record.get("timestamp"),
        "event": record.get("event"),
        "user_id": user_id,
        "metadata": metadata
    }

Step 4: Building Arrow Arrays

records = [normalize_record(r) for r in raw_json_records]

timestamps = pa.array([r["timestamp"] for r in records], pa.string())
events = pa.array([r["event"] for r in records], pa.string())
# Construct union arrays manually
user_ids = pa.UnionArray.from_sparse(
    type_ids=[0 if r["user_id"][0] == "int" else 1 for r in records],
    children=[
        pa.array([r["user_id"][1] if r["user_id"][0] == "int" else None for r in records], pa.int64()),
        pa.array([r["user_id"][1] if r["user_id"][0] == "str" else None for r in records], pa.string())
    ]
)

This explicitly preserves type information.

Step 5: Querying the Data Efficiently

Once in Arrow format, you can:

• Filter
• Aggregate
• Convert to Parquet

Example:

import pyarrow.compute as pc

# Filter only login events
filtered = table.filter(pc.equal(table["event"], "login"))

Step 6: Writing to Parquet

import pyarrow.parquet as pq

pq.write_table(table, "logs.parquet")

Parquet preserves nested structures efficiently and is compatible with tools like Spark, DuckDB, and Snowflake.

Performance Benefits

Why this approach works well:

• Columnar layout → faster scans
• Explicit typing → no ambiguity at runtime
• Zero-copy reads → integrates with analytics engines
• Compression → smaller storage footprint

Compared to pandas:

• Lower memory usage
• Faster filtering and aggregation
• Better handling of nested data

Real-World Use Cases

This pattern is especially useful for:

• Observability pipelines (logs, traces, metrics)
• Security analytics (variable payloads)
• Event-driven architectures
• Data lakes with evolving schemas

Key Takeaways

• JSON logs are inherently messy; embrace schema variability
• PyArrow enables strong typing for semi-structured data
• Variant (union) types preserve information without flattening
• Converting early to Arrow/Parquet unlocks performance

Final Thoughts

Instead of fighting JSON’s flexibility, PyArrow lets you model it properly. Using union/variant types, you can build pipelines that are both robust and high-performance — a necessity for modern data systems.