Evolution — JSON to msgspec

How Python Data Handling Evolved

When Python applications exchange data, the journey usually starts with JSON.

It is simple, universal, human-readable, and works almost everywhere. But as applications grow, we often want more than just:

“Parse this JSON into a dictionary.”

We want:

• safer access to nested fields
• clearer data models
• type validation
• support for optional and evolving fields
• fast encoding and decoding
• compact binary formats where needed
• minimal boilerplate

Over time, different Python tools have solved different parts of this problem.

This article walks through that evolution:

Our Example Json

We will use the same small nested order throughout the article.

raw_json = """
{
    "order_id": 1001,
    "customer": {
        "name": "Alice",
        "email": "alice@example.com"
    },
    "items": [
        {
            "product": "Laptop",
            "quantity": 1,
            "price": 999.99
        },
        {
            "product": "Mouse",
            "quantity": 2,
            "price": 25.50
        }
    ]
}
"""

This example is simple enough to follow, but it still gives us:

• nested objects
• lists
• different data types
• optional fields
• possible schema changes

Stage 1: JSON — Simple and Universal

Python already gives us the built-in json module.

import json
order = json.loads(raw_json)
print(order)

This converts the JSON into normal Python dictionaries and lists.

To read values safely, we can use .get().

order_id = order.get("order_id")
customer = order.get("customer", {})
customer_name = customer.get("name")
print(order_id)
print(customer_name)

For nested lists:

items = order.get("items", [])
if items:
    first_item = items[0]
    print(first_item.get("product"))

This is safer than directly writing:

order["customer"]["name"]

because direct dictionary indexing can raise a KeyError if a field is missing.

With .get():

customer = order.get("customer", {})
email = customer.get("email")
print(email)

a missing value simply becomes None.

The First Problem: JSON Does Not Know Our Business Rules

Consider this input:

bad_json = """
{
    "order_id": "ABC",
    "customer": {
        "name": "Alice"
    },
    "items": []
}
"""

Load it:

order = json.loads(bad_json)
print(order.get("order_id"))

The output is:

ABC

But we intended order_id to be an integer. Neither JSON nor json.loads() knows that.

JSON gives us data. It does not give us a Python data model.

Stage 2: Dataclasses — Giving the Data Structure

Python dataclasses improve the situation by letting us describe the shape of our data.

from dataclasses import dataclass

@dataclass
class Customer:
    name: str
    email: str | None = None

@dataclass
class Item:
    product: str
    quantity: int
    price: float

@dataclass
class Order:
    order_id: int
    customer: Customer
    items: list[Item]

Now our model is much clearer.

Instead of:

customer.get("name")

we eventually want to write:

order.customer.name

This gives us:

• named fields
• type hints
• better editor support
• clearer application code

But there is still another problem. json.loads() still returns dictionaries. So we need to convert those dictionaries into dataclasses.

data = json.loads(raw_json)

customer_data = data.get("customer", {})

customer = Customer(
    name=customer_data.get("name", ""),
    email=customer_data.get("email"),
)

items = [
    Item(
        product=item.get("product", ""),
        quantity=item.get("quantity", 0),
        price=item.get("price", 0.0),
    )
    for item in data.get("items", [])
]

order = Order(
    order_id=data.get("order_id", 0),
    customer=customer,
    items=items,
)

It works. But now we have a lot of manual mapping code.

Dataclass Type Hints Are Not Runtime Validation

Dataclasses support type annotations. We can even add metadata using Annotated.

from dataclasses import dataclass
from typing import Annotated

@dataclass
class Order:
    order_id: Annotated[int, "Order identifier"]

That looks strongly typed. But try this:

order = Order(order_id="ABC")
print(order)
print(type(order.order_id))

Python happily creates:

Order(order_id='ABC')
<class 'str'>

This is an important distinction.

Dataclass annotations help:

• IDEs
• static type checkers
• developers understand the model

But normal dataclasses do not automatically enforce those types at runtime. So we now have:

CapabilityDataclassStructure ✅

Type hints. ✅

Readable access ✅

Runtime validation ❌

Automatic JSON conversion ❌

That leads naturally to the next stage.

Stage 3: Pydantic — Structure Plus Validation

Pydantic takes a model-oriented approach and adds runtime parsing and validation.

from pydantic import BaseModel

class Customer(BaseModel):
    name: str
    email: str | None = None

class Item(BaseModel):
    product: str
    quantity: int
    price: float

class Order(BaseModel):
    order_id: int
    customer: Customer
    items: list[Item]

Now we can parse our JSON directly:

order = Order.model_validate_json(raw_json)
print(order.customer.name)

This is a major improvement.

Pydantic Catches Invalid Data

Try the earlier invalid value:

bad_json = """
{
    "order_id": "ABC",
    "customer": {
        "name": "Alice"
    },
    "items": []
}
"""

Then:

order = Order.model_validate_json(bad_json)

Now we get a validation error instead of silently allowing "ABC" through the application.

Without validation:

With validation:

Failing at the application boundary is usually much easier to debug than failing somewhere deep inside business logic.

Pydantic Can Also Coerce Values

Suppose an API sends:

{
    "order_id": "1001"
}

Pydantic can often convert compatible input into the required type.

raw_json = """
{
    "order_id": "1001",
    "customer": {
        "name": "Alice"
    },
    "items": []
}
"""
order = Order.model_validate_json(raw_json)
print(order.order_id)
print(type(order.order_id))

The resulting value can be an integer. This is useful when we want validation and normalisation together. Strict modes are also available when coercion is not desirable.

What Happens When JSON Evolves?

Real APIs rarely remain unchanged forever.

Suppose the producer adds:

{
    "order_id": 1001,
    "priority": "HIGH"
}

Our existing consumer may not know about priority.

This is schema evolution.

There are several ways a consumer can react.

We can choose to:

• ignore unknown fields
• preserve them
• reject them

Pydantic lets us configure this behaviour.

For example:

from pydantic import BaseModel, ConfigDict

class Order(BaseModel):
    order_id: int
    customer: Customer
    items: list[Item]

model_config = ConfigDict(extra="forbid")

Now unexpected fields become validation errors. The important point is that schema evolution is a design choice.

Stage 4: orjson — Focusing on JSON Performance

So far we have mainly focused on modelling and correctness. Another requirement appears when applications process large volumes of messages:

Serialization speed.

Python’s standard json module is convenient, but faster alternatives exist.

One popular option is orjson.

import orjson
data = orjson.loads(raw_json)
encoded = orjson.dumps(data)
print(encoded)

orjson.dumps() returns bytes.

print(type(encoded))

Output:

<class 'bytes'>

If we decode JSON:

data = orjson.loads(raw_json)
print(type(data))

we still normally get:

<class 'dict'>

This distinction matters. orjson focuses primarily on fast JSON serialization and deserialization. It is not primarily a typed model-validation system. At this point we are dealing with two different concerns.

Stage 5: MessagePack — What If We Do Not Need Text?

JSON is human-readable. That is a major advantage. But human-readable text is not always the most compact representation for service-to-service communication.

Get SP Barrow’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Now enters msgpack (Python Wrapper for MessagePack). MessagePack is a binary serialization format.

import msgpack
data = {
    "order_id": 1001,
    "customer": {
        "name": "Alice"
    }
}
packed = msgpack.packb(data)
print(packed)

Decode it:

unpacked = msgpack.unpackb(packed)
print(unpacked)

And because the result is dictionary-based:

customer = unpacked.get("customer", {})
print(customer.get("name"))

MessagePack solves a different problem. It gives us a compact binary representation. But by itself, it does not give us our typed application model.

At This Point, We Have Several Pieces

By now, the ecosystem looks something like this:

Different libraries have solved different concerns:

JSON gives us interoperability

dataclasses give us structure

Pydantic gives us validation

orjson gives us faster JSON

MessagePack gives us compact binary serialization

But what if we want all of these concerns closer together? We want:

• typed Python structures
• validation
• nested models
• JSON encoding and decoding
• MessagePack encoding and decoding
• speed
• minimal boilerplate

This is where msgspec becomes interesting.

Stage 6: msgspec

With msgspec, we define our structures like this:

import msgspec

class Customer(msgspec.Struct):
    name: str
    email: str | None = None

class Item(msgspec.Struct):
    product: str
    quantity: int
    price: float

class Order(msgspec.Struct):
    order_id: int
    customer: Customer
    items: list[Item]

The model is compact and easy to read.

Decode JSON Directly Into Typed Objects

Create a typed decoder:

decoder = msgspec.json.Decoder(Order)

Then decode:

order = decoder.decode(raw_json)
print(order)

Access the nested value normally:

print(order.customer.name)

Instead of thinking in terms of several separate stages:

we can move toward:

That is one of the key ideas behind msgspec.

Nested Validation

Now deliberately break one of the nested fields.

bad_json = """
{
    "order_id": 1001,
    "customer": {
        "name": "Alice"
    },
    "items": [
        {
            "product": "Laptop",
            "quantity": "ONE",
            "price": 999.99
        }
    ]
}
"""

Then:

order = decoder.decode(bad_json)

The decoder understands the nested model.

It knows that quantity belongs inside an Item, which belongs inside items, which belongs inside an Order.

That is much safer than relying on every part of the application to remember what type each nested value should be.

Optional Fields Remain Simple

Our customer model says:

class Customer(msgspec.Struct):
    name: str
    email: str | None = None

That means email may be absent. For example:

{
    "name": "Alice"
}

The resulting model can still be used normally.

print(order.customer.email)

The result becomes:

None

The model itself documents the contract.

Handling Evolving Messages

Suppose the producer later sends:

{
    "name": "Alice",
    "email": "alice@example.com",
    "loyalty_level": "GOLD"
}

but our consumer currently understands only:

class Customer(msgspec.Struct):
    name: str
    email: str | None = None

This becomes a forwards-compatibility question.

In distributed systems, it can be useful for older consumers to continue working even when producers add new fields. By default, msgspec.Struct ignores unknown fields when decoding.

This means a producer can add fields without immediately breaking older consumers.

When you want strict schemas

Sometimes ignoring new fields is not desirable. You may want unexpected fields to fail validation instead. With msgspec, you can configure the struct with forbid_unknown_fields=True:

class Customer(
    msgspec.Struct,
    forbid_unknown_fields=True
):
    name: str
    email: str | None = None

Now:

decoder = msgspec.json.Decoder(Customer)
decoder.decode(
    b'{"name":"Alice","loyalty_level":"GOLD"}'
)

raises a decoding validation error because loyalty_level is not part of the schema.

Encoding Back to JSON

Encoding is equally straightforward.

encoded = msgspec.json.encode(order)
print(encoded)

To display it as text:

print(encoded.decode())

Our round trip becomes:

The same model can drive both directions.

Switching to MessagePack

This is one of the strongest parts of the msgspec API. The model does not change. For JSON:

json_data = msgspec.json.encode(order)

For MessagePack:

msgpack_data = msgspec.msgpack.encode(order)

Typed decoding works similarly:

decoder = msgspec.msgpack.Decoder(Order)
decoded = decoder.decode(msgpack_data)
print(decoded)

The important point is:

Our Python model does not have to change because the transport format changed.

Comparing Payload Size

Because MessagePack is binary, it can often produce a smaller representation than JSON. We can measure this directly.

json_bytes = msgspec.json.encode(order)
msgpack_bytes = msgspec.msgpack.encode(order)
print("JSON:", len(json_bytes), "bytes")
print("MessagePack:", len(msgpack_bytes), "bytes")

The exact difference depends on the payload, so it is better to measure your own data than assume a fixed percentage improvement.

I’ll cover performance and memory in a follow-up article, comparing encoding/decoding speed, payload size, object creation overhead, and peak memory usage across these approaches.

Where msgspec Fits

At this point, our requirement tree becomes much simpler.

msgspec combines:

typed structures

runtime validation

JSON

MessagePack

fast encoding

fast decoding

This makes it especially interesting for message-heavy systems.

Does msgspec Replace Pydantic?

Not necessarily. Pydantic has a broad ecosystem and is particularly useful in application and API development. It provides capabilities such as:

extensive validation behaviour

configurable coercion

custom validators

JSON Schema generation

mature framework integration

strong FastAPI support

msgspec, on the other hand, becomes especially compelling when the hot path looks like this:

Usecases include:

• event-processing systems
• APIs with high serialization volume
• queues
• streaming applications
• caches
• internal microservice communication
• data pipelines

When encoding and decoding happen millions of times, serialization overhead matters.

The Evolution in One Picture

The full story can be summarised like this:

The important idea is not that one tool made every previous tool obsolete.

Instead, different requirements gradually converged:

• JSON solved portability
• dataclasses solved structure
• Pydantic solved validation
• orjson solved JSON performance
• MessagePack solved compact binary transport
• msgspec brings several of these concerns together

Verdict

I like to remember the evolution in six lines:

JSON gave us interoperability.
Dataclasses gave our dictionaries structure.
Pydantic added runtime validation and convenient parsing.
orjson focused heavily on fast JSON serialization.
MessagePack gave us a compact binary representation.
msgspec combines typed structures, validation and high-performance
serialization behind a small API.

That is the real reason msgspec is interesting. Instead of a longer pipeline:

we can often move toward:

Fewer moving parts. A smaller API. And for workloads where serialization performance matters, a useful balance between speed, type safety, and developer simplicity.