External Data Representation& Marshalling

A technical comparison of CORBA CDR, Java Serialisation, XML, Protocol Buffers, MessagePack, and Apache Avro — with real-world code.

When two processes on different machines need to exchange data, they face a deceptively hard problem: every machine has its own memory layout, byte order, type system, and object model. External Data Representation (EDR) is the discipline of defining a common wire format so that a message produced by one system can be faithfully reconstructed by another — regardless of language, OS, or hardware architecture.

Marshalling is the act of serialising an in-memory object graph into that wire format. Unmarshalling is the reverse. The choice of format has profound consequences for performance, interoperability, versioning, and debugging — and different distributed contexts demand different trade-offs.

In this article we compare six formats across three generations of distributed computing: CORBA CDR (object-oriented middleware), Java Object Serialisation (JVM ecosystems), XML/SOAP (web services), and three modern binary formats — Protocol Buffers, MessagePack, and Apache Avro.

1 · CORBA CDR — Common Data Representation

CORBA · 1991 Common Data Representation (CDR)

CORBA CDR is the wire encoding defined by the Object Management Group (OMG) as part of the General Inter-ORB Protocol (GIOP). It underpins communication between distributed objects in a CORBA system, where a client holds a remote object reference and invokes methods as if the object were local.

CDR solves the heterogeneity problem at the binary level. Data types — integers, floats, strings, enums, structs, sequences — are encoded in a tag-length-value style with explicit byte-order negotiation. The receiver inspects a byte-order flag in the message header and decodes accordingly, avoiding unnecessary byte-swapping on matching architectures.

The Interface Definition Language (IDL) describes the contract between client and server. IDL compilers generate stub and skeleton code in any supported language (C++, Java, Python), automating marshalling and unmarshalling entirely.

Real-World Scenario — Telecom Billing System

A telecom operator’s billing engine (C++) invokes a rating service (Java) via CORBA to calculate call charges. CDR encodes the call record on the wire; both ORBs negotiate little-endian byte order at session start. The IDL contract ensures both sides agree on the CallRecord structure without sharing source code.

// billing.idl — shared contract between C++ client and Java server
module Billing {

  struct CallRecord {
    string   caller_id;
    string   callee_id;
    long     duration_secs;
    double   rate_per_min;
  };

  interface RatingService {
    // Returns charge in cents — marshalled as CDR double on the wire
    double calculateCharge(in CallRecord record);
  };
};

// C++ billing engine — stub generated by IDL compiler
Billing::CallRecord record;
record.caller_id     = "94771234567";
record.callee_id     = "94712345678";
record.duration_secs = 180;
record.rate_per_min  = 1.50;

// CDR marshals 'record' automatically before sending over IIOP
CORBA::Double charge = rating_service->calculateCharge(record);
std::cout << "Charge: " << charge << " cents\n";

Strengths

• Compact binary encoding — minimal wire overhead
• Language-neutral via IDL compilation
• Explicit byte-order negotiation handles endianness
• Strong type safety enforced at the IDL level

Weaknesses

• Complex toolchain — IDL compiler required
• Rigid schema — interface changes break compatibility
• Not human-readable — hard to debug without tooling
• CORBA ecosystem is largely obsolete in modern stacks

2 · Java Object Serialisation

Java · 1997 Java Object Serialisation

Java Object Serialisation flattens an entire object graph — including nested references and cyclic dependencies — into a byte stream. Any class implementing java.io.Serializable participates automatically. The JVM walks the object graph at runtime using reflection, encoding class metadata, field names, types, and values.

The serialVersionUID is a 64-bit fingerprint of the class structure. If sender and receiver have different versions of a class, a mismatch in serialVersionUID throws an InvalidClassException, preventing silent data corruption. Developers can declare it explicitly to allow controlled evolution of serialised classes.

Real-World Scenario — E-Commerce Shopping Cart (JMS Queue)

An order service publishes a ShoppingCart object to an ActiveMQ JMS queue. The fulfilment service on another node consumes it. Both services are Java — serialisation handles the nested List<CartItem> graph automatically, including the BigDecimal price fields.

import java.io.Serializable;
import java.math.BigDecimal;
import java.util.List;

public class ShoppingCart implements Serializable {

    // Explicit UID — changing this intentionally breaks old payloads
    private static final long serialVersionUID = 2024_06_01L;

    private String          customerId;
    private List<CartItem>  items;
    private BigDecimal      totalAmount;

    public static class CartItem implements Serializable {
        private static final long serialVersionUID = 1L;
        private String     sku;
        private int        quantity;
        private BigDecimal unitPrice;
        // getters / setters omitted for brevity
    }
}

import java.io.*;

// ── Serialise (Order Service) ──────────────────────────────
ShoppingCart cart = buildCart("CUST-9912");

ByteArrayOutputStream baos = new ByteArrayOutputStream();
try (ObjectOutputStream oos = new ObjectOutputStream(baos)) {
    oos.writeObject(cart);          // full object graph → bytes
}
byte[] payload = baos.toByteArray();
jmsProducer.send(session.createBytesMessage(payload));

// ── Deserialise (Fulfilment Service) ──────────────────────
byte[] received = jmsConsumer.receive();
ObjectInputStream ois = new ObjectInputStream(
    new ByteArrayInputStream(received)
);
ShoppingCart restored = (ShoppingCart) ois.readObject();

// SECURITY: Always use a filter to whitelist expected classes
// ois.setObjectInputFilter(ObjectInputFilter.Config.createFilter(
//     "com.shop.*;java.math.BigDecimal;java.util.ArrayList;!*"));

Strengths

• Zero boilerplate — just implement Serializable
• Handles complex object graphs and cyclic references natively
• Deep JVM integration — works with RMI, JMS, HTTP sessions
• serialVersionUID enables controlled versioning

Weaknesses

• JVM-only — no cross-language interoperability
• Verbose format — produces large payloads
• Critical security vulnerability — gadget chain deserialization attacks
• Slow due to reflection-based encoding

3 · XML — Self-Describing Textual Format

W3C · 1998 XML & SOAP Web Services

XML represents data as a tree of tagged elements and attributes. Its self-describing nature — every field carries its own name — makes it human-readable and tooling-friendly. SOAP (Simple Object Access Protocol) uses XML as its envelope format for web service calls, wrapping a structured payload with a header for metadata (authentication, transaction ID) and a body for the actual message.

WSDL (Web Services Description Language) plays the same contract role as IDL in CORBA — it describes available operations, message shapes, and bindings. WS-Security, WS-ReliableMessaging, and WS-AtomicTransaction extend SOAP with enterprise-grade features unmatched by any binary format.

Real-World Scenario — Bank Payment Gateway (SOAP)

A retail POS system integrates with a bank’s legacy payment gateway via SOAP over HTTPS. The XML envelope carries the transaction, a WS-Security header with a signed token, and the response carries an authorisation code. The verbosity is acceptable — transactions are infrequent and the audit trail benefits from human-readable logs.

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope
  xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
  xmlns:pay="http://bank.example.com/payment/v2">

  <soapenv:Header>
    <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/...">
      <wsse:UsernameToken>
        <wsse:Username>pos-terminal-042</wsse:Username>
        <wsse:Password>hashed-token-value</wsse:Password>
      </wsse:UsernameToken>
    </wsse:Security>
  </soapenv:Header>

  <soapenv:Body>
    <pay:ProcessPayment>
      <pay:cardNumber>4111111111111111</pay:cardNumber>
      <pay:amount>4750.00</pay:amount>
      <pay:currency>LKR</pay:currency>
      <pay:merchantId>MID-20045</pay:merchantId>
    </pay:ProcessPayment>
  </soapenv:Body>

</soapenv:Envelope>

Strengths

• Human-readable — trivial to inspect and debug
• Language and platform agnostic
• Rich ecosystem: XSD, XSLT, XPath, WS-* standards
• Self-describing — no separate schema needed to read

Weaknesses

• Verbose — tag overhead inflates payload 3–10× compared to binary
• Slow parsing — text-based parsing is CPU-intensive
• SOAP complexity — WS-* stack has a steep learning curve
• Poor fit for high-throughput, low-latency systems

4 · Protocol Buffers — Google’s Binary IDL

Google · 2008 Protocol Buffers (protobuf)

Get Jashvanth Balakirushnan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Protocol Buffers encode data as compact binary using field numbers rather than field names. A .proto schema defines message types; the protoc compiler generates typed, efficient code in Go, Python, Java, C++, Rust, and more. Forward and backward compatibility is built-in: fields are identified by number, unknown fields are preserved, and optional fields may be absent without breaking decoders.

protobuf is the serialisation layer behind gRPC — the RPC framework used at Google, Netflix, and most cloud-native microservice architectures. The binary encoding is typically 3–10× smaller than equivalent JSON or XML, and decoding is significantly faster.

Real-World Scenario — Ride-Sharing Location Updates (gRPC)

A ride-sharing app streams driver GPS coordinates to a dispatch server every 500ms across thousands of active drivers. Protocol Buffers encode each update in ~20 bytes vs ~120 bytes for JSON — a 6× reduction. Over a million daily active drivers, this saves terabytes of bandwidth and halves mobile battery drain from network I/O.

// location.proto
syntax = "proto3";
package dispatch;

message DriverLocation {
  string  driver_id  = 1;   // field number 1 — stable across versions
  double  latitude   = 2;
  double  longitude  = 3;
  int64   timestamp  = 4;   // Unix ms
  float   speed_kmh  = 5;
  // Add field 6 later — old decoders will simply ignore it
}

service DispatchService {
  rpc StreamLocation (stream DriverLocation) returns (Ack);
}

from location_pb2 import DriverLocation
import time

# ── Encode (driver mobile app) ────────────────────────────
update = DriverLocation(
    driver_id  = "DRV-00451",
    latitude   = 6.9271,    # Colombo, Sri Lanka
    longitude  = 79.8612,
    timestamp  = int(time.time() * 1000),
    speed_kmh  = 32.5,
)

wire_bytes = update.SerializeToString()
print(f"Payload size: {len(wire_bytes)} bytes")  # ≈ 40 bytes

# ── Decode (dispatch server) ──────────────────────────────
received = DriverLocation()
received.ParseFromString(wire_bytes)
print(f"Driver {received.driver_id} at ({received.latitude}, {received.longitude})")

Strengths

• Extremely compact — field numbers, not names, on the wire
• Strong schema evolution — add or deprecate fields safely
• Multi-language code generation — typed, no reflection
• Native gRPC integration with streaming RPC support

Weaknesses

• Not human-readable — requires tooling to inspect payloads
• No self-description — schema required to decode
• The .proto compile step adds toolchain complexity

5 · MessagePack — Binary JSON

Open · 2008 MessagePack

MessagePack encodes JSON-like structures (maps, arrays, strings, numbers, booleans, null) in a compact binary format. It requires no schema or code generation — you pass in a dictionary or object and receive back bytes. This makes it the pragmatic choice when you want significantly better performance than JSON but cannot afford the toolchain overhead of protobuf or Avro.

MessagePack is schema-less: field names are encoded on the wire (like JSON), so the receiver does not need an out-of-band schema to decode. This adds some overhead compared to protobuf but enables dynamic use cases where message shapes are not known at compile time.

Real-World Scenario — IoT Sensor Network (MQTT)

A smart factory deploys 10,000 vibration sensors publishing readings every 100ms to an MQTT broker. JSON payloads saturate the network; switching to MessagePack halves bandwidth without requiring firmware teams to manage .proto schemas. The Python analytics backend deserialises readings with a single msgpack.unpackb() call.

import msgpack, time, paho.mqtt.client as mqtt

# ── Sensor node — publish encoded reading ─────────────────
reading = {
    "sensor_id": "VIB-Factory2-Line7-042",
    "ts":        int(time.time() * 1000),
    "axis_x":   0.032,
    "axis_y":   -0.017,
    "axis_z":   1.001,
    "temp_c":   68.4,
}

payload = msgpack.packb(reading, use_bin_type=True)
# JSON equivalent: ~130 bytes  |  MessagePack: ~65 bytes

client.publish("factory/line7/vibration", payload)

# ── Analytics backend — decode ────────────────────────────
def on_message(client, userdata, msg):
    data = msgpack.unpackb(msg.payload, raw=False)
    if data["axis_x"] > 0.5:   # anomaly threshold
        trigger_alert(data["sensor_id"])

Strengths

• No schema or codegen — drop-in JSON replacement
• Approximately 50% smaller than JSON, 2–3× faster to parse
• Self-describing — receiver needs no external schema
• Wide language support — 50+ implementations

Weaknesses

• Larger than protobuf — field names still on the wire
• No formal schema — no compile-time type safety
• No built-in versioning or evolution strategy
• Binary — not human-readable without tooling

6 · Apache Avro — Schema-Driven for Big Data

Apache · 2009 Apache Avro

Avro is a row-oriented binary serialisation format designed for Hadoop and streaming data pipelines. Its key design insight is schema separation: the wire payload contains no field names or type tags — just raw values in schema-defined order. The reader resolves fields by matching the writer’s schema against its own reader schema. This allows full schema evolution: fields can be added with defaults, removed, or renamed, without breaking existing producers or consumers.

Avro schemas are defined in JSON and stored in a Schema Registry (e.g. Confluent Schema Registry). Kafka producers embed a schema ID in each message; consumers fetch the schema by ID and deserialise accordingly. This combination enables safe, decentralised schema evolution across hundreds of microservices.

Real-World Scenario — Event Streaming Pipeline (Kafka)

A fintech platform streams TransactionEvent records from 15 microservices into Kafka. The risk analytics team adds a new field merchant_category to the schema. Because Avro supports additive evolution with defaults, old producers keep writing; new consumers read the new field; old consumers default it to UNKNOWN — zero coordination required.

{
  "type":      "record",
  "name":      "TransactionEvent",
  "namespace": "com.fintech.events",
  "fields": [
    { "name": "txn_id",     "type": "string" },
    { "name": "account_id", "type": "string" },
    { "name": "amount_usd", "type": "double" },
    { "name": "timestamp",  "type": "long"   },
    {
      "name":    "merchant_category",
      "type":    ["null", "string"],  // union — nullable new field
      "default": null                 // old readers safely default to null
    }
  ]
}

from confluent_kafka.avro import AvroProducer, AvroConsumer
from confluent_kafka import avro

schema = avro.loads(open("transaction_event.avsc").read())

# ── Producer ──────────────────────────────────────────────
producer = AvroProducer({
    "bootstrap.servers":   "kafka:9092",
    "schema.registry.url": "http://schema-registry:8081",
}, default_value_schema=schema)

producer.produce(
    topic="transactions",
    value={
        "txn_id":             "TXN-8812934",
        "account_id":         "ACC-00451",
        "amount_usd":         249.99,
        "timestamp":          1717200000000,
        "merchant_category":  "ELECTRONICS",   # new field
    }
)

# ── Consumer ──────────────────────────────────────────────
consumer = AvroConsumer({
    "bootstrap.servers":   "kafka:9092",
    "schema.registry.url": "http://schema-registry:8081",
    "group.id":            "risk-analytics",
})
consumer.subscribe(["transactions"])

msg = consumer.poll(10)
txn = msg.value()              # Avro auto-deserialised dict
print(txn["merchant_category"]) # "ELECTRONICS"

Strengths

• Smallest payload — no field names or type tags on the wire
• Native schema evolution with default values
• First-class Kafka and Hadoop ecosystem integration
• Schema Registry enables independent team evolution

Weaknesses

• Schema Registry required — adds operational overhead
• Harder to debug — payload is meaningless without the schema
• Row-oriented — poor fit for analytical column scans
• Less tooling support outside the JVM ecosystem

Side-by-Side Comparison

There is no universally best serialisation format — only the right tool for the context. Choose based on who the consumers are, how fast the schema will change, and how much bandwidth you can afford.

Key Takeaways

01 Marshalling is a design decision, not an afterthought. The format you choose determines payload size, decoding speed, schema evolution strategy, and which languages can participate in your system.

02 CORBA CDR was the first serious attempt at language-neutral binary encoding. Its IDL-driven approach directly inspired protobuf and Thrift — understanding it makes modern formats easier to grasp.

03 Java Serialisation is convenient but dangerous. It handles complex object graphs automatically, but its reflection-based deserialization is a well-documented attack surface. Use ObjectInputFilter in any production JMS or RMI system.

04 XML/SOAP is verbose but not obsolete. Its rich WS-* ecosystem (security, transactions, reliable messaging) is unmatched. For regulated industries — banking, healthcare, government — it remains the dominant choice.

05 Protocol Buffers win on performance and type safety. For cloud-native microservices communicating over gRPC, protobuf is the default choice — compact, fast, and strongly typed across languages.

06 MessagePack is the pragmatic middle ground. When you need better performance than JSON but cannot afford schema management overhead, MessagePack is a frictionless upgrade with no codegen step.

07 Apache Avro was designed for schema evolution at scale. Its Schema Registry integration makes it the natural choice for Kafka-based event streaming platforms where schemas evolve independently across many teams.

Conclusion

External data representation is one of the most consequential low-level decisions in distributed systems design. The formats covered here span three decades of evolution — from CORBA’s IDL-compiled binary encoding to Avro’s schema-registry-driven streaming model — and each reflects the constraints and priorities of its era.

Modern distributed systems increasingly layer these formats by context: XML/SOAP for legacy enterprise integration, protobuf for internal microservice communication, Avro for event streaming pipelines, and MessagePack for low-overhead IoT or gaming payloads. Understanding each format’s trade-offs is what separates an engineer who can implement from one who can architect.

#DistributedSystems#Serialisation#ProtocolBuffers#ApacheAvro#CORBA#Marshalling#Kafka#gRPC