Network Protocol Pentesting — What & How

*A practical field guide for testing proprietary and embedded network protocols*

Introduction

I have been working on protocol pentesting for more than 2 years now. And believe me, protocol pentesting is a different beast compared to typical web or API security testing. There’s often no dissector, no public documentation, and no clean REST interface like to poke at — just raw bytes flowing between a server and a device(client). This post walks through a structured methodology for approaching protocol-level penetration testing, from building foundational knowledge to fuzzing & exploiting real bugs.

Bird’s Eye View

1. Learning the ropes: building the knowledge base you need
2. Recon: gathering everything available about the target protocol
3. Understanding the behavior: how the protocol actually works in practice
4. Reverse engineering: mapping the protocol and building a spec (reference to scapy snippets/blogs)
5. The actual test: AuthN/AuthZ, opcode/flow abuse, encryption checks, file upload abuse, DoS, logging issues, etc.
6. Fuzzing: field type/length mutation, DoS and rate-limit testing
7. Tooling: common open source tools

Learning the Ropes (Knowledge Base)

Before touching a target, you need a solid foundation:

• Networking fundamentals: OSI/network layers, TCP/IP, UDP, Ethernet, routing, sockets
• Cryptography: symmetric & asymmetric encryption, signature/certificate validation, TLS/SSL, CRC, checksums
• Tools: Wireshark/Tcpdump, TCPreplay & tcprewrite, Ghidra/IDA
• C/C++ literacy: bases, structs & pointers, memory offset calculation, types, and execution flow tracing
• Python scripting: for replay/spoofing, packet modification, sniffing

Without these fundamentals, reverse engineering a proprietary protocol becomes guesswork rather than methodical analysis.

Here is a comparison between HTTP and a assumed custom protocol for easy understanding:

A minimal TCP client/server pair will look like below, with a custom port:

Server →

import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.bind(("0.0.0.0", 9000))
server.listen(5)
print("listening on 9000")
while True:
    conn, addr = server.accept()
    data = conn.recv(4096)
    print(f"{addr} sent {data!r}")
    conn.sendall(b"ack")
    conn.close()

Client →

import socket

client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("127.0.0.1", 9000))
client.sendall(b"hello")
response = client.recv(4096)
print(response)
client.close()

Notes:

• AF_INET = IPv4, SOCK_STREAM = TCP. Swap to SOCK_DGRAM for UDP (no listen/accept, just sendto/recvfrom).
• recv(4096) doesn't guarantee a full message in one call — for a custom protocol you would typically read a fixed-size header first (to get a length field), then loop recv until you've read that many bytes.

Quick recap on the Offset with an example:

/* uint8_t → anywhere
uint16_t → even offsets (0,2,4,6,8,…)
uint32_t → multiples of 4 (0,4,8,12,…)
uint64_t → multiples of 8 (0,8,16,…)
Example: */

typedef struct {
    uint8_t  a; // offset 0,  size 1
    // padding 1 byte
    uint16_t b; // offset 2,  size 2
    uint8_t  c; // offset 4,  size 1
    // padding 3 bytes
    uint32_t d; // offset 8,  size 4
    uint8_t  e; // offset 12, size 1
    // padding 3 bytes
    uint32_t f; // offset 16, size 4
    // padding 4 bytes
    uint64_t g; // offset 24, size 8
    uint16_t h; // offset 32, size 2
    // padding 6 bytes
} example_t; // sizeof = 40, not 34


/*
+-------+----------+------+-----------+--------+
| Field |   Type   | Size | Alignment | Offset |
+-------+----------+------+-----------+--------+
| a     | uint8_t  |    1 |         1 |      0 |
| b     | uint16_t |    2 |         2 |      2 |
| c     | uint8_t  |    1 |         1 |      4 |
| d     | uint32_t |    4 |         4 |      8 |
| e     | uint8_t  |    1 |         1 |     12 |
| f     | uint32_t |    4 |         4 |     16 |
| g     | uint64_t |    8 |         8 |     24 |
| h     | uint16_t |    2 |         2 |     32 |
+-------+----------+------+-----------+--------+
*/

Recon

Start by figuring out what you actually have:

Do you have a Wireshark dissector, source code, or an firmware build file? Your entire approach depends on one of these.

Then dig into what’s publicly available (search egnines):

1. RFCs, SDK documentation/blogs, publications, patents
2. Protocol version and any associated CVEs
3. Public enumeration/simulator/exploit/monitoring scripts targeting the protocol
4. Ask LLMs to “teach/tell me everything about <protocol-name>”
5. Wireshark dissectors
6. Search over org’s internal documentation platform — atlassian, notion, etc.

Examine at the network-level (just the basics for now):

1. Is the device provisioned via DHCP (from a remote server) or does it use a static IP?
2. What ports are used — fixed or random — and what info/banners are exposed on open ports (Nmap)?
3. Does it respond to ICMP packets (ping)?
4. Do you see a TCP, UDP or TLS (version?) header in the packet?
5. Does it communicate over an Ethernet (layer 2) or Layer 3–4?

More on this, in Wireshark, inspect a packet’s Packet Details pane. The protocol stack is shown from lower to higher layers:

• Layer 2 (Data Link): Ethernet II, 802.1Q VLAN, ARP, MAC addresses.
  Example: Ethernet II, Src: …, Dst: …
• Layer 3 (Network): Internet Protocol Version 4/6, ICMP, IP addresses.
  Example: IPv4, Src: 192.168.1.10, Dst: 8.8.8.8
• Layer 4 (Transport): TCP, UDP, SCTP, port numbers.
  Example: TCP, Src Port: 12345, Dst Port: 443

A typical packet hierarchy looks like:

Frame
└─ Ethernet II ← Layer 2
 └─ Internet Protocol (IPv4) ← Layer 3
 └─ Transmission Control Protocol (TCP) ← Layer 4
 └─ TLS / HTTP / DNS… ← Application layer

Useful display filters:

eth # Ethernet / Layer 2
arp # ARP, commonly treated as L2/L2.5
vlan # 802.1Q tagging at Layer 2
ip or ipv6 # Layer 3
icmp or icmpv6 # Layer 3 control traffic
tcp or udp or sctp # Layer 4

For a protocol you do not recognize, expand it and check its placement:
1. Directly inside Ethernet II usually means Layer 2.

2. Inside IPv4 or IPv6 means Layer 3 payload/control traffic.

3. Inside TCP or UDP means Layer 4/application payload.

Also check the Protocol column, but use Packet Details as the authoritative view because it shows encapsulation.

And many more ways to recon and understand the protocol before diving in.

Understanding the Behavior

This is where you start forming a mental model of how the protocol actually operates:

Ask: Is it & Does it questions -

1. Architecture: client(s)/server(s), publisher/subscriber, or peer-to-peer
2. Packet variations just TCP-syn, rst, etc.: discovery/broadcast, connect/disconnect, single vs. continuous data transfer, and management operations (factory reset, field rename, firmware upgrade, restart/sleep/shutdown, etc.)
3. Session management: depends on sequence number, checksums, nonces (timestamp-based, incremental, or random), priority flags or something else.
4. Session liveness: heartbeats, keep-alives, and other liveness mechanisms

Additional behavioral questions worth answering:

1. Is server/client config (metadata) pre-loaded into each other (devices, server or applications) — a form of “reserved” authentication?
2. Who initiates the connection?
3. What’s the loading/response time for specific actions like restart or any other operation? because when you’re working on protocol level you need to know exactly how much time you need to wait until executing your next test or exploit.
4. Does the device or service have an auto-sleep mode?
5. Is communication full-duplex or half-duplex, and is it real-time?
6. After setup, how is identification handled between client(s) & server — UIDs, strings, special headers?

Reverse Engineering + Mapping/Building a Spec

This step is scenario-dependent, based on what you have.

Now: If you have firmware access

Reverse engineer the firmware (plenty of resources available over the internet to get started) to extract:

1. Packet structure
2. Opcodes / function codes
3. Functional logic — authentication, data collection/processing/send/receive, and everything else the protocol supports can be seen
4. And almost all the details about it.

Now, start calculating offsets manually and start testing using Scapy with Python slicing techniques (read below attached blogs).

Next: If you do not have Wireshark dissector or source code

Start capturing packets for every behavior/operation/action using Wireshark

1. Capture broadly — don’t filter at the port level, only filter by source & destination IP (ip.addr with OR operator). Because, you never know a device/server might be using different ports for different actions (usually happens in an initial connection stage)

2. Enable the feature remotely or start the device/service

3. Trigger every supported opcode/functions and save each resulting pcap separately

4. Document the sequence for each triggered action (just like TCP sequence flow), along with sequence numbers, nonces, etc.

5. Compare different operations’ pcaps using pcap_diff (byte level)

6. This reveals field/variable offsets (even without knowing their names)

7. From here, you can begin testing with Scapy’s slicing techniques

Must read →

Modifying Strings in .pcap Files Using Python Scapy Without Errors

Network Scanning with Scapy: A Beginner’s Guide to ICMP Scanning

[SOLVED] Network Scanner build using Scapy only scans itself

The Actual Test

Now, if you are familiar with common OWASP vulnerabilities you should proceed easily.

Before you get started, few things you need to take care of

Step 1 — Convert captured packets into proper Python objects (using a dissector if available) with the help of pyshark

Get 0xpalindrome’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Step 2 — Decide how you’ll monitor the target application/server/device’s health (liveliness):

1. A dedicated monitoring application from project itself
2. Continuous ping/ICMP checks (does it support?)
3. Sending valid packets at a fixed interval (NTP synched)
4. Install a background service (acts as an agent) to send live health data to you (a normal timestamp data, etc.)

Step 3 (most important) — Before executing any crafted packet, always capture traffic again in Wireshark (makes reuse and debugging much easier for yourself as well as for client/dev-team)

Important Note: Attacks can occur either at the protocol parser level, or at the end-application layer that processes the actual data field values (protocol compiler or client/server-app?)

Replaying

Simple packet replay often isn’t enough. When replaying captured traffic need to be mindful about :

1. Changing source/destination IP, MAC, and port numbers (try to keep the source port identical to the original)
2. Administrative/root permission is required to replay the layer 2 packets
3. TCP-based protocols depend heavily on few of the below headers/fields:

• Sequence numbers (SEQ)
• Acknowledgments (ACK)
• Nonces
• Checksums
• Timestamps, length, and connection handshakes, adjusting the TTL (based on the test environment setup hops)

Simply resending old packets to a server/target device usually fails because the connection state is out of sync (TCP based) — you need to properly reconstruct these values (refer above listed blog posts).

Once you understand and can properly reconstruct valid packet, proceed:

Authentication

1. Try to authenticate or establish a session by reusing captured packets, playing either the client or server role

2. Close the connection, then re-execute a valid opcode packet to see if the target still accepts it with the old auth/session

3. Check if session IDs can be guessed (Base64 encoding, XOR patterns, random-looking IDs) — is “authentication” really just a sequence number, it wants any standard or custom header with fixed value?

4. Is there a single shared secret used across all connected devices, or per-device unique secret?

5. Test certificate validation — does it accept expired, self-signed, or wrong-hostname certificates?

and many more well known authentication test cases …

Authorization

Check whether roles are properly enforced:

1. Spoof the source IP/MAC of another device
2. Reuse a session token or certificate from another device/client
3. Use another device’s UID to send a valid opcode packet (since unique identifiers are often assigned per device)
4. Modify user or session IDs (IDOR) or attempt to execute privileged opcodes directly

and many more well known authorization test cases…

Opcode/Flow Abuse (Business Logic)

Business logic flaws are often the easiest to find and hardest to catch in review in protocol testing:

1. Send the “final” opcode right at the start of a sequence. Understanding: consider an operation execution that may look like this — execute opcode 1, and then opcode 2 to review the data or perform any action. In HTTP, we have OPTIONS verb (here its opcode) preflight in Oauth flow.
2. Send the same opcode packet more than once
3. Send a close-connection opcode, then continue sending valid packets anyway
4. Send partial/incomplete messages or payloads
5. Send a downgraded protocol version at the initial handshake (modify the header or version field from the body)
6. Skip a packet in the expected sequence and observe the target’s reaction
7. Combine wrong field values with a mismatched opcode

and many more…

Encryption

Don’t assume something is encrypted just because it looks unreadable:

1. Determine the encryption scope — headers or payload/body only, or headers + payload both?
2. Check if plaintext data can still be read from payload fields
3. Follow the stream (TCP/UDP) packets on Wireshark
4. Remember: a poorly built dissector may fail to render ASCII text properly — seeing hex doesn’t automatically mean the data is encrypted
5. Ask whether the data is actually encoded rather than encrypted

and many more…

File Upload

Manually chunk or segment the file’s binary data and place it into the raw payload layer of packets.

1. Chunks: divide the file into fixed-size blocks, with or without a sequence/session ID per packet

Sample chunking — fixed-size blocks with a sequence ID →

from scapy.all import IP, UDP, Raw, send
import struct

CHUNK_SIZE = 1024
DST_IP = "192.168.1.11"
DST_PORT = 9999

def chunk_file(path, chunk_size=CHUNK_SIZE):
    with open(path, "rb") as f:
        while True:
            data = f.read(chunk_size)
            if not data:
                break
            yield data

def send_chunks(path, dst_ip=DST_IP, dst_port=DST_PORT):
    for seq, data in enumerate(chunk_file(path)):
        # 4-byte sequence id prepended - receiver can reorder/reassemble
        payload = struct.pack("!I", seq) + data
        pkt = IP(dst=dst_ip) / UDP(dport=dst_port) / Raw(load=payload)
        send(pkt, verbose=False)

if __name__ == "__main__":
    send_chunks("example.bin")

2. Segmentation: a more structured approach, expecting a specific header in each packet alongside the actual file data

Sample: Define an explicit header (session ID, sequence, total chunks, length, flags) instead of a bare sequence number →

from scapy.all import IP, UDP, Raw, send
import struct, os, uuid

HEADER_FMT = "!IIIHB"  # session_id, seq_num, total_chunks, payload_len, flags
HEADER_SIZE = struct.calcsize(HEADER_FMT)
CHUNK_SIZE = 1024

FLAG_MORE = 0x01
FLAG_LAST = 0x02

def build_segments(path, session_id, chunk_size=CHUNK_SIZE):
    size = os.path.getsize(path)
    total_chunks = (size + chunk_size - 1) // chunk_size

    with open(path, "rb") as f:
        for seq in range(total_chunks):
            data = f.read(chunk_size)
            flags = FLAG_LAST if seq == total_chunks - 1 else FLAG_MORE
            header = struct.pack(HEADER_FMT, session_id, seq, total_chunks, len(data), flags)
            yield header + data

def send_segments(path, dst_ip="192.168.1.11", dst_port=9999):
    session_id = uuid.uuid4().int & 0xFFFFFFFF
    for segment in build_segments(path, session_id):
        pkt = IP(dst=dst_ip) / UDP(dport=dst_port) / Raw(load=segment) # stacking  - refer above blogs
        send(pkt, verbose=False)

# Receiver-side parsing sketch (rough)
def parse_segment(raw_bytes):
    session_id, seq, total_chunks, payload_len, flags = struct.unpack(
        HEADER_FMT, raw_bytes[:HEADER_SIZE]
    )
    payload = raw_bytes[HEADER_SIZE:HEADER_SIZE + payload_len]
    return session_id, seq, total_chunks, flags, payload

and many more…

RCE (Remote Code Execution)

Example attack chain:

1. Use a file upload opcode (e.g., opcode 123) to upload a malicious executable payload
2. Call a follow-up opcode (e.g., opcode 124) to load/execute the uploaded file
3. Sometimes, after uploading it automatically run the executable file (just like many web apps)

And many more…

More Bugs to Hunt For

• Rate limiting / DoS / handling of multiple simultaneous connection requests
• Large packet payloads (oversized data)
• Logging of sensitive data — at the device level and/or application level (master/slave)
• Injection flaws — first and second order (including buffer overflows)

The above-listed bug categories are more impactful and prominent when it comes to protocol testing.

Fuzzing: Field Type & Length, DoS/Rate-Limit

Fuzzer needs its own blog series to explain how it is built and how it works.

But for now, a Fuzzer is generally made up of three core parts:

1. Instrumentation — where to inject payloads and how to monitor target device health

2. Fuzzing engine — generates the actual payload mutations at runtime

3. Supporting features — reporting, crash triage, etc.

Common fuzzing tools/frameworks:

• Boofuzz
• Enterprise Defensics (with Suite Wizard, SDK Express)

Categories of Bugs to Watch for When Fuzzing

Fuzzing against a protocol parser or embedded target can surface a huge range of memory-safety and logic bugs, including:

• Bounds violations: out-of-bounds read/write, buffer/heap/stack overflow or underflow, off-by-one/two/null errors
• Lifetime violations: use-after-free, use-after-return, use-after-scope, double free, dangling pointer dereference
• Initialization bugs: uninitialized stack/heap reads, uninitialized variables or structure fields
• Integer bugs: overflow/underflow (signed & unsigned), truncation, narrowing conversions, offset/size calculation errors
• Pointer bugs: NULL/wild/arbitrary pointer dereference, pointer overflow/underflow/wraparound, invalid casts
• Type safety bugs: type confusion, bad casts, vtable/RTTI corruption
• Copy/move bugs: memcpy/strcpy/sprintf overflows, serialization/deserialization corruption
• Allocation bugs: undersized allocation, realloc misuse, structure size miscalculation
• Heap metadata bugs: chunk header, free list, tcache/fastbin/smallbin/largebin corruption
• Control-flow corruption: return address overwrite, function/vtable pointer overwrite, jump table corruption
• Concurrency bugs: data races, TOCTOU, atomicity violations, missing synchronization
• Information disclosure: stack/heap/address/pointer leaks, ASLR bypass leaks
• Protocol/parser-specific bugs: length/offset field overflow, parser state corruption, opcode validation failure, bounds check bypass

These map closely to categories flagged by common sanitizers: heap-buffer-overflow, stack-buffer-overflow, heap-use-after-free, double-free, null-dereference, signed/unsigned-integer-overflow, and more.

Yeah, a fuzzer is very helpful, but only when it properly understands the protocol structure, how to build and maintain the connection, and how to monitor the target correctly.

Purpose-built tool help streamline the workflow:

P-SEQ — Visualizes PCAP conversations directly in your browser, generating sequence diagrams, allowing packet inspection, and supporting PNG export for documentation.

A proposal worth considering before the test

One practical recommendation for teams doing this kind of testing repeatedly: make Wireshark dissector for the target protocol a mandatory prerequisite before starting the pentest. Without a dissector, testers spend disproportionate time on manual offset calculation (if source code or reverse engineered the firmware/protocol-parser) and packet comparison rather than actual vulnerability discovery — investing in tooling/resources upfront pays off across the whole engagement.

Closing Thoughts

Protocol pentesting rewards patience and methodical documentation more than almost any other pentesting discipline. Success hinges on:

• Building strong fundamentals (networking, crypto, C/C++, scripting)
• Doing thorough recon before touching the target (proper documentation at every step)
• Carefully mapping protocol behavior and building your own “spec” when none exists
• Systematically testing AuthN/AuthZ, business logic, encryption assumptions, and memory safety, etc.
• Using fuzzing to catch what manual testing misses

The bytes on the wire will tell you everything — if you take the time to listen carefully.

Let me know your thoughts on this blog — https://www.linkedin.com/in/okpalindrome/