IDAStealth v1.3 - VMProtect and improved Debugger Support

The new IDAStealth version v1.3 includes support for anti-debugging techniques used e.g. by VMProtect. More specifically, the stealth driver as well as the user-mode hooks now support two additional process information class parameters: ProcessDebugObjectHandle and ProcessDebugFlags for the NtQueryInformationProcess API. VMProtect actually uses the ProcessDebugFlags information class to detect a debugger (and a few additional techniques).

Apart from that, v1.3 now has some features to support you while debugging. It allows for halting the debugger automatically as soon as an SEH handler is invoked by the OS. Additionally, the debugger can also be halted after SEH kicked in but as soon as the OS re-schedules the faulting thread with a (possibly) changed instruction pointer (or context in general). Both events can also just be logged without stopping the debugger.

Debugging Support

To illustrate the new debugger support features a bit, let's analyze the following assembler program and see to what extent IDAStealth can assist us in doing so.
The code uses a few tricks to complicate the analysis a bit. In fact, the code is quite trivial but it's still good enough to fool IDA with standard settings. Naturally, you can just turn off analysis and generate code at two places and IDA disassembles everything correctly. Also note, that these tricks are targeting recursive disassemblers so OllyDbg isn't heavily affected since it apparently uses an algorithm based on the simple linear-sweep disassembling technique (I might be wrong about this, though).

    call lbl_1
    dd 25360ABEh ; random junk
    ; seh handler starts here
    xor eax, eax
    jnz lbl_1 + 3 ; fake jump
    jz lbl_2
    dd 25ff6645h ; random junk
 
lbl_1:
    add dword ptr [esp], 4 ; skip 4 junk bytes
    push fs:[0]
    mov fs:[0], esp
    int 3
    jmp strings + 2 ; fake jump
 
strings:
    szText ttl,"1337"
    szText msg, "Hello!"
 
lbl_2:
    mov ecx, [esp+12] ; get context pointer
    assume  ecx:ptr CONTEXT
    mov [ecx].regEip, offset quit-123456789
    add [ecx].regEip, 123456789 ; never use direct offsets :)
    ret
    jmp lbl_2 + 2 ; fake jump
 
quit:
    pop fs:[0]
    add esp, 4
    invoke MessageBox, 0, addr msg, addr ttl, 0
    invoke ExitProcess, 0
    ret

Actually, this code is really simple, but nevertheless it is able to confuse most (recursive) disassemblers (and specifically IDA).
The first call implicitly pushes the address of the 4 junk bytes onto the stack and dispatches to lbl_1. The code at lbl_1 fixes the address pushed by the call to lbl_1 to point past the junk bytes. Then it adds an SEH record to the thread environment block, using the corrected address as the SEH handler. Finally, the code raises an exception by means of the int 3 instruction in line 13. As a consequence, the OS redirects control flow to the recently installed SEH handler.

The handler in turn uses a very simple trick to confuse the disassembly process employed by recursive traversal disassemblers: one of the successor addresses points to invalid code by jumping "into the middle" of an instruction, thereby confusing the disassembler.
The important point here is to know how a recursive traversal disassembler explores a binary program: it starts to disassemble until an instruction is encountered which causes a change in control flow. Then, the disassembler has to determine the successors, i.e. the possible branch targets of this instruction. After the successor addresses have been determined, the algorithm starts over.

So the disassembly process can be disturbed if there is one successor which points to invalid code but this code is actually never reached at runtime.
The simple trick here consists of xor'ing eax with itself in order to erase the zero flag. Although the jnz branch is never taken, the disassembler has to assume that it might be taken. Since the successor address of that instruction points into the middle of an instruction, the disassembler creates an incorrect disassembly listing because it tries to disassemble invalid code but does not know that this program point is actually unreachable.
In this simple case however, a disassembler could determine that this is indeed unreachable code by performing a simple data flow analysis, e.g. conditional constant propagation.

The junk bytes right behind the jump instructions (line 7) are crafted in a way such that if disassembled, the resulting instruction overlaps with the next valid instruction starting at line 10.
The last trick used at lbl_1 is the jump instruction after the interrupt. Again, this is unreachable code since the control flow won't reach the instruction following the interrupt. The effect however is, that the strings are treated as code.

After the exception, control flow resumes at line 4 and branches to lbl_2.
The exception handler then modifies the CPU context and adjusts the instruction pointer. In order to further complicate the disassembling process, the instruction pointer is indirectly modified by computing its value instead of simply assigning it.
Finally, the handler returns zero which signals the OS to continue execution with the modified context.
So in the end, the code reaches the quit label, uninstalls the SEH handler, displays a message box and terminates the process gracefully.

If we assemble the code and let IDA analyze the executable we can see that the tricks worked pretty well:

So how do we analyze this code? By manually creating an instruction at 401013h we can at least analyze the functionality used to install the SEH handler and to generate the int3 exception. Note the impact of the fake jump after the interrupt: the instructions from 40102A until 401038 are actually the embedded strings which are passed later to the MessageBox API.

However, we still need to manually create new code at 401009h in order to reveal the instructions belonging to the installed exception handler. Without doing so, we are unable to follow the flow of execution.
The picture below shows the first few instructions of the SEH handler.

The end of the SEH handler modifies the instruction pointer in order to let the OS continue the thread after the exception at the quit label. Obviously, IDA can not easily figure out that these calculations actually result in a valid code section offset and consequently completely fails to disassemble any further instructions.

In order to follow the control flow from here we either need a good understanding of how SEH works or guess that these calculations yield some interesting address and start over from there. By doing so, we end up with the situation shown in the picture below.

Seems like a lot of work - and this is still an extremely trivial example. Wouldn't it be nice if we could automatically halt the debugger at the beginning of the SEH handler, or better yet, at the new position after the handler modified the instruction pointer?
Well, just download the new IDAStealth and enable the two corresponding options in the tab Other Options.

If you let the debugger run, it will automatically halt at 401009h (line 4) and at 401054h (line 29), respectively. You can also tell IDAStealth to just log SEH events to IDAs output window without having the debugger stopped each time.
You can download the test executable along with the source code from here.