wbenny / hvpp

hvpp is a lightweight Intel x64/VT-x hypervisor written in C++ focused primarily on virtualization of already running operating system

hvpp

hvpp is a lightweight Intel x64/VT-x hypervisor written in C++ focused primarily on virtualization of already running operating system.

Motivation

Although several open-source research hypervisors aimed at simplicity already exist, in my opinion this field is still somewhat unexplored and needs more open-source projects. This can especially help those who have just started exploring virtualization technologies and are looking for small/reference projects. If you're one of them, my bets are that you're really disappointed right now, because all you've got are barely dozen of (great!) projects and huge pile of Intel Manual pages.

C++ has been chosen as a language for this project because of two reasons:

• The virtualization architecture can be better expressed in OOP concepts (with such objects as VCPU, EPT).
• I didn't find other projects which would use modern C++17 features, except for bareflank. Although bareflank is compilable under Windows, Linux and UEFI, on Windows, it uses cygwin to cross-compile the hypervisor. Since the hypervisor is a self contained ELF binary, the Windows kernel is missing the debug symbols for the hypervisor, which prevents easy debugging with WinDbg.

Even though this project is primarily developed for Windows, I've decided to not use traditional Windows Driver naming convention (aka DrvCamelCase). Instead, traditional C++ snake_case is used. The reason is that hypervisor is very "stand-alone" and doesn't depend on many OS functions. Therefore I decided to treat it as a regular C++ project.

If you want to challenge yourself in low-level programming, my advice would be to try and write a simple hypervisor. During the process you'll get invaluable knowledge and experience and you'll probably discover many new things. For instance here's a selection of some things I've learned thanks to writing this project:

• what are MTRR good for
• NT APIs like
  • MmGetPhysicalMemoryRanges() (and \REGISTRY\MACHINE\HARDWARE\RESOURCEMAP)
  • KeIpiGenericCall - which can be used instead of undocumented KeGenericCallDpc (but at the cost of higher IRQL)
• that TraceLogging API (built upon ETW) exist
• how Meltdown vulnerability has been mitigated in Windows
• that VMWare uses not only I/O port 0x5658 for backdoor, but also port 0x5659 and it uses almost all GP registers for both input and output parameters
• that bugs in other hypervisors exist (and found a way how to fix them)
• that VirtualBox uses ring 1 for software-based virtualization (i.e. when VT-x/AMD-V isn't available)
• how guest memory has been virtualized before EPT (by so called Shadow-paging, trapping page-faults and mov cr3, <source> instructions in the guest)

Also - as obvious as it might sound - I'd like to point out that if you decide to write your own VT-x hypervisor, you'll NEED Intel® 64 and IA-32 architectures software developer’s manual combined volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D, and 4. So download the PDF - together with Adobe Acrobat Reader - because trust me, you don't want to read and navigate through 5000 pages with browser's built-in PDF reader.

Features

• EPT with identity mapping with usage of 2MB pages for the first 512GB of physical memory (see ept.cpp). This results in faster translations of the memory. It also means splitting particular 2MB pages into 4kb pages might be desired if EPT hooking is required. This process is actually not complicated at all and this repository includes example on how to achieve that.
• Simple pass-through VM-exit handler, which can handle:
  • exceptions or NMIs
  • CPUID, (WB)INVD, INVLPG, RDTSC(P), MOV CR, MOV DR, IN/OUT, RDMSR, WRMSR, SGDT, SIDT, LGDT, LIDT, SLDT, STR, LLDT, LTR, XSETBV and INVPCID instructions
  • VMCALL instruction (used for termination of hvpp)
  • VMCLEAR, VMLAUNCH, VMRESUME, VMPTRLD, VMPTRST, VMREAD, VMWRITE, VMFUNC, VMXOFF, VMXON, INVEPT and INVVPID instructions raise #UD (invalid opcode exception)
• Ability to run in VMWare (tested even with KVM - I advise to turn off Hyper-V Enlightenments, as it can cause conflicts). VMWare's nested virtualization makes development and debugging of hypervisors much easier.
• Simple custom memory manager (see mm.cpp). The need for custom memory manager emerges from the fact that you should think twice before calling any kernel function from VM-exit handler, because many of them can be called at IRQL <= DISPATCH_LEVEL (such as ExAllocatePoolWithTag). But in VM-exit handler, interrupts are disabled and your IRQL is effectively HIGH_LEVEL.
• Detailed code comments, which should explain what the code does and why - sometimes with direct references to Intel Manual for further reading.
• TraceLogging API (which builds on ETW) - the main benefit is it can be used for really high frequency logging (10'000+ per second) and it can be used from any IRQL - which makes it a perfect candidate even for logging in VM-exit handlers.
  • hvpp already includes tracing VM-exit handler (see vmexit_stats.cpp).
• Various reimplemented classes and functions - such as bitmaps and spinlocks - to avoid calling kernel functions.
• Included simple application (hvppctrl) which should demonstrate CPUID instruction interception, hiding hooks in user-mode applications via EPT and communication with hvpp via VMCALL

Code workflow

Note: hvpp is compiled as a static library, which is linked with the hvppdrv project.

• Bootstrap of the driver (hvpp, driver.cpp)
  • preallocate enough memory and initialize the hvpp memory manager
  • initialize the logger
• Bootstrap of the hypervisor (hvppdrv, main.cpp)
  • create vmexit_handler instance
• Start the hypervisor with provided VM-exit handler: hypervisor::start(vmexit_handler& handler)
  • initialize virtual cpu (VCPU) for each logical processor
  • assign provided vmexit_handler instance to each VCPU
  • launch all VCPUs via IPI (inter-processor interrupt): vcpu_t::start()
    • setup VMXON region and VMCS: vcpu_t::vmx_enter()
    • vmexit_handler::setup() is called, which allows anyone to initialize the VM-exit handler and/or modify the VMCS before the launch (see vmexit_custom_handler::setup() in hvppdrv, vmexit_custom.cpp)
• Hypervisor is now running and handling VM-exits via provided VM-exit handler
• Stop the hypervisor: hypervisor::stop()
  • destroy each VCPU via IPI: vcpu_t::stop()
    • vmexit_handler::teardown() is called and switches into VMX mode (vmexit_passthrough_handler::teardown() does it by VMCALL instruction)
    • in VMX mode, vcpu_t::vmx_leave() is called - it leaves VMX mode with VMXOFF instruction

Compilation

Compile hvpp using Visual Studio 2017. Solution file is included. The only required dependency is WDK.

Usage

You can run hvpp on Windows 7 or higher. Windows 10 is recommended though, because it supports TraceLogging.

Enable Test-Signing boot configuration option (note that you'll need administrative privileges to use bcdedit and sc commands):

bcdedit /set testsigning on

Register driver with Service Control Manager (yes, it's important to leave these spaces):

sc create hvpp type= kernel binPath= "C:\full\path\to\hvppdrv.sys"

Now you should restart your computer for testsigning to take effect, otherwise you'll be unable to start the driver. But before you do, you might want to prepare DebugView from SysInternals and traceview.exe tool from the WDK (note that traceview will work properly only on Windows 10).

After restart, launch DebugView and TraceView. In TraceView:

• go to File -> Create New Log Session, click on Add Provider
  • pick Manually Entered Control GUID or Hashed Name
  • paste 916fcd3e-673b-4610-aaba-0b71e28acd40 (arbitrarily chosen, see lib/win32/tracelog.cpp)
  • click OK
• in the next dialog, leave the Source Of WPP Format Information set to Auto
  • click OK
• after that, click Next, which will bring you to Log Session Options dialog
  • in Log Session Name editbox you can give this logging session any name you like, e.g. HvppSession or you can leave it as it is
  • if you desire to analyze this logging session when it's stopped (e.g. with Windows Performance Analyzer) you can optionally enable Log Trace Event Data To File, which saves whole logging session into an .ETL file
  • click Finish

TraceView is now set-up and ready to show tracelogs from hvpp. You can launch hvpp now:

sc start hvpp

hvpp now performs various checks and enters VMX mode if they pass. In case of success you should see message hvpp started in the DebugView.

Run hvppctrl:

hvppctrl.exe

• hvppctrl performs CPUID instruction with EAX = 0x70707668 ('hvpp') which hvpp should intercept and return string hello from hvpp in EAX, EBX, ECX and EDX registers (see vmexit_custom.cpp). hvppctrl should print this string.

• hvppctrl tries to "stealthily" hook ntdll!ZwClose function using EPT. The exact process is described further below.

• hvppctrl performs IOCTL, which should instruct hvpp to set one-time breakpoint when IN/OUT instruction manipulating with port 0x64 (keyboard) is executed.

Description of "stealth hooking" process

• locates ZwClose function in ntdll.dll
  • disassembles first 16 bytes of this function and prints them
    • printed instructions should indicate that this function is NOT hooked yet
  • calls this function (with NULL parameter, this function call will most likely fail with some NTSTATUS error code, which it ignores)
  • prints value of HookCallCount and it's expected value (explained below)
• hooks ntdll!ZwClose fuction using Detours
  • disassembles first 16 bytes of this function and prints them
    • printed instructions should now indicate that the function IS hooked (by jmp being first instruction)
  • calls this function (with NULL parameter)
    • instead of original function, the hook function will be called - on each call, it increments variable HookCallCount
  • prints value of HookCallCount and it's expected value - it should be 1 now, as the hooked function has been called for the first time now
• calls hvpp by VMCALL instruction and RCX = 0xc1 (arbitrarily chosen), RDX = AddressOfReadPage and R8 = AddressOfExecutePage - this instructs hvpp to hide the hook
  • disassembles first 16 bytes of this function and prints them
    • printed instructions should now indicate that the function hook is HIDDEN (by showing original content of memory - no jmp)
  • calls this function (with NULL parameter)
    • despite what the memory returned when we read it, the hook function will be called again and the HookCallCount will be incremented again
  • prints value of HookCallCount and it's expected value - it should be 2
• calls hvpp by VMCALL instruction and RCX = 0xc2 (arbitrarily chosen) - this instructs hvpp to unhide the hook
  • disassembles first 16 bytes of this function and prints them
    • printed instructions should now indicate that the function hook is NOT HIDDEN and it should show jmp as a first instruction again
  • calls this function (with NULL parameter)
    • because the function is still hooked, the hook function will be called and HookCallCount will be incremented again
  • prints value of HookCallCount and it's expected value - it should be 3
• unhooks ntdll!ZwClose function
  • disassembles first 16 bytes of this function and prints them
    • printed instructions should now indicate that the function is NOT hooked - it should show the same content as when this function wasn't hooked
  • calls this function (with NULL parameter)
    • original function will be called, therefore the HookCallCount should not be incremented now
  • prints value of HookCallCount and it's expected value - it should be still 3

At the same time you should see tracelog messages in the TraceView - they are generated on each VMCALL and on each EPT Violation.

When you decide you want to turn off the hvpp, just execute:

sc stop hvpp

Remarks

• hvpp is designed to virtualize already running OS - i.e. it's not capable of running multiple guests like VMWare or VirtualBox. It also lacks support for any nested VMX operations.
• hvpp is designed to run only on 64bit Intel processors, which support VT-x and EPT. This makes the code more simple and less bloated.
• hvpp is designed to run only on Windows - future work might focus on Linux.
• hvpp currently doesn't exit VMX mode on sleep or hibernate (S3 and S4 power states) - Intel Manual says we should do so - this is known limitation.

License

This software is open-source under the MIT license. See the LICENSE.txt file in this repository.

Detours is licensed under MIT license (a copy of the license is included here).

udis86 is licensed under the terms of the 2-clause "Simplified BSD License" (a copy of the license is included here).

Similar work

SimpleVisor: https://github.com/ionescu007/SimpleVisor

HyperPlatform: https://github.com/tandasat/HyperPlatform

HyperBone: https://github.com/DarthTon/HyperBone

Bareflank: https://github.com/Bareflank/hypervisor

ksm: https://github.com/asamy/ksm

MoRE: https://github.com/ainfosec/MoRE

hyperdbg: https://github.com/rmusser01/hyperdbg

virtdbg: https://github.com/upring/virtdbg

BluePill: https://invisiblethingslab.com/resources/bh07/nbp-0.32-public.zip

Phrack #69: http://www.phrack.org/issues/69/15.html

NOVA Microhypervisor: https://github.com/udosteinberg/NOVA

Finally, I'd especially like to suggest reading interesting writings from Satoshi Tanda (github, twitter):

• http://tandasat.github.io/HyperPlatform/
• http://tandasat.github.io/HyperPlatform/userdocument/index.html
• http://standa-note.blogspot.com/2015/08/writing-hypervisor-for-kernel-mode-code.html

And notes from LordNoteworthy (github, twitter):

• https://github.com/LordNoteworthy/cpu-internals
• https://github.com/LordNoteworthy/windows-internals

If you find this project interesting, you can buy me a coffee

  BTC 3GwZMNGvLCZMi7mjL8K6iyj6qGbhkVMNMF
  LTC MQn5YC7bZd4KSsaj8snSg4TetmdKDkeCYk

hvpp's Issues

how to port to 32-bit Windows?

I only have 32-bit Windows 7 vmware vm. I plan to use this project here. How to port it to 32-bit OS?

Can't build on latest vs2019, LNK4210 and C2131

I've tried cloning and building this in vs2019 and I'm getting the following errors:

3>device_custom.cpp
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(107,31): error C2131: expression did not evaluate to a constant
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(107,31): message : failure was caused by a read of an uninitialized symbol
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(107,31): message : see usage of 'ia32::vmx::interrupt_info_t::<unnamed-tag>::nmi_unblocking'
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(113,33): error C2131: expression did not evaluate to a constant
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(113,33): message : failure was caused by a read of an uninitialized symbol
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(113,33): message : see usage of 'ia32::vmx::interrupt_info_t::<unnamed-tag>::nmi_unblocking'
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(119,42): error C2131: expression did not evaluate to a constant
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(119,42): message : failure was caused by a read of an uninitialized symbol
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(119,42): message : see usage of 'ia32::vmx::interrupt_info_t::<unnamed-tag>::nmi_unblocking'
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(125,46): error C2131: expression did not evaluate to a constant
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(125,46): message : failure was caused by a read of an uninitialized symbol
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(125,46): message : see usage of 'ia32::vmx::interrupt_info_t::<unnamed-tag>::nmi_unblocking'
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(132,38): error C2131: expression did not evaluate to a constant
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(132,38): message : failure was caused by a read of an uninitialized symbol
3>C:\Users\User\source\repos\hvpp\src\hvpp\hvpp\interrupt.h(132,38): message : see usage of 'ia32::vmx::interrupt_info_t::<unnamed-tag>::nmi_unblocking'
2>Generating code
2>Finished generating code
2>hvpp.lib(hvpp.cpp.obj) : warning LNK4210: .CRT section exists; there may be unhandled static initializers or terminators
2>LINK : error LNK1218: warning treated as error; no output file generated

The first error seems to be here:

static constexpr auto nmi =
      interrupt_t {
        vmx::interrupt_type::nmi,
        exception_vector::nmi_interrupt
      };

This looks related: https://stackoverflow.com/questions/63558400/vs2019-latest-update-constexpr-expression-did-not-evaluate-to-a-constant-for and if I apply the fix it seems to solve the problem.

As for the second fault, no idea what's causing this (LNK4210)

How to complete ForEachLogicalCore () in the driver

` vcpu_t* vcpu_list = hvpp::hypervisor::get_vcpu_list();

for (uint32_t i = 0; i < mp::cpu_count(); i++)
{
  // Split the 2MB page where the code we want to hook resides.
  vcpu_list[i].ept().split_2mb_to_4kb(page_exec & ept_pd_t::mask, page_exec & ept_pd_t::mask);
  // Set execute-only access on the page we want to hook.
  vcpu_list[i].ept().map_4kb(page_exec, page_exec, epte_t::access_type::execute);
  // We've changed EPT structure - mappings derived from EPT need to be invalidated.
  vmx::invept_single_context(vcpu_list[i].ept().ept_pointer());
}`

Hello, I had some problems while modifying your code as a toy. I plan to port the function ForEachLogicalCore () to the kernel so that when the program exits, the driver can automatically recover the hidden memory. But the code I wrote has an exception DRIVER_IRQL_NOT_LESS_OR_EQUAL

Use 2MB EPT pages by default

It would be nice to initially cover whole memory with 2MB pages and eventually split desired pages into 4KB pages on demand. There are several benefits:

• walking through such page hierarchy is faster
• this solution will require less memory
• ept_t::map_identity() will finish much faster

Multiple issues with debug register access handling

1. DR7.general_detect (GD) should be cleared

hvpp does not clear the DR7.GD bit in the VMCS up on #DB. This causes an unintended #DB when the processor attempt to access debug registers and likely to cause a bug check. As 17.2.4 Debug Control Register (DR7) states (The processor clears the GD flag upon entering to the debug exception handler (...)), the processor automatically the DR7.GD bit on #DB. Since hvpp does not emulate this behaviour and leaves that bit set in the VMCS, any DR access results in #DB, while it should have been allowed if GD is properly cleared. The proper fix will be to add such a #DB handler, though clearing it within handle_mov_dr when #DB is injected can be an lazy option too.

2. Access to DR4 and 5 should be treated in the same manner as DR6 and 7 access are

hvpp behaves differently when DR4,5 and DR6,7 are accessed, resulting in setting to or getting of incorrect values. For example, write to DR5 is emulated with just the mov instruction while it should be handled in the same manner as write to DR7, which needs writes to VMCS with the upper bits check. This issue only manifests when CR4.DE is cleared, which is not the default case on Windows as far as I can tell.

3. Always 1 and 0 bits in the DR6 and 7 should be emulated

The processor appears to ignore writes to the bits described as 0 or 1 in Figure 17-1. Debug Registers on the non-root mode. It is, however, the processor can indeed modify those bits on the root-mode, and so, the current hpvv's emulate that behaviour. I was unable to find a description about this behaviour in the Intel SDM, but VMware and KVM(Link) emulate this too.

If you would like to have three separated issues, let me know. I will do so.

Syntax error occurs in vmexit_passthrough.cpp file

Severity	Code	Description	Project	File	Line	Suppression State
Error	C2143	syntax error: missing ')' before '}'	hvpp
Error	C2143	syntax error: missing ';' before '}'	hvpp

The error occrus in here:

static const uint8_t* skip_prefixes(const uint8_t* first, const uint8_t* last) noexcept
{
  //
  // Return the first byte of the opcode that is not a prefix.
  //
  return std::find_if(first, last, [&](uint8_t byte){		  
	  //
	  // List of prefix types that should be skipped, LOCK is not included as it'd #UD.
	  //
	  static constexpr uint8_t skip_table[] = {
			  0xF2, 0xF3, 0x2E, 0x36, 0x3E, 0x26, 0x64, 0x65, 0x2E, 0x3E, 0x66, 0x67
	  };
	  return std::find(std::begin(skip_table), std::end(skip_table), byte) == std::end(skip_table);
  }
}

But Visual Studio doesn't notify any syntax error. I have no idea what's the problem.

INT1 INT3

INT1 INT3 example？

KERNEL_SECURITY_CHECK_FAILURE (139)

BSOD randomly

debug Screenshot:

!analyze -v

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

KERNEL_SECURITY_CHECK_FAILURE (139)
A kernel component has corrupted a critical data structure.  The corruption
could potentially allow a malicious user to gain control of this machine.
Arguments:
Arg1: 0000000000000004, The thread's stack pointer was outside the legal stack
	extents for the thread.
Arg2: ffffac0a81dbd110, Address of the trap frame for the exception that caused the bugcheck
Arg3: ffffac0a81dbd068, Address of the exception record for the exception that caused the bugcheck
Arg4: 0000000000000000, Reserved

Debugging Details:
------------------


"C:\Windows\System32\KERNELBASE.dll" was not found in the image list.
Debugger will attempt to load "C:\Windows\System32\KERNELBASE.dll" at given base 00000000`00000000.

Please provide the full image name, including the extension (i.e. kernel32.dll)
for more reliable results.Base address and size overrides can be given as
.reload <image.ext>=<base>,<size>.

KEY_VALUES_STRING: 1


PROCESSES_ANALYSIS: 1

SERVICE_ANALYSIS: 1

STACKHASH_ANALYSIS: 1

TIMELINE_ANALYSIS: 1


DUMP_CLASS: 1

DUMP_QUALIFIER: 0

BUILD_VERSION_STRING:  17763.1.amd64fre.rs5_release.180914-1434

DUMP_TYPE:  0

BUGCHECK_P1: 4

BUGCHECK_P2: ffffac0a81dbd110

BUGCHECK_P3: ffffac0a81dbd068

BUGCHECK_P4: 0

TRAP_FRAME:  ffff18a01182070f -- (.trap 0xffff18a01182070f)
Unable to read trap frame at ffff18a0`1182070f

EXCEPTION_RECORD:  ffffac0a81dbd068 -- (.exr 0xffffac0a81dbd068)
ExceptionAddress: fffff80413a3b7df (nt!RtlpGetStackLimits+0x0000000000147c7f)
   ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 0000000000000004
Subcode: 0x4 FAST_FAIL_INCORRECT_STACK

CPU_COUNT: 4

CPU_MHZ: e10

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 9e

CPU_STEPPING: 9

CPU_MICROCODE: 6,9e,9,0 (F,M,S,R)  SIG: B4'00000000 (cache) B4'00000000 (init)

BUGCHECK_STR:  0x139

PROCESS_NAME:  debug_me.exe

CURRENT_IRQL:  0

DEFAULT_BUCKET_ID:  FAIL_FAST_INCORRECT_STACK

WATSON_BKT_EVENT:  BEX

ERROR_CODE: (NTSTATUS) 0xc0000409 - <Unable to get error code text>

EXCEPTION_CODE: (NTSTATUS) 0xc0000409 - <Unable to get error code text>

EXCEPTION_CODE_STR:  c0000409

EXCEPTION_PARAMETER1:  0000000000000004

ANALYSIS_SESSION_HOST:  windywhli-PC1

ANALYSIS_SESSION_TIME:  11-19-2019 20:19:23.0224

ANALYSIS_VERSION: 10.0.18362.1 amd64fre

BAD_STACK_POINTER:  ffffac0a81dbc628

LAST_CONTROL_TRANSFER:  from fffff80413aa2d72 to fffff804139cd390

STACK_TEXT:  
ffffac0a`81dbc628 fffff804`13aa2d72 : 00000000`00000004 00000000`00000003 ffffac0a`81dbc790 fffff804`1396d380 : nt!DbgBreakPointWithStatus
ffffac0a`81dbc630 fffff804`13aa24f7 : 00000000`00000003 ffffac0a`81dbc790 fffff804`139d9660 00000000`00000139 : nt!KiBugCheckDebugBreak+0x12
ffffac0a`81dbc690 fffff804`139c5837 : 00000000`00000000 cccccccc`cccccccc 00000000`00000000 00001f80`00cc00cc : nt!KeBugCheck2+0x957
ffffac0a`81dbcdb0 fffff804`139d6e69 : 00000000`00000139 00000000`00000004 ffffac0a`81dbd110 ffffac0a`81dbd068 : nt!KeBugCheckEx+0x107
ffffac0a`81dbcdf0 fffff804`139d7210 : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : nt!KiBugCheckDispatch+0x69
ffffac0a`81dbcf30 fffff804`139d5608 : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : nt!KiFastFailDispatch+0xd0
ffffac0a`81dbd110 fffff804`13a3b7df : 00000000`00000000 ffffac0a`81dbd550 cccccccc`cccccccc cccccccc`cccccccc : nt!KiRaiseSecurityCheckFailure+0x308
ffffac0a`81dbd2a0 fffff804`13926e7b : cccccccc`cccccccc cccccccc`cccccccc cccccccc`00000003 cccccccc`cccccccc : nt!RtlpGetStackLimits+0x147c7f
ffffac0a`81dbd2d0 fffff804`13834ac4 : ffffac0a`81dbdd08 ffffac0a`81dbda50 ffffac0a`81dbdd08 ffffac0a`81db6000 : nt!RtlDispatchException+0x6b
ffffac0a`81dbd520 fffff804`139d6f42 : 00000000`00000000 fffff804`138ec094 ffff18a0`1182070f fffff804`14283af6 : nt!KiDispatchException+0x144
ffffac0a`81dbdbd0 fffff804`139d178e : ffffac0a`81dbddc0 fffff804`1aab3e85 00000000`00000000 00000020`00000000 : nt!KiExceptionDispatch+0xc2
ffffac0a`81dbddb0 fffff804`1aab34c1 : 00000000`00000000 00000000`00484ea3 cccccccc`cccccccc cccccccc`cccccccc : nt!KiInvalidOpcodeFault+0x30e
ffffac0a`81dbdf40 fffff804`1aac3be5 : 00000000`0019fa90 00000000`004023fd 00000000`00010202 00000000`00000005 : win32dk!hvpp::vcpu_t::entry_host+0x161 [D:\MyProjects\VSProjects\hvpp\src\hvpp\hvpp\vcpu.cpp @ 849]
ffffac0a`81dbdfb0 00000000`00402402 : 004020f3`00401714 00739670`004020f3 0019fa98`00401564 0040131f`0019fae0 : win32dk!hvpp::vcpu_t::entry_host_+0x58 [D:\MyProjects\VSProjects\hvpp\src\hvpp\hvpp\vcpu.asm @ 231]
00000000`0019fa90 004020f3`00401714 : 00739670`004020f3 0019fa98`00401564 0040131f`0019fae0 00000007`004020f3 : debug_me+0x2402
00000000`0019fa98 00739670`004020f3 : 0019fa98`00401564 0040131f`0019fae0 00000007`004020f3 4150083c`c0000000 : 0x004020f3`00401714
00000000`0019faa0 0019fa98`00401564 : 0040131f`0019fae0 00000007`004020f3 4150083c`c0000000 40180000`00000000 : 0x00739670`004020f3
00000000`0019faa8 0040131f`0019fae0 : 00000007`004020f3 4150083c`c0000000 40180000`00000000 4150083b`40000000 : 0x0019fa98`00401564
00000000`0019fab0 00000007`004020f3 : 4150083c`c0000000 40180000`00000000 4150083b`40000000 00000007`0073e308 : 0x0040131f`0019fae0
00000000`0019fab8 4150083c`c0000000 : 40180000`00000000 4150083b`40000000 00000007`0073e308 004020f3`00739670 : 0x00000007`004020f3
00000000`0019fac0 40180000`00000000 : 4150083b`40000000 00000007`0073e308 004020f3`00739670 0041c7f0`0019fb14 : 0x4150083c`c0000000
00000000`0019fac8 4150083b`40000000 : 00000007`0073e308 004020f3`00739670 0041c7f0`0019fb14 004b5710`02416dc8 : 0x40180000`00000000
00000000`0019fad0 00000007`0073e308 : 004020f3`00739670 0041c7f0`0019fb14 004b5710`02416dc8 004b5710`02416dc8 : 0x4150083b`40000000
00000000`0019fad8 004020f3`00739670 : 0041c7f0`0019fb14 004b5710`02416dc8 004b5710`02416dc8 004b5710`0040baa0 : 0x00000007`0073e308
00000000`0019fae0 0041c7f0`0019fb14 : 004b5710`02416dc8 004b5710`02416dc8 004b5710`0040baa0 fffffffe`ffffffff : 0x004020f3`00739670
00000000`0019fae8 004b5710`02416dc8 : 004b5710`02416dc8 004b5710`0040baa0 fffffffe`ffffffff 76a96c42`0019fb40 : 0x0041c7f0`0019fb14
00000000`0019faf0 004b5710`02416dc8 : 004b5710`0040baa0 fffffffe`ffffffff 76a96c42`0019fb40 000007d8`004012c3 : 0x004b5710`02416dc8
00000000`0019faf8 004b5710`0040baa0 : fffffffe`ffffffff 76a96c42`0019fb40 000007d8`004012c3 0019fb80`004120b9 : 0x004b5710`02416dc8
00000000`0019fb00 fffffffe`ffffffff : 76a96c42`0019fb40 000007d8`004012c3 0019fb80`004120b9 02416dc8`02416dc8 : 0x004b5710`0040baa0
00000000`0019fb08 76a96c42`0019fb40 : 000007d8`004012c3 0019fb80`004120b9 02416dc8`02416dc8 0040baa0`0019fc20 : 0xfffffffe`ffffffff
00000000`0019fb10 000007d8`004012c3 : 0019fb80`004120b9 02416dc8`02416dc8 0040baa0`0019fc20 02416dc8`0040baa0 : 0x76a96c42`0019fb40
00000000`0019fb18 0019fb80`004120b9 : 02416dc8`02416dc8 0040baa0`0019fc20 02416dc8`0040baa0 00000000`00000001 : 0x000007d8`004012c3
00000000`0019fb20 02416dc8`02416dc8 : 0040baa0`0019fc20 02416dc8`0040baa0 00000000`00000001 0047f1d2`0019fc14 : 0x0019fb80`004120b9
00000000`0019fb28 0040baa0`0019fc20 : 02416dc8`0040baa0 00000000`00000001 0047f1d2`0019fc14 004153f9`ffffffff : 0x02416dc8`02416dc8
00000000`0019fb30 02416dc8`0040baa0 : 00000000`00000001 0047f1d2`0019fc14 004153f9`ffffffff 0019fb80`000007d8 : 0x0040baa0`0019fc20
00000000`0019fb38 00000000`00000001 : 0047f1d2`0019fc14 004153f9`ffffffff 0019fb80`000007d8 004153ca`00000000 : 0x02416dc8`0040baa0
00000000`0019fb40 0047f1d2`0019fc14 : 004153f9`ffffffff 0019fb80`000007d8 004153ca`00000000 0019fb80`000007d8 : 0x1
00000000`0019fb48 004153f9`ffffffff : 0019fb80`000007d8 004153ca`00000000 0019fb80`000007d8 0040badb`00000000 : 0x0047f1d2`0019fc14
00000000`0019fb50 0019fb80`000007d8 : 004153ca`00000000 0019fb80`000007d8 0040badb`00000000 0019fb80`000007d8 : 0x004153f9`ffffffff
00000000`0019fb58 004153ca`00000000 : 0019fb80`000007d8 0040badb`00000000 0019fb80`000007d8 00000000`00000000 : 0x0019fb80`000007d8
00000000`0019fb60 0019fb80`000007d8 : 0040badb`00000000 0019fb80`000007d8 00000000`00000000 16010002`52010001 : 0x004153ca`00000000
00000000`0019fb68 0040badb`00000000 : 0019fb80`000007d8 00000000`00000000 16010002`52010001 00000000`00000000 : 0x0019fb80`000007d8
00000000`0019fb70 0019fb80`000007d8 : 00000000`00000000 16010002`52010001 00000000`00000000 00486218`00000000 : 0x0040badb`00000000
00000000`0019fb78 00000000`00000000 : 16010002`52010001 00000000`00000000 00486218`00000000 004b8118`0047de32 : 0x0019fb80`000007d8


THREAD_SHA1_HASH_MOD_FUNC:  9878726627229c0c7c7b6cf8cacb076f99901365

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  6b7b4724f584776c4f57d46af500ba565cf0d156

THREAD_SHA1_HASH_MOD:  6ae1ede5fed85cf92790998645c6060afa1c331a

FOLLOWUP_IP: 
win32dk!hvpp::vcpu_t::entry_host+161 [D:\MyProjects\VSProjects\hvpp\src\hvpp\hvpp\vcpu.cpp @ 849]
fffff804`1aab34c1 440f79a720800000        vmwrite r12,qword ptr [rdi+8020h]

FAULT_INSTR_CODE:  a7790f44

FAULTING_SOURCE_LINE:  D:\MyProjects\VSProjects\hvpp\src\hvpp\hvpp\vcpu.cpp

FAULTING_SOURCE_FILE:  D:\MyProjects\VSProjects\hvpp\src\hvpp\hvpp\vcpu.cpp

FAULTING_SOURCE_LINE_NUMBER:  849

SYMBOL_STACK_INDEX:  c

SYMBOL_NAME:  win32dk!hvpp::vcpu_t::entry_host+161

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: win32dk

IMAGE_NAME:  win32dk.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  5dce5bd8

STACK_COMMAND:  .thread ; .cxr ; kb

BUCKET_ID_FUNC_OFFSET:  161

FAILURE_BUCKET_ID:  0x139_MISSING_GSFRAME_STACKPTR_ERROR_win32dk!hvpp::vcpu_t::entry_host

BUCKET_ID:  0x139_MISSING_GSFRAME_STACKPTR_ERROR_win32dk!hvpp::vcpu_t::entry_host

PRIMARY_PROBLEM_CLASS:  0x139_MISSING_GSFRAME_STACKPTR_ERROR_win32dk!hvpp::vcpu_t::entry_host

TARGET_TIME:  2019-11-19T12:18:30.000Z

OSBUILD:  17763

OSSERVICEPACK:  0

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  272

PRODUCT_TYPE:  1

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

OSEDITION:  Windows 10 WinNt TerminalServer SingleUserTS

OS_LOCALE:  

USER_LCID:  0

OSBUILD_TIMESTAMP:  2005-12-02 00:58:59

BUILDDATESTAMP_STR:  180914-1434

BUILDLAB_STR:  rs5_release

BUILDOSVER_STR:  10.0.17763.1.amd64fre.rs5_release.180914-1434

ANALYSIS_SESSION_ELAPSED_TIME:  fbfd

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:0x139_missing_gsframe_stackptr_error_win32dk!hvpp::vcpu_t::entry_host

FAILURE_ID_HASH:  {46030f82-f280-8494-bba2-d63fa3a0fafa}

Followup:     MachineOwner
---------

simple of syscall interception,What shall I do?(EFER HOOK or SysCall HOOK)

I'm sorry,I know it might be silly,But I don't know what to do.
I want to use EFER HOOK or SysCall HOOK,I see the code, vmexit_passthrough_handler::handle_emulate_syscall
Settings may be required efer.Bits.sce = false....

hooking cpu instructions

So, i see CPUID hook. It is good work. But how i can add hook, for example, __rdtsc? add handle function to vmexit_custon, and?

Errors in Visual Studio editor

Hi. I noticed that Visual Studio shows in editor a lot of errors but it compiles fine. I don't have this problem with other projects. Reinstalling VS, WDK etc. doesn't give any results. Is it a problem with the whole project or my setup? I'm using the newest VS 2017 Community and the newest WDK

Implement simple example of syscall interception

BSOD on r3 memory access

I've been exploring the hypervisor features but I ran into a problem when accessing r3 memory from the vmcall execute handler. I tried to read my r3 struct from the vmcall handler and got "IRQL NOT LESS OR EQUAL". I can also get this error when reading the PTE I passed as rdx. I tested on Windows 10 (1903 ), but I don't think it matters

Is this project actively being worked on still?

Just curious if this great project still has an active developer or community. Or has it been superseeded by other projects based off of it?

kernel_security_check_failure

Hi ! First of all thank you for your work on this !
I'm trying to run your latest code but when starting the service my system crash with kernel_security_check_failure error everytime.
Windows 10 latest version, started with testsigning off and also tested with Patch Guard off.
Any hint on this ?

[hvppdrv] Unloading is currently broken

Hey there,
Commit a59f7cc seems to have broken HV unloading,
its still working in 4404207.

Crash: SYSTEM_PTE_MISUSE
VM: VMware Workstation 15.0.2 build-10952284
Guest OS: Windows 8.1 Pro, 64-bit (Build 9600) 6.3.9600
Guest CPU: i9 7980XE, 4 vCores
Guest RAM: 6.1GB(Intentionally using 6208MB instead of 6144MB)

BSOD when running vmware 15.0.2

I'm getting a BSOD CLOCK_WATCHDOG_TIMEOUT when starting a vmware vm whilst hvpp is running.
Is this unsupported?

Issue with multiple hidden pages

Hi.
I've been studying ept hooking with hvpp.
So far, it is great and work as I expected.
However, If I try to make multiple hidden pages, it just doesnt work.
I primary work on hvppdrv but it doesnt work on hvppdrv_c either.
Am I missing? or is it bug?

Vm handler problem

void vmexit_passthrough_handler::handle_vm_fallback(vcpu_t& vp) noexcept
{

//inject hardware exception must be write vmentryinslen
vp.inject(
interrupt_info_t(vmx::interrupt_type::hardware_exception,
exception_vector::invalid_opcode));
//wtf this???
//inject ud exceptions does not require adding ins rip
vp.suppress_rip_adjust();

}

void vcpu_t::entry_host() noexcept{
........////
if (!suppress_rip_adjust_)
{
exit_context_.rip += exit_instruction_length();
}
//if set Eflags.tf must be inject #DB exception
if (exit_context_.rflags.trap_flag) {

      dr6_t dr6 = read<dr6_t>();
      dr6.single_instruction = true;
      write<dr6_t>(dr6);

      inject(interrupt_info_t(vmx::interrupt_type::hardware_exception,
        exception_vector::debug));
  
    }

   // anti blocking_movss
    vmx::interruptibility_state_t state = guest_interruptibility_state();
    state.blocking_by_mov_ss = false;
    state.blocking_by_sti = false;
    guest_interruptibility_state(state);

.....///////
}

Make hvpp work on Linux

This might be challenging, as the Linux headers and C++ don't like each other very much. I assume compiling OS-related stuff in separate *.c files and link/wrap them with *.cpp files together should work, though.

Strange BSOD with multiple shadowed pages

Hi guys. I am having weird issue I wonder if anyone else have experience. Sorry for my poor English.

I am making multiple hook on Kernel function. Hook is work OK. I am make multiple shadow page and all is work OK. But when I am try to hide certain combination of page, BSOD with KERNEL_SECURITY_CHECK_FAILURE and no idea why.

From my Debug Output:

The combination of last 2 are causing BSOD. Other combination OK. But 3 and 4 in picture together make BSOD. But all Debug addresses look normal to me? I am not understanding why they cannot work together.

@wbenny Can you advice me master? 👍 🥇

Cannot trap rdtsc

Hello. I try to trap rdtsc

I add to vmexit_custom.h

void handle_execute_rdtsc(vcpu_t& vp) noexcept override;

and to vmexit_custom.cpp

void vmexit_custom_handler::handle_execute_rdtsc(vcpu_t& vp) noexcept
{
  DbgPrint("rdtsc");
  base_type::handle_execute_rdtsc(vp);
}

But am never catch the instruction. What am do wrong sir?

Possible bug in vcpu_t::entry_host()

You capturing rip rsp and rflags before vmexit handler:
` auto captured_rsp = exit_context_.rsp;
auto captured_rflags = exit_context_.rflags;

{
  exit_context_.rsp    = guest_rsp();
  exit_context_.rip    = guest_rip();
  exit_context_.rflags = guest_rflags();

  //
  // WinDbg will show full callstack (hypervisor + interrupted application)
  // after these two lines are executed.
  // See vcpu.asm for more details.
  //
  // Note that machine_frame.rip is supposed to hold return address.
  // exit_instruction_length() is added to the guest_rip() to create
  // this value.
  //
  stack_.machine_frame.rip = exit_context_.rip + exit_instruction_length();
  stack_.machine_frame.rsp = exit_context_.rsp;

  {
    handler_->handle(*this);`

As result any modifications to rip, rsp and rflags in vmexits have not any effect. So possible solution is:
` memory_manager::allocator_guard _;

auto captured_rsp    = exit_context_.rsp;
auto captured_rflags = exit_context_.rflags;

{
  handler_->handle(*this);

  exit_context_.rsp    = guest_rsp();
  exit_context_.rip    = guest_rip();
  exit_context_.rflags = guest_rflags();`

Invalid use of invd

In the following:
https://github.com/wbenny/hvpp/blob/master/src/hvpp/hvpp/vmexit.cpp#L289

You execute invd, which will invalidate the cache of your VMM including the guest which would result in corruption if this exit ever occurred. Instead, when this exit occurs a wbinvd should be executed instead, or the instruction can be ignored.

my system lags out completely

hi，build c was ok,but can not unload hvpp,and build c++ my system lags out completely.

syscall hooking freezes Windows when stopping the hypervisor

Hi, I'm enabling syscall hooking via:

auto efer = msr::read<msr::efer_t>();
efer.syscall_enable = false;
msr::write(efer);

auto exception_bitmap = vp.exception_bitmap();
exception_bitmap.invalid_opcode = true;
vp.exception_bitmap(exception_bitmap);

Whenever I sc.exe stop my service the entire OS freezes

About DebugBuggin

@wbenny Thought you should be aware that the user DebugBuggin who has been pestering you for help over the past few weeks is a Walmart game cheat author who pastes anything he can get his hands on and is taking credit for your work himself. You probably already noticed there was something wrong with him from his entitled attitude though.

ZW hooking

The usermo example works great. Would kernel hooking be difficult to do with this project? Also just a side unrelated note I had disable CPU overclocking in my bios to stop some random bluescreens that would show.

Implement simple example of hooking kernel pages via EPT

Estimate how much memory will be needed for hvpp on startup

hvpp now has hardcoded size of memory, which is preallocated on the startup. By all means this is rather unfortunate and used as a temporary solution.

HostCr3 Bug

The initialization routine using Ipi will run in a random context. So it can cause a blue screen. You should save cr3 in the system context(DriverEntry ) and set it in vcpu init

void vcpu_t::setup_host() noexcept
{
...///
host_cr0(read<cr0_t>());
host_cr3(read<cr3_t>());->> host_cr3(dq_savedSystemCr3);
host_cr4(read<cr4_t>());

cr3_t cr3 = read<cr3_t>();
DbgPrint("Host Cr3:%p\n", cr3.flags);

proper use of mtf

hi wbenny, thanks for awesome project :) i am having a bit of trouble to use mtf correctly and hoped you might be able to help.

in ept violation handler sometimes i set mtf so i can change permissions again but i seem to enter an endless loop of ept violations and page permissions get screwed up. am i using it incorrectly?

//set permissions
vp.ept().map_4kb(.....)
     
//enable mtf 
auto procbased_ctls = vp.processor_based_controls();
procbased_ctls.monitor_trap_flag = true;
vp.processor_based_controls(procbased_ctls);

vp.suppress_rip_adjust();

return;

void vmexit_custom_handler::handle_monitor_trap_flag(vcpu_t& vp) noexcept
{
 //restore permissions
 vp.ept().map_4kb(...);

 //disable mtf
 auto procbased_ctls = vp.processor_based_controls();
 procbased_ctls.monitor_trap_flag = false;
 vp.processor_based_controls(procbased_ctls);
}

Vm Exit Eflags error

Your hypervisor will change the original Eflags on exit.
For example, eflags==246 cpuid->>vmexit -->eflags 206.
You can test on x64dbg~

Crashing in VMWare 15

Hi. I have installed the newest available version of VMWare and it seems to be crashing at the moment of entering VMX mode. I hadn't any problems with older version of VMWare that supported only Workstation 14.x, now it doesn't want to work even with Workstation 12.x. Anyone knows how to solve it?

Won't compile with Visual Studio 2017

I upgraded to 2017 and got the latest SDK ( 10.0.17763.0 ) and compiling this project is tons of errors and 3 out of 4 projects fail. I have to target to latest SDK and that doesn't fix anything, lot of constexpr errors and other things. This is such a fascinating project and was really looking forward to testing it out, if you know of way to get this compiling please share.

How to send buffer to driver while the hypervisor is running?

Hello,

I've been struggling with this issue for days and I've looked everywhere I knew would help, however, I'm still having a problem with this

I want to send a small buffer (~1KB) to the driver while the hypervisor is running.
I tried creating an IOCTL code specific for this along with the ioctl_enable_io_debugbreak_t::code which is already present.
I also tried implementing the on_write() driver function to get the buffer.

no matter the method, the buffer from the user app is always corrupted and the expected data is not present. (the correct size is passed) the buffer is mostly filled with 0xcccccccc along with some other random numbers.
I also tried the routine in copy_from_user function where IoAllocateMdl is used then MmProbeAndLockPages. the same kind of garbage is always returned.

any ideas?

What version of SDK and WDK was used for compilation?

VS2017
SDK -10.0.17763.0
WDK - Windows 10, version 1809

Compile- tons of error

Can anyone tell me the development environment?

hvppdrv.sys file isn't created

After building the hvpp.cpp file on visual studio on test mode, the hvppdrv.sys file is not created and not found on the folder. Help me in solving this issue

EPT lags out PC; while it's working on VM

Sorry to bother again. Enabling EPT (and only mapping identity) severely degrades system performance when run on my PC. I also noticed the "System Interrupts" process occasionally has high CPU load (~20%) when hvpp is running. I don't have this issue on VMWare. What could be the cause of this?

KMODE_EXCEPTION_NOT_HANDLED

Crash when I try to load the driver:

Windows 10 Rs4
Intel i3-22230 CPU

A bugfix I would like to contribute back

@wbenny
Related to issue: #37

For some reason on 6th gen Intels and newer this does not cause any issue, however on older Intel CPUs it will cause system instability/freezing. The function below is responsible for the determining the memory cache type that each page on the EPT is configured to. The bug in the code lies in fact that the original code's default result is "memory_type::uncacheable" even though a valid mttr descriptor for that memory range was found.

I spent weeks looking into every line of hvpp until I finally stumbled upon this one bug that was fixed by adding just 3 lines.

In hvpp/lib/mttr_descriptor.h:

     memory_type type(pa_t pa) const noexcept
      {
        memory_type result = memory_type::invalid;

        for (auto mtrr_item : *this)
        {
          if (mtrr_item.range.contains(pa))
          {
            if (is_fixed(mtrr_item) || mtrr_item.type == memory_type::uncacheable)
            {
              result = mtrr_item.type;
              break;
            }

            if ( result == memory_type::write_back && (result == memory_type::write_through || mtrr_item.type == memory_type::write_through))
            {
              result = memory_type::write_through;
            } else
            {
                result = mtrr_item.type; // TODO bruhh
            }
          }
        }

        if (result == memory_type::invalid)
        {
          result = default_memory_type_;
        }

        return result;
      }

Other than that I've found that hvpp is a really well written hypervisor and has a lot of potential. Thanks!

How do I use this for x86 process?

Nvm

bugcheck on sc stop

sc start and hvppctrl.exe work but sc stop explodes. Even if I just start and immediately stop afterwards.

Seems this assert fails:

void vcpu_t::entry_host() noexcept
{
  hvpp_assert(state_ == state::running); <<<---

Any ideas?

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

KERNEL_SECURITY_CHECK_FAILURE (139)
A kernel component has corrupted a critical data structure.  The corruption
could potentially allow a malicious user to gain control of this machine.
Arguments:
Arg1: 0000000000000004, The thread's stack pointer was outside the legal stack
	extents for the thread.
Arg2: ffffd28f4a7470d0, Address of the trap frame for the exception that caused the bugcheck
Arg3: ffffd28f4a747028, Address of the exception record for the exception that caused the bugcheck
Arg4: 0000000000000000, Reserved

Debugging Details:
------------------

KEY_VALUES_STRING: 1

STACKHASH_ANALYSIS: 1

TIMELINE_ANALYSIS: 1

DUMP_CLASS: 1

DUMP_QUALIFIER: 401

BUILD_VERSION_STRING:  17763.1.amd64fre.rs5_release.180914-1434

DUMP_TYPE:  1

BUGCHECK_P1: 4

BUGCHECK_P2: ffffd28f4a7470d0

BUGCHECK_P3: ffffd28f4a747028

BUGCHECK_P4: 0

TRAP_FRAME:  cccccccccccccccc -- (.trap 0xcccccccccccccccc)
Unable to read trap frame at cccccccc`cccccccc

EXCEPTION_RECORD:  cccccccccccccccc -- (.exr 0xcccccccccccccccc)
Cannot read Exception record @ cccccccccccccccc

CPU_COUNT: 8

CPU_MHZ: fa0

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 3c

CPU_STEPPING: 3

CPU_MICROCODE: 6,3c,3,0 (F,M,S,R)  SIG: 24'00000000 (cache) 24'00000000 (init)

BLACKBOXBSD: 1 (!blackboxbsd)

BUGCHECK_STR:  0x139

PROCESS_NAME:  System

CURRENT_IRQL:  e

DEFAULT_BUCKET_ID:  FAIL_FAST_INCORRECT_STACK

WATSON_BKT_EVENT:  BEX

ERROR_CODE: (NTSTATUS) 0xc0000409 - The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.

EXCEPTION_CODE: (NTSTATUS) 0xc0000409 - The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.

EXCEPTION_CODE_STR:  c0000409

EXCEPTION_PARAMETER1:  0000000000000004

ANALYSIS_SESSION_HOST:  MACHINE

ANALYSIS_SESSION_TIME:  04-14-2019 02:48:27.0458

ANALYSIS_VERSION: 10.0.17763.132 amd64fre

BAD_STACK_POINTER:  ffffd28f4a746da8

LAST_CONTROL_TRANSFER:  from fffff80737c6de69 to fffff80737c5c730

STACK_TEXT:  
ffffd28f`4a746da8 fffff807`37c6de69 : 00000000`00000139 00000000`00000004 ffffd28f`4a7470d0 ffffd28f`4a747028 : nt!KeBugCheckEx
ffffd28f`4a746db0 fffff807`37c6e210 : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : nt!KiBugCheckDispatch+0x69
ffffd28f`4a746ef0 fffff807`37c6c608 : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : nt!KiFastFailDispatch+0xd0
ffffd28f`4a7470d0 fffff807`37cd27df : 00000000`00000000 ffffd28f`4a747510 cccccccc`cccccccc cccccccc`cccccccc : nt!KiRaiseSecurityCheckFailure+0x308
ffffd28f`4a747260 fffff807`37bbde7b : cccccccc`cccccccc cccccccc`cccccccc cccccccc`00000003 cccccccc`cccccccc : nt!RtlpGetStackLimits+0x147c7f
ffffd28f`4a747290 fffff807`37acbac4 : ffffd28f`4a747cc8 ffffd28f`4a747a10 ffffd28f`4a747cc8 ffff8201`692f1bc8 : nt!RtlDispatchException+0x6b
ffffd28f`4a7474e0 fffff807`37c6df42 : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : nt!KiDispatchException+0x144
ffffd28f`4a747b90 fffff807`37c67c7b : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : nt!KiExceptionDispatch+0xc2
ffffd28f`4a747d70 fffff802`81311883 : fffff802`813115f9 cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : nt!KiBreakpointTrap+0x2fb
ffffd28f`4a747f00 fffff802`813115f9 : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : hvppdrv!ia32_asm_int3+0x3 [c:\dev\gits\hvpp\src\hvpp\hvpp\ia32\win32\asm.h @ 15] 
ffffd28f`4a747f08 fffff802`8132322f : cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : hvppdrv!debugger::breakpoint+0x9 [c:\dev\gits\hvpp\src\hvpp\hvpp\lib\debugger.h @ 12] 
ffffd28f`4a747f38 fffff802`81327672 : ffffd28f`4a740000 cccccccc`cccccccc cccccccc`cccccccc cccccccc`cccccccc : hvppdrv!hvpp::vcpu_t::entry_host+0x1f [c:\dev\gits\hvpp\src\hvpp\hvpp\vcpu.cpp @ 690] 
ffffd28f`4a747fb8 00000000`00000005 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : hvppdrv!hvpp::vcpu_t::entry_host_+0x35 [C:\dev\gits\hvpp\src\hvpp\hvpp\vcpu.asm @ 220] 
00000000`00000002 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x5


THREAD_SHA1_HASH_MOD_FUNC:  ce6782b2a839ccb27881c377e3e13f8067ec0b59

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  f5608505d956203b03081191b4ab1c878e79ec10

THREAD_SHA1_HASH_MOD:  8e2b9f3283621d081c9906e4f551d738704c1b34

FOLLOWUP_IP: 
hvppdrv!ia32_asm_int3+3 [c:\dev\gits\hvpp\src\hvpp\hvpp\ia32\win32\asm.h @ 15]
fffff802`81311883 c3              ret

FAULT_INSTR_CODE:  ccccccc3

FAULTING_SOURCE_LINE:  c:\dev\gits\hvpp\src\hvpp\hvpp\ia32\win32\asm.h

FAULTING_SOURCE_FILE:  c:\dev\gits\hvpp\src\hvpp\hvpp\ia32\win32\asm.h

FAULTING_SOURCE_LINE_NUMBER:  15

FAULTING_SOURCE_CODE:  
    11: 
    12: inline void ia32_asm_int3() noexcept
    13: {
    14:   __debugbreak();
>   15: }
    16: 
    17: //
    18: // CPUID.
    19: //
    20: 


SYMBOL_STACK_INDEX:  9

SYMBOL_NAME:  hvppdrv!ia32_asm_int3+3

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: hvppdrv

IMAGE_NAME:  hvppdrv.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  5cb281ef

STACK_COMMAND:  .thread ; .cxr ; kb

BUCKET_ID_FUNC_OFFSET:  3

FAILURE_BUCKET_ID:  0x139_MISSING_GSFRAME_STACKPTR_ERROR_hvppdrv!ia32_asm_int3

BUCKET_ID:  0x139_MISSING_GSFRAME_STACKPTR_ERROR_hvppdrv!ia32_asm_int3

PRIMARY_PROBLEM_CLASS:  0x139_MISSING_GSFRAME_STACKPTR_ERROR_hvppdrv!ia32_asm_int3

TARGET_TIME:  2019-04-14T00:44:28.000Z

OSBUILD:  17763

OSSERVICEPACK:  0

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  272

PRODUCT_TYPE:  1

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

OSEDITION:  Windows 10 WinNt TerminalServer SingleUserTS

OS_LOCALE:  

USER_LCID:  0

OSBUILD_TIMESTAMP:  2005-12-02 08:58:59

BUILDDATESTAMP_STR:  180914-1434

BUILDLAB_STR:  rs5_release

BUILDOSVER_STR:  10.0.17763.1.amd64fre.rs5_release.180914-1434

ANALYSIS_SESSION_ELAPSED_TIME:  f39

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:0x139_missing_gsframe_stackptr_error_hvppdrv!ia32_asm_int3

FAILURE_ID_HASH:  {999c9d6e-1a59-f6d4-bbb1-8a741040c989}

Followup:     MachineOwner

Hypervisor detect

this article shows few methods to detect hypervisor. https://rayanfam.com/topics/defeating-malware-anti-vm-techniques-cpuid-based-instructions/

Inside HvppHandleExecuteCpuid I do the following to set the 31st bit to 0

else if(Context->Eax == 1)
	{
		 Context->Ecx |= 0UL << 30;
	}

but this breaks chrome, firefox and many others, they won't connect to the internet. Do you why and a way to fix it?

update: appears it breaks apps regardless if I set anything, I have to cal HvppVmExitPassthrough(Passthrough); but then the context registers aren't set.

Is there anyway to hook external process?

inside ia32::cr3_t kernel_cr3
auto kprocess = reinterpret_cast<NT_KPROCESS*>(PsGetCurrentProcess());

I've found that if I KeStackAttachProcess to external process, therefore PsGetcurrentProcess returns external process

In this way it can hook external process, but it bsod sometime (when unloading driver etc..)

How can I hook external process?

Crash sometimes on hvpp stop

https://www.dropbox.com/s/8pem9gwlsnvvj54/Crash.rar?dl=0

Here is my .sys, a mini dump (I don't know why a minidump is being generated if I have checked the memory dump) and my .pdb

the error is SYSTEM_THREAD_EXCEPTION_NOT_HANDLED

Add safe handling memory read/write instead of naive CR3 switch

Get rid of initialize() and destroy() method

These methods should be replaced with constructors/destructors. Since constructors may fail - and using exceptions isn't very much of an option - error code should be checked via some special function "is_initialized() ?" after each created instance.

Trying vmcall from a wow64 process

Hello,
Maybe it is my inexperience in hypervisors, but I'm not sure if I'm doing this ok.

I'm trying to make your TestHook() hide in a x86 process. I have ported the asm portion of ia32_asm_vmx_call to this to adjust the asm to x86:

.MODEL flat
.CODE
_ia32_asm_vmx_vmcall PROC
vmcall
ret
_ia32_asm_vmx_vmcall ENDP
END

(As you can see, I add _ before the function, if not, the linker wont find it and add the .MODEL flat on the top, that I'm not sure what is for, but I need it because the compiler tells me to add it)

When the application calls the hide operation, it gives me a 0xC000001D: Illegal Instruction after the call to vmcall. In x64 works perfect, so I suppose what I'm trying is not possible or I'm doing it wrong.

Also, __cpuid is detecting the HV on x86.

Any advice?

doesn't seem to support multiple ept hidden pages

Done lot of testing and hiding 1 ept page works fine but if I add any others it doesn't work properly, am I do things improperly or was this a short sight for the example? i'm testing the hppdrv_c