microsoft / usersim

User-mode Simulation of Windows Kernel APIs

This project is designed to allow a Windows kernel driver developer to compile their code as a user-mode DLL for testing purposes, so that it can be used for code coverage, fuzzers, etc.

Using

To use this repository from another project:

1. Include this repository as a git submodule from your project.
2. Build your driver files as a DLL. That is, create a new project or directory, and have it include the same source files as your driver does, but set the configuration type to be a Dynamic Library. See the "sample" project as an example, which builds the KMDF sample driver as a DLL (the sample project can be built either using Visual Studio or using cmake).
3. Set the following property on the project: Configuration Properties -> General -> Platform Toolset to Visual Studio 2022 (v143).
4. Add a reference from your DLL project to the usersim project and the usersim_dll_skeleton project.
5. Define _AMD64_;_WIN32_WINNT=0x0a00 in the project properties preprocessor defines.
6. Add to AdditionalIncludeDirectories: $(WindowsSdkDir)Include\10.0.22621.0\km;$(WindowsSdkDir)Include\wdf\kmdf\1.15
7. Disable warning 4324 (structure was padded due to alignment) in the project properties, by adding 4324 to Configuration Properties -> C/C++ -> Advanced -> Disable Specific Warnings.
8. Make sure you are not linking with any kernel libs, since they aren't usable by DLLs.
9. Give usersim.lib preference over ntdll.lib as follows: Under Configuration Properties -> Linker -> Input -> Ignore Specific Default Libraries, add ntdll.lib. Under Configuration Properties -> Linker -> Input -> Additional Dependencies, add usersim.lib before ntdll.lib.

Leak Detection

To detect memory leaks on exit, define the environment variable CXPLAT_MEMORY_LEAK_DETECTION=true

Fault Injection

To use fault injection, define the environment variable CXPLAT_FAULT_INJECTION_SIMULATION=4 where the value (4 in this example) is the number of stack frames to use to determine whether a call stack is unique. Fault injection will cause one call into the UserSim library to fail, for every unique call stack.

The failed call stacks are dumped into a file in the current directory with the name <exe name>.fault.log where <exe name> is the name of the original executable. To reset fault injection for a test executable, simply delete all <exe name>.*.log files from the current directory.

The .\scripts\Test-FaultInjection.ps1 powershell script can be used to test fault injection for a Catch2-based test executable by making successive runs that fail each successive path, until the test executable ends with all tests passing, indicating that no more code paths were found to fail. This can be done as follows.

First, navigate into the directory containing the test executable. Then do:

powershell <path to script>\Test-FaultInjection.ps1 <dump path> <test timeout> ".\<test exe name>.exe" <depth>

where:

• <path to script> is the path to the Test-FaultInjection.ps1 file.
• <dump path> is the directory to put any crash dumps in.
• <test timeout> is the maximum number of seconds per test iteration.
• <test exe name> is the filename of the test executable.
• <depth> is the value to use for CXPLAT_FAULT_INJECTION_SIMULATION.

Each iteration will result in more stacks being added to the same <exe name>.fault.log file, after a line # Iteration: <number> where <number> is the iteration number. If a crash occurs, it can then be reproduced and diagnosed by deleting the last iteration's content from the file, and then running the test executable under a debugger.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Building this repository

The steps above will build the UserSim project components needed by a dependent project, but not the tests for UserSim itself. For contributors wanting to contribute to the UserSim project, it can be built and tested as follows.

1. As a one-time step, from a Visual Studio Developer Command Prompt, do: cmake -G "Visual Studio 17 2022" -S external\catch2 -B external\catch2\build -DBUILD_TESTING=OFF
2. Build usersim.sln from the Visual Studio UI or using msbuild.
3. cxplat_test.exe -d yes and usersim_tests.exe -d yes can then be executed to run the standard tests.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

usersim's Issues

Add work item as an argument to the work_item_routine

https://learn.microsoft.com/en-us/previous-versions/windows/desktop/legacy/ms687396(v=vs.85) has the work item as an argument.
cxplat_allocate_preemptible_work_item should likewise use a work_item_routine signature that passes it to the callback so that the callback can choose to free it if desired.
Today the caller has to store it in some context structure, but the cxplat code that calls the callback already has the work item pointer.

~_usersim_emulated_dpc should hold mutex while modifying state

~_usersim_emulated_dpc notifies the worker thread that it needs to terminate.

The issue is that it sets terminate -> true without holding the mutex.

This can result in a failed wakeup.

Possible KeBugCheck improvements

Currently the implementation throws a C++ exception, which cannot be caught by Catch2 across a C API.

https://learn.microsoft.com/en-us/cpp/cpp/using-setjmp-longjmp?view=msvc-170 says setjmp/longjmp can be used in both C and C++ and may provide a better mechanism. We should investigate if that might work with Catch2.

cxplat_test CMakeList is missing several test cpp files

https://github.com/microsoft/usersim/blob/main/cxplat/cxplat_test/CMakeLists.txt

Including ntddk.h in user mode generates lots of compiler warnings

Update the README with how to deal with the following:

C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km\wdm.h(24384): warning C28301: No annotations for first declaration of 'POOL_FLAGS'. See c:\program files (x86)\windows kits\10\include\10.0.22621.0\km\wdm.h(17293). 
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km\wdm.h(24547): warning C28160: Error annotation: Must succeed pool allocations are forbidden. Allocation failures cause a system crash.
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km\wdm.h(24538): warning C6387: 'return' could be '0'.
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km\wdm.h(24615): warning C28160: Error annotation: Must succeed pool allocations are forbidden. Allocation failures cause a system crash.
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km\wdm.h(24606): warning C6387: 'return' could be '0'.
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km\wdm.h(24646): warning C6387: 'return' could be '0'.
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km\wdm.h(24684): warning C28160: Error annotation: Must succeed pool allocations are forbidden. Allocation failures cause a system crash.
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km\wdm.h(24674): warning C6387: 'return' could be '0'.
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km\wdm.h(28457): warning C28252: Inconsistent annotation for 'MmGetSystemRoutineAddress': return/function has 'SAL_maxIRQL(0)' on the prior instance. See c:\program files (x86)\windows kits\10\include\10.0.22621.0\km\wdm.h(13307). 
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km\wdm.h(28457): warning C28253: Inconsistent annotation for 'MmGetSystemRoutineAddress': return/function has 'SAL_maxIRQL(1)' on this instance. See c:\program files (x86)\windows kits\10\include\10.0.22621.0\km\wdm.h(13307). 
c:\program files (x86)\windows kits\10\include\10.0.22621.0\km\ntddk.h(22357): warning C28230: The type of '_Param_(1)' has no member 'RecordLength'.
c:\program files (x86)\windows kits\10\include\10.0.22621.0\km\ntddk.h(22357): warning C28285: For function 'WheaErrorRecordBuilderAddPacket' '_Param_(1)' syntax error in 'SAL_writableTo(byteCount(__formal(0,Record)->RecordLength))' near 'RecordLength))'.
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km\ntstrsafe.h(6765): warning C28196: The requirement that '_Param_(1)->MaximumLength>0?_Param_(1)->Length>0?_Param_(1)->Buffer==_Param_(2)&&(_Param_(1)->Buffer==0||_Param_(1)->Buffer!=0):1:1' is not satisfied. (The expression does not evaluate to true.)
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\km\ntstrsafe.h(6823): warning C28196: The requirement that '_Param_(1)->MaximumLength>0?_Param_(1)->Length>0?_Param_(1)->Buffer==_Param_(2)&&(_Param_(1)->Buffer==0||_Param_(1)->Buffer!=0):1:1' is not satisfied. (The expression does not evaluate to true.)

Probably need to create an ntddk.h wrapper to disable these while processing ntddk.h and its dependencies.

Missing Ob* APIs

The following APIs are needed by ebpf-for-windows:

• ObfReferenceObject
• ObfDereferenceObject
• ObReferenceObjectByHandle
• ObCloseHandle

Add DPC APIs

• KeInitializeDpc
• KeSetTargetProcessorDpc
• KeRemoveQueueDpc
• KeInsertQueueDpc
• KeFlushQueuedDpcs

Ke*Timer apis are covered by issue #34

CXPlat must initialize all sub-components prior to enabling fault injection

        [2]              : "cxplat_fault_injection_inject_fault + 67 D:\a\ebpf-for-windows\ebpf-for-windows\external\usersim\cxplat\src\cxplat_winuser\fault_injection.cpp 408" [Type: std::basic_string<char,std::char_traits<char>,std::allocator<char> >]
        [3]              : "cxplat_allocate + 111 D:\a\ebpf-for-windows\ebpf-for-windows\external\usersim\cxplat\src\cxplat_winuser\memory_winuser.cpp 89" [Type: std::basic_string<char,std::char_traits<char>,std::allocator<char> >]
        [4]              : "_get_processor_group_info + 174 D:\a\ebpf-for-windows\ebpf-for-windows\external\usersim\cxplat\src\cxplat_winuser\processor_winuser.cpp 29" [Type: std::basic_string<char,std::char_traits<char>,std::allocator<char> >]
        [5]              : "cxplat_winuser_initialize_processor_info + 94 D:\a\ebpf-for-windows\ebpf-for-windows\external\usersim\cxplat\src\cxplat_winuser\processor_winuser.cpp 48" [Type: std::basic_string<char,std::char_traits<char>,std::allocator<char> >]
        [6]              : "cxplat_initialize + 598 D:\a\ebpf-for-windows\ebpf-for-windows\external\usersim\cxplat\src\cxplat_winuser\cxplat_winuser.cpp 162" [Type: std::basic_string<char,std::char_traits<char>,std::allocator<char> >]
        [7]              : "usersim_platform_initiate + 68 D:\a\ebpf-for-windows\ebpf-for-windows\external\usersim\src\platform_user.cpp 56" [Type: std::basic_string<char,std::char_traits<char>,std::allocator<char> >]

The above stack shows how #143 caused a regression. The cxplat_winuser_initialize_processor_info itself is getting fault-injected and cxplat is failing to initialize. It goes to cxplat cleaup path - where it is trying to wait on the rundown_ref object which is not intialized and hence throws an exception.

01 VCRUNTIME140D!_CxxThrowException
02 usersim!_rundown_ref_table::wait_for_rundown_ref
03 usersim!cxplat_wait_for_rundown_protection_release
04 usersim!cxplat_wait_for_preemptible_work_items_complete
05 usersim!usersim_platform_terminate
06 usersim!usersim_platform_initiate
07 usersim!DllMain

Add fault injection to README and CI/CD

Fault injection and leak detection are both supported, but some tasks remain which were done in the ebpf-for-windows project, but not in this repo yet:

• list the environment variables in the README (done in PR #39)
• add leak detection to CI/CD (done in PR #39)
• add fault injection to CI/CD (done in PR #118)
• explain how to use the fault injection log in the README (done in PR #118)

cxplat_free_cache_aligned exists in winkernel but not winuser

We could just delete the API, but that would conflict with #111.
Today there are no tests for this API so maybe we remove it in the meantime.

Data corruption in usersim

Moved from eBPF-for-Windows repo issue: microsoft/ebpf-for-windows#2807
More appropriate to solve here.

Support for KeInitializeSpinLock/KeAcquireSpinLockRaiseToDpc/KeReleaseSpinLock cxplat equivalent

Support for KeInitializeEvent/KeSetEvent/KeResetEvent cxplat equivalent

The solution fails to build on latest Visual Studio 17.8, c++ standards are misaligned.

See:

• microsoft/ebpf-for-windows#3094
• microsoft/ebpf-for-windows#3091

Add support for MmGetMdlPfnArray

Needed by ebpf-for-windows

IofCompleteRequest support

The following are needed by ebpf-for-windows:

• ProbeForWrite (PR #72)
• IofCompleteRequest (stub exists as of PR #76)
• KfRaiseIrql (PR #73)
• RtlStringVPrintfWorkerA

PKTIMER emulation doesn't honor KeSetTargetProcessorDpc

When creating a timer, a DPC is provided.

Expected behavior:
DPC runs on the CPU specified in the call to KeSetTargetProcessorDpc.

Actual behavior:
DPC runs on an arbitrary CPU.

Move usersim to a dynamic library

To allow testing multiple other driver DLLs at once, usersim should be a shared dynamic library (DLL) that they can each call into, rather than a static library duplicated into each one.

Add DLL skeleton wrapping calls to DriverEntry/Unload

• Add sample driver (fixed in PR #14)
• Add dll skeleton library (fixed in PR #14)
• update readme (fixed in PR #15)
• add a test that runs the sample.dll (fixed in PR #15)

Syntax of memory-freeing APIs

Today, APIs that free memory follow the POSIX standard model for free(), etc., namely:

1. The argument is the pointer to free, and
2. null is permitted

In microsoft/ebpf-for-windows#2783 (comment), @gtrevi asks:

Could we merge nulling the pointer into cxplat_free_preemptible_work_item?

That is, Gianni asks about changing point 1 above, to take an [in,out] pointer to the pointer, and set it to null after freeing.

In #109, @nibanks also asks about changing point 2 above. There, Dave Thaler responded:

When we went through the decision process, we came to the opposite conclusion (ALL free apis should accept nullptr), for mainly the following reasons:

1. Consistency with the POSIX standard. This is the industry consensus on good API design, which is why free() is defined to accept a null ptr. Thus, programmers familiar with standard API conventions aren't surprised.
2. Minimizes number of code paths (reducing code complexity and increasing code coverage), since otherwise many places, instead of 1, have to test for null.

The first point applies to Gianni's question as well.

Use tags for all internally allocated memory

Currently some places just do things like cxplat_allocate with no tag.

The use of Directory.Build.Props breaks consuming modules attempting to do ASAN builds

This file Directory.Build.props overrides all the settings from the any Directory.Build.props higher in the directory hierarchy.

eBPF-For-Windows has its own "Directory.Build.props" file that controls address sanitizer options, but setting this in submodule overrides this.

The result is that the consuming module is build with ASAN, but this project is not.

@dthaler You added the Directory.Build.props to this project. Do you have any suggestions on how to resolve this?

Etw* API support

The following are needed by ebpf-for-windows:

• EtwWriteTransfer (stub exists as of PR #73)
• EtwRegister
• EtwUnregister
• EtwSetInformation (stub exists as of PR #73)

Create a generic test executable that can be used with various driver DLLs

Today the test executables hard code knowledge of which driver DLL they are testing.
However, a generic test executable can be created that can do things like:

1. LoadLibrary/FreeLibrary on one or more driver DLLs
2. invoke usersim_*() APIs to trigger various events

Such a test executable could only do simple generic things, but may still be useful.
rundll32.exe can do 1 in the meantime, but there are probably generic things in category 2 that would be useful.

Make fault injection be DEBUG only

Currently cxplat/src/cxplat_winuser/cxplat_winuser.cpp always enables fault injection and leak detection.

For performance, these should only be enabled on DEBUG builds.

ifdefs per platform

In #82, Nick writes:

We can discuss more in a follow up PR, but I don't want each consumer of cxplat to be required to have platforms specific logic to change the include path. They should include one path, and just #include <cxplat_platform.h> and be good to go.

vcxproj file won't load on xbox

In #82, Nick writes regarding cxplat/src/cxplat_winkernel/cxplat_winkernel.vcxproj:

We can do this in a follow up PR, but this project file needs some work. It won't load on xbox for instance.

Add analysis job to CI/CD

Define extern C macros

In #82, Nick suggested:

To simplify this stuff, we can define EXTERN_C (or EXTERN_C_START) and EXTERN_C_END in cxplat_common.h for this. It would reduce the duplication and line count a bit. It would also hide the { and } to eliminate any visible/explicit need to tab things over.

Optimize cxplat_winuser/rundown_winuser.cpp

In PR #94, Nick writes:

Whis is this so complicated, and not simply a ref count and a completion event? Is it purely for debugging reasons? If so, we need a hyper optimized release mode version.

Dave writes:

we should move any workarounds for "no free API in kernel mode" over to the usersim directory and keep the complexity out of the cxplat directory.

Fault injection should only inject faults in the calling module

CXPLAT_FAULT_STACK_CAPTURE_FRAMES_TO_SKIP is used to know how many frames to skip before gathering relevant stack frames.

Now that it's in its own DLL, it makes more sense to examine the stack address and skip any frames that match usersim.dll

Potential issues with dispatch IRQL emulation

From code review, I spotted two potential issues with usersim's emulation of code running at dispatch IRQL:

1. Two separate usersim threads of execution at dispatch level on the same processor can be interleaved by the OS dispatcher. This is prohibited in kernel dispatch level code, by definition.
2. A usersim thread running at dispatch level can be preempted by another thread of execution running at lower IRQL due to various priority boosting strategies (e.g., IO completion, anti-priority-inversion, etc.) in Windows. It appears that code running at a realtime priority is immune to dispatcher boosts.

support SeAccessCheck

The following are needed by ebpf-for-windows:

• SeCaptureSubjectContext
• SeLockSubjectContext
• SeAccessCheck
• SeUnlockSubjectContext
• SeQueryAuthenticationIdToken
• RtlValidRelativeSecurityDescriptor

Add Ke*Timer APIs

• KeInitializeTimer
• KeSetTimer
• KeCancelTimer

cxplat: use of inline functions and macros

In #89, comments ehre:

Nick: General question though: Can/should we move stuff to be header only (via macros or possibly inline functions) instead of functions defined in separate files? Ideally the linker should do a good job of eliminating overhead, but it doesn't always do it perfect.

Dave: The downside of macros/inlines is that if they're non-trivial and used a lot, then they can increase the footprint and memory requirements. So I would say: yes for trivial functions, no for complex functions. Can do a pass in a separate PR.

Add cache-aligned allocation

• ExAllocatePoolUninitialized(NonPagedPoolNxCacheAligned, ...) -> _aligned_malloc
• ExFreePool(...) -> _aligned_free

Support for KeInitializeDpc/KeSetTargetProcessorDpc/KeInsertQueueDpc/KeFlushQueuedDpcs cxplat equivalent

inc headers should not include src headers

Currently a number of public headers include "..\src\platform.h"
However, files under src should be internal to the usersim DLL

Handle multiple calls to cxplat_initialize/cleanup

What happens if the caller calls these multiple times?
Should also add test cases for this behavior.

Create nuget package

Create a binary release in the form of a nuget package so other repos need not compile usersim themselves.

Support NtQuerySystemInformation with SystemCodeIntegrityInformation

needed by ebpf-for-windows

Support for KeQueryMaximumProcessorCountEx equivalent

Support for KeInitializeTimer/KeSetTimer/KeSetTimer cxplat equivalent

KeInitializeSemaphore leaks handles

The kernel version of KeInitializeSemaphore does not require an OS allocation and doesn't have a cleanup function.

The functionality can be implemented with a spinlock and WaitOnAddress/WakeByAddressAll.

Multiple cxplat_initialize() calls and fault injection

The list of modules for fault injection is in a std::set.
The problem is that if cxplat_initialize() is called multiple times, the first cxplat_cleanup() rather than the last one, will remove it and stop fault injection in that module.

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Support for multiple devices per usersim driver

Add support for multiple devices per driver

Optimize memory_winuser.cpp

cxplat_allocation_header_t should be DEBUG only.
To do that, we need to split the cxplat_free() api and expose separate apis for cache-aligned vs unaligned, and require callers to know which one to call.

Expose ioctl API

Today we have DllMain that wraps DriverEntry.
But there's not already a way to expose the IOCTL interface that a driver exposes.
Strawman is that the DLL skeleton wrapping the sys code might export from the DLL a function for a DeviceIoControl interface that the DLL skeleton implements and calls the normal "kernel" APIs in the wrapped driver code.

Support for ebpf_get_current_cpu equivalent in cxplat