microsoft / xdp-for-windows

XDP speeds up networking on Windows

XDP For Windows

"XDP for Windows" is a Windows interface similar to XDP (eXpress Data Path), used to send and receive packets at high rates by bypassing most of the OS networking stack.

Documentation

• Usage
• Development
• AF_XDP API
• Release and Support
• FAQ

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

xdp-for-windows's Issues

Define NDIS polling dependencies

Figure out what we need from the NDIS polling API.

• Add XDP work indication to NDIS out parameters
• NDIS poll API provides serialized XDP queue state in callback?
• AF_XDP busy polling extension
• (AF_)XDP backpressure indication

Support EWDK 2022 builds

A partner is trying to build the project using the 2022 EWDK, which is failing. Add EWDK build support.

Finish support for fragmented frames

We support fragmented frames at the NIC layer for RX. Add support for TX and expose fragments to AF_XDP.

MSB4044: The "Validate NTTargetVersion" task was not given

Hi,

I'm following the build instructions and hit the error below.
I have VS2019, MSVC 14.29.30133, install the WDK 10.0.22621.0 and SDK on Windows server 2019 datacenter.
Which NTTargetVersino should I set? and where should I set it? Thank you!

PS C:\xdp-for-windows> .\tools\build.ps1
 ApiValidator:
         Validating 'Universal' driver using ApiValidator.exe
         cmd.exe /D /C "C:\Users\Administrator\AppData\Local\Temp\2\tmp1e01928a7773408cb76c66e14a26379e.cmd"
         "C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64\ApiValidator.exe" -DriverPackagePath:C:\xdp-for-w
         indows\artifacts\bin\x64_Debug\xdpmp.sys -SupportedApiXmlFiles:"C:\Program Files (x86)\Windows Kits\10\build\\
         universalDDIs\x64\UniversalDDIs.xml" -ModuleWhiteListXmlFiles:"C:\Program Files (x86)\Windows Kits\10\build\\u
         niversalDDIs\x64\ModuleWhiteList.xml" -ApiExtractorExePath:"C:\Program Files (x86)\Windows Kits\10\bin\10.0.22
         621.0\x64"
         Driver is 'Universal'.
       DriverPackageTarget:
         Packaging up the following projects for the following configurations:

         C:\xdp-for-windows\src\nmr\nmr.vcxproj Configuration='Debug' Platform='x64'

         C:\xdp-for-windows\test\fakendis\fndis.vcxproj Configuration='Debug' Platform='x64'


         The following files will be packaged:

         File to package:      C:\xdp-for-windows\artifacts\bin\x64_Debug\fndis.sys.
         Location in Package:  \fndis.sys.
         Requested by project: C:\xdp-for-windows\test\fakendis\fndis.vcxproj


         File to package:      C:\xdp-for-windows\artifacts\bin\x64_Debug\xdpmp.sys.
         Location in Package:  \xdpmp.sys.
         Requested by project:


         File to package:      C:\xdp-for-windows\build\x64_Debug\obj\xdpmp\xdpmp.inf.
         Location in Package:  \xdpmp.inf.
         Requested by project:


         Creating directory "C:\xdp-for-windows\artifacts\bin\x64_Debug\xdpmp".
         Creating directory "C:\xdp-for-windows\artifacts\bin\x64_Debug\xdpmp".
         Copying file from "C:\xdp-for-windows\artifacts\bin\x64_Debug\fndis.sys" to "C:\xdp-for-windows\artifacts\bin\
         x64_Debug\xdpmp\fndis.sys".
         Copying file from "C:\xdp-for-windows\artifacts\bin\x64_Debug\xdpmp.sys" to "C:\xdp-for-windows\artifacts\bin\
         x64_Debug\xdpmp\xdpmp.sys".
         Copying file from "C:\xdp-for-windows\build\x64_Debug\obj\xdpmp\xdpmp.inf" to "C:\xdp-for-windows\artifacts\bi
         n\x64_Debug\xdpmp\xdpmp.inf".
       FinalizeBuildStatus:
         Deleting file "C:\xdp-for-windows\build\x64_Debug\obj\xdpmp\xdpmp.tlog\unsuccessfulbuild".
         Touching "C:\xdp-for-windows\build\x64_Debug\obj\xdpmp\xdpmp.tlog\xdpmp.lastbuildstate".
    22>Done Building Project "C:\xdp-for-windows\test\xdpmp\xdpmp.vcxproj" (default targets).
     1>Done Building Project "C:\xdp-for-windows\xdp.sln" (Clean;Build target(s)) -- FAILED.
       "C:\xdp-for-windows\xdp.sln" (Clean;Build target) (1) ->
       "C:\xdp-for-windows\src\xdpapi\xdpapi.vcxproj" (default target) (15:9) ->
       (ValidateNTTargetVersion target) ->
         C:\Program Files (x86)\Windows Kits\10\build\WindowsDriver.common.targets(314,5): error MSB4044: The "Validate
       NTTargetVersion" task was not given a value for the required parameter "ValidNTTargetVersions". [C:\xdp-for-wind
       ows\src\xdpapi\xdpapi.vcxproj]


       "C:\xdp-for-windows\xdp.sln" (Clean;Build target) (1) ->
       "C:\xdp-for-windows\src\xdp\xdp.vcxproj" (default target) (26:2) ->
       "C:\xdp-for-windows\src\xdpetw\xdpetw.vcxproj" (default target) (2:4) ->
         C:\Program Files (x86)\Windows Kits\10\build\WindowsDriver.common.targets(314,5): error MSB4044: The "Validate
       NTTargetVersion" task was not given a value for the required parameter "ValidNTTargetVersions". [C:\xdp-for-wind
       ows\src\xdpetw\xdpetw.vcxproj]
  0 Warning(s)
    2 Error(s)

Several test failures with ` Timer.IsExpired() not false` on 2019 functional test run

@mtfriesen it seems like we've got several test failures with Timer.IsExpired() not false on the 2019 functional test run. Is there an open bug for this already?

Originally posted by @nibanks in #82 (comment)

improve logging experience and documentation

Make it simpler to perform common logging actions and add documentation.

Compiling issue: xdpetw.h no such file

hi, all,

not sure how to fix this. There is no such .h xdpetw.h in the repo.

1>C:\cygwin\home\marku\xdp-for-windows.git\src\xdplwf\precomp.h(35,10): fatal error C1083: Cannot open include file: 'xdpetw.h': No such file or directory (compiling source file generic.c)
1>C:\cygwin\home\marku\xdp-for-windows.git\src\xdplwf\precomp.h(35,10): fatal error C1083: Cannot open include file: 'xdpetw.h': No
...

Build started...
1>------ Build started: Project: xdplwf, Configuration: Release x64 ------
1>Building 'xdplwf' with toolset 'WindowsKernelModeDriver10.0' and the 'Universal' target platform.
1>bind.c
1>dispatch.c
1>ec.c
1>generic.c
1>native.c
1>offload.c
1>oid.c
1>recv.c
1>rss.c
1>send.c
1>C:\cygwin\home\marku\xdp-for-windows.git\src\xdplwf\precomp.h(35,10): fatal error C1083: Cannot open include file: 'xdpetw.h': No such file or directory (compiling source file generic.c)
1>C:\cygwin\home\marku\xdp-for-windows.git\src\xdplwf\precomp.h(35,10): fatal error C1083: Cannot open include file: 'xdpetw.h': No such file or directory (compiling source file bind.c)
1>C:\cygwin\home\marku\xdp-for-windows.git\src\xdplwf\precomp.h(35,10): fatal error C1083: Cannot open include file: 'xdpetw.h': No such file or directory (compiling source file native.c)
1>C:\cygwin\home\marku\xdp-for-windows.git\src\xdplwf\precomp.h(35,10): fatal error C1083: Cannot open include file: 'xdpetw.h': No such file or directory (compiling source file ec.c)
1>C:\cygwin\home\marku\xdp-for-windows.git\src\xdplwf\precomp.h(35,10): fatal error C1083: Cannot open include file: 'xdpetw.h': No such file or directory (compiling source file rss.c)
1>C:\cygwin\home\marku\xdp-for-windows.git\src\xdplwf\precomp.h(35,10): fatal error C1083: Cannot open include file: 'xdpetw.h': No such file or directory (compiling source file recv.c)
1>C:\cygwin\home\marku\xdp-for-windows.git\src\xdplwf\precomp.h(35,10): fatal error C1083: Cannot open include file: 'xdpetw.h': No such file or directory (compiling source file oid.c)
1>C:\cygwin\home\marku\xdp-for-windows.git\src\xdplwf\precomp.h(35,10): fatal error C1083: Cannot open include file: 'xdpetw.h': No such file or directory (compiling source file offload.c)
1>C:\cygwin\home\marku\xdp-for-windows.git\src\xdplwf\precomp.h(35,10): fatal error C1083: Cannot open include file: 'xdpetw.h': No such file or directory (compiling source file dispatch.c)
1>C:\cygwin\home\marku\xdp-for-windows.git\src\xdplwf\precomp.h(35,10): fatal error C1083: Cannot open include file: 'xdpetw.h': No such file or directory (compiling source file send.c)
1>Done building project "xdplwf.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========

RssOffloadError test failure

The RSS offload error test fails due to a race condition with a previous test's interface restart: the upper protocol hasn't established the RSS configuration prior to the test attempting to partially override the settings, which is not supported yet.

XdpRssSet(InterfaceHandle.get(), RssConfig.get(), RssConfigSize) failed, 0x80070015
at StopTest() in D:\a\1\s\test\functional\taef\tests.cpp:line 23
at OffloadRssError() in D:\a\1\s\test\functional\lib\tests.cpp:line 4266

Handle offload initialization race

This race condition applies to all offloads.

XdpGenericRssInitialize:
//
// Attempt to query the indirection table.
//
// TODO: how do we avoid a race with the protocol driver plumbing RSS? The
// OID might already be in flight before this LWF has a chance to inspect it
// and might still be in flight after we query NDIS.
//

Cross-Platform AF_XDP API or Not?

Currently, the FAQ makes the following statement:

It is not directly source-compatible with the Linux API, though it is possible to build a thin shim layer to abstract any differences.

This begs the following questions:

How important is cross-platform AF_XDP API usage?

Do we expect that a single application, built for multiple platforms, would want to support XDP on both? My personal take is Yes, applications that support XDP on one platform will most likely want to leverage it on another if possible.

If cross-platform support is important, should XDP for Windows have exactly the same API as on Linux?

Assuming cross-platform support is an eventual goal, should XDP for Windows be constrained to what Linux has/does or should we instead propose a shim on top of both? I'm not sure.

Should XDP for Windows have the sole responsibility for driving this cross-platform support?

Whether it is decided to match APIs exactly (or perhaps close enough that only minimal differences exist) or a separate shim layer is written, I believe more input will be needed from the community. If the shim approach is taken, it doesn't really make sense for either platform to explicitly own the shim, but should be a combined ownership, or completely separate.

Query RSS processor limit on newer Windows builds

Found in https://dev.azure.com/mscodehub/WindowsXDP/_git/xdp/pullrequest/19220

Add support for querying NdisGetRssProcessorInformation from LWFs

XdpGenericRssInitialize:
//
// RSS allows more processors than receive queues, but the RSS
// implementation in TCPIP constrains the number of processors to the
// number of receive queues. The NDIS API to query the actual maximum
// processor count is unavailable to LWFs, so simply use the number of
// receive queues instead.
//
// TODO: can we query the max processor count some other way? (It can be
// queried via miniport and protocol driver handles, and WMI.)
//

API usage triggers driver demand start/stop

Start the XDP driver only if apps are using it.

Add scalable poke+wait multi socket API

A scalable (good performance w/ lots idle sockets) poke+wait multi socket API may be useful. Should be based on the ProcessSocketNotifications API and similarly support IOCPs. This may also support poking multiple sockets in one syscall.

Create Dev Kit job fails for PRs

Example: https://github.com/microsoft/xdp-for-windows/runs/6510474185?check_suite_focus=true#step:4:5

Write-Error: D:\a\xdp-for-windows\xdp-for-windows\tools\common.ps1:27
Line |
  27 |              $BranchName = Get-CurrentBranch
     |                            ~~~~~~~~~~~~~~~~~
     | Failed to get branch from git

Error: Process completed with exit code 1.

Relationship to xdp-tools

https://github.com/xdp-project/xdp-tools is the main XDP library/api/tools for using XDP.

In the eBPF for Windows project, we've been enabling the same bpftool source (with appropriate refactoring/ifdefs/etc) to run on eBPF for Windows, and have been working towards parity with the libbpf.h / bpf.h header files in the libbpf project (and have a github issue for stronger integration).

Hence filing this issue to track integration with the xdp-tools project.

Test Failure: MpRxFlush(GenericMp) failed, 0x80070005

https://github.com/microsoft/xdp-for-windows/runs/7929096731?check_suite_focus=true

MpRxFlush(GenericMp) failed, 0x80070005
  Failed GenericXskWaitAsyncTx [23 s]
  Error Message:
   Assert failed. Stop test execution.
  Stack Trace:
  	at StopTest() in D:\a\xdp-for-windows\xdp-for-windows\test\functional\taef\tests.cpp:line 23
	at <lambda_6d57901003d55c3814e1a57279ad254b>::operator() in D:\a\xdp-for-windows\xdp-for-windows\test\functional\lib\tests.cpp:line 3582
	at GenericXskWaitAsync() in D:\a\xdp-for-windows\xdp-for-windows\test\functional\lib\tests.cpp:line 3620
  Standard Output Messages:
 MpRxFlush(GenericMp) failed, 0x80070005

Implement layout offload

Design, implement, and test frame layout metadata.

XDP performance counters

Expose perf/diagnostics counters. Query using XDP APIs and/or expose through Windows perf counter framework?

AF_XDP versioning

Figure out a versioning story for AF_XDP and prepare for V1.

Support non-Admin Apps

In order to get more wide-spread usage of XDP for Windows, we should see if it's possible for a non-Admin app to use this interface. It's really a pretty big security issue if an app that upgrades to using XDP now has to run as an admin. What if it has a bug, and it's exploitable over the network? Now the attacker gets admin access. FWIW, I thought about this while reading an article about the large number of cloud attacks/bugs that have happened over the last few months, and I don't want to introduce any new issues here.

Implement in-order TX completion at AF_XDP layer

XDP supports in-order TX completion at the NIC layer. Add this optimization at the AF_XDP layer.

Implement USO offload

Design, implement, and test USO offload.

SpinXsk Failure in CI (failed with -2147483645)

Recent CI run on main failed: https://github.com/microsoft/xdp-for-windows/runs/6636837015?check_suite_focus=true

Exception: D:\a\xdp-for-windows\xdp-for-windows\tools\spinxsk.ps1:148
Line |
 148 |              throw "SpinXsk failed with $LastExitCode"
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | SpinXsk failed with -2147483645

I'm not positive, but that error (0x80000003) might be E_INVALIDARG.

Logs: https://github.com/microsoft/xdp-for-windows/suites/6696645318/artifacts/254375706

Internal OneBranch Builds don't Like XDP Kit Test/Tool Binaries

A recent automated PR for MsQuic on our internal mirror exposed the following build errors:

##[error]1. BinSkim Error BA2007 - File: artifacts/bin/xdp/pktcmd.exe.  
  Tool: BinSkim: Rule: BA2007 (EnableCriticalCompilerWarnings). https://github.com/microsoft/binskim/blob/main/docs/BinSkimRules.md#rule-BA2007EnableCriticalCompilerWarnings
  'pktcmd.exe' was compiled at too low a warning level (effective warning level 1 for one or more modules). Warning level 3 enables important static analysis in the compiler to flag bugs that can lead to memory corruption, information disclosure, or double-free vulnerabilities. To resolve this issue, compile at warning level 3 or higher by supplying /W3, /W4, or /Wall to the compiler, and resolve the warnings emitted. An example compiler command line triggering this check: -c -ID:\a\1\s\test\pkthlp -ID:\a\1\s\published\external -Zi -nologo -W1 -WX- -diagnostics:column -O2 -GL -DCODE_ANALYSIS -DUSER_MODE=1 -DPOOL_NX_OPTIN_AUTO=1 -DPOOL_ZERO_DOWN_LEVEL_SUPPORT=1 -DFNDIS=1 -Gm- -EHs -EHc -MT -GS -fp:precise -Zc:wchar_t -Zc:forScope -Zc:inline -external:W1 -Gd -TC -wd26812 -analyze:projectdirectoryD:\a\1\s\test\pktcmd -analyze:rulesetdirectory"D:\a\1\s\test\pktcmd\;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Team Tools\Static Analysis Tools\Rule Sets;" -analyze:ruleset"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Team Tools\Static Analysis Tools\Rule Sets\NativeRecommendedRules.ruleset" -analyze:quiet -analyze:plugin"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\EspXEngine.dll" -FC -errorreport:queue -I"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include" -I"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\atlmfc\include" -I"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\VS\include" -I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um" -I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\shared" -I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\winrt" -I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\cppwinrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\Include\um" -external:I"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include" -external:I"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\atlmfc\include" -external:I"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\VS\include" -external:I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\ucrt" -external:I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um" -external:I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\shared" -external:I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\winrt" -external:I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\cppwinrt" -external:I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\Include\um" -X
Modules triggering this check: pktcmd.obj [warning level: 1]
pkthlp.obj (pkthlp_um.lib) [warning level: 1]

##[error]2. BinSkim Error BA2008 - File: artifacts/bin/xdp/pktcmd.exe.  
  Tool: BinSkim: Rule: BA2008 (EnableControlFlowGuard). https://github.com/microsoft/binskim/blob/main/docs/BinSkimRules.md#rule-BA2008EnableControlFlowGuard
  'pktcmd.exe' does not enable the control flow guard (CFG) mitigation. To resolve this issue, pass /guard:cf on both the compiler and linker command lines. Binaries also require the /DYNAMICBASE linker option in order to enable CFG.

##[error]3. BinSkim Error BA2007 - File: artifacts/bin/xdp/xskbench.exe.  
  Tool: BinSkim: Rule: BA2007 (EnableCriticalCompilerWarnings). https://github.com/microsoft/binskim/blob/main/docs/BinSkimRules.md#rule-BA2007EnableCriticalCompilerWarnings
  'xskbench.exe' was compiled at too low a warning level (effective warning level 1 for one or more modules). Warning level 3 enables important static analysis in the compiler to flag bugs that can lead to memory corruption, information disclosure, or double-free vulnerabilities. To resolve this issue, compile at warning level 3 or higher by supplying /W3, /W4, or /Wall to the compiler, and resolve the warnings emitted. An example compiler command line triggering this check: -c -ID:\a\1\s\published\external -Zi -nologo -W1 -WX- -diagnostics:column -O2 -GL -DCODE_ANALYSIS -DUSER_MODE=1 -DPOOL_NX_OPTIN_AUTO=1 -DPOOL_ZERO_DOWN_LEVEL_SUPPORT=1 -DFNDIS=1 -Gm- -EHs -EHc -MT -GS -fp:precise -Zc:wchar_t -Zc:forScope -Zc:inline -external:W1 -Gd -TC -wd26812 -analyze:projectdirectoryD:\a\1\s\test\xskbench -analyze:rulesetdirectory"D:\a\1\s\test\xskbench\;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Team Tools\Static Analysis Tools\Rule Sets;" -analyze:ruleset"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Team Tools\Static Analysis Tools\Rule Sets\NativeRecommendedRules.ruleset" -analyze:quiet -analyze:plugin"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\EspXEngine.dll" -FC -errorreport:queue -I"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include" -I"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\atlmfc\include" -I"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\VS\include" -I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um" -I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\shared" -I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\winrt" -I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\cppwinrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\Include\um" -external:I"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\include" -external:I"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\atlmfc\include" -external:I"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\VS\include" -external:I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\ucrt" -external:I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um" -external:I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\shared" -external:I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\winrt" -external:I"C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\cppwinrt" -external:I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\Include\um" -X
Modules triggering this check: xskbench.obj [warning level: 1]

##[error]4. BinSkim Error BA2008 - File: artifacts/bin/xdp/xskbench.exe.  
  Tool: BinSkim: Rule: BA2008 (EnableControlFlowGuard). https://github.com/microsoft/binskim/blob/main/docs/BinSkimRules.md#rule-BA2008EnableControlFlowGuard
  'xskbench.exe' does not enable the control flow guard (CFG) mitigation. To resolve this issue, pass /guard:cf on both the compiler and linker command lines. Binaries also require the /DYNAMICBASE linker option in order to enable CFG.

I see two possible solutions here:

1. Remove/separate these test binaries from the kit, so consumers that don't need these binaries don't have to worry about this.
2. Follow the instructions above and increase the warning level on the tools' builds.

P.S. @mtfriesen I'm not quite sure how you'd like to label this issue. Probably feature request if we go for option 1, or a bug if we go for option 2.

Usage doc doesn't list what version(s) of Windows the drivers will work on

Need to update doc to say what versions of Windows this works on.

Add support for promiscuous mode

Add control path to put the NIC into promiscuous RX mode.

Document how to Develop on top of XDP for Windows

Currently there is documentation on how to build and run the tests for XDP for Windows in development.md, but there is no documentation on how a new app might started consuming the AF_XDP APIs.

The best temporary solution I may offer for anyone looking for documentation is for them to take a look at the current usage in MsQuic as a guide.

Publish main CI build artifacts to prerelease

To enable rapid sharing and testing of main builds, publish devkit and runtime artifacts to a prerelease release.

Add AF_XDP wait cancellation

Currently there is no API for AF_XDP wait cancellation - only thread termination will reliably abort a wait. Apps can also close the handle, but that introduces a race condition between the wait thread and closehandle thread.

Consider a separate IOCTL that cancels the current/next wait on the socket.

Add profile guided optimization (PGO)

Add PGO support.

XSK: reliably release polling contexts when interface is detached

XSK is (or was?) missing robust handling of polling handles in the interface removal code path.

Add openssf best practices badge

https://bestpractices.coreinfrastructure.org/en can be used to get a badge for display on the main page.

Add loopback TX

A partner requested the ability to loop back TX onto an RX queue.

Add L2FWD kernel XDP program action for RX_ACTION_TX testing

Wire up a dummy RX_ACTION_TX action.

Implement checksum offload

Design, implement, and test checksum offload.

Add latency, CPU usage, throughput measurements

• Measure L2FWD latency
• CPU usage

Request for Binaries

There are users reporting build issues.

Q: Can you guys provide binaries ?

Integrate Microsoft-internal CI

We must have test coverage with driver verifier, and we should have coverage prerelease Windows builds. The internal CI provides that coverage but needs to be wired up to PR (pre-merge) gates.

Generic XDP coexistence with NBL offloads

Generic XDP needs to be compatible with:

RSC offload - send OID to disable when non-RSC-aware programs/XSKs are attached
Checksum offload - fix up checksum header fields when *ChecksumValueInvalid bits are set in OOB.

Replace Existing Program/Rule Engine with eBPF

Currently, XDP for Windows has a custom set of rules that can make up a "program" that an app pushes down to configure how packets should be handled. This custom logic isn't super performant, nor is it very configurable. This is a great opportunity to partner with https://github.com/microsoft/ebpf-for-windows and integrate with that instead of the custom logic currently supported.

Add asynchronous IO completion API

We have task #33 tracking a PSN/epoll/kqueue style API, but we can't do that without Windows publishing IOCP kernel APIs. As an interim step, add support for an asynchronous version of XskNotifySocket.

Tool to test RX actions

We need a simple tool that allows users to filter traffic and apply RX actions.

RSS settings sync

When XDP transitions from independently managing RSS for each edge to letting upper layer components manage the miniport RSS (RemoveLowerEdgeRssSetting), XDP must perform 2 actions:

1. Plumb the upper edge settings down to the miniport

2. Update the RSS indirection table managed by the generic rss module (XdpGenericRssUpdateIndirection)

Both of these code paths are prone to allocation failures, which could cause 2 types of issues:

A) Action 1 succeeds and action 2 fails, so the miniport and the generic module go out of sync regarding their RSS settings

B) Action 1 fails, so upper layer components and the miniport remain out of sync regarding their RSS settings

The impact of issue A is a broken generic XDP datapath. To solve this issue, one possibility is to eliminate the allocations done in XdpGenericRssUpdateIndirection. We know the maximum number of queues and maximum indirection table size as returned from OID_GEN_RECEIVE_SCALE_CAPABILITIES.

The impact of issue B is that the upper layer datapaths may be broken or degraded. This issue is harder to solve, as we don't have control over failures in the OID path. We need to add retry logic as a best-effort measure to avoid this issue.

Define Linux compatibility goals

How do we support Linux apps? Lowest common denominator of Windows/Linux features? If Windows XDP adds a new feature, can we avoid divergence if Linux adds a similar feature later?

spinxsk stress tests unreliable in longhaul CI

We're seeing machines lose connectivity in the 24-hour CI, as well as a handful of failures during shorter main and PR tests. Investigate.

Test multi-processor-group configurations

We should consider using bcdedit to force multiple processor groups in our functional and stress tests

e.g. bcdedit.exe /set groupsize 2 and reboot

Since TCPIP doesn't support multi-group very well <=WS2022, we should probably start with that setting only on the MsQuic pool, and they already run a pre-test provisioning script that e.g. enables testsigning.

Please work with NickB to add groupsize to corenet-ci - it should help with their QUIC tests, too.

We can explore setting groupsize in the WS2016,2019,2022 pools but the ROI is lower. The easy part is making the setting change (we also have pre-boot provisioning) but we'd have to adjust test logic or upsize our VMs to have more vCPUs since we still need to party on one group due to TCPIP, and hyperthreading, and ...

Logs Don't Include Test Events

The logs that are collected from test failures should include at least a minimal number of events from the test code, such as test failures. I think the easiest way to achieve this is to update LogTestFailure and LogTestWarning to write the same console output to ETW/WPP.

Without this, the unfamiliar (me) cannot figure out where to look for failures. For instance, in this test failure:

XskNotifySocket(Xsk.Handle.get(), NotifyFlags, WaitTimeoutMs, &NotifyResult) not equal to HRESULT_FROM_WIN32(ERROR_TIMEOUT)
  Failed GenericXskWaitRxTx [1 s]
  Error Message:
   Assert failed. Stop test execution.
  Stack Trace:
  	at StopTest() in D:\a\1\s\test\functional\taef\tests.cpp:line 24
	at GenericXskWait() in D:\a\1\s\test\functional\lib\tests.cpp:line 3551
	at xdpfunctionaltests::GenericXskWaitRxTx() in D:\a\1\s\test\functional\taef\tests.cpp:line 217
  Standard Output Messages:
 XskNotifySocket(Xsk.Handle.get(), NotifyFlags, WaitTimeoutMs, &NotifyResult) not equal to HRESULT_FROM_WIN32(ERROR_TIMEOUT)

I downloaded the logs (xdpfunc.txt) but cannot figure out where to find what the actual return of XskNotifySocket is, let alone why it was that.

Add API to tear down XDP queue from interface

For NetVSC, interfaces will need to dynamically close queues. Add an XDP callback to initiate queue teardown.

Partially implemented: see XdpTxQueueNotify.

XDPMP undercounts RX activity when native XDP isn't bound

We don't increment RX frames and RX bytes when non-native-XDP-inspected indications occur.

spinxsk stress tests unreliable in GitHub Actions

We're seeing a variety of CI stress failures in GitHub - watchdogs are firing and machines are losing connectivity. Is this an XDP bug or are the VMs undersized? We need 4 vCPUs for the default spinxsk configuration IIRC.