How I started working on network device drivers about blog

[emmericp]

I've also considered writing a driver for the Intel 82599 for similar reasons in around 2012 (?); but I was scared by the perceived complexity after I've found some documents on the proprietary version of DPDK.

DPDK was running on bare metal back then (and that still shows in parts of the API today). I was like "nope, don't want to write a whole OS". Turns out that approach is neither necessary nor useful and user space drivers are easy.

Anyways, thanks for Snabb! I've learned a lot about user space drivers and LuaJIT from your code :)

[jsnell]

"Some new requirements in the pipeline and some concerns about the future" is perhaps understating things a bit. By far the best opportunity we had in the sales pipeline needed 4 pairs of 1G copper cards + bypass, and we did not think they'd accept some Rube Goldberg network intergration multiple switches for one machine. So it was more like corporate life or death.

My recollection is that the timeline was something like:

• Mid-January you wrote the initial POC. Took about a week from first code to two NICs ping-ponging an initial packet that was manually inserted to the descriptor ring.
• End of January a slightly modified version of that POC was installed as an alternate IO backend in the product, but only capable of passing traffic through unmodified.
• End of February we had a fully productionized version with all kinds of niceties like zero-copy even for packets we buffered for indefinite amounts of time. (And even today not doing a zero copy on packet retention would cost that product about 20% performance; I know that for Snabb you can treat memory copies as basically free, but for whatever reason that's not the case here.)
• Sometime in March it was installed for a live trial with half the operator's traffic.

One reason it ended up being such a smooth process is that the stub kernel driver could still be used to support ifconfig, ethtool, libnl, etc. So all of the existing O&M infrastructure just worked with no changes. It was almost a classic data plane vs. control plane split, except with the roles inverted :) Exporting the NICs as normal kernel interfaces, just ones that do no traffic, is to my mind still the right way to do things.

And what about the life or death situation? Well, the sale happened, which was nice. But it turns out the customer had a habit of paying invoices months or years late. And we were on a strict runway: there was an exact date at which we'd need to start irrevocably winding down the company. In the end we were paid with a couple of weeks of runway left. If writing, integrating and testing those user-space drivers from scratch had taken any more than the ~6 weeks it did, it would not have been a happy ending.

So in retrospect that project was probably madness; the schedule was far too tight, we just didn't realize it and got lucky. And there was a lot of stupid crap over the years that could be traced to this decision, just like predicted. (Like the hilarious time we crashed all of a WiMax operator's base stations due to a MAC stripping bug in the userspace driver, or the mystery of the processes that ran at half speed if and only if they had direct /dev/mem mappings both above and below address 0x400000000, or depending on hardware features we really would have been better off without just because we could). But in all honesty the density of stupid crap was roughly the same even after that system was switched to using DPDK.

[lukego]

Nostalgia mode... I found this old correspondence from 2011:

@lukego: I reflected that it seems a bit odd to be so impressed with Myricom. as far as I know their hardware is no better than anybody else's and the only major difference is that they wrote a device driver in user-space. I've written some device drivers and they are not really that hard. Shouldn't proper hackers like us be able to whip up our own drivers for e.g. the DL380's Broadcom chips or Intel's 10G chips and get the same performance as Myricom?

@jsnell: Implementing our own fully userspace 10G driver seems like madness to me, but I could be overestimating the amount of stupid crap that you'll end up having to deal with :-)

Could be the jury is still out on this ;-)

[lukego]

@emmericp You're welcome! I love what you are doing with ixy now too :). I'd love a universe where we compete to reduce line count rather than to absorb new vendor code the fastest :).

I know you are billing it as educational but ixy looks like the beginnings of something that we in Snabb-land could contribute to and potentially use as a vehicle to share drivers with other projects. Our trouble with existing drivers isn't that they are written in C - that's no obstacle for us - but that they want to dictate our core data structures, go wild with feature creep, act as a vector for vendors to inject code into our applications without meaningful review, etc.

But we are not in a hurry to retire our built-in Lua drivers either and in these early days it may be best to have multiple "competing" approaches to feed each other with ideas.

[lukego]

@jsnell The D-I-Y pitch could be "Writing drivers is much less crazy than running thinly-capitalized startup companies, and people do that all the time." 😄

It's really hard to assess the buy-vs-build trade offs. Do the complex frameworks really get you off the hook with hard problems or does it just mean you will be debugging them in production with a large unfamiliar codebase? If it were the former then it would probably be worth holding your nose and drinking the kool-aid.

I reported one extremely nasty bug on DPDK that I reckon would have been beyond my capabilities to find in that gigantic codebase in a production setting. It was memory corruption in an external process triggered by a very rare race condition ultimately due to a missing memory barrier. The sort of thing that might happen once per month and could have been anywhere (e.g. in the vswitch, in QEMU, in the kernel inside a VM, etc.)

I spent an easter weekend tracking that down even in the tiny Snabb code base and with a controlled test environment. I wondered if using DPDK would have saved me that pain but, nope, it would have amplified it because they had exactly the same bug in a much more complex codebase.

I hope I saved somebody some grey hairs by reporting that one before it reached production :).

EDIT: I misremembered the details of that bug. It was a hang rather than a memory corruption. The principle is the same though :).

[lukego]

I was scared by the perceived complexity after I've found some documents on the proprietary version of DPDK.

This is such a huge barrier, right? People see complex driver code and say "wow, hardware is complex!" But the main source of the visible complexity is the software frameworks that the drivers have to interface with e.g. Linux kernel and DPDK. This is a little known secret and has been for decades already.

I was influenced by contributing to Openfirmware on OLPC. There you always have a simple Forth driver in the firmware (used mostly for hardware debugging) and a complex driver in Linux (used by general applications.) The firmware drivers were nice because you write them and then they are done. The kernel drivers looked more like an endless task of keeping in sync with upstream, debugging "is it hardware or is it software?" problems in a complex environment, etc.

The hardware was sometimes funny to program too :). My firmware audio driver needed to play the startup jingle when you press the power button. I wanted to do this without special interrupt handlers but the hardware could only loop a sample and not automatically stop when done. I couldn't poll because the CPU was needed for loading the kernel etc. I think that I ultimately grumbled about interrupt-centric hardware and did some polling for completion on a generic timer interrupt :).

[emmericp]

I didn't want to plug my projects in your thread. It's really only meant for education because that just allows us to just avoid implementing lots of boring stuff and keep the driver and example apps simple. Example apps like DPDK's l2fwd are just horrible if you only want to see how it works.
I'm also trying to find students to re-implement my ixgbe driver in some other languages; I'm currently thinking of OCaml (because of MirageOS), Rust, some JVM language and maybe Python.

Regarding Snabb drivers: I've wondered how Snabb would work with DPDK as a backend.
Yes, that sounds like a horrible abomination ;)
I've a student doing a bachelor's thesis implementing a proof of concept for a "DPDKDeviceApp" and wrappers for the intel apps. Goal is to run lwaftr with DPDK drivers without modifications to lwaftr, probably out of scope/too much work for the thesis.