flag error: -l is required but that's not documented about netboot

[danderson]

Wow, talk about an embarrassing mistake :(. Thanks for the report!

[arnehormann]

Anytime! I also got some problems with the cli. Not on it right now, but from memory, netstat says it's not listening on :67 for the api subcommand and I had to use the compatibility mode. Something in the new dhcp4 package maybe - or another unset default.
Still, it's booting CoreOS for BIOS and UEFI on three systems since Friday with the press of a single button. That's almost all I want, so thanks again!

[danderson]

Is this on linux? The missing :67 in netstat could be normal. Did booting fail in that mode? If so, I'd be interested in the output of pixiecore debug tcpdump <interface> <capture filename> (just launches tcpdump with appropriate filter).

On linux, Pixiecore will use raw sockets and kernel packet filtering to snoop DHCP packets, without having to bind to port 67. This means Pixiecore can coexist on the same machine as a regular DHCP server. The downside is that this listening mode is invisible to userspace, because it's using raw sockets.

I've been meaning to make that mode an explicit opt-in (currently, on linux, it tries the raw+filter mode first, and falls back to binding if that fails), because it's quite confusing if you're not expecting it.

[danderson]

Submitted f061a78 to not do the weird bind() bypass thing by default on linux, so the behavior is less surprising.

That said, could you please test using pixiecore boot or pixiecore api with --dhcp-no-bind in this new version? On linux, that should definitely work, so if booting fails, please record a pcap with pixiecore debug tcpdump <interface> <pcap filename> and open a new bug with that pcap attached, so I can try to figure out what's up? Thanks!

[arnehormann]

Yes, it was Linux. And I only checked with netstat when I couldn't boot in any other mode (until I found compat mode in the sources).
I ran dhcpdump, it showed no packet from pixiecore, only our primary dhcp server. There was also nothing on stdout / stderr, though I entered the debug flag (neither for a BIOS nor for a regular boot). I have to say my server configuration may be weird, though.
I got em1 and an em1:1 alias to a different ip/subnet.
Concerning pixiecore debug tcpdump ..., is that an additional process or boot or api mode? I can check myself, but you'll be able to answer that faster.

I just read through the dhcp4 sources again; Maybe here https://github.com/google/netboot/blob/master/dhcp4/conn_linux.go#L83 s/17/67/?

[danderson]

So, the compat mode doesn't really affect any of the core logic of Pixiecore, it just configures the server using different flags, but the boot logic is the same. This is why I'm confused, because afaict switching to the legacy CLI arguments should make no difference.

The line you pointed out in conn_linux.go is correct - that's creating a raw socket that listens for IP protocol 17, i.e. all UDP traffic. Then it attaches a BPF in-kernel filter to only keep UDP traffic to port 67. In any case, if you build from the latest code, that codepath should be disabled. Let's first check if you can boot without that weird codepath.

To troubleshoot more, I need need a boot attempt with a bunch of logging and packet captures enabled. Could you build from the latest source, and run these 2 commands (at the same time):

sudo pixiecore boot -t -d https://alpha.release.core-os.net/amd64-usr/current/coreos_production_pxe.vmlinuz https://alpha.release.core-os.net/amd64-usr/current/coreos_production_pxe_image.cpio.gz --cmdline='coreos.autologin'

sudo pixiecore debug tcpdump em1 bug.pcap

Then, start the machine you're trying to boot, and wait for it to fail/time out/whatever. Then, please give me both the output of the pixiecore boot command, and the bug.pcap file created by the pixiecore debug command (packet capture of DHCP, PXE and TFTP packets). Either in this bug, or mail them to [email protected] .

[arnehormann]

When I started in compat mode, port 67 was shown in netstat. In api mode,
it was not. I won't get to test it today, I need my notebook and didn't
bring it. Tomorrow...
Thanks for working on this!

Am 29.08.2016 08:52 schrieb "Dave Anderson" [email protected]:

Reopened #6 #6.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#6 (comment), or mute the
thread
https://github.com/notifications/unsubscribe-auth/AA9u1U5eGV5d4iUETMLEyo-32CYwILfaks5qkoGkgaJpZM4Jt3cS
.

[arnehormann]

Besides - I can't find any docs for the net colon notation in Listen...,
only in Dial. Is this an undocumented api?

Am 29.08.2016 09:17 schrieb "Arne Hormann" [email protected]:

When I started in compat mode, port 67 was shown in netstat. In api mode,
it was not. I won't get to test it today, I need my notebook and didn't
bring it. Tomorrow...
Thanks for working on this!

Am 29.08.2016 08:52 schrieb "Dave Anderson" [email protected]:

Reopened #6 #6.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#6 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA9u1U5eGV5d4iUETMLEyo-32CYwILfaks5qkoGkgaJpZM4Jt3cS
.

[danderson]

Nope, it's documented functionality, see: https://golang.org/pkg/net/#Listen

The exact meaning of foo:bar for different types of listener is more or less well documented (e.g. "ip:17" for "IP protocol 17 i.e. UDP"), but they are all official APIs.

[danderson]

And it's very strange that the old-style CLI flags made port 67 appear in netstat, there was nothing in the code to trigger that (until yesterday, when I disabled the fancy socket-bypass code by default). The fact that it listened on :67 suggests that it failed to set up the raw packet snooping socket, because the fallback was to just open a normal socket. Anyway, please test on the latest codebase and let me know how things work. It sounds like there's a bug hiding somewhere in the new DHCP code, it'd be great to find it :)

[arnehormann]

I don't see it, or at least I don't read it as you do. It only mentions
that the net argument must be stream oriented and mentions examples
without a colon. Then it talks about the address and directs to Dial for
that (the second argument).
Btw, should we close this issue (sorry for inadvertently hijacking it) and
move to a fresh one with a better subject line?

Am 30.08.2016 01:14 schrieb "Dave Anderson" [email protected]:

Nope, it's documented functionality, see: https://golang.org/pkg/net/#
Listen

The exact meaning of foo:bar for different types of listener is more or
less well documented (e.g. "ip:17" for "IP protocol 17 i.e. UDP"), but they
are all official APIs.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#6 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA9u1flBsUul6zUvwL0ZIZ9Nj9X0YQ9Zks5qk2fqgaJpZM4Jt3cS
.

[arnehormann]

Concerning the documentation, it looks to me like it works from reading the sources. Not from the docs, though.

I'll run the capture when I'm at the office.
For now, I got this (old version before your changes):

Running ./pixiecore api http://127.0.0.1:4242/ --port 70 --debug -l 0.0.0.0 --log-timestamps I get

$ netstat -putlan | grep pixiecore
tcp6       0      0 :::70                   :::*                    LISTEN      30330/pixiecore
udp6       0      0 :::4011                 :::*                                30330/pixiecore
udp6       0      0 :::69                   :::*                                30330/pixiecore

And running ./pixiecore -api=http://127.0.0.1:4242/ I get

$ netstat -putlan | grep pixiecore
tcp6       0      0 :::70                   :::*                    LISTEN      30342/pixiecore
udp        0      0 0.0.0.0:67              0.0.0.0:*                           30342/pixiecore
udp6       0      0 :::4011                 :::*                                30342/pixiecore
udp6       0      0 :::69                   :::*                                30342/pixiecore

Maybe it's also related to the usage of IPv6, IPv4 should be prefered according to the stdlib. I don't have a clue what's happening there - yet.

[arnehormann]

It's working in the new version. If anything else comes up, I'll open a new issue. I sent some capture files anyway...
Thanks again!