Dial from a port we're listening on about go-libp2p-quic-transport

[marten-seemann]

I’m not exactly sure how the reuseport-transport works, but I think there’s an even better solution with QUIC. Since QUIC connections are identified by a connection ID, we can use a single port for all outgoing connections.

I’ve been working on a set of changes to make this possible in quic-go (quic-go/quic-go#1324), and I’m expecting to merge these next week or so.

[Stebalien]

Nice!

So, my point about reuseport-transport is that it can handle the case where we're listening on multiple ports. That is, users may listen on localhost:123 and my_public_addr:345. We'd want to use port 345 when dialing out. reuseport-transport has a bunch of logic for picking a reasonable source port based on the destination address.

[marten-seemann]

Ok, that makes sense.
We should use that logic once quic-go supports running a listener and a dialer on the same packet conn (quic-go/quic-go#561). It's a kind of complicated issue, so I don't expect to fix it within the next few weeks.

[Stebalien]

Thinking through this a bit, the approach we're currently using in go-reuseport-transport won't work. If a libp2p node is configured to listen on some unicast IP X and then tries to dial a unicast IP Y, that may fail with both #34 and #44 as there may be no route from X to Y. go-reuseport-transport works around this by:

1. Picking the best port it can by looking at address types.
2. Binding to 0.0.0.0:PORT instead of X:PORT.
3. Dialing.

The trick is dialing from 0.0.0.0 instead of X. Unfortunately, we can't reuse this same trick for QUIC because we don't want to open a new file descriptor bound to 0.0.0.0.

---

The only solution I can think of is to do this the right way and use netlink. See: libp2p/go-reuseport-transport#2

A partial solution would be to reuse the source port if and only if we're listening on 0.0.0.0.

[Stebalien]

It may also be possible to use /proc/net/route along with the list of interfaces. Unfortunately, netlink will only work on Linux.

On windows, we'll have to use the GetBestInterface function (from the windows API), lookup that interface with go, and then figure out if we're listening on any of those interface's addresses.

[cannium]

If a libp2p node is configured to listen on some unicast IP X and then tries to dial a unicast IP Y, that may fail with both #34 and #44 as there may be no route from X to Y.

Could you elaborate it a bit? In my knowledge the routing table is system-wide despite network namespace, I don't understand why there's no route from X to Y when a route between 0.0.0.0 and Y exists.

[Stebalien]

IP address X is actually bound to a specific interface. There may be no route through that interface to IP address Y.

For example, X could be 10.1.2.3 bound to some VPN interface depending on the VPN's configuration, traffic entering that VPN may not be able to exit it.

[marten-seemann]

Related: quic-go/quic-go#1736

[Stebalien]

@marten-seemann is that actually an issue? That is, does QUIC care about the source IP address?

[marten-seemann]

It might cause some problems during the handshake, since migration is only allowed after the handshake. After the handshake, this would just lead to a migration to the new IP address, so as long as the host is also reachable under the new address, things should be fine. All of this requires the mapping to be stable though, if the kernel keeps choosing source IPs by random, things would get very messy.

[Stebalien]

I see. Unfortunately, the only clean solution I can think of is:

1. Always use reuseport when listening on 0.0.0.0.
2. When responding to a packet received on IP X, open a socket with a source port X (if not already open) and use that instead of 0.0.0.0.

[hackman]

so would you consider the proposed solution by @ironsteel with IP_PKTINFO ?

[Stebalien]

so would you consider the proposed solution by @ironsteel with IP_PKTINFO ?

That looks like it should work but it also looks harder to implement given the current functions exposed by the standard library.

[hackman]

How would you like it to be implemented? We can make a pull request with the actual implementation.

[lnykww]

is there any update on this issue?

[Stebalien]

@hackman sorry, missed the notification. Looking at @ironsteel's proposal, that actually looks correct. I just didn't think that would be possible without dropping down to manual syscalls (which don't play well with golang's networking stack).

@lnykww Status is in quic-go/quic-go#1736.

[lnykww]

@Stebalien
The Issue(quic-go/quic-go#1736) is to resolve the source address selection for response. but our problem is the source address selection for dial. Can the issue really resolve our problem?

And Can we use the logic of go-reuseport-transport to solve this problem?

[Stebalien]

@lnykww you're right. Sorry, I haden't fully paged this issue back in. The best solution I can think of for this issue is the netlink approach.

And no, there haven't been any updates.

[lnykww]

@Stebalien @marten-seemann
Does QUIC can work If accept new connection via Dial socket?
For Example:
If we have a machine with ips 4.4.4.4 and 192.168.1.2. And has quic-transport listen on socket1 with address (0.0.0.0:3344).
We want dial 5.5.5.5:3344，then we use netlink to choose the source address and we get source address (4.4.4.4). then we listen on socket2 with address (4.4.4.4:3344) and send data via socket2. This is no problem.
But For kernel, if packet send to 4.4.4.4:3344, this packet will deliver to socket2 rather than socket1. For example, 6.6.6.6:3344 send packet to 4.4.4.4:3344 for a new QUIC connection, this packet will deliver to socket2. But QUIC listen on socket1 rather than socket2. I am not sure if QUIC will work properly under such circumstances.

[Stebalien]

Without reuseport, we wouldn't even be able to bind to 4.4.4.4. With reuseport, well, I'm not sure.

Really, that shouldn't be an issue here. Here, we can:

1. Check the expected source address using netlink.
2. If we're listening on that address, use that socket.
3. If we're not, check to see if we have any open 0.0.0.0 listeners. If we do, use one of them.
4. Otherwise, open (or use an existing) a "dial only" socket on a random port.