Tracking down netlink bugs about babeld-hacking

[dtaht]

So the ideal behavior is like this - for ipv4. Looking at it in ip mon, it is working correctly.

#!/bin/sh

IFACE=eno1

ip route replace unreachable 172.26.0.0/16 metric 128
ip route replace 172.26.0.0/16 metric 128 dev $IFACE

ip route replace unreachable 172.26.0.0/16 metric 128
ip route add unreachable 172.26.0.0/16 metric 128

RTNETLINK answers: File exists - expected

ip route replace unreachable 172.26.0.0/16 metric 128
ip route replace 172.26.0.0/16 metric 128 dev $IFACE
ip route del 172.26.0.0/16 metric 128 dev $IFACE
echo proto tests
ip route replace unreachable 172.26.0.0/16 metric 128 proto 52
ip route replace 172.26.0.0/16 metric 128 dev $IFACE proto 52
ip route replace unreachable 172.26.0.0/16 metric 128 proto 52
ip route del 172.26.0.0/16 metric 128 dev $IFACE

RTNETLINK answers: No such process: expected

ip route replace 172.26.0.0/16 metric 128 dev $IFACE proto 52
ip route del 172.26.0.0/16 metric 128 dev $IFACE

[dtaht]

The moral equivalent of this for ipv6 does report an error for the unreachable insert, but atomic replaces DO work according to ip mon

#!/bin/sh
echo nexthop test

IFACE=eno1
NEXTHOP="fe80::225:90ff:fec1:6252 dev eno1"
DEVICE="" # dev eno1"
#fe80::1
ROUTE=fc::/64

ip -6 route replace unreachable $ROUTE metric 128 $DEVICE
ip -6 route replace $ROUTE metric 128 via $NEXTHOP

ip -6 route replace unreachable $ROUTE metric 128 $DEVICE
ip -6 route add unreachable $ROUTE metric 128
# RTNETLINK answers: File exists - expected
ip -6 route replace unreachable $ROUTE metric 128 $DEVICE
ip -6 route replace $ROUTE metric 128 via $NEXTHOP
ip -6 route del $ROUTE metric 128 via $NEXTHOP
echo proto tests
ip -6 route replace unreachable $ROUTE metric 128 $DEVICE proto 52
ip -6 route replace $ROUTE metric 128 via $NEXTHOP proto 52
ip -6 route replace unreachable $ROUTE metric 128 $DEVICE proto 52
ip -6 route del $ROUTE metric 128 via $NEXTHOP
 # RTNETLINK answers: No such process: expected
ip -6 route replace $ROUTE metric 128 via $NEXTHOP proto 52
ip -6 route del $ROUTE metric 128 via $NEXTHOP

[dtaht]

so: always with a nexthop on ipv6, you need to specify both the device and the fe80 address for a route, but with unreach you do not. However you can indeed keep doing just replace" and it works... even if you specify the nexthop or not....

I have no idea what the error is for here, but:

unreachable fc::/64 dev lo metric 128 error 4294967183 pref medium
fc::/64 via fe80::225:90ff:fec1:6252 dev eno1 metric 128 pref medium
unreachable fc::/64 dev lo metric 128 error 4294967183 pref medium
unreachable fc::/64 dev lo metric 128 error 4294967183 pref medium
fc::/64 via fe80::225:90ff:fec1:6252 dev eno1 metric 128 pref medium
Deleted fc::/64 via fe80::225:90ff:fec1:6252 dev eno1 metric 128 pref medium
unreachable fc::/64 dev lo proto 52 metric 128 error 4294967183 pref medium
fc::/64 via fe80::225:90ff:fec1:6252 dev eno1 proto 52 metric 128 pref medium
unreachable fc::/64 dev lo proto 52 metric 128 error 4294967183 pref medium
Deleted fc::/64 via fe80::225:90ff:fec1:6252 dev eno1 proto 52 metric 128 pref medium
fc::/64 via fe80::225:90ff:fec1:6252 dev eno1 proto 52 metric 128 pref medium
Deleted fc::/64 via fe80::225:90ff:fec1:6252 dev eno1 proto 52 metric 128 pref medium

[dtaht]

ok, now in poking through the babel code I really need to sit down and decode the netlink that iproute2 generates. I need to also test via iproute these tests for source specific route replacement.

rtm->rtm_scope = RT_SCOPE_UNIVERSE;
if(metric < KERNEL_INFINITY) {
    rtm->rtm_type = RTN_UNICAST;
    rtm->rtm_flags |= RTNH_F_ONLINK;

[dtaht]

From watching babel do it's thing, it doesn't do a modify sanely in the first place. This version of my code does a staged replace

72.22.0.1 dev eno1 lladdr e2:91:f5:be:a3:53 REACHABLE
2603:3024:1536:86f0::/60 via fe80::f6f2:6dff:feb6:a01c dev eno1 proto babel metric 1 pref medium
Deleted 2603:3024:1536:86f0::/60 via fe80::46d9:e7ff:fe93:822e dev eno1 proto babel metric 1024 pref medium
2603:3024:1536:86f0::/60 via fe80::f6f2:6dff:feb6:a01c dev eno1 proto babel metric 1024 pref medium
Deleted 2603:3024:1536:86f0::/60 via fe80::f6f2:6dff:feb6:a01c dev eno1 proto babel metric 1 pref medium
default from 2603:3024:1536:86f0::/60 via fe80::f6f2:6dff:feb6:a01c dev eno1 proto babel metric 1 pref medium
Deleted default from 2603:3024:1536:86f0::/60 via fe80::46d9:e7ff:fe93:822e dev eno1 proto babel metric 1024 pref medium
default from 2603:3024:1536:86f0::/60 via fe80::f6f2:6dff:feb6:a01c dev eno1 proto babel metric 1024 pref medium
Deleted default from 2603:3024:1536:86f0::/60 via fe80::f6f2:6dff:feb6:a01c dev eno1 proto babel metric 1 pref medium
default from 2603:3024:1536:8600:f6f2:6dff:feb6:a01d via fe80::f6f2:6dff:feb6:a01c dev eno1 proto babel metric 1 pref medium
Deleted default from 2603:3024:1536:8600:f6f2:6dff:feb6:a01d via fe80::46d9:e7ff:fe93:822e dev eno1 proto babel metric 1024 pref medium
default from 2603:3024:1536:8600:f6f2:6dff:feb6:a01d via fe80::f6f2:6dff:feb6:a01c dev eno1 proto babel metric 1024 pref medium
Deleted default from 2603:3024:1536:8600:f6f2:6dff:feb6:a01d via fe80::f6f2:6dff:feb6:a01c dev eno1 proto babel metric 1 pref medium
default via 172.22.0.2 dev eno1 proto babel metric 1 onlink
Deleted default via 172.22.0.2 dev eno1 proto babel metric 1 onlink
fd89:cb20:8854::/48 via fe80::46d9:e7ff:fe93:822e dev eno1 proto babel metric 1 pref medium
Deleted fd89:cb20:8854::/48 via fe80::230:18ff:fec9:de9c dev eno1 proto babel metric 1024 pref medium

[dtaht]

So the core insight here, is that babel issues a deletion first then attempts to insert the unreachable route, instead of just making it unreachable. Time to look further up the stack.

Deleted fdb8:92ec:da7d::1 via fe80::f6f2:6dff:feb6:a01c dev eno1 proto babel metric 1024 pref medium
unreachable fdb8:92ec:da7d::1 dev lo proto babel metric 1024 error 4294967183 pref medium

[christf]

yes. there even is a comment about that in the code in kernel_socket.c in kernel_route() - line 454. The CHANGES file reveals that in 2012/2013 atomic updates were disabled in BSD / OSX. It seems worthwile to figure out whether the reasons for these changes are still around.

[dtaht]

well, the above note was that a ROUTE_MODIFY was not even being issued for the above inefficiency.

It really is my hope that whatever bug existed back then has long since been squashed. I'm sort of running now in a branch (atomic) that is sort of running better, but was focused on some other stuff that I just posted on the mailing list. I am encouraged though about what progress I could make with a week's worth of focus on the various nagging problems.

[dtaht]

bird's code:

/*

• We could use NL_OP_REPLACE, but route replace on Linux has some problems:
• 1. Does not check for matching rtm_protocol
• 2. Has broken semantics for IPv6 ECMP
• 3. Crashes some kernel version when used for IPv6 ECMP
• So we use NL_OP_DELETE and then NL_OP_ADD. We also do not trust the old
• route value, so we do not try to optimize IPv6 ECMP reconfigurations.
  */

[dtaht]

But I swear olsr and quagga now do do atomic updates. And what I have is working halfway decently, except that it's higher up in the babel call stack that's not doing the route_modify in the right place....