Spark Error (lxd) about openr

[saifhhasan]

Can you give output for following commands ?

ip addr
breeze lm links --all

Also update issue title to describe Spark Error.

One more side question for my education. Does this error harming you from your testing in anyway ?

[bodleytunes]

`1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: gre0@NONE: mtu 1476 qdisc noop state DOWN group default qlen 1000
link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
4: ip_vti0@NONE: mtu 1332 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
5: zt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether 1e:97:fc:54:40:fd brd ff:ff:ff:ff:ff:ff
inet 10.55.0.10/24 brd 10.55.0.255 scope global zt0
valid_lft forever preferred_lft forever
inet6 fc7b:5e01:5e9f:7538:3f4::1/40 scope global
valid_lft forever preferred_lft forever
inet6 fe80::1c97:fcff:fe54:40fd/64 scope link
valid_lft forever preferred_lft forever
45: eth0@if46: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:77:22:ad brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.10.99.164/24 brd 10.10.99.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe77:22ad/64 scope link
valid_lft forever preferred_lft forever
root@vrf-zerotier:~# breeze lm links --all

Interface Status Overloaded Metric Override ifIndex Addresses

---

eth0 Up 45 10.10.99.164
fe80::216:3eff:fe77:22ad
`

[bodleytunes]

Also:

`#!/bin/bash

Copyright (c) 2014-present, Facebook, Inc.

This source code is licensed under the MIT license found in the

LICENSE file in the root directory of this source tree.

Ports 60000 - 60100 needs to be reserved in system so that they are always

free for openr to use. On linux you can do following to reserve.

> sysctl -w net.ipv4.ip_local_reserved_ports=60000-60100

Default OpenR configuration

Override the ones you need in /etc/sysconfig/openr for custom configuration

on the node

OpenR binary path or command name present on bin paths

OPENR=openr

Domain. Nodes will not form adjacencies to other nodes with different domain.

One node can be part of only one domain. Useful when want to run OpenR on

adjacent nodes but don't want to form adjacencies with them.

DOMAIN=wizznet.co.uk

List of comma separated list of prefixes to announce

e.g. "face:cafe::1/128,face:b00c::/64"

PREFIXES="10.10.0.0/16"

Used to assign elected address if prefix allocator is enabled

LOOPBACK_IFACE="lo,zt0,eth0"

Announce all global addresses of interfaces into the network

REDISTRIBUTE_IFACES="zt0,eth0"

dryrun => Do not program routes in dryrun mode

DRYRUN=false

Enable RTT metric on links. RTTs are computed dynamically and then used as

cost for links. If disabled then hop count will be used as a cost of path

ENABLE_RTT_METRIC=true

Enable v4

ENABLE_V4=true

Enable health-checker

ENABLE_HEALTH_CHECKER=yes
HEALTH_CHECKER_PING_INTERVAL_S=3

Interface prefixes to perform neighbor discovery on. All interfaces whose

names start with these are used for neighbor discovery

IFACE_PREFIXES="eth0"

Logging verbosity

VERBOSITY=1

PrefixAllocator parameter

ENABLE_PREFIX_ALLOC=false # Enable automatic election of prefixes for nodes
SEED_PREFIX="" # Master prefix to allocate subprefixes of nodes
ALLOC_PREFIX_LEN=128 # Length of allocated prefix
SET_LOOPBACK_ADDR=false # Assign choosen prefix for node on loopback
# with prefix length as /128 (and address being first
# in elected network block
OVERRIDE_LOOPBACK_ADDR=false # Overrides other existing loopback addresses on
# loopback interface

Spark Configuration

How long to keep adjacency without hearing hello from neighbor

SPARK_HOLD_TIME_S=30

How often to send hello packeks to neighbors

SPARK_KEEPALIVE_TIME_S=3

How fast to perform initial neighbor discovery

SPARK_FASTINIT_KEEPALIVE_TIME_MS=100

Enable in build Fib service handler

ENABLE_NETLINK_FIB_HANDLER=yes
FIB_HANDLER_PORT=60100

Enable in built System service handler

ENABLE_NETLINK_SYSTEM_HANDLER=true

set decision debounce time

DECISION_DEBOUNCE_MIN_MS=10
DECISION_DEBOUNCE_MAX_MS=250

enable performance measurements

ENABLE_PERF_MEASUREMENT=true
`

[bodleytunes]

Ah new messages if I put the zt0 and eth0 in :

Announce all global addresses of interfaces into the network

REDISTRIBUTE_IFACES="zt0,eth0"

`Interface Status Overloaded Metric Override ifIndex Addresses

---

eth0 Up 45 10.10.99.164
fe80::216:3eff:fe77:22ad
zt0 Up 5 10.55.0.10
fe80::1c97:fcff:fe54:40fd`

Nov 28 08:35:55 vrf-zerotier openr[21997]: I1128 08:35:55.385100 22032 LinkMonitor.cpp:472] InterfaceDb Sync is successful Nov 28 08:35:58 vrf-zerotier openr[21997]: I1128 08:35:58.141927 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:00 vrf-zerotier openr[21997]: I1128 08:36:00.941509 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:03 vrf-zerotier openr[21997]: I1128 08:36:03.548934 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:05 vrf-zerotier openr[21997]: I1128 08:36:05.378876 21997 NetlinkSystemHandler.cpp:178] Re-syncing Netlink DB Nov 28 08:36:05 vrf-zerotier openr[21997]: I1128 08:36:05.379459 21997 NetlinkSystemHandler.cpp:180] Completed re-syncing Netlink DB from Netlink Subscriber Nov 28 08:36:05 vrf-zerotier openr[21997]: I1128 08:36:05.387539 22032 LinkMonitor.cpp:472] InterfaceDb Sync is successful Nov 28 08:36:06 vrf-zerotier openr[21997]: I1128 08:36:06.400626 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:09 vrf-zerotier openr[21997]: I1128 08:36:09.269106 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:12 vrf-zerotier openr[21997]: I1128 08:36:12.756955 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:15 vrf-zerotier openr[21997]: I1128 08:36:15.389602 22032 LinkMonitor.cpp:472] InterfaceDb Sync is successful Nov 28 08:36:15 vrf-zerotier openr[21997]: I1128 08:36:15.735888 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:18 vrf-zerotier openr[21997]: I1128 08:36:18.724366 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:20 vrf-zerotier openr[21997]: I1128 08:36:20.114837 22032 LinkMonitor.cpp:446] LinkMonitor: processing LinkMonitor command Nov 28 08:36:20 vrf-zerotier openr[21997]: I1128 08:36:20.115170 22032 LinkMonitor.cpp:1234] Dump Links requested, replying with 2 links Nov 28 08:36:22 vrf-zerotier openr[21997]: I1128 08:36:22.190623 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:24 vrf-zerotier openr[21997]: I1128 08:36:24.861799 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:25 vrf-zerotier openr[21997]: I1128 08:36:25.386286 21997 NetlinkSystemHandler.cpp:178] Re-syncing Netlink DB Nov 28 08:36:25 vrf-zerotier openr[21997]: I1128 08:36:25.387398 21997 NetlinkSystemHandler.cpp:180] Completed re-syncing Netlink DB from Netlink Subscriber Nov 28 08:36:25 vrf-zerotier openr[21997]: I1128 08:36:25.392123 22032 LinkMonitor.cpp:472] InterfaceDb Sync is successful Nov 28 08:36:27 vrf-zerotier openr[21997]: I1128 08:36:27.490969 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:30 vrf-zerotier openr[21997]: I1128 08:36:30.279563 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:33 vrf-zerotier openr[21997]: I1128 08:36:33.341143 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:35 vrf-zerotier openr[21997]: I1128 08:36:35.397189 22032 LinkMonitor.cpp:472] InterfaceDb Sync is successful Nov 28 08:36:36 vrf-zerotier openr[21997]: I1128 08:36:36.492441 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:39 vrf-zerotier openr[21997]: I1128 08:36:39.911485 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:42 vrf-zerotier openr[21997]: I1128 08:36:42.569042 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:45 vrf-zerotier openr[21997]: I1128 08:36:45.334821 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:45 vrf-zerotier openr[21997]: I1128 08:36:45.391568 21997 NetlinkSystemHandler.cpp:178] Re-syncing Netlink DB Nov 28 08:36:45 vrf-zerotier openr[21997]: I1128 08:36:45.392102 21997 NetlinkSystemHandler.cpp:180] Completed re-syncing Netlink DB from Netlink Subscriber Nov 28 08:36:45 vrf-zerotier openr[21997]: I1128 08:36:45.399906 22032 LinkMonitor.cpp:472] InterfaceDb Sync is successful Nov 28 08:36:47 vrf-zerotier openr[21997]: I1128 08:36:47.998008 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:50 vrf-zerotier openr[21997]: I1128 08:36:50.925370 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:53 vrf-zerotier openr[21997]: I1128 08:36:53.450906 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:55 vrf-zerotier openr[21997]: I1128 08:36:55.404263 22032 LinkMonitor.cpp:472] InterfaceDb Sync is successful Nov 28 08:36:56 vrf-zerotier openr[21997]: I1128 08:36:56.798099 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:36:59 vrf-zerotier openr[21997]: I1128 08:36:59.701334 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:37:02 vrf-zerotier openr[21997]: I1128 08:37:02.531883 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:37:05 vrf-zerotier openr[21997]: I1128 08:37:05.314360 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:37:05 vrf-zerotier openr[21997]: I1128 08:37:05.392982 21997 NetlinkSystemHandler.cpp:178] Re-syncing Netlink DB Nov 28 08:37:05 vrf-zerotier openr[21997]: I1128 08:37:05.393541 21997 NetlinkSystemHandler.cpp:180] Completed re-syncing Netlink DB from Netlink Subscriber Nov 28 08:37:05 vrf-zerotier openr[21997]: I1128 08:37:05.407763 22032 LinkMonitor.cpp:472] InterfaceDb Sync is successful Nov 28 08:37:08 vrf-zerotier openr[21997]: I1128 08:37:08.071864 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:37:11 vrf-zerotier openr[21997]: I1128 08:37:11.441593 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:37:13 vrf-zerotier openr[21997]: I1128 08:37:13.403115 22030 ThreadManager.tcc:374] ThreadManager::add called with numa == true, but not a NumaThreadManager Nov 28 08:37:15 vrf-zerotier openr[21997]: I1128 08:37:15.025465 22020 Spark.cpp:990] Neighbor hub2-proxhub2-prox.i.wizznet.co.uk on iface zt0 from iface zt0 has not heard from us yet Nov 28 08:37:15 vrf-zerotier openr[21997]: I1128 08:37:15.412070 22032 LinkMonitor.cpp:472] InterfaceDb Sync is successful

Could this be local firewall related on the other side?

I do have iptables running but I thought I opened the correct ports. 60,000 : 60100 tcp

[bodleytunes]

yes it was the firewall. Works when I turn it off

Can you confirm if these should be the correct ports?

[saifhhasan]

You also need to open firewall for UDP port 6666 on which neighbor discovery is performed by Spark module. Can you give it a try.

You can ignore the spark error. The error was happening because

• Spark is receiving all multicast packets on port 6666
• But Spark doesn't know about interface zt0 (with ifIndex=5). Once you set IFACE_PREFIXES=eth,zt then spark will learn about zt0

Note that REDISTRIBUTE_IFACES is for announcing interface addresses into the network.

Closing the task as error is resolved. If you have more comment or questions regarding usage of OpenR (and not code related issue) you can discuss it over Facebook Group - https://www.facebook.com/groups/openr/