The current read_nvram function requires hardcoding the length of the TLS root cert. This is brittle and shouldn't be neccessary.

I don't know if this can be fixed in the read_nvram function or if we need to write the length into nvram as well.

If the latter, ideally we would just write the length as a prefix in the same nvram index. Then do a partial read to get the length and then a full read.

Requiring separate handles is ugly.
@zanebeckwith interested in your thoughts

See #8 for background.

The mark will be used to ensure that tunnel traffic is not routed back into the tunnel interface itself.

The current process for provisioning TLS keys and certs for backend servers is very cumbersome during customer demos and training sessions. It requires copying PEM between machines and moving them into the location and names expected by enftun (or modifying the enftun config file to match the actual locations). Getting all these "fiddly bits" right during a demo or training take valuable time and doesn't impart much useful knowledge.

Currently the enfcli is used to generate server credentials (an x509 key and cert) and upload the cert to the ENF for a particular IP. The enfcli is supposed to be run on an engineer's computer, so the resulting cert and key must be manually copied to and installed on the VM running enftun. Installing enfcli on each remote machine is not desirable.

An easier approach to provisioning certs for the "one-off" servers is needed.

Requirements

1. enfcli should not be required on the host running enftun.
2. The key and cert should be generated locally on the host running enftun.
3. Additional authentication mechanisms (e.g., XTT cipher suites) should not be added just for this purpose.

Assume that (local or remote) console access is available on the host running enftun.

Suggested Approach

Extend enftun with an additional tool called enftun-keygen that can generate the keys and upload the cert to the ENF using the IAM APIs. This tool can read the existing enftun config files to determine where to place the certs. The user will have to provide API credentials (user name and password), but these are easy to provide to a CLI tool.

Usage:

# Create config for enf0
cp /usr/share/doc/enftun/example/server.conf /etc/enftun/enf0.conf

# Provision keys
# Either a full IPv6 address or a /64 prefix can be specified.  If a /64 prefix is specified, the IAM server will arbitrarily generate a new IPv6 address in that /64.
enftun-keygen [--config /etc/enftun/enf0.conf] [--user email] <address>|auto
> Enter password:

# Start enftun
systemctl start enftun@enf0

Rejected Approaches

1. Add an additional attestation mechanism in the form of a single-use, time-limited token. The token would be generated by the enfcli and then manually copied onto the VM. enftun would generate a key pair and use the token to authenticate with the ENF (either using the IAM APIs or a new XTT cipher suite) to upload the cert.

   This approach is easy to use. The token is just a small ASCII string that can be easily copy and pasted between machines. However, it requires adding an authentication mechanism that is essentially a not particularly secure preshared key (mitigated somewhat by the single-use, time-limited aspect).

2. Teach enfcli to directly SCP the generated certs and keys into the appropriate location on the remote host. This is difficult implement in a way that works across the variety of operating systems, SSH clients, and SSH configurations that our customers might be using.

Implementation

• Implement in C, because C compilers are widely available on the platforms on which enftun runs.
• Use libssl/libcrypto (openSSL) for key/cert generation, since enftun already uses openSSL.
• Use libcurl for the HTTP requests and libjansson for JSON, since both are widely available on the platforms on which enftun runs.
• Make this a separate CMake subproject
  • Files in ./keygen (keygen/src, keygen/CMakeLists.txt, etc.)
  • In the top-level ./CMakeLists.txt, an an option to BUILD_KEYGEN. Optionally include the keygen subdirectory, just like the example and test directories
• The main steps will be
  1. Send username/password to the xauth API to get back an auth token for subsequent API requests
  2. Generate a new ECDSA private key using openSSL.
  3. Post the new key to the xiam API API as a NEW_ENDPOINT_AUTH object.
  4. Generate a new certificate with a 1 year validity period using openSSL using the IPv6 address returned by the prior API call as the common name (CN) and sign it using the private key.
  5. Store the private key in the file pointed to by the config file.
  6. Store the certificate in the file pointed to by the config file.

enftun writes the TLS cert and key generated during the XTT handshake to files specified in the configuration. The permissions on these should be explicitly set. In particular, the key file should not be world readable.

• cert and key files should be owned by the same user than enftun is running as
• the cert file should have permissions 644
• the key file should have permissions 600

Most of this will be implemented in xaptum/xtt#114.

• Use xtt_save_key_to_file and xtt_save_cert_to_file instead of the plain xtt_save_to_file.

The host OS must be configured to route traffic through the enftun interface.

What to Configure

1. The tun device itself must be created.
2. The IP address assigned by the XTT handshake must be assigned to the interface.
3. Traffic must be routed through the interface. Which traffic is situation dependent:
   • For devices, all traffic should be routed through the enftun except the tunnel traffic itself.
     Also, traffic to a local physical network should not go through the tunnel, in most cases.
   • For backend servers, only the traffic for the ENF subnet should go through the tunnel.

There are several goals requirements relating to this configuration.

1. Traffic which should transit the ENF must not transit a regular interface, even when the tunnel is down.
   This means the interface must be created and the routing policy configured independent of establishment of the tunnel. In fact, they should be configured before any other services which use the network are allowed to start.
   It also means that the interface and routing policy must not be lost if the enftun client crashes and restarts.

2. The user should be able to easily configure the routing policy required, with a conservative (from a security perspective) policy provided by default.
   Using familiar tools like iproute2 will minimize user mistakes, compared to inventing our own configuration syntax.

3. The enftun program should require minimal privileges. Ideally, it could be run as a fully unprivileged user, not root.
   The CAP_NET_ADMIN capability may be required, since almost all network administration tasks (setting IP addresses, etc.) require it.

Where/When to Configure
The IP address is most easily applied by the enftun client itself, after the xtt handshake.

There are several options for routing policy configuration:

• enftun could apply the routing policy itself. This is the approach taken by the OpenVPN client. It does not satisfy the first two requirements above.

• enftun could run user-provided scripts before and after establishing the tunnel. This is the approach taken by the tinc VPN client. It does not satisfy the first requirement above --- configuring the routes before any other network services are up.

• The configuration is left entirely up to the user and init system. We can provide example configurations (shell scripts or systemd services). This approach is operationally the simplest, is the most configurable, handles all our requirements (assuming CAP_NET_ADMIN), and minimizes the lines of code in enftun. We choose this approach.

Routing Policies
For backend servers, routing just the ENF traffic to the tun device is a simple routing rule:
ip -6 route 2607:8f80:8000::/33 dev enf0

For devices, all traffic (except the tunnel itself) should be routed through the ENF. Doing this properly is involved. The Wireguard documentation has a good overview of several approaches.

The biggest pain point is ensuring that the tunnel traffic (the TCP/TLS tunnel traffic to the ENF) is not routed back into the tunnel itself.

The default approaches require adding an explicit route for the ENF remote addresses to the physical interface. This requires knowing the physical interface a priori. Since this can change as the device transitions between various forms of last-mile connectivity, this approach doesn't work well for us.

Instead, the "Improved Rule-Based Routing" is better. We mark the tunnel packets (via the SO_MARK option on the underlying TCP socket) and configure the kernel to route unmarked packets via the tunnel interface. The marked tunnel packets will transit the physical interface.

Deliverables

• #12 Set SO_MARK on tunnel sockets
• Set address on interface
• A (set of) systemd service file(s) to create the tun device and route all traffic through it.
• A (set of) systemd service file(s) to create the tun device and route just ENF traffic through it.
• A wrapper script that does the same, and can be easily run from the command line for ad hoc usage of the enftun by developers and in demos.

When enftun is run on the router card, the IPv6 address should be assigned to the interface on the host, not the interface on the router card.

To support this, enftun should expose a config option tun.set_address to control setting of the IP address. The default value should be true.

Blocks #73

Router advertisements

enftun configures routes for the ENF by running on iproute2 commands. This will not work for the router card, where enftun is not running on the host. enftun should instead send IPv6 router advertisements containing these routes.

• Send router advertisement when first connected and periodically thereafter (say 600 seconds).
• Respond to router solicitations with a router advertisement.
• Configure the router card to not accept RAs on enf0. Otherwise it will try to interpret the RA itself.

The router advertisements should contain the following:

• Header
  • src IP of fe80::1
  • dst IP of ff02::1
• RA
  • Flags: MANAGED
  • Router Preference: High
  • MTU: 1280
  • Router Default Lifetime: 0 # do not advertise as a default router
• Route Option - one for each route listed in the route.prefixes config
  • Route Lifetime: 3 * period of the RAs
  • Route Preference: High
• Source Link Layer Address (optional, see the "Neighbor Advertisement" section below)
  • a MAC address specified in the config (tun.lladdr)

I think that a Prefix option is not needed, since this tunnel is point-to-point (there's no LAN prefix) and we don't support SLAAC. However, if we want to send it, the following options should be correct:

• Prefix Option
  • Flags: not on-link, not autonomous
  • Valid Lifetime: 30 days

Neighbor Advertisements

The router card is exposed on the host as an L2 interface, so it needs to learn the MAC address corresponding to the router IP (fe80::1).

Generically, it will issue neighbor solicitations to the router IP (fe80::) to determine the MAC address of the router card (usb0 interface). The router card would need to respond with a neighbor advertisement. This would require implementing support for these message types.

Option 1

However, the router advertisement can itself can contain its own link-layer address. I propose we use this, to avoid the need to support the two neighbor messages.

Most uses of enftun are L3 only, so they won't specify a tun.lladdr and that option won't be included in the RAs. For the router card, the MAC address of usb0 can be hard-coded (passed to the g_ether module and in the enftun config.)

Option 2

enftun is intentionally L3-only, so having it include a link-layer address is a bit odd. So a different approach to remove link-layer addresses entirely from the enftun. Don't include them in the RA and don't handle NS in enftun.

Instead, handle ND with a different service running on the router card, e.g., inside of xbridge. Before passing a packet from usb0 to enf0, check if it is a NS for fe80::1. If so, send a NA to usb0 instead.

Implementation

These features should be implemented in a src/ndp.c module, which should send an RA when triggered.

Triggering an RA

Sending a packet is asynchronous, so we separate the concepts of sending vs scheduling an RA.
Only a single router advertisement should be in flight at a time.

An RA should be scheduled when an RS is received or the periodic timer (a libuv timer) expires. This scheduling is idempotent: If an RA is already scheduled, that particular trigger has no effect. Don't schedule multiple RAs out into the future.

If no RA is in-flight, the scheduled RA should be sent immediately. If an RA is already in-flight, it should be sent after the previous one finishes.

Sending an RA

The enftun_packet struct is a buffer for network packet data (similar to the sk_buff in the Linux kernel). The enftun_crb struct is used to schedule a packet for reading or writing (similar to the USB request urb in the Linux kernel). enftun_crb_write(crb, chan) will asynchronously write the
the specified CRB to the specified channel. The callback attached to the CRB will be called upon completion.

Only a single RA should be in-flight at a time, so the enftun_ndp struct can hold a single, statically allocated packet and CRB. The enftun_context struct holds the tunnel channel as a member tunchan.

Intercepting RSs

This module should expose a filter hook that is registered is called in the enftun.c:chain_egress_filter() function. The hook should filter out RSs and schedule the sending of an RA.

NS and NA

Neighbor discovery can be handled in a similar fashion to RAs and RSs, if "Option 1" above is chosen.

Currently enftun advertises routes using route information options in the main RA. Support for these options can be disabled at compile time in the Linux kernel. This is true for some IoT gateways like the Eurotech RG-20-25.

The original RA specification did include support for advertising the default route via the default lifetime parameter. We currently set this to 0 to disable, and advertised the default route (if configured) via the aforementioned option.

To support both current and legacy systems, we should change how the default route is advertised.

• specify C99
• specify all warnings

For local TCP sockets, we:

• subscribe to netlink notifications for routing table changes
• when the routing table changes, we check if the preferred route has change by
  • opening and "connecting" a UDP socket
  • checking if the local address for that socket is the same as the active TCP tunnel
  • if addresses are different, our route has changed, so reconnect the tunnel (call to reconect_cb)

When connected via HSS, we want to detect if local TCP (wifi) becomes available:

• subscribe to netlink notifications for routing table changes
• when the routing table change, check if it is possible to connect locally (via wifi) by
  • opening and "connecting" a UDP socket
  • if that succeeds, reconnect the tunnel (call to reconnect_cb)

When connected va HSS, we want to detect if the host (HSS kernel driver) route has changed:

• TODO

test/router.py is a simple server for local testing of enftun that responds to any ICMP pings (echo requests). It only supports TLS (not XTT).

test/router.crt.pem contains the "CA" certificate" that it uses. That validity period is 2018-08-31 through 2019-08-31. Expired in real life and not yet valid on the router card (which doesn't have a reliable clock.)

We need to generate a new cert with standard "infinite" lifetime we use for the production ENF. The existing private key (test/router.key.pem) should still be usable.

Observed with HSS driver.

The enftun-setup script configures the routes and firewall to ensure that all traffic from the host runs through the enf. Only the tunnels to the ENF itself are allowed to egress directly over the physical network interface. One of the techniques used is to apply a special routing table for the ENF tunnels by setting a fwmark ("firewall mark") on those TCP streams. See Improved Rule-Based Routing.

The default config from enftun-setup is to block all IPv4 traffic without this fwmark, so the udp socket always fails to connect.

We want the UDP socket used to test the local address to be subject to the same routing rules as the TCP stream to the ENF. Therefore, the same fwmark must be set on the udp socket.

• Propagate the fwmark config option through to conn_state, like it is through the enftun_tls object.
• Change the enftun_udp_connect_addr signature to take the int fwmark parameter, like the enftun_tcp_connect_* functions.
• Update the enftun_udp_connect_addr implementation to set the firewall mark.

The instructions for installing on Debian are missing some steps. These will be obvious to experienced Debian users, but cause confusion during demos and trainings with others.

The "default" routing policy configured by enftun-setup routes all traffic via the tunnel, but makes an exception for local traffic:

ip -4 rule add table main suppress_prefixlength 0
ip -6 rule add table main suppress_prefixlength 0

Local traffic is any traffic where the matching routing rule has a prefix length greater than 0, i.e., is not a default route.

However, our intent with the "default" routing policy is to route all traffic via the tunnel (except the tunnel packets themselves, which is taken care of by the fwmark rules), effectively disallowing local communication.

Only DHCP and ICMPv6, for IP address assignment and router discovery, are required. I believe that both of these bypass the routing table, since they are intended to be used on a specific interface that does not yet have an IP address.

This needs to be confirmed, but I believe that the local traffic exceptions can be removed from the "default" routing policy.

Testing

This should be an easy one to test in the office, where we have local IPv4 and IPv6 support.

On a gateway connected to ethernet and configured with the enftun service:

1. Remove the "suppress_prefixlength" lines from the enftun-setup script.
2. Reboot the gateway.
3. Check if it receives IPv4 and IPv6 addresses on the ethernet interface and if the enf0 tunnel successfully connects.

Repeat the test with the wifi interface, instead of ethernet.

Scenario: the enftun application detects a packet with an invalid source IP:

<7>DROP [ egress]: invalid src IP

This results in further communication through enf0 being blocked.

check_preferred_route will return an error code if the udp socket fails to connect, but on_poll just ignores this error. Instead, I think we should force a reconnect.

An error in check_preferred_route indicates that something is wrong with the network, so trying to reconnect is the fail-safe option.

The router cards will support connecting to the ENF using the internet connection on the host. This is done using a custom socket protocol called psock for proxying TCP sockets from the host to the router card.

Phase 1

• Support compile time selection of either normal TCP or proxy TCP. The xaprc001 router card will only ever use the latter, so this is sufficient to support that card.

Phase 2

• Support both normal TCP and proxy TCP at the same time . Switch between the two depending on which is available. This will allow the xaprw001 router card with WiFi to fallback to the host internet when the wifi isn't available or configured.

  Initial Algorithm Proposal

  • Try TCP first
  • If TCP fails, try SCM
  • When on SCM, use the device netlink notifications to detect when wifi might be back. And restart tunnel

  Still to Figure Out:

  • How to detect when SCM socket has failed (For TCP, we use netlink notifications)

When the network connectivity changes, the TCP connection to the ENF might become unusable, but this can take a while to detect.

To ensure availability, monitor netlink events and force a reconnect on any actual connectivity change.

Reconnection is not necessary on every netlink event (interface state changed, interface IP address changed, route changed, etc.). It is only necessary if either

1. the preferred route (i.e., local IP address) has changed, for example, when a interface with higher priority comes online.

2. the TCP connection has been broken by the change, for example, when the interface carrying the connection goes offline.

The first can be detected by connecting a new UDP socket; if the local IP of this socket is different from the local IP of the active connection's socket, the connection should be reestablished. The second can be detected by pinging the ENF; if no response is received within a timeout period, the connection has been broken.

This task consists of the following phases, which incrementally optimize the number of reconnections.

• Monitor for all interface/route/address changes and reconnect on any of them
• Check if the route has changed. Do not reconnect if it hasn't.
• Trigger a "ping" test. Do not reconnect if it passes (#6).

Currently when there is an error in enftun we either:

• call stop_all and reconnect
• log the error, but nothing else is done

If we are reconnecting, when the error could be handled in another way, we lose availability. How this is solved may be different for each case, but it would be good to take a look at how we can improve error handling. The motivation for this issue comes from this comment.

Currently the same code repeats in both xtt.c and tls.c to initialize the enftun_tcp_native and enftun_tcp_psock structs. This can be combined into one function.

It would be ideal to be able to initialize either one so that we can support both tcp and psock simultaneously (falling back to the psock connection when tcp is not available).

Connected to #73

The tunnel client should respond to IPv6 router soliticiations (RSs) with router advertisments (RAs).

The RAs should

• include /64 as route
• disallow SLAAC

Configure CMake to install the systemd unit files. Make this behavior configurable via a BUILD_SYSTEMD option.

As a ping service that can be used to determine if the connection to the ENF is still alive.

Currently the server DUID is hard-coded, so all instances of enftun will present the same DUID. This reuse definitely violates the intent and spirit of the DUID concept, although it might not cause any actual issues.

There are a couple of options to consider to be more "compliant":

1. Generate the DUID randomly each time enftun starts up. I like that this approach is stateless, but not that the DUID will change on each boot (which is not necessary).

2. Generate the DUID randomly on the first time enftun starts up and store it for future reuse. This approach requires storing state.

   Since we already store the TLS credentials anyway, storing another file with the DUID seems fine.

   If for some reason the file can't be read (or written), then fallback to regenerating a new DUID, like case 1.

User-space apps do not have access to our kernel header files, the sock IDs are defined in the arm-buildroot-linux-gnueabihf project (the userspace C libraries for the device). We can overlay our own patch, fork and modify or just keep defining it here for now.

Tasks:

• #14 install systemd file
• create debian packaging files

The various header processing code (filter.c, ndp.c, etc.) generate unaligned memory traps on older ARM architectures.

To reproduce on x86, compile with -fsanitizer=alignment in CMakeLists.txt

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 6262665..17f1cf8 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -32,7 +32,8 @@ if(BUILD_SCM)
     add_definitions(-DUSE_SCM)
 endif()
 
-add_compile_options(-std=c99 -Wall -Wextra -Wno-missing-field-initializers)
+add_compile_options(-std=c99 -Wall -Wextra -Wno-missing-field-initializers -fsanitize=alignment)
+set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fsanitize=undefined")
 set(CMAKE_C_FLAGS_RELWITHSANITIZE "${CMAKE_C_FLAGS_RELWITHSANITIZE} -O2 -g -fsanitize=address,undefined -fsanitize=unsigned-integer-overflow")
 set(CMAKE_C_FLAGS_DEV "${CMAKE_C_FLAGS_RELEASE} -Werror")
 set(CMAKE_C_FLAGS_DEVDEBUG "${CMAKE_C_FLAGS_DEBUG} -Werror")

enftun/src/icmp.c:118:31: runtime error: member access within misaligned address 0x7fff0707b10a for type 'struct nd_router_advert', which requires 4 byte alignment

enftun/src/icmp.c:119:31: runtime error: member access within misaligned address 0x7fff0707b10a for type 'struct nd_router_advert', which requires 4 byte alignment

enftun/src/icmp.c:120:31: runtime error: member access within misaligned address 0x7fff0707b10a for type 'struct nd_router_advert', which requires 4 byte alignment

enftun/src/icmp.c:121:31: runtime error: member access within misaligned address 0x7fff0707b10a for type 'struct nd_router_advert', which requires 4 byte alignment

enftun/src/icmp.c:122:31: runtime error: member access within misaligned address 0x7fff0707b10a for type 'struct nd_router_advert', which requires 4 byte alignment

enftun/src/icmp.c:123:31: runtime error: member access within misaligned address 0x7fff0707b10a for type 'struct nd_router_advert', which requires 4 byte alignment

enftun/src/icmp.c:124:31: runtime error: member access within misaligned address 0x7fff0707b10a for type 'struct nd_router_advert', which requires 4 byte alignment

enftun/src/icmp.c:125:31: runtime error: member access within misaligned address 0x7fff0707b10a for type 'struct nd_router_advert', which requires 4 byte alignment

enftun/src/icmp.c:66:29: runtime error: member access within misaligned address 0x7fff0707b11a for type 'struct nd_opt_mtu', which requires 4 byte alignment

enftun/src/icmp.c:67:29: runtime error: member access within misaligned address 0x7fff0707b11a for type 'struct nd_opt_mtu', which requires 4 byte alignment

enftun/src/icmp.c:68:29: runtime error: member access within misaligned address 0x7fff0707b11a for type 'struct nd_opt_mtu', which requires 4 byte alignment

enftun/src/icmp.c:69:29: runtime error: member access within misaligned address 0x7fff0707b11a for type 'struct nd_opt_mtu', which requires 4 byte alignment

enftun/src/icmp.c:90:30: runtime error: member access within misaligned address 0x7fff0707b122 for type 'struct nd_opt_route_info', which requires 4 byte alignment

enftun/src/icmp.c:91:30: runtime error: member access within misaligned address 0x7fff0707b122 for type 'struct nd_opt_route_info', which requires 4 byte alignment

enftun/src/icmp.c:92:30: runtime error: member access within misaligned address 0x7fff0707b122 for type 'struct nd_opt_route_info', which requires 4 byte alignment

enftun/src/icmp.c:93:30: runtime error: member access within misaligned address 0x7fff0707b122 for type 'struct nd_opt_route_info', which requires 4 byte alignment

...

Simplest solution is to ensure that these headers are word aligned by reserving the appropriate offset in the packet.c buffers.

The current firewall policy allows all ICMPv6 except echo requests in on the local network. We could instead drop by default and only allow the types that are known to be required:

https://resources.sei.cmu.edu/tools/downloads/vulnerability-analysis/assets/IPv6/ip6tables_rules.txt

The ENF tunnel service is available via both IPv4 and IPv6 addresses. The enftun client must be capable of use both, since it may move between IPv4 and IPv6 last mile networks.

• Extend config to take a list of remote hosts, not just one
• Try all hosts in the order specified in the config file

The default route in is not used for IPv6 multicast traffic, so enftun-setup must specially configure routes for multicast traffic.

Background

The Linux kernel adds a route for multicast IPv6 traffic to the local table for each network interface.

$ ip -6 route show table local
...
ff00::/8 dev eth0 metric 256  pref medium
ff00::/8 dev enf0 metric 256  pref medium
ff00::/8 dev wlan0 metric 256  pref medium

These routes are used instead of the default route in the main table, for multicast (ff00::/8) traffic, because local table takes precedence over the main table.

All the ff00::/8 routes added by the kernel have the metric (256) and preference (medium). The interface used is arbitrary, depending on the order in which they were added.

Controlling the routing of multicast traffic requires adding routes with more specificity, lower metric, or high preference to the local table.

Requirements

The default route as configured by enftun-setup must be routed all IPv6 multicast traffic through the enf0 interface.

Any specific multicast prefixes specified by the user must be added to the local table, not the main table.

Tasks

• Extend the "default" support in enftun-setup to add a ff00::/8 dev enf0 metric 50 pref high route to the local table.
• When adding a route for a specific prefix, check if it is a multicast prefix (starts with ff). If so, add it to the local table, not the default main table.

The intent with enftun is that all traffic must be routed through the tunnel.

The "default" routing policy configured by enftun-setup accomplishes this for outgoing traffic to non-local addresses, but it doesn't prevent incoming traffic from other interfaces from reaching local services.

The configuration is stronger if the host firewall is configured to block all non-tunnel traffic on all interfaces except the tunnel interfaces, with the following exceptions:

• DHCP to configure the IPv4 address of the local interface. This traffic bypasses the firewall anyway.
• ICMPv6/DHCPv6 to configure the IPv6 address of the local interface. These use link local addresses with bypass the routing table.

I think the following rules are desirable:

• allow TCP or UDP traffic on ports 443 and 444 to/from the Xaptum ENF IPs (which are found by running $ENFTUN -c "$1" -p remote.host)
• allow all traffic in/out of the ENF interface ($INTERFACE in the enftun-setup script).
• drop all other traffic.

So the implementation tasks are

• add a function add_firewall() to enftun-setup that configures these rules (analogous to add_default() configuring the routing rules).
• add a function del_firewall() to enftun-setup that removes these rules (analogous to del_default()).
• call add_firewall in add_routes() if add_default() is called.
• call del_firewall in del_routes() if del_default() is called.

Add Travis config to build the main configurations (with and without XTT support) in Dev, Release, DevDebug, and Debug types, similar to the XTT Travis config. The Dev* variants should enable -Werror.

There is a static function do_connect in both tcp_psock.c and tcp.c. Both do almost the same things with the differences of the address family used to create the socket, whether or not they define socket options for the keepalives and firewall marks.

This can be handled in one function with more parameters to define which options.

/etc/enftun/enf.cacert.pem is overridden to provide the default CA cert, but the operation leaves any trailing file data intact. Below is an example of what may be left on the machine

-----BEGIN CERTIFICATE-----
MIIBJzCBzwIIYUquXYZXXdswCgYIKoZIzj0EAwIwGzEZMBcGA1UEAwwQWEFQVFVN
Uk9PVDMzNTgxNzAiGA8wMDAwMDEwMTAwMDAwMFoYDzk5OTkxMjMxMjM1OTU5WjAb
MRkwFwYDVQQDDBBYQVBUVU1ST09UMzM1ODE3MFkwEwYHKoZIzj0CAQYIKoZIzj0D
AQcDQgAEVld7eTb+BOcTex9GBqZ9hr3eRhIjuprj5igLXb+F/ANnRY6yvDvYzZCt
2wRBivtW4Rpsi0SEEafTHuRWQTjq6jAKBggqhkjOPQQDAgNHADBEAiBiBwVHgHOq
ncWZEqZq31lQ4Y9Ks622arJuncMk5XHcMwIgJzuqN5MTpeWGBQ+RrvzB/zBlzrl5
ob2zZNkZ6sfIa8s=
-----END CERTIFICATE-----

BAMCA0gAMEUCIDT1nEl1
mYF9PWMEHFku4q9bupVrjpTkpBuES3t+7U6pAiEA214UME1YWo/EH9sS1mhVCRxD
HhpBG9YdhAuFk9bBSu0=
-----END CERTIFICATE-----

If the user provides an invalid network or address, the resulting error message is not very helpful:

[chimayo  0 ✓ (keygen|✔) build/keygen]$ sudo ./enftun-keygen -a 2607:8f80:8080:7::/65 -c /etc/enftun/enf0.conf -u xap@demo
Enter API Password: 
ENFTUN KEYGEN
Login Email: xap@demo
IP Network: 2607:8f80:8080:7::/65
Enftun Config: /etc/enftun/enf0.conf
Cert target file: /etc/enftun//enf0/enf0.crt.pem
Key target file: /etc/enftun//enf0/enf0.key.pem
Enter y to continue, n to exit: y

Logging in... Successful.
Generating Key... Successful.
Preparing IAM request... Successful.
ep_auth_resp_unmarshal ERR: Could not parse JSON string error string or '}' expected near '&'iam_send_ep_auth unable to marshal response: {"xiam_error":500,"reason":"Internal Server Error","http_error":500}
Requesting IP address... Failed. Exiting.

Because the error message return by the XIAM API is also not very helpful, I suggest the following two fixes:

1. enftun-keygen does a basic validation that the provided parameter looks like a syntactically valid ::/64 subnet or full IP address. If not, display an error message that the address or network is invalid.

2. If the XIAM API returns a 500 (instead of 2xx) response code, don't try to parse the API response. Instead just print a message that an error occurred and to double check the inputs.

Support rate limiting of outgoing traffic to protect data usage on LTE router cards.

Issue #29 will block non-tunnel traffic from all interfaces, including local traffic. This is the default behavior that we desire.

However, in some situations local traffic should be allowed. For example, in our bacnet setup, there are two physical interfaces, wlan0 connected to the internet and eth0 connected to the local BACnet/IP installation. Local traffic should be blocked on wlan0, but should be allowed on the "trusted" eth0 interface.

Here is my suggested approach, for now.

Generally local traffic should only be allowed on trusted networks. These will be local, on-prem networks that are not connected to the internet themselves (like the BACnet/IP network in our demo).

Each interface on a host will either be connected to a trusted network (eth0 in our bacnet demo) or an untrusted network (wlan0 in our bacnet demo).

So, we can allow local traffic for any interfaces that are trusted. This can be accomplished by accepting a list of trusted interfaces in the enftun config file and configuring a suppress_ifgroup rule for them.

Tasks

• Add a trusted_interfaces parameter to the config that takes a list of interface names.
• In enftun-setup, create an [interface group](https://baturin.org/docs/iproute2/#Link%20group%20management containing these interfaces.
• In enftun-setup, create a suppress_ifgroup rule for those interfaces, similar to the the suppress_prefixlength rule prior to the fix of #29.

Running enftun-keygen without an explicit -e argument results in a buffer overflow.

This line is incorrect. sizeof(enftun_def_path) returns the size of the char* pointer, not the length of the string. We need to use strlen or one of its siblings instead.

On my Debian machine the cache was stored on /var/lib/dhcp

Add support for using XTT to provision the identity certificate and key. Currently only preconfigured static certs are supported.

• add configuration section for xtt options, e.g.:
  xtt.enable (default true)
  xtt.tcti_type (default device | socket)
  xtt.device (default /dev/tpm0)
  xtt.socket
• add enftun_xtt.{h,c} files that implement an enftun_xtt_handshake(...) function or similar. See enftun_tls.{h.c} and enftun_tun.{h.c} for an example layouts.
• integrate the enftun_xtt_handshake function into the main enftun_connect(...) function.
• support compiling enftun without xtt support

Algorithm for Running XTT and TLS

The configuration file specifies the file names for the certificate and private key files (these can be PEM or DER encoded). The following logic should be used in enftun_connect(...) to regarding these files.

If the files exist, attempt to establish the TLS tunnel.

If the files don't exist or the TLS tunnel fails (due to an invalid certificate/key, not a network error) and XTT is enabled, try to run the XTT handshake.

If the XTT handshake succeeds, save the cert and key to the files and try to connect the TLS tunnel again.

If XTT fails or the second TLS attempt fails, return an error. (The outer enftun_main loop will sleep for a second and try the whole enftun_connect again.)

Operating without XTT support

Some systems like backend servers will not use xtt, just pre-distributed certificates. To avoid requiring unnecessary dependencies like xtt, ecdaa, and xaptum-tpm, it should be possible to compile enftun without xtt support.

Here are two options for implementing this.

ifdefs

An easy option is to #ifdef out the xtt-specific code in enftun.c and not compile the enftun_xtt.c. The config parsers should return an error if the "xtt" section is found.

This approach results in ugly code and potentially brittle ifdef trees that are hard to reason about.

abstraction

An alternative is to abstract out the strategy for obtaining the TLS credentials.

The base strategy used without xtt can simply return the cert and key from disk. The xtt strategy first runs xtt and then returns the cert and key.

The strategies can be implemented as functions and accessed via a function pointer, so that the calling code is not away which strategy it is calling.

This approach could result in cleaner code that is easier to maintain.

The exact API needs to be worked out.

Currently we must manually enforce the code formatting conventions in this project. This labor uses up programmer and code reviewer time. clang-format can automated this tedious work.

• Add a .clang-format style file for the formatting conventions in this project. I recommend starting with the LLVM style and tweaking from there. See xaptum/xaprc/captived) for an example; note that it is for a different formatting convention.
• Add a script to run clang-format on the code (see xaptum/xaprc/captived).
• Integrate a clang-format check into CircleCI to validate the code formatting on each pull request (see xaptum/xaprc/captived).

DHCPv6

enftun configures the IPv6 address of the enfX interfaces by running on iproute2 commands. This will not work for the router card, where enftun is not running on the host. enftun should instead responsd to DHCPv6 solicits and requests.

• Respond to DHCPv6 solicits with an an advertise.
• Respond to DHCPv6 requests with a reply containing the IPv6 for the tunnel.

No other DHCPv6 message types need to be implemented at this time.

The DUID in the request should be ignored.

Implementation

Implementation should follow the same basic approach laid out for router advertisements here.