tritondatacenter / sdc-net-agent

Triton agent for the internal networking API.

sdc-net-agent

This repository is part of the Joyent Triton project. See the contribution guidelines and general documentation at the main Triton project page.

The SDC Networking agent is a library for keeping track of per VM NIC changes on a Triton data center. There is one Networking agent installed per Compute Node. NIC changes trigger updates on NAPI so data is persisted.

Development

Typically sdc-net-agent development is done by:

• making edits to a clone of sdc-net-agent.git on a Mac (likely Linux too, but that's untested) or a SmartOS development zone,

    git clone [email protected]:joyent/sdc-net-agent.git
    cd sdc-net-agent
    git submodule update --init   # not necessary first time
    vi
  

• building:

    make all
    make check
  

• syncing changes to a running SDC (typically a COAL running locally in VMWare) via: ./tools/rsync-to coal

• then testing changes in that SDC (e.g. COAL). See "Testing" below for running the test suite.

Testing

At the moment, sdc-net-agent testing is done by running the VMAPI test suite. SSH into a running SDC and run the following commands:

touch /lib/sdc/.sdc-test-no-production-data
/zones/`vmadm lookup -1 alias=vmapi0`/root/opt/smartdc/vmapi/test/runtests

The net-agent SMF service log can be inspected while running the VMAPI tests by calling:

tail -f `svcs -L net-agent` | bunyan

sdc-net-agent's Issues

Instance loses NIC after sdc-migrate - again

Just experienced this in our DC with PI 20190912T054018Z and [email protected]:

Steps to reproduce:

1. Set a tag on an instance
2. migrate the instance to another CN
3. finalize
4. remove the tag

Here net-agent logs the following (duplicates removed):

{
    "name": "net-agent",
    "hostname": "cn03",
    "pid": 9497,
    "component": "instance",
    "uuid": "62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24",
    "level": 30,
    "msg": "Removing NIC 90:b8:d0:14:b2:13 on VM 62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24",
    "time": "2019-09-25T19:55:50.855Z",
    "v": 0
} {
    "name": "net-agent",
    "hostname": "cn03",
    "pid": 9497,
    "component": "vmadm-events",
    "level": 20,
    "msg": "VM 62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24 fields updated (nics.*) - setting needsUpdate",
    "time": "2019-09-25T19:55:51.269Z",
    "v": 0
} {
    "name": "net-agent",
    "hostname": "cn03",
    "pid": 9497,
    "level": 30,
    "num_vms": 6,
    "msg": "updateVMs",
    "time": "2019-09-25T19:55:51.270Z",
    "v": 0
} {
    "name": "net-agent",
    "hostname": "cn03",
    "pid": 9497,
    "component": "instance",
    "uuid": "62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24",
    "level": 30,
    "msg": "NIC 90:b8:d0:14:b2:13 removed from VM 62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24",
    "time": "2019-09-25T19:55:51.270Z",
    "v": 0
} {
    "name": "net-agent",
    "hostname": "cn03",
    "pid": 9497,
    "component": "instance",
    "uuid": "62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24",
    "level": 30,
    "payload": {
        "uuid": "62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24",
        "resolvers": [],
        "set_routes": {},
        "remove_routes": []
    },
    "msg": "Updating networking information for VM",
    "time": "2019-09-25T19:55:51.273Z",
    "v": 0
} {
    "name": "net-agent",
    "hostname": "cn03",
    "pid": 9497,
    "component": "instance",
    "uuid": "62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24",
    "level": 30,
    "msg": "Rebooting VM 62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24",
    "time": "2019-09-25T19:55:51.630Z",
    "v": 0
} {
    "name": "net-agent",
    "hostname": "cn03",
    "pid": 9497,
    "level": 30,
    "num_vms": 6,
    "msg": "updateVMs",
    "time": "2019-09-25T19:55:51.630Z",
    "v": 0
} {
    "name": "net-agent",
    "hostname": "cn03",
    "pid": 9497,
    "component": "vmadm-events",
    "level": 20,
    "msg": "VM 62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24 fields updated (resolvers.*,resolvers.*) - setting needsUpdate",
    "time": "2019-09-25T19:55:51.841Z",
    "v": 0
} {
    "name": "net-agent",
    "hostname": "cn03",
    "pid": 9497,
    "component": "instance",
    "uuid": "62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24",
    "level": 30,
    "payload": {
        "uuid": "62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24",
        "resolvers": [],
        "set_routes": {},
        "remove_routes": []
    },
    "msg": "Updating networking information for VM",
    "time": "2019-09-25T19:55:51.915Z",
    "v": 0
} {
    "name": "net-agent",
    "hostname": "cn03",
    "pid": 9497,
    "component": "vmadm-events",
    "level": 20,
    "msg": "VM 62ce42b8-048b-6eb3-e36f-ad7b3a3e2a24 fields updated (state) - setting needsUpdate",
    "time": "2019-09-25T19:55:52.733Z",
    "v": 0
} {
    "name": "net-agent",
    "hostname": "cn03",
    "pid": 9497,
    "component": "nic",
    "mac": "90:b8:d0:14:b2:13",
    "level": 30,
    "msg": "Stopped tracking NIC",
    "time": "2019-09-25T19:55:54.450Z",
    "v": 0
} {
    "name": "net-agent",
    "hostname": "cn03",
    "pid": 9497,
    "level": 30,
    "num_vms": 6,
    "msg": "updateVMs",
    "time": "2019-09-25T19:55:54.451Z",
    "v": 0
}

And somehow all NICs on that instance disappear. I'd be happy if I can be of any help to debug this further.

EDIT: on another instance the NICs got removed without the tag add/remove action, so maybe it's just the finalize? that instance's UUID is: 40659ae0-0328-ed41-db18-d484f80b0361

Filtered logs in this gist - I'll guess we'll need more log space here.
https://gist.github.com/teutat3s/646a8ffcbcb71515f84ed23feb2b649f