faucetsdn / daq Goto Github PK

Currently requires a specific startup_cmds line -- less that intuitive. Make it auto-run when ext_ofip is defined.

In user mode, cmd/run calls the docker registry to check for latest images.
If the internet connection has less priority than the interface that DAQ uses, then it is not possible to connect to the internet and DAQ fails.

Example to fix this on debian/ubuntu based systems:

# apt install ifmetric
# route -n
# ifmetric <internet_interface> 50
# route -n 

It would be better if cmd/run doesn’t have to get the docker layers when an internet connection is not available and there are already docker layers downloaded.

Please modify bin/setup_base to reflect the below:

# Docker versions after this introduced a bug that would prevent "service docker start" from working. DOCKER_VERSION='=18.06.1~ce~3-0~debian'

Previous version: DOCKER_VERSION='=18.06.1~ce~3-0~ubuntu'

Improve the stability of test_many, which fails sometimes due to Faucet load.

bcast traffic incoming upstream into a tier-2 switch should be limited appropriately, rather than being sent to all hosts.

When doing setup_base and setup_dev on the Ubuntu 18.04 LTS release, I need to remove realpath from the package list as this package has been dropped in the latest version of Ubuntu.
Can an OS check be added to the setup_base and setup_dev scripts so that under Ubuntu realpath is not included in the list of packages to be installed by apt?

Ping test is failing from test switch interrogation docker container to switch over control plane.

Suspect routing between docker container and switch control plane is at fault.

Even though the startup_cmds=misc/startup_phy.cmd is defined and the bin/physical_sec is running and is able to ping the switch. You can see this working as part of the inst/cmdrun.log

Expand to scan other ports, especially the ones relevant for building systems.

6633 6654??!?!

The event time in the faucet logs does not represent the actual time.

To configure IP connected building control devices, it would be extremely useful to use DAQ to put them in a reachable network state reachable from the host.
The -n option already creates the network, but the devices under testing are not accessible from the DAQ machine.

The use case is to open a browser and reach the device on port 80/443 or open an ssh connection on port 22 from the DAQ machine.
Other ports might be useful in the future.

Thanks!

Works OK with a 6-8 port switch setup, but using a full 48-port switch causes performance problems. Likely related to an O(n^2) algorithm somewhere in Faucet or OVS. Workaround is to keep sec_port set to something small, but eventually this needs to be sorted.

When using cmd/exrun it is possible to use Ctrl+C to stop the program, but when running cmd/run this does not seem to be possible. At the moment I use

docker stop daq-runner

or

docker kill daq-runner

from a separate terminal window to stop the execution.
Being able to use Ctrl+C in the same terminal would be useful.

Can the system allow to specify devices in group without having to create MUD files for them?

• Report to include how far along sequence before aborted.
• Report to include exception as to why sequence aborted.
• Report should be generated regardless of whether device failed or not.
• If device fails old report is left behind from previous test.

This feature request is about enabling hierarchical naming of tests, for instance switch.port.link

There could be an option when running the test that allows to add a comment to describe the device configuration for the test.

For troubleshooting and configuration purposes, it would be extremely useful to see in the firebase app the IP address of the devices being tests/connected.
At the moment I'm using docker exec to get a terminal on the containers, but that would be more immediate.
Thanks!

Specifically need multi-tiered hierarchy rather than just "secondary"

Use some configuration file indirection (like the credential files do).

It would be nice for Daq to have a common data store that tests are able to read and write from.

This may trigger additional features in further tests.

i.e. results of nmap scan would then feed into the brute test, that would then enable brute force tests on those ports and protcols found.

After the last source code pull, I run into this problem when connecting to the GCP app

sudo cmd/run -s

Loading container configuration from /home/francesco/Code/faucetsdn/daq/local//system.conf
0.8.1: Pulling from daqf/runner
Digest: sha256:ae436ab18d1c922d3a12e0ba112be8bac01e7e20297ff47bb95db542d8f98a2c
Status: Image is up to date for daqf/runner:0.8.1
Configuring apparmor...
sudo: apparmor_parser: command not found
Starting daqf/runner:0.8.1 -s...
Release version 0.8.1
Loading daq run configuration from /root/daq/local/system.conf
cleanup='echo cleanup'
cmdarg=
gcp_cred=local/bos-daq-testing-cf03a659eff0.json
network_config=/root/daq/misc/faucet.yaml
site_description='Arup DAQ Testing - Francesco - Debian Stretch'
value=-s

• Starting Docker: docker
  ...done.
  ovsdb-server is not running
  ovs-vswitchd is not running
• /etc/openvswitch/conf.db does not exist
• Creating empty database /etc/openvswitch/conf.db
• Starting ovsdb-server
• system ID not configured, please use --system-id
• Configuring Open vSwitch system IDs
• Starting ovs-vswitchd
• Enabling remote OVSDB managers
  Loading faux configuraiton from /root/daq/local/system.conf
  Launching faux ...
  DAQ autoclean docker kill daq-faux
  Removing old interface faux
  Adding new interface to 300...
  Done with faux device launch.
  Using python exe /usr/bin/python
  Executing python -s...
  INFO:daq:configuration map: {'single_shot': True, 'site_description': '"Arup DAQ Testing - Francesco - Debian Stretch"', 'gcp_cred': 'local/bos-daq-testing-cf03a659eff0.json'}
  INFO:daq:pid is 385
  INFO:gcp:Loading gcp credentials from local/bos-daq-testing-cf03a659eff0.json
  INFO:gcp:Initialized gcp pub/sub bos-daq-testing:daq-arup
  INFO:gcp:Initialized gcp firestore bos-daq-testing:daq-arup
  Exception in thread Thread-MonitorBatchPublisher:
  Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
  self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
  self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/publisher/batch/thread.py", line 239, in monitor
  return self._commit()
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/publisher/batch/thread.py", line 204, in _commit
  self._messages,
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", line 325, in publish
  return self._publish(request, retry=retry, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/gapic_v1/method.py", line 139, in call
  return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
  on_error=on_error,
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/retry.py", line 195, in retry_target
  last_exc)
  File "/usr/lib/python2.7/dist-packages/six.py", line 718, in raise_from
  raise value
  RetryError: Deadline of 600.0s exceeded while calling <functools.partial object at 0x7f0685507208>, last exception: 504 Deadline Exceeded

It looks like a timeout related problem, but would like to have some help debugging it.
Thanks!

Reports should allow saving different reports on same device instead of overwriting.
Reports should include mac address + testing machine ID + timestamp in name of test file.

When IP address changes (b/c new test run), the BACnet simulator needs to restart!

When running

bin/mudacl local/device_specs.json 

I get an output like this

BUILD SUCCESSFUL in 0s
2 actionable tasks: 2 up-to-date
Running regression test.
com.google.daq.orchestrator.mudacl.MudAclGenerator$ExceptionMap: 1 type errors
 java.lang.RuntimeException: While processing type bacnet_controller rule ntp-frdev
   java.lang.IllegalArgumentException: Should not include NTP ports in ACLs
Writing errors to /home/francesco/Code/pisuke/daq/mudacl/build/mud_errors.json
com.google.daq.orchestrator.mudacl.MudAclGenerator$ExpectedException: com.google.daq.orchestrator.mudacl.MudAclGenerator$ExceptionMap: 1 type errors

Eventually I figured out there was a problem file by searching for NTP in all the files inside the mud_files/ directory, but the output can be even more useful if pointing at the problematic file.

Make a way to just rebuild one test, not everything.

I've don a git pull to upgrade DAQ to the latest code on the master branch.
Then issued the following commands

sudo cmd/clean
sudo bin/clean_dev
sudo bin/setup_base
sudo bin/setup_dev
cmd/run -s 

So far, so good.
Then, running

sudo cmd/build 

the build script execution stops with this output:

Loading build configuraiton from local/system.conf
Build docker/Dockerfile.aardvark into daq/aardvark, log to build/docker_build.aardvark...
E: Unable to locate package tcpdump
E: Unable to locate package strace
Retry 2 of cmd apt-get failed with exit code 100
E: Unable to locate package net-tools
E: Unable to locate package iproute2
E: Unable to locate package iputils-ping
E: Unable to locate package tcpdump
E: Unable to locate package strace
Retry 3 of cmd apt-get failed with exit code 100
The command '/bin/sh -c $AG update && $AG install net-tools bash iproute2 iputils-ping tcpdump strace' returned a non-zero code: 100
Build failed, see build/docker_build.aardvark for complete log.

Here is the output of the build/docker_build.aardvark log file

Sending build context to Docker daemon 105.4MB
Step 1/5 : FROM ubuntu:xenial
---> 52b10959e8aa
Step 2/5 : WORKDIR /root
---> Running in bf8c666f4121
Removing intermediate container bf8c666f4121
---> 5de4177e6b7c
Step 3/5 : COPY bin/retry_cmd bin/
---> 8fa513a51e35
Step 4/5 : ENV AG="bin/retry_cmd apt-get -qqy --no-install-recommends -o=Dpkg::Use-Pty=0"
---> Running in 8d8618616145
Removing intermediate container 8d8618616145
---> ef1ddd9e0a78
Step 5/5 : RUN $AG update && $AG install net-tools bash iproute2 iputils-ping tcpdump strace
---> Running in 8fb940bc53e6
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
E: Unable to locate package net-tools
E: Unable to locate package iproute2
E: Unable to locate package iputils-ping
E: Unable to locate package tcpdump
E: Unable to locate package strace
Retry 1 of cmd apt-get failed with exit code 100
E: Unable to locate package net-tools
E: Unable to locate package iproute2
E: Unable to locate package iputils-ping
E: Unable to locate package tcpdump
E: Unable to locate package strace
Retry 2 of cmd apt-get failed with exit code 100
E: Unable to locate package net-tools
E: Unable to locate package iproute2
E: Unable to locate package iputils-ping
E: Unable to locate package tcpdump
E: Unable to locate package strace
Retry 3 of cmd apt-get failed with exit code 100
The command '/bin/sh -c $AG update && $AG install net-tools bash iproute2 iputils-ping tcpdump strace' returned a non-zero code: 100

Hope this can be useful to get it fixed.
Thanks,
Francesco

Implement a subtest of the dhcp module (suggested name dhcp.shortlease), where DAQ sets initially a 1 minute lease and then checks that the device re-requests DHCP after 1 min. Then DAQ sets a long term lease and starts the next tests.

Sometimes the system becomes unstable and hangs at the “INFO:runner:Entering main event loop.” stage.
In order to make it work again it is necessary to restart the VM / computer or to unplug/plug the adapter so that the kernel restarts the adapter.

"Non-local run mode not supported for physical switches yet"

That should error-out and stop the process.

When using the local/system.conf host_tests variable it would appear that the parsing of local/local_tests.conf file isnt correct as notifed after using command cmd/build in terminal message:> No such file or directoryst_tests.conf

local/system.conf
host_tests=local/local_tests.conf

local/local_tests.conf
include misc/host_tests.conf

The workaround for now is to manipulate tests through misc/host_tests.conf

If the Dockerfile.test_xxxxxx name length too long then cmd/build fails.

Workaround for now to keep test names short.

If there's a configuration error in the faucet.yaml file, it should be exposed through the DAQ command line.

Have more generic options in the local/system.conf such as intf_names=

This is to provide a generic subset of options so the user can see what a standard system configuration file would look like. This could include both options for a single device and also a switch but simply have them commented out so it's not as expansive as some of the templates in the misc/ directory.

Would it be possible to add the report date/timestamp. To the report output text?
At the moment, the date is only included in the filename.

Add a device config file input by the tester/end user that contains device make / device manufacturer / firmware version in addition to MAC address (these should be provided manually by the end user in a configuration text file).

Possibly, this information can also be derived programmatically via nmap scripts like this one.
https://github.com/nmap/nmap/blob/master/scripts/bacnet-info.nse

Get the documentation into a more sequential state to better perform test in order of operation.

New Order:

-System Requirements
-Build Setup
-Configuration
-Network Topologies
-Quickstart
-Report Generation
-Debugging
-Network Taps
-Qualification Dashboard
-Containerized Tests

Reason for re-order:
Need some clarification on the build and configuration beforing running cmd/run. By understanding some of the key components and their setup, eg Network Topology (using single device or switch?), then you can more easily get the setup running and be able to start troubleshooting if necessary.

I'm running DAQ in local mode and the startup.pcap file generated by DAQ is empty.

Also, another thing related is that the location of the pcap files is no longer inst/run-port-01/nodes/gw01/tmp/startup.pcap as reported in the example, but something like inst/run-port-01/scans/startup.pcap.

NMAP test fails:
ERROR:runner:Test Failure: ['03:nmap:1']
daq/inst folder and other files attached as .zip
nmap-error-reports-logs.zip

Occasionally after successive daq run/cmd I am receiving terminal message:>
Local build hash does not match, or not found. Please run cmd/build, or if you know what your doing:

The workaround for now is to perform cmd/clean followed by cmd/build which takes 5minutes> everytime to build docker containers, slow process.

I understand that the following should work to merge changes from the following forked wiki repo
https://github.com/pisuke/daq.wiki.git
according to the instruction provided at this link.

The owner of the original repo (https://github.com/faucetsdn/daq.wiki.git) should do the following.

$ git clone https://github.com/faucetsdn/daq.wiki.git
$ cd https://github.com/faucetsdn/daq.wiki.git

# squashing all forked repo changes
$ git pull https://github.com/pisuke/daq.wiki.git master

$ git push origin master

If one device's test finishes, it should linger until the entire group is finished (mirror startup)

Running the command: sudo cmd/clean creates following error:

bridgehead@us-mtv-c440-bridgehead-lab:~/daq$ sudo cmd/clean
"docker rmi" requires at least 1 argument.
See 'docker rmi --help'.

Usage:  docker rmi [OPTIONS] IMAGE [IMAGE...]

Remove one or more images
"docker rmi" requires at least 1 argument.
See 'docker rmi --help'.

Usage:  docker rmi [OPTIONS] IMAGE [IMAGE...]

Remove one or more images
Total reclaimed space: 0B
bridgehead@us-mtv-c440-bridgehead-lab:~/daq$

This happens regardless of running the cmd/build previously or not.

1. Use of "startup_cmds" -- as per misc/system_phy.conf
2. Debug output in test_ping activation logs.

https://github.com/faucetsdn/daq/blob/master/docs/validator.md#validator-setup
Current:
There's currently two schemas available, defined in the validator/schemas/ subdirectory:

Directory correction:
daq/schemas