fg-networking / mgen-tools Goto Github PK

Tools to Analyze MGEN Log Files:
================================

This repository contains tools that can be used to analyze log files created by
the Multi-Generator (MGEN) network test tool.  The tools are implemented in Awk
and require a POSIX compatible Awk interpreter.  They are expected to work with
both GNU awk (gawk) and mawk.


List of Tools:
--------------

average-bandwidth         compute overall average bandwidth and other statistics
                          (aggregated information over all flows)
flow-info-csv             create CSV table (with header) of info about the
                          streams included in the log like src/dst IPs, creates
                          lists of values for more dynamic streams (or multiply
                          used stream IDs)
flow-latency-analyzer     calculate min, max, and average latency of flows
flow-outages              print per flow per outage times and lost packets
flow-statistics           compute per-flow statistics
ignore-sequence_numbers   keep packets with sequence numbers in [low,high]
ignore-startup_phase      remove the first n receive events from every flow
last-flow-result          print the last mgen-tools result message for each flow
                          (this tool works on output of other mgen-tools, not
                          on MGEN logs)
packet-dup-loss-analyzer  print lost or duplicated packets, one line per packet
packet-loss-analyzer      count number of received packets and print statistics
                          for each flow and a summary over all flows
packet-loss-filter        print a message for each lost or too late packet
packet-reorder-filter     print a message for each reordered packet
recv-to-csv               create CSV table (with header) of receive events
                          (time stamps are converted into seconds)


Controlling Tool Operation with Variables:
------------------------------------------

Some tools support setting variables via Awk option "-v VARIABLE=VALUE":
 - flow-info-csv           uses the variable "sep" to specify the separator
                           string (default: sep=,) and the variable "lstsep"
                           to specify the list separator for the potential
                           lists of values (default: lstsep=' ')
 - ignore-sequence_numbers uses the variables "low" and "high" to ignore receive
                           events with sequence numbers outside the closed
                           interval [low,high] (defaults: low=0, high=infinity)
 - ignore-startup_phase    uses the variable "skip" to ignore the first skip+1
                           receive events per flow (default: skip=92)
 - recv-to-csv             uses the variable "sep" to specify the separator
                           string (default: sep=,)


Delayed or Immediate Output:
----------------------------

Some tools read all input before any output, others produce output immediately:
 - Immediate output:
   - ignore-sequence_numbers
   - ignore-startup_phase
   - packet-loss-filter
   - packet-reorder-filter
   - recv-to-csv
 - Output generation after reading input completely:
   - average-bandwidth
   - flow-info-csv
   - flow-latency-analyzer
   - flow-outages
   - flow-statistics
   - last-flow-result
   - packet-dup-loss-analyzer
   - packet-loss-analyzer

Interrupting (CTRL+C) a tool that needs to read all input results in no output.


Use Cases:
----------

High-Level Overview:

  Sometimes, a high-level view of the information in MGEN log files is needed,
  e.g., when sending many data flows with the expectation that the network
  under test cannot deliver all the data.

  Tools for a high-level overview:
   - average-bandwidth

Flow-Level Information:

  Sometimes, a flow-level view of the information in MGEN log files is needed,
  e.g., when one expects different flows to take different paths through the
  network under test.  In this case one would expect only a subset of the flows
  to be affected by, e.g., a localized failure event.

  Tools for flow-level information:
   - flow-info-csv
   - flow-latency-analyzer
   - flow-outages
   - flow-statistics
   - packet-loss-analyzer

Packet-Level Information:

  Sometimes, a packet-level view of the information in MGEN log files is needed,
  e.g., when the network under test is expected to deliver every packet, but
  is delaying, losing, reordering, or duplicating a small number of packets.

  Tools for packet-level information:
   - packet-dup-loss-analyzer
   - packet-loss-filter
   - packet-reorder-filter

Analyzing mgen-tools output:

  Sometimes, the output of one of the mgen-tools needs to be analyzed further.
  E.g., the network under test is experiencing several events one after the
  other where each event should affect only some flows, and packet level
  information is collected.  In this case, a simple way to check the time of
  the last reported problem (result message) per flow can be useful.

  Tools to analyze mgen-tools output:
   - last-flow-result


Data Conditioning:
------------------

Sometimes, the network under test is expected to exhibit an initial settling
phase before reaching nominal behavior.  Or some part of an MGEN log file is
known to coincide with unusual network conditions and shall be excluded from
analysis.

  Data conditioning tools:
   - ignore-sequence_numbers
   - ignore-startup_phase


Format Conversion:
------------------

Instead of writing specific MGEN log file data importers for third-party
applications, MGEN log file data can be converted to common formats, e.g., CSV.

  Format conversion tools:
   - recv-to-csv


Third Party Software:
---------------------

Tools producing CSV output can be used in combination with third party software,
e.g., LibreOffice Calc, GNU datamash, q - Text as Data, or Miller.

To analyze only a subset of flows from (a collection of) MGEN log files, "grep"
can be used to extract the intended flows, using the output of grep as input
to the respective tool.

To select a subset of flows from tools that create hierarchical output, i.e.,
a line naming the flow without indentation followed by indented flow-specific
information, "section" can be used.


Links:
------

Multi-Generator (MGEN) Network Test Tool URL:
https://www.nrl.navy.mil/Our-Work/Areas-of-Research/Information-Technology/NCS/MGEN/

Older MGEN URLs that redirect to the above:
 - http://cs.itd.nrl.navy.mil/work/mgen/index.php
 - https://www.nrl.navy.mil/itd/ncs/products/mgen

The MGEN source code is available on GitHub:
https://github.com/USNavalResearchLaboratory/mgen

Links to tools mentioned in this README that are not part of POSIX:
 - GNU datamash:      https://www.gnu.org/software/datamash/
 - LibreOffice:       https://www.libreoffice.org/
 - Miller:            https://github.com/johnkerl/miller/
 - q - Text as Data:  https://harelba.github.io/q/
 - section:           https://www.unix-ag.uni-kl.de/~auerswal/section/

Free software implementations of the POSIX tools "awk" and "grep" can be
obtained from the GNU project:
 - GNU awk (gawk):    https://www.gnu.org/software/gawk/
 - GNU grep:          https://www.gnu.org/software/grep/

[The above links to third party software are valid as of 2021-07-05.
 Please note that links may change, as well as the content behind links.]