fg-networking / mgen-tools Goto Github PK
View Code? Open in Web Editor NEWmgen-tools - tools to analyze MGEN logs
License: MIT License
mgen-tools - tools to analyze MGEN logs
License: MIT License
Tools to Analyze MGEN Log Files: ================================ This repository contains tools that can be used to analyze log files created by the Multi-Generator (MGEN) network test tool. The tools are implemented in Awk and require a POSIX compatible Awk interpreter. They are expected to work with both GNU awk (gawk) and mawk. List of Tools: -------------- average-bandwidth compute overall average bandwidth and other statistics (aggregated information over all flows) flow-info-csv create CSV table (with header) of info about the streams included in the log like src/dst IPs, creates lists of values for more dynamic streams (or multiply used stream IDs) flow-latency-analyzer calculate min, max, and average latency of flows flow-outages print per flow per outage times and lost packets flow-statistics compute per-flow statistics ignore-sequence_numbers keep packets with sequence numbers in [low,high] ignore-startup_phase remove the first n receive events from every flow last-flow-result print the last mgen-tools result message for each flow (this tool works on output of other mgen-tools, not on MGEN logs) packet-dup-loss-analyzer print lost or duplicated packets, one line per packet packet-loss-analyzer count number of received packets and print statistics for each flow and a summary over all flows packet-loss-filter print a message for each lost or too late packet packet-reorder-filter print a message for each reordered packet recv-to-csv create CSV table (with header) of receive events (time stamps are converted into seconds) Controlling Tool Operation with Variables: ------------------------------------------ Some tools support setting variables via Awk option "-v VARIABLE=VALUE": - flow-info-csv uses the variable "sep" to specify the separator string (default: sep=,) and the variable "lstsep" to specify the list separator for the potential lists of values (default: lstsep=' ') - ignore-sequence_numbers uses the variables "low" and "high" to ignore receive events with sequence numbers outside the closed interval [low,high] (defaults: low=0, high=infinity) - ignore-startup_phase uses the variable "skip" to ignore the first skip+1 receive events per flow (default: skip=92) - recv-to-csv uses the variable "sep" to specify the separator string (default: sep=,) Delayed or Immediate Output: ---------------------------- Some tools read all input before any output, others produce output immediately: - Immediate output: - ignore-sequence_numbers - ignore-startup_phase - packet-loss-filter - packet-reorder-filter - recv-to-csv - Output generation after reading input completely: - average-bandwidth - flow-info-csv - flow-latency-analyzer - flow-outages - flow-statistics - last-flow-result - packet-dup-loss-analyzer - packet-loss-analyzer Interrupting (CTRL+C) a tool that needs to read all input results in no output. Use Cases: ---------- High-Level Overview: Sometimes, a high-level view of the information in MGEN log files is needed, e.g., when sending many data flows with the expectation that the network under test cannot deliver all the data. Tools for a high-level overview: - average-bandwidth Flow-Level Information: Sometimes, a flow-level view of the information in MGEN log files is needed, e.g., when one expects different flows to take different paths through the network under test. In this case one would expect only a subset of the flows to be affected by, e.g., a localized failure event. Tools for flow-level information: - flow-info-csv - flow-latency-analyzer - flow-outages - flow-statistics - packet-loss-analyzer Packet-Level Information: Sometimes, a packet-level view of the information in MGEN log files is needed, e.g., when the network under test is expected to deliver every packet, but is delaying, losing, reordering, or duplicating a small number of packets. Tools for packet-level information: - packet-dup-loss-analyzer - packet-loss-filter - packet-reorder-filter Analyzing mgen-tools output: Sometimes, the output of one of the mgen-tools needs to be analyzed further. E.g., the network under test is experiencing several events one after the other where each event should affect only some flows, and packet level information is collected. In this case, a simple way to check the time of the last reported problem (result message) per flow can be useful. Tools to analyze mgen-tools output: - last-flow-result Data Conditioning: ------------------ Sometimes, the network under test is expected to exhibit an initial settling phase before reaching nominal behavior. Or some part of an MGEN log file is known to coincide with unusual network conditions and shall be excluded from analysis. Data conditioning tools: - ignore-sequence_numbers - ignore-startup_phase Format Conversion: ------------------ Instead of writing specific MGEN log file data importers for third-party applications, MGEN log file data can be converted to common formats, e.g., CSV. Format conversion tools: - recv-to-csv Third Party Software: --------------------- Tools producing CSV output can be used in combination with third party software, e.g., LibreOffice Calc, GNU datamash, q - Text as Data, or Miller. To analyze only a subset of flows from (a collection of) MGEN log files, "grep" can be used to extract the intended flows, using the output of grep as input to the respective tool. To select a subset of flows from tools that create hierarchical output, i.e., a line naming the flow without indentation followed by indented flow-specific information, "section" can be used. Links: ------ Multi-Generator (MGEN) Network Test Tool URL: https://www.nrl.navy.mil/Our-Work/Areas-of-Research/Information-Technology/NCS/MGEN/ Older MGEN URLs that redirect to the above: - http://cs.itd.nrl.navy.mil/work/mgen/index.php - https://www.nrl.navy.mil/itd/ncs/products/mgen The MGEN source code is available on GitHub: https://github.com/USNavalResearchLaboratory/mgen Links to tools mentioned in this README that are not part of POSIX: - GNU datamash: https://www.gnu.org/software/datamash/ - LibreOffice: https://www.libreoffice.org/ - Miller: https://github.com/johnkerl/miller/ - q - Text as Data: https://harelba.github.io/q/ - section: https://www.unix-ag.uni-kl.de/~auerswal/section/ Free software implementations of the POSIX tools "awk" and "grep" can be obtained from the GNU project: - GNU awk (gawk): https://www.gnu.org/software/gawk/ - GNU grep: https://www.gnu.org/software/grep/ [The above links to third party software are valid as of 2021-07-05. Please note that links may change, as well as the content behind links.]
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.