YANG is the data modeling language chosen by the Telco industry to represent network devices' configurations and states.
The data, modeled using YANG, is gathered (or sent) from (to) the devices over the network using protocols like NETCONF/RESTCONF and typically encoded using JSON or XML. The data sent/received via NETCONF is usually going over SSH (or, more generic, TLS).
In 2016 Google released a new RPC framework called gRPC which is now adopted by all the main Vendors to retrieve/send data to the network. Currently the main implementations are gNMI and gRPC Dial-in/Dial-out
-
gRPC is generally faster to develop with: it's using Protocol Buffers as the Interface Description Language and an ad-hoc compiler to automagically generate the associated skeleton code.
-
gRPC is out-of-the-box supporting multiple programming languages and gives you the freedom to choose the one which fits best your skills, independently from the existing implementations (the Protobuff file is pre-defining the specs for both client and server).
-
gRPC is efficient and scalable: it's taking advantage of the efficiency coming from HTTP/2 and thanks to Protocol Buffers, the exchanged data is binary encoded which considerably reduces the message size.
gNMI (gRPC Network Management Interface) is using the gRPC framework and a standardized Protobuff file to implement a solution to fully operate the network.
With what concerning gRPC dial-in/dial-out The data-stream is always pushed-out from the router, however in case of Dial-in the connection is initiated by the collector, vice-versa with Dial-out the connection is initiated by the router. The biggest benefit of Dial-in over Dial-out is that you're gonna have a single channel usable for both telemetry & routers configuration.
The main aim of this project it to use the gRPC framework to implement a multi-vendor gRPC Dial-out server. Currently the application is implementing the Cisco's gRPC Dial-out .proto file , the Juniper's gRPC Dial-out .proto file and the Huawei's gRPC Dial-out .proto file.
With some restrictions, due to the technical limitations imposed by the Vendors' implementations, the supported encodings are JSON, GPB-KV and GPB-Comapct (Huawei openconfig-interfaces only).
Vendor | OS Version | Encoding | .proto file |
---|---|---|---|
Cisco | XR (7.4.1@NSC-540) | JSON, GPB-KV | XR Telemetry .proto |
Cisco | XE (17.06.01prd7@C8000V) | GPB-KV | XE Telemetry .proto |
Juniper | Junos (20.4R3-S2.6@mx10003) | JSON-GNMI | Junos Telemetry .proto (Download section) |
Huawei | VRP (V800R021C10SPC300T@NE40E) | JSON | VRP Telemetry .proto |
Huawei | VRP (V800R021C10SPC300T@NE40E) | GPB-Compact | OpenConfig Interfaces .proto |
-
It's recommended to compile gRPC and all the associated libraries from scratch. The gRPC's Quick start guide is describing in detail the compile/install procedure. If you're running a Debian derived Linux distribution you can also refer to the Alfanetti documentation.
-
Libraries like jsoncpp & librdkafka are also required.
Debian$ sudo apt install libjsoncpp-dev librdkafka-dev libconfig++-dev libspdlog-dev
Redhat$ sudo yum install jsoncpp-devel librdkafka-devel libconfig-devel spdlog-devel
$ cd /opt
$ git clone https://github.com/scuzzilla/mdt-dialout-collector.git
$ cd mdt-dialout-collector
$ mkdir build
$ cmake ../
$ make
$ ./bin/mdt-dialout-collector
If you refer to the Alfanetti documentation to compile gRPC, you might want to configure the $PATH variable to include your local "libs/header" folder.
$ export MY_INSTALL_DIR=$HOME/.local
$ export PATH="$MY_INSTALL_DIR/bin:$PATH"
On CentOS you might need to modify the pkg-config path to allow cmake to find all required libraries.
$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig/
On CentOS the "spdlog" library should use the external 'fmt' library.
uncomment #define SPDLOG_FMT_EXTERNAL from /usr/include/spdlog/tweakme.h before
running the make command
The default configuration location is following the FHS recommendation, therefore:
- the application itself is located within the "/opt/mdt-dialout-collector/bin" folder
- its configuration is located within the "/etc/opt/mdt-dialout-collector" folder
Here below is an example of commented configuration:
#### mdt-dialout-collector - main
## physical interface where to bind the daemon
iface = "eth0";
## socket dedicated to the cisco's data-stream
ipv4_socket_cisco = "0.0.0.0:10007";
## socket dedicated to the huawei's data-stream
ipv4_socket_huawei = "0.0.0.0:10008";
## socket dedicated to the juniper's data-stream
ipv4_socket_juniper = "0.0.0.0:10009";
## network replies: fine control on the amount of messages received within a single
## session - valid range: "0" < replies < "1000" - (default = "0" => unlimited)
replies_cisco = "10";
replies_juniper = "100";
replies_huawei = "1000";
## workers (threads) per vendor - default = "1"
cisco_workers = "1";
juniper_workers = "1";
huawei_workers = "1";
## logging:
## Syslog support: default => syslog = "flase" | facility (static) default => LOG_USER
syslog = "true";
## Console support: default => console_log = "true"
console_log = "false";
## Severity level: default => spdlog_level = "info" | supported [debug, info, warn, error, off]
spdlog_level = "debug";
#### mdt-dialout-collector - data-flow manipulation
## simplified JSON after GPB/GPB-KV decoding - default = "true"
enable_cisco_gpbkv2json = "false";
## standard JSON after GPB/GPB-KV deconding - default = "false"
enable_cisco_message_to_json_string = "true";
## data-flow enrichment with node_id/platform_id - default = "false"
## for additional details refer to "csv/README.md" or "ptm/README.md"
## CSV format - default = "false"
enable_label_encode_as_map = "false";
## label_map_csv_path - default = "/opt/mdt_dialout_collector/csv/label_map.csv"
label_map_csv_path = "/define/here/your/custom_path.csv"
## PTM format (pmacct's pretag) - default = "false"
enable_label_encode_as_map_ptm = "true";
## label_map_ptm_path - default = "/opt/mdt_dialout_collector/ptm/label_map.ptm"
label_map_ptm_path = "/define/here/your/custom_path.ptm"
#### mdt-dialout-collector - kafka-producer
# https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
bootstrap_servers = "kafka.brockers.net:9093";
topic = "json.topic";
enable_idempotence = "true";
client_id = "mdt-dialout-collector";
# valid options are either plaintext or ssl
security_protocol = "ssl";
ssl_key_location = "/opt/mdt-dialout-collector/cert/collectors.key";
ssl_certificate_location = "/opt/mdt-dialout-collector/cert/collectors.crt";
ssl_ca_location = "/opt/mdt-dialout-collector/cert/sbd_root_ca.crt";
log_level = "0";
Libraries / .proto files | License |
---|---|
JsonCpp | MIT License |
librdkafka | BSD 2-Clause License |
Modern C++ Kafka API | Apache License Version 2.0 |
libconfig | LGPL v2.1 |
rapidcsv | BSD-3-Clause license |
spdlog | MIT License |
gRPC | BSD 3-Clause License |
Cisco dial-out .proto | Apache License Version 2.0 |
Cisco telemetry .proto | Apache License Version 2.0 |
Junos Telemetry .proto (Download section) | Apache License Version 2.0 |
Huawei dial-out .proto | N/A |
Huawei telemetry .proto | N/A |
mdt-dialout-collector | MIT License |