stratum / stratum

Stratum is an open source silicon-independent switch operating system for software defined networks.

Stratum - Enabling the era of next-generation SDN

Stratum is an open source silicon-independent switch operating system for software defined networks. It is building an open, minimal production-ready distribution for white box switches. Stratum exposes a set of next-generation SDN interfaces including P4Runtime and OpenConfig, enabling interchangeability of forwarding devices and programmability of forwarding behaviors. Current support includes Barefoot Tofino and Broadcom Tomahawk devices, as well as the bmv2 software switch.

Build status (main):

Documentation

Getting Started with SDN / P4 / Stratum

If you're completly new to the world of P4 and are looking for step-by-step instructions with batteries included, the NG-SDN tutorial is a good starting point. For material more focused on P4 as a language, have a look at the p4lang tutorials. Finally, there is the "Software-Defined Networks: A Systems Approach" book (HTML) for comprehensive coverage of SDN. Also see the chapter about Stratum in it.

Table of Contents

Development

• Code of Conduct
• Contribution Guidelines
• IDE Setup
• Release Plan
• Logging Guide
• PHAL Development Tutorial
• Stratum Configurators Design Document
• Stratum PHAL Overview (Physical Hardware Access Layer)

Platforms

• Tofino
  • Building Stratum for Barefoot Tofino based switches
  • Running Stratum on a Barefoot Tofino based switch
  • Testing Stratum on a Barefoot Tofino based switch
  • Pushing a pipeline to a Barefoot Tofino based switch
• Broadcom
  • How to Add a Dataplane Feature to Stratum (BCM)
• Intel NP4
• bmv2
• Dummy Switch

Tools

• gNMI CLI Tool
• Tofino Pipeline Builder
• Stratum-Enabled Mininet
• P4Runtime write request replay tool
• ChassisConfig Migrator
• PHAL CLI Tool
• ONLP CLI Tool
• ONLP PHAL CLI Tool

Other Documentation

• SD-Fabric
• Trellis + Stratum example
• Continuous Certification Program (CCP)
• Community Channels
• Talks
• Test Vectors
• Stratum example pipelines
  • Main FPM pipeline
• Supported gNMI paths
• Config Monitoring Service
• Transponder Abstraction Interface (TAI)
• P4C backend
• CDLang gNMI Compliance Definition Language
• gNMI/OpenConfig Contract Definition Language tools
• gNMI Test Scenarios
• SSL/TLS Support and Security

Component Overview

P4Runtime provides a flexible mechanism for configuring the forwarding pipeline on a network switch.

gNMI is a framework for network device management that uses gRPC as the transport mechanism.

SDKLT is used to program fixed-pipeline switches using the Tomahawk chip from Broadcom.

Supported Devices

Tofino

• APS Networks BF2556X-1T
• APS Networks BF6064X-T
• Delta AG9064v1
• Edgecore Wedge100BF-32QS
• Edgecore Wedge100BF-32X
• Edgecore Wedge100BF-65X
• Inventec D5254
• Inventec D5264Q28B
• Inventec D10056
• Inventec D10064
• Netberg Aurora 610
• Netberg Aurora 710
• Netberg Aurora 750

Note: Pre-built Stratum binaries for Tofino are available for recent Barefoot SDE releases in the Releases page. Vendors, customers, and researchers with access to the Barefoot SDE can build Stratum themselves. For access to the SDE, please contact an Intel representative.

Broadcom

• Tomahawk (SDKLT, OpenNSA, SDK6)
  • Edgecore AS7712-32X
  • Edgecore AS7716-24XC Cassini
  • Dell Z9100
  • Inventec D7032
  • QCT QuantaMesh T7032-IX1
• Tomahawk+ (SDK6)
  • Edgecore AS7716-24SC Cassini
  • Inventec D7054
• Tomahawk2 (OpenNSA, SDK6)
• Tomahawk3 (SDK6)
• Trident2 (OpenNSA, SDK6)
  • Edgecore AS6712
  • Inventec D6254

Note: Pre-built Stratum binaries for SDKLT and OpenNSA are available in the Releases page. Vendors and customers with a Broadcom SLA can build Stratum themselves using the Broadcom Switch SDK (SDK6); versions 6.5.17 and 6.5.19 are tested. If you would like to use Stratum with SDK6 and you do not have an SLA with Broadcom, contact a Broadcom representative or your switch vendor.

Smart NIC

• Intel PAC N3000

Software Switches

• bmv2
• dummy

Source code

This repository contains source code for a reference implementation of the P4Runtime and gNMI services, that serves as the hardware abstraction layer for a network switch. It has been successfully prototyped at Google, running on production hardware on a data center network subsystem.

Development environment

We provide a script to create a Docker development environment for Stratum.

To start the container, run the following in your stratum directory:

./setup_dev_env.sh

The script will build a Docker image using Dockerfile.dev and run a bash session in it. This directory will be mounted in the Docker image and you will be able to run git, edit code, and build Stratum / run tests using Bazel.

For more information, you can see a complete list of arguments by running: setup_dev_env.sh -h

If you are using Docker on Linux, make sure that you can use Docker as a non-root user, otherwise you will not be able to run setup_dev_env.sh:

sudo usermod -aG docker $USER

stratum's Issues

Create document on how to add a new BCM feature

Check if bazel caching in CI can be improved

We are not using --disk_cache which is the best option for caching.

Error when Trying to run the Binary

When I was trying to run the binary with BSP, the following errors occur:
bf-sde> 01-22 09:35:42.123896 [onlp_platform_defaults] The default implementation of onlp_platformi_get() has been called.
01-22 09:35:42.123984 [onlp_platform_defaults] This can happen for the following reasons, all fatal:
01-22 09:35:42.124032 [onlp_platform_defaults] * The ONLP build configuration is incorrect.
01-22 09:35:42.124063 [onlp_platform_defaults] * The ONLP platform library for this platform does not contain the onlp_platformi_get() symbol.
01-22 09:35:42.124093 [onlp_platform_defaults] * The ONLP platform shared libraries are not setup properly before we executed.
01-22 09:35:42.124123 [onlp_platform_defaults] * The platform cannot continue until this issue is resolved.

When I installed Stratum, I didn't set the variable: ONLP_INSTALL, so ONLP should be installed by bazel. I also searched for the existence of onlp_platformi_get() but could not find anything.

How could I solve this problem?

Merge BcmChassisMaps

Instead of having a separate chassis map file for each platform, the code already supports reading from one big list and picking the correct one by an Id:

stratum/stratum/hal/lib/bcm/bcm_chassis_manager.cc

Lines 1489 to 1502 in 83c0b32

	// Read the proto from the path given by base_bcm_chassis_map_file flag.
	BcmChassisMapList bcm_chassis_map_list;
	RETURN_IF_ERROR(ReadProtoFromTextFile(FLAGS_base_bcm_chassis_map_file,
	&bcm_chassis_map_list));
	base_bcm_chassis_map->Clear();
	bool found = false;
	for (const auto& bcm_chassis_map : bcm_chassis_map_list.bcm_chassis_maps()) {
	if (bcm_chassis_map_id.empty() ||
	bcm_chassis_map_id == bcm_chassis_map.id()) {
	*base_bcm_chassis_map = bcm_chassis_map;
	found = true;
	break;
	}
	}

We could consolidate our maps into one file and ship that. Selection happens in the chassis config.
Currently this feature is hidden behind the google_config field, but we could add the same field directly in vendor_config:

stratum/stratum/hal/lib/bcm/bcm_chassis_manager.cc

Lines 511 to 517 in 83c0b32

	// Load base_bcm_chassis_map before anything else if not done before.
	std::string bcm_chassis_map_id = "";
	if (config.has_vendor_config() &&
	config.vendor_config().has_google_config()) {
	bcm_chassis_map_id =
	config.vendor_config().google_config().bcm_chassis_map_id();
	}

Race condition on Subscribe handler of gNMI sample subscriptions

The following describes how gNMI subscriptions are implemented:

• Sampling subscriptions are using the TimerDaemon to trigger periodic TimerEvents inside GnmiPublisher
• TimerDaemon keeps track of all triggers in a heap of std::weak_ptrs. Events are canceled by freeing the owing std::shared_ptr. Triggers run in a separate thread.
• GnmiPublisher also keeps a list of all subscriptions in EventHandlerList.
• ConfigMonitoringService::DoSubscribe returns too early and causes gRPC to free the streams resources, which are still referenced in the eventhandlers and timer triggers.
• When using installing a sampling subscription with a very short interval (<100ms), the callback basically runs continuously.

This is how the bug occurs:

0. Client installs sampling subscription with interval of 1ms
1. The event callback runs and checks if the weak_ptr to subscription is still valid, which it is.
2. The handler traverses the leafs and collects the relevant values. This takes some time.
3. Client closes stream
4. ConfigMonitoringService::DoSubscribe returns, frees gRPC stream
5. Handler tries to write results to stream, causes segfault or use-after-free in ASAN build

Logs, notice the timer after stream close:

I1009 16:31:21.450788 18707 config_monitoring_service.cc:401] Initial Subscribe request from ipv4:10.1.8.133:56836 over stream 0x7fc33effc680.
I1009 16:31:21.450845 18707 config_monitoring_service.cc:403] SubscribeRequest: subscribe { subscription { path { elem { name: "interfaces" } elem { name: "interface" key { key: "name" value: "4/0" } } elem { name: "state" } elem { name: "counters" } } mode: SAMPLE sample_interval: 1 } updates_only: true }
I1009 16:31:21.459323 18707 timer_daemon.cc:143] Registered timer.
I1009 16:31:21.535076 18644 timer_daemon.cc:84] Timer has been triggered!
I1009 16:31:21.610148 18644 timer_daemon.cc:84] Timer has been triggered!
[...]
I1009 16:31:23.655316 18644 timer_daemon.cc:84] Timer has been triggered!
I1009 16:31:23.719635 18707 config_monitoring_service.cc:593] Subscribe stream 0x7fc33effc680 from ipv4:10.1.8.133:56836 has been closed.
I1009 16:31:23.730439 18644 timer_daemon.cc:84] Timer has been triggered!

Set bazel copts and/or linkopts on Stratum targets

Setting flags on all our targets is preferable to global flags via .bazelrc, because the later applies to ALL code, including external. This limits the options we can actually use. Enabling all warnings for example results in an useless torrent of compiler output from code we have no influence over. External code also does not always support instrumentation and breaks if compiled with, e.g., ASAN.

Abseil does it this way:

cc_library(
    name = "compressed_tuple",
    hdrs = ["internal/compressed_tuple.h"],
    copts = ABSL_DEFAULT_COPTS,
    linkopts = ABSL_DEFAULT_LINKOPTS,
    deps = [
        "//absl/utility",
    ],
)

This is a huge change LoC wise, but could probably be automated fairly easily.

Some of the flags we could set:

• -Wno-unused-function
• -Wno-unused-lambda-capture
• -Werror=thread-safety-reference
• -Werror=enum-compare
• -Werror=deprecated-declarations

Link to C++ style code in CONTRIBUTING file

https://google.github.io/styleguide/cppguide.html

Publish `stratum_bcm` binary on github

We build stratum_bcm as part of CI. The runtime platform we support is fairly constrained as well, so a common pre-build binary should work for most users.

I tied a docker-build binary and it worked on the AS7712.

Cleanup Dummy switch target

• Remove the dummy phal or make it become a phal backend.
• Rewrite dummy CLI with C++ (or implement it as an embedded CLI)

Is there any plan about SONiC + Stratum?

SONiC could use Stratum as an implementation of SAI , is there any plan?

Issue in installing BMV2

While trying to install the bmv2 , I am getting following error

src/pi_imp.cpp: In function ‘pi_status_t _pi_init(int*, void*)’:
src/pi_imp.cpp:65:13: error: conflicting declaration of C function ‘pi_status_t _pi_init(int*, void*)’
pi_status_t _pi_init(int *abi_version, void extra) {
^~~~~~~~
In file included from src/pi_imp.cpp:26:0:
/usr/local/include/PI/target/pi_imp.h:30:13: note: previous declaration ‘pi_status_t _pi_init(void)’
pi_status_t _pi_init(void *extra);
^~~~~~~~
src/pi_imp.cpp:68:18: error: ‘PI_ABI_VERSION’ was not declared in this scope
*abi_version = PI_ABI_VERSION;
^~~~~~~~~~~~~~
src/pi_imp.cpp:68:18: note: suggested alternative: ‘_SC_VERSION’
*abi_version = PI_ABI_VERSION;
^~~~~~~~~~~~~~
_SC_VERSION
Makefile:537: recipe for target 'src/pi_imp.lo' failed
make[2]: *** [src/pi_imp.lo] Error 1
make[2]: *** Waiting for unfinished jobs....
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -I../include -I../include -isystem../third_party/jsoncpp/include -isystem../third_party/spdlog -I../targets/simple_switch -DWITH_SIMPLE_SWITCH -isystem/include -Wall -Wextra -pthread -g -O2 -MT src/group_selection.lo -MD -MP -MF src/.deps/group_selection.Tpo -c src/group_selection.cpp -o src/group_selection.o >/dev/null 2>&1
libtool: compile: g++ -DHAVE_CONFIG_H -I. -I.. -I../include -I../include -isystem../third_party/jsoncpp/include -isystem../third_party/spdlog -I../targets/simple_switch -DWITH_SIMPLE_SWITCH -isystem/include -Wall -Wextra -pthread -g -O2 -MT src/pi_clone_imp.lo -MD -MP -MF src/.deps/pi_clone_imp.Tpo -c src/pi_clone_imp.cpp -o src/pi_clone_imp.o >/dev/null 2>&1
make[2]: Leaving directory '/home/deba/bmv2_install/bmv2/PI'
Makefile:454: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/deba/bmv2_install/bmv2'
Makefile:384: recipe for target 'all' failed
make: *** [all] Error 2

Need cmake and library realize

 I am a newer interesting in stratum, I have tried the stratum p4runtime function，build and run in my barefoot device. so，I have two suggests as follows,
Frist my device os building system is based on buildroot， I hope you can offer a common cmake compile method，not now bazel only.
Second，all the stratum code build as only a binary bin，but some users may have their own grpc server realize，they hope stratum project supply  APIs ，not only a binay binary bin。So, in future, I hope some librarys which supply API for the p4runtime.

Can't see bmv2-stratum-switch full log output

When I send a packet to the router, the only message I see (regarding packet forwarding) in the log is this:

[11:52:25.183] [bmv2] [E] [thread 55] [5.0] [cxt 0] Checksum 'cksum_1' is not correct

I realised that in this line on the top of the log file

stratum_bmv2 -device_id=1 -chassis_config_file=/tmp/s1/chassis-config.txt -forwarding_pipeline_configs_file=/tmp/s1/pipe.txt -persistent_config_dir=/tmp/s1 -initial_pipeline=/root/dummy.json -cpu_port=255 -external_stratum_urls=0.0.0.0:50001 -local_stratum_url=localhost:41164 -max_num_controllers_per_node=10 -write_req_log_file=/tmp/s1/write-reqs.txt -logtosyslog=false -logtostderr=true

The -logtosyslog=false and -logtostderr=true. So maybe that is why I only have error message regarding wrong checksum (intentionally bad written). I changed the parameter in the stratum.py script in the mininet docker so now I see this in the log output:

stratum_bmv2 -device_id=1 -chassis_config_file=/tmp/s1/chassis-config.txt -forwarding_pipeline_configs_file=/tmp/s1/pipe.txt -persistent_config_dir=/tmp/s1 -initial_pipeline=/root/dummy.json -cpu_port=255 -external_stratum_urls=0.0.0.0:50001 -local_stratum_url=localhost:41164 -max_num_controllers_per_node=10 -write_req_log_file=/tmp/s1/write-reqs.txt -logtosyslog=true -logtostderr=true

But still I only see the error message.

I would like to see full log. Any clues about how to do it?

Thanks in advance!

Enable cpplint and clang_format checks in CI

At least on newly committed files.

Fix ChassisConfig in YangParseTreeTest

Currently, the YangParsetreeTest generates the tree manually. The chassis config will be empty if we perform gNMI set operation (see https://github.com/stratum/stratum/blob/master/stratum/hal/lib/common/yang_parse_tree_test.cc#L341)

In some cases, the set functor modify the chassis config (e.g., set port speed and mac). The code coverage will drop if we use empty chassis config in the test.

what bazel version should i use to build the stratum

Describe the bug
A clear and concise description of what the bug is.
I met different kinds of errors when I have tried use different bazel version to build stratum for tofino SDE.
Environment
Help us understand by giving us some context of your setup.
when I use bazel 2.0 , the error is :

ERROR: /home/ubuntu/asterfusion/stratum/stratum/hal/bin/barefoot/BUILD:24:1: file '//bazel:rules.bzl' does not contain symbol 'stratum_cc_binary'
ERROR: Skipping '//stratum/hal/bin/barefoot:stratum_bf': no such target '//stratum/hal/bin/barefoot:stratum_bf': target 'stratum_bf' not declared in package 'stratum/hal/bin/barefoot' defined by /home/ubuntu/asterfusion/stratum/stratum/hal/bin/barefoot/BUILD
WARNING: Target pattern parsing failed.
ERROR: no such target '//stratum/hal/bin/barefoot:stratum_bf': target 'stratum_bf' not declared in package 'stratum/hal/bin/barefoot' defined by /home/ubuntu/asterfusion/stratum/stratum/hal/bin/barefoot/BUILD

when i use bazel 0.20 ，the error is

ERROR: error loading package '': in /home/ubuntu/.cache/bazel/_bazel_ubuntu/69559fb472981a06aaa6cb6210fe0eb9/external/com_github_grpc_grpc/bazel/grpc_python_deps.bzl: Encountered error while reading extension file 'requirements.bzl': no such package '@grpc_python_dependencies//': pip_import failed: Collecting coverage>=4.0 (from -r /home/ubuntu/.cache/bazel/_bazel_ubuntu/69559fb472981a06aaa6cb6210fe0eb9/external/com_github_grpc_grpc/requirements.bazel.txt (line 2))
 (  Could not find a version that satisfies the requirement coverage>=4.0 (from -r /home/ubuntu/.cache/bazel/_bazel_ubuntu/69559fb472981a06aaa6cb6210fe0eb9/external/com_github_grpc_grpc/requirements.bazel.txt (line 2)) (from versions: )
No matching distribution found for coverage>=4.0 (from -r /home/ubuntu/.cache/bazel/_bazel_ubuntu/69559fb472981a06aaa6cb6210fe0eb9/external/com_github_grpc_grpc/requirements.bazel.txt (line 2))
)
ERROR: error loading package '': in /home/ubuntu/.cache/bazel/_bazel_ubuntu/69559fb472981a06aaa6cb6210fe0eb9/external/com_github_grpc_grpc/bazel/grpc_python_deps.bzl: Encountered error while reading extension file 'requirements.bzl': no such package '@grpc_python_dependencies//': pip_import failed: Collecting coverage>=4.0 (from -r /home/ubuntu/.cache/bazel/_bazel_ubuntu/69559fb472981a06aaa6cb6210fe0eb9/external/com_github_grpc_grpc/requirements.bazel.txt (line 2))
 (  Could not find a version that satisfies the requirement coverage>=4.0 (from -r /home/ubuntu/.cache/bazel/_bazel_ubuntu/69559fb472981a06aaa6cb6210fe0eb9/external/com_github_grpc_grpc/requirements.bazel.txt (line 2)) (from versions: )
No matching distribution found for coverage>=4.0 (from -r /home/ubuntu/.cache/bazel/_bazel_ubuntu/69559fb472981a06aaa6cb6210fe0eb9/external/com_github_grpc_grpc/requirements.bazel.txt (line 2))
)

If building from source:
bazel version
2.0.0
0.20.0

Switch:

tofine 
SDE-9.0.0

To Reproduce
Steps to reproduce the behavior:

1. Use this config '....'
2. Run this command '....'
3. See error

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Update CONTRIBUTING file

We should guide developers on how to make a PR, what tools to run before commit and how to test.

Is there any detailed guides or documents?

Hi all, I want to study the stratum project, but I think it lacks detailed documents. Can you tell how to start or where to find more guides?

Unable to add reviewers to PRs

I don't know if I'm the only one seeing this issue, but when submitting a PR I cannot request reviews as the Github button/input form on the right is not shown/clickable. Perhaps, we need to update permissions on GitHub to allow anyone to request reviews? Otherwise, I think we should change this line in the contributor's guidelines:

Request review. You can add anyone you think is helpful, but only Collaborators and above can give final approval.

Ping @pudelkoM

Add SDPX license header to source files

We can also drop the full license text from the top of each file.

Support relative path names

Stratum should support relative path names in its flags.

Given the following startup command:

./stratum_bcm -base_bcm_chassis_map_file=./stratum_configs/x86-64-accton-as7712-32x-r0/chassis_map.pb.txt [...]

Stratum fails with:

E0121 18:44:33.150527 11245 utils.cc:120] StratumErrorSpace::ERR_FILE_NOT_FOUND: ./stratum_configs/x86-64-accton-as7712-32x-r0/chassis_map.pb.txt not found.

Relevant code:

stratum/stratum/lib/utils.h

Lines 155 to 159 in 455ede1

	// Checks to see if a path exists.
	inline bool PathExists(const std::string& path) {
	struct stat stbuf;
	return (stat(path.c_str(), &stbuf) >= 0);
	}

PathExists uses stat which should work with relative paths: https://stackoverflow.com/questions/18460849/can-we-use-stat-with-relative-path-to-a-file-in-c
I suspect cwd is overwritten somewhere.

Run stratum failed on brcm Tomahawk platfrom

Hello,everyone
I am new for this. I try to run stratum on brcm Tomahawk platform,But it's failed. as below log.
In this test, pipeline_cfg.pb.txt and slx_serdes_db.pb.txt is empty file.
Does anyone help me check this issue? Thank you very much.

I create a stratum.flags:

-external_hercules_urls=0.0.0.0:28000
-persistent_config_dir=/etc/stratum
-base_bcm_chassis_map_file=/root/sdklt/config/slx/slx_chassis_map.pb.txt
-chassis_config_file=/root/sdklt/config/slx/slx_chassis_config.pb.txt
-bcm_sdk_config_file=/root/sdklt/config/slx/slx_32x100.config.yml
-bcm_hardware_specs_file=/root/sdklt/config/slx/bcm_hardware_specs.pb.txt
-forwarding_pipeline_configs_file=/root/sdklt/config/pipeline_cfg.pb.txt
-write_req_log_file=/tmp/stratum/p4_writes.pb.txt
-bcm_serdes_db_proto_file=/root/sdklt/config/slx/slx_serdes_db.pb.txt
-bcm_sdk_checkpoint_dir=/tmp/stratum/bcm_chkpt
-colorlogtostderr
-alsologtostderr
-logtosyslog=true
-v=1

root@localhost# ./stratum_bcm -flagfile=stratum.flags
I1204 10:39:21.604415 3305 main.cc:89] Starting Stratum in STANDALONE mode for a Broadcom-based switch...

Message from syslogd@localhost at Dec 4 10:39:21 ...
stratum_bcm[3305]: INFO main.cc:89] Starting Stratum in STANDALONE mode for a Broadcom-based switch...
I1204 10:39:21.605490 3305 timer_daemon.cc:101] The timer daemon has been started.
I1204 10:39:21.605700 3305 hal.cc:133] HAL sanity checks all passed.

Message from syslogd@localhost at Dec 4 10:39:21 ...
stratum_bcm[3305]: INFO timer_daemon.cc:101] The timer daemon has been started.
I1204 10:39:21.605769 3305 hal.cc:139] Setting up HAL in COLDBOOT mode...

Message from syslogd@localhost at Dec 4 10:39:21 ...
stratum_bcm[3305]: INFO hal.cc:133] HAL sanity checks all passed.
I1204 10:39:21.605957 3305 config_monitoring_service.cc:116] Pushing the saved chassis config read from /root/sdklt/config/slx/slx_chassis_config.pb.txt...

Message from syslogd@localhost at Dec 4 10:39:21 ...
stratum_bcm[3305]: INFO hal.cc:139] Setting up HAL in COLDBOOT mode...

Message from syslogd@localhost at Dec 4 10:39:21 ...
stratum_bcm[3305]: INFO config_monitoring_service.cc:116] Pushing the saved chassis config read from /root/sdklt/config/slx/slx_chassis_config.pb.txt...
I1204 10:39:21.623787 3305 bcm_switch.cc:492] Chassis config verified successfully.

Message from syslogd@localhost at Dec 4 10:39:21 ...
stratum_bcm[3305]: INFO bcm_switch.cc:492] Chassis config verified successfully.
I1204 10:39:21.624691 3305 bcm_chassis_manager.cc:1020] Portmap:
Panel, logical (PORT_ID), physical (PC_PHYS_PORT_ID)
1, 1, 65
2, 2, 69

Message from syslogd@localhost at Dec 4 10:39:21 ...
stratum_bcm[3305]: INFO bcm_chassis_manager.cc:1020] Portmap:#012Panel, logical (PORT_ID), physical (PC_PHYS_PORT_ID)#12 1, 1, 65#012 2, 2, 69
I1204 10:39:21.625116 3305 bcm_sdk_wrapper.cc:817] Found 1 device.

Message from syslogd@localhost at Dec 4 10:39:21 ...
stratum_bcm[3305]: INFO bcm_sdk_wrapper.cc:817] Found 1 device.
I1204 10:39:22.483228 3305 bcm_sdk_wrapper.cc:884] Unit 0 is assigned to SOC device BCM56960 found on PCI bus 1, PCI slot 0.

Message from syslogd@localhost at Dec 4 10:39:22 ...
stratum_bcm[3305]: INFO bcm_sdk_wrapper.cc:884] Unit 0 is assigned to SOC device BCM56960 found on PCI bus 1, PCI slot 0.
I1204 10:39:22.783887 3305 bcm_sdk_wrapper.cc:1338] Unit 0 initialized successfully (warm_boot: NO).

Message from syslogd@localhost at Dec 4 10:39:22 ...
stratum_bcm[3305]: INFO bcm_sdk_wrapper.cc:1338] Unit 0 initialized successfully (warm_boot: NO).
BCMLT.0> I1204 10:39:23.504276 3305 bcm_chassis_manager.cc:2040] Successfully set the following options for SingletonPort (slot: 1, port: 1, unit: 0, logical_port: 1, speed: 100G): (enabled: false, blocked: true)

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO bcm_chassis_manager.cc:2040] Successfully set the following options for SingletonPort (slot: 1, port: 1, unit: 0, logical_port: 1, speed: 100G): (enabled: false, blocked: true)
I1204 10:39:23.558394 3305 bcm_chassis_manager.cc:2040] Successfully set the following options for SingletonPort (slot: 1, port: 2, unit: 0, logical_port: 2, speed: 100G): (enabled: false, blocked: true)

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INdevice knet-0-0 entered promiscuous mode
FO bcm_chassis_manager.cc:2040] Successfully set the following options for SingletonPort (slot: 1, port: 2, unit: 0, logical_port: 2, speed: 100G): (enabled: false, blocked: true)
I1204 10:39:23.559119 3305 bcm_sdk_wrapper.cc:4120] Added VLAN 2 on unit 0.

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO bcm_sdk_wrapper.cc:4120] Added VLAN 2 on unit 0.
I1204 10:39:23.559166 3309 bcm_sdk_wrapper.cc:236] Unit: 0 Port: 1 Link: DOWN.
I1204 10:39:23.559545 3305 bcm_sdk_wrapper.cc:4204] VLAN profile 2 does not exist.
I1204 10:39:23.559578 3309 bcm_sdk_wrapper.cc:236] Unit: 0 Port: 2 Link: DOWN.

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO bcm_sdk_wrapper.cc:236] Unit: 0 Port: 1 Link: DOWN.
I1204 10:39:23.559846 3305 bcm_sdk_wrapper.cc:4258] VLAN 2 has VLAN profile 2

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO bcm_sdk_wrapper.cc:4204] VLAN profile 2 does not exist.

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO bcm_sdk_wrapper.cc:236] Unit: 0 Port: 2 Link: DOWN.

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO bcm_sdk_wrapper.cc:4258] VLAN 2 has VLAN profile 2
I1204 10:39:23.560104 3305 bcm_sdk_wrapper.cc:2370] Created a new L3 drop egress intf: (port: 0, vlan: 1, router_intf_id: 0, dst_mac: 00:00:00:00:00:00, egress_intf_id: 1) on unit 0.

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO bcm_sdk_wrapper.cc:2370] Created a new L3 drop egress intf: (port: 0, vlan: 1, router_intf_id: 0, dst_mac: 00:00:00:00:00:00, egress_intf_id: 1) on unit 0.
W1204 10:39:23.561180 3305 bcm_sdk_wrapper.cc:4824] Currently not explicitly enabling/disabling ACL stages for packets ingressing on internal and external ports.
W1204 10:39:23.561468 3305 bcm_sdk_wrapper.cc:4839] Enabling intra-slice double wide is not supported.
W1204 10:39:23.561529 3305 bcm_sdk_wrapper.cc:4844] Stats collection hardware read-through is not supported.
I1204 10:39:23.561609 3305 bcm_acl_manager.cc:375] ACL manager successfully configured ACLs for node with ID 1 mapped to unit 0.

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO bcm_acl_manager.cc:375] ACL manager successfully configured ACLs for node with ID 1 mapped to unit 0.
I1204 10:39:23.633458 3305 bcm_packetio_manager.cc:758] KNET interface knet-0-0 created for node with ID 1 (unit: 0, purpose: BCM_KNET_INTF_PURPOSE_CONTROLLER, vlan: 1, cpu_queue: 0, netif_id: 1, netif_index: 36, rx_thread_id: 139721546000128).
I1204 10:39:23.633632 3305 bcm_switch.cc:76] Chassis config pushed successfully.
I1204 10:39:23.633803 3326 bcm_chassis_manager.cc:1755] State of SingletonPort (node_id: 1, port_id: 1, slot: 1, port: 1, unit: 0, logical_port: 1, speed: 100G): DOWN
I1204 10:39:23.633889 3326 bcm_chassis_manager.cc:1755] State of SingletonPort (node_id: 1, port_id: 2, slot: 1, port: 2, unit: 0, logical_port: 2, speed: 100G): DOWN
I1204 10:39:23.634012 3307 gnmi_events.h:618] Handling N7stratum3hal25PortOperStateChangedEventE
I1204 10:39:23.634070 3307 gnmi_events.h:618] Handling N7stratum3hal25PortOperStateChangedEventE
I1204 10:39:23.634253 3305 gnmi_events.h:618] Handling N7stratum3hal24ConfigHasBeenPushedEventE

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO bcm_packetio_manager.cc:758] KNET interface knet-0-0 created for node with ID 1 (unit: 0, purpose: BCM_KNET_INTF_PURPOSE_CONTROLLER, vlan: 1, cpu_queue: 0, netif_id: 1, netif_index: 36, rx_thread_id: 139721546000128).
I1204 10:39:23.634320 3305 gnmi_publisher.h:220] Configuration has changed.

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO bcm_switch.cc:76] Chassis config pushed successfully.

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO bcm_chassis_manager.cc:1755] State of SingletonPort (node_id: 1, port_id: 1, slot: 1, port: 1, unit: 0, logical_port: 1, speed: 100G): DOWN

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO bcm_chassis_manager.cc:1755] State of SingletonPort (node_id: 1, port_id: 2, slot: 1, port: 2, unit: 0, logical_port: 2, speed: 100G): DOWN

Message from syslogd@localhost at Dec 4 10:39:23 ...
stratum_bcm[3305]: INFO gnmi_events.h:618] Handling N7stratum3hal25PortOperStateChangedEventE

Message from sysI1204 10:39:23.643627 3305 p4_service.cc:130] Pushing the saved forwarding pipeline configs read from /root/sdklt/config/pipeline_cfg.pb.txt...

Message from syslogd@localhost at Dec 4 10:39:24 ...
stratum_bcm[3305]: INFO p4_service.cc:130] Pushing the saved forwarding pipeline configs read from /root/sdklt/config/pipeline_cfg.pb.txt...
½04 10:39:24.016986 3305 p4_service.cc:151] Empty forwarding pipeline configs file: /root/sdklt/config/pipeline_cfg.pb.txt.
E1204 10:39:24.019216568 3305 server_chttp2.cc:40] {"created":"@1575455964.019178318","description":"Temporary failure in name resolution","errno":-3,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/resolve_address_posix.cc","file_line":108,"os_error":"Temporary failure in name resolution","syscall":"getaddrinfo","target_address":""}
E1204 10:39:24.019464 3305 hal.cc:236] Stratum external facing services are listening to , localhost:28000...
*** Aborted at 1575455964 (unix time) try "date -d @1575455964" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 3305 (TID 0x7f1389dc0c00) from PID 0; stack trace: ***
@ 0x7f13892b10e0 (unknown)
@ 0x7f138a6e96c0 stratum::hal::Hal::Run()
@ 0x7f138a05c8d7 stratum::hal::bcm::Main()
@ 0x7f138a05cbb9 main
@ 0x7f1387e562e1 __libc_start_main
@ 0x7f138a05c32a _start
@ 0x0 (unknown)
Segmentation fault

Assigning interfaces to bmv2

While trying to instanciate a bmv2 switch (in the docker container created using setup_dev_env.sh ) using the command below both using setcap and run stratum_bmv2 as root.

./bazel-bin/stratum/hal/bin/bmv2/stratum_bmv2 -device_id=1 -chassis_config_file=/stratum/stratum/hal/bin/bmv2/sample_config.proto.txt -forwarding_pipeline_configs_file=/tmp/s1/pipe.txt -persistent_config_dir=/tmp/s1 -initial_pipeline=/stratum/stratum/hal/bin/bmv2/dummy.json -cpu_port=255 -external_stratum_urls=0.0.0.0:50001 -local_stratum_url=localhost:60705 -max_num_controllers_per_node=10 -write_req_log_file=/tmp/s1/write-reqs.txt -logtosyslog=false -logtostderr=true -bmv2_log_level=warn

I am getting following error:

E0108 07:54:15.370785 73 main.cc:133] Starting bmv2 simple_switch and waiting for P4 pipeline
I0108 07:54:15.375005 73 timer_daemon.cc:101] The timer daemon has been started.
I0108 07:54:15.375104 73 hal.cc:139] Setting up HAL in COLDBOOT mode...
I0108 07:54:15.375197 73 config_monitoring_service.cc:116] Pushing the saved chassis config read from /stratum/stratum/hal/bin/bmv2/sample_config.proto.txt...
I0108 07:54:15.379084 73 bmv2_chassis_manager.cc:488] Registered port status callbacks successfully for node 1.
I0108 07:54:15.379130 73 bmv2_chassis_manager.cc:73] Adding port 1 to node 1
[07:54:15.379] [bmv2] [E] [thread 73] Add port operation failed
E0108 07:54:15.379379 73 bmv2_chassis_manager.cc:82] StratumErrorSpace::ERR_INTERNAL: Error when binding port 1 to interface veth0 in node 1.
I0108 07:54:15.379621 73 bmv2_chassis_manager.cc:73] Adding port 2 to node 1
[07:54:15.379] [bmv2] [E] [thread 73] Add port operation failed
E0108 07:54:15.379721 73 bmv2_chassis_manager.cc:82] StratumErrorSpace::ERR_INTERNAL: Error when binding port 2 to interface veth2 in node 1.
E0108 07:54:15.379748 73 bmv2_switch.cc:59] Return Error: bmv2_chassis_manager_->PushChassisConfig(config) failed with StratumErrorSpace::ERR_INTERNAL: Error when binding port 1 to interface veth0 in node 1. Error when binding port 2 to interface veth2 in node 1.
E0108 07:54:15.379767 73 error_buffer.cc:42] (config_monitoring_service.cc:145): Pushing saved chassis config failed: Error when binding port 1 to interface veth0 in node 1. Error when binding port 2 to interface veth2 in node 1.
E0108 07:54:15.379781 73 error_buffer.cc:45] StratumErrorSpace::ERR_INTERNAL: (config_monitoring_service.cc:145): Pushing saved chassis config failed: Error when binding port 1 to interface veth0 in node 1. Error when binding port 2 to interface veth2 in node 1.
E0108 07:54:15.379801 73 hal.cc:147] Return Error: config_monitoring_service_->Setup(FLAGS_warmboot) failed with StratumErrorSpace::ERR_INTERNAL: Error when binding port 1 to interface veth0 in node 1. Error when binding port 2 to interface veth2 in node 1.
E0108 07:54:15.379812 73 main.cc:180] Error when setting up Stratum HAL (but we will continue running): Error when binding port 1 to interface veth0 in node 1. Error when binding port 2 to interface veth2 in node 1.
E0108 07:54:15.380566336 73 server_chttp2.cc:40] {"created":"@1578470055.380535361","description":"Only 1 addresses added out of total 2 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":357,"referenced_errors":[{"created":"@1578470055.380532177","description":"Address family not supported by protocol","errno":97,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":383,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::1]:60705"}]}
E0108 07:54:15.380703 73 hal.cc:234] Stratum external facing services are listening to 0.0.0.0:50001, localhost:60705...

Any help?

Problems installing kernel modules when trying to run stratum in Broadcom Tomahawk virtual device

I am trying to build and run stratum on the Broadcom Tomahawk switch by following these instructions.
I am using a VM with Ubuntu 16.04 provided by Broadcom (link).
Everything works fine when building the Stratum binary. But I have problems installing dependencies to run the pre-built binaries for the kernel, specifically the kernel modules "linux_ngbde.ko" and "linux_ngknet.ko". When I try to install these I got the following error:

insmod: ERROR: could not insert module linux_ngbde.ko: Invalid module format

Which is probably related to the mismatch regarding the kernel version (the VM in use has kernel version "4.15.0-72" and the binaries require "4.14.49"). So I downgraded the kernel to the expected version ("4.14.49"), using an Ubuntu 18.04 VM. However, I get the same "insmod" error message when installing the "ngbde" and "ngknet" kernel modules.

Any ideas about how to fix this issue? Maybe you could provide an exhaustive list with the specific environment to use and changes to perform; or share the environment you used so that it has the pre-installed dependencies to compile and run stratum for Broadcom? Maybe I'm just missing something else?

Hope you can help with this.

Thanks in advance.

EDIT: Updated the title to be more consistent with the environment we are using

Create canonical P4 pipeline

Move over the main.p4 program to pipelines folder. It should contain the canonical P4 pipeline that works on all Stratum targets (Tofino, BCM, bmv2).

Rename `phal_config_path` to `phal_config_file`

The path points to a single file, not a directory.

Also fix all startup scripts and other references.

Revert (now) unnecessary workarounds

Some changes done in, e.g. a0cbf35 (sorry @craigsdell), are unnecessary now that we fixed the underlying issues.

Example: a0cbf35#diff-544b79186d8682de1dfea855225dc05bR46-R49
Fixed by moving this snippet to logging.h:

// ostream overload for std::nullptr_t for C++11
// see: https://stackoverflow.com/a/46256849
#if __cplusplus == 201103L
#include <cstddef>
#include <iostream>
namespace std {
::std::ostream& operator<<(::std::ostream& s, ::std::nullptr_t) {
    return s << static_cast<void *>(nullptr);
}
}
#endif

Add TLS support

We need to start a design document for TLS support and implement it.
The TLS should be enabled by default and can be disabled (not recommended) optionally.

Improve StatusBuilder class

We only got a pretty limited version in the original code dump from Google, and I tacked it onto the Status class later on.
StatusBuilder should be in a separate file with its own header and tests. Luckily Google open sourced their implementations. A good place to start is here.

Fix object lifetime violation in StatusBuilder

./stratum/glue/status/status.h:379:15: warning: initializing pointer member 'file_' to point to a temporary object whose lifetime is shorter than the lifetime of the constructed object [-Wdangling-gsl]
        file_(location.file_name().c_str()),
              ^~~~~~~~~~~~~~~~~~~~
./stratum/glue/status/status.h:433:21: note: pointer member declared here
  const char* const file_;
                    ^

Probably enough to use std::string and make a copy.

Create a Stratum container image without kernel module

Some devices use different Linux-based system with different Kernel (e.g. 4.x generic)

Currently, we put kernel module file (.ko) to the container and loads the kernel module when we run the entry point script.

To make Stratum works with different Kernel, we can load the custom kernel module before we start the container and starts the Stratum directly in the container without loading the kernel module.

Update default config files for different platforms

The default config should enable all ports with max speed, for example:
Wedge100 BF 32x: 32x100G ports

The current default config file creates 0 ports when Stratum started.

How to debug stratum_bmv2

I am using the bmv2 as p4 switch simulator. I have followed the steps described here https://github.com/stratum/stratum/blob/master/stratum/hal/bin/bmv2/README.md .

Whenever I try to debug a bmv2 switch (running as mininet switch) using eclipse it never succeeds. I have used Eclipse IDE. Threads of the bmv2 switch can be seen in Eclipse. But I can not actually debug the bmv2 codes. I tried to attach the stratum_bmv2 binary as remote process to be debugged in eclipse.

Seems I am missing something. Is there any other way to debug the bmv2? Or any other better way can be used to debug it?

error : Node 0 is not configured or not known

Describe the bug
Node 0 Error messages appear on console when pushed the config from ONOS server.
2020-02-19 18:27:51.258252 BF_SYS ERROR - ASSERTION FAILED: "0" (0) from port_mgr_tof1_ha_warm_init_serdes_this_port:292
2020-02-19 18:27:51.272880 BF_SYS ERROR - ASSERTION FAILED: "0" (0) from port_mgr_tof1_ha_warm_init_serdes_this_port:292
2020-02-19 18:27:51.296220 BF_SYS ERROR - ASSERTION FAILED: "0" (0) from port_mgr_tof1_ha_warm_init_serdes_this_port:292
E0219 18:27:52.421530 52 bf_chassis_manager.cc:374] StratumErrorSpace::ERR_INVALID_PARAM: 'port_id_to_config != nullptr' is false. Node 0 is not configured or not known.
E0219 18:27:52.421614 52 bf_chassis_manager.cc:429] Return Error: GetPortConfig( request.port_speed().node_id(), request.port_speed().port_id()) at stratum/hal/lib/barefoot/bf_chassis_manager.cc:429
E0219 18:27:52.421797 52 bf_switch.cc:205] StratumErrorSpace::ERR_INTERNAL: Not supported yet
E0219 18:27:52.421998 52 bf_switch.cc:205] StratumErrorSpace::ERR_INTERNAL: Not supported yet
E0219 18:27:52.422183 52 bf_chassis_manager.cc:374] StratumErrorSpace::ERR_INVALID_PARAM: 'port_id_to_config != nullptr' is false. Node 0 is not configured or not known.
E0219 18:27:52.422226 52 bf_chassis_manager.cc:412] Return Error: GetPortConfig( request.port_speed().node_id(), request.port_speed().port_id()) at stratum/hal/lib/barefoot/bf_chassis_manager.cc:412

ASSERTION messages are expected on tofino 2T devices from bf-sde code. since not all ports are used in tofino pipelines.
Node 0 related messages appear on the console when pushed config from ONOS.
Environment
running stratum docker container from dockerhub
stratumproject/stratum-bf:9.0.0-4.14.49-OpenNetworkLinux

Switch:
root@localhost:# uname -a
Linux localhost 4.14.49-OpenNetworkLinux #1 SMP Tue Jan 21 05:45:24 UTC 2020 x86_64 GNU/Linux
root@localhost:# cat /etc/onl/platform
x86-64-inventec-d10056-r0
root@localhost:~# lsmod
Module Size Used by
bf_kdrv 28672 2
x86_pkg_temp_thermal 16384 0
vpd 20480 0
swps 172032 0
inv_cpld 16384 0
inv_psoc 24576 0
inv_platform 16384 0
i2c_gpio 16384 0
gpio_ich 16384 3

root@localhost:~# docker version
Client:
Version: 18.06.0-ce
API version: 1.38
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:10:47 2018
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.06.0-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:08:51 2018
OS/Arch: linux/amd64
Experimental: false

To Reproduce
Steps to reproduce the behavior:
ONOS server side:
run onos server
onos-app localhost reinstall! fabric-tofino-1.1.0.oar
onos localhost

app deactivate org.onosproject.mobility
app deactivate org.onosproject.openflow
app deactivate org.onosproject.fwd
app activate org.onosproject.segmentrouting
app activate org.onosproject.pipelines.fabric
app activate org.onosproject.hostprovider
app activate org.onosproject.lldpprovider
app activate org.onosproject.drivers.barefoot

netcfg
"devices": {
"device:InventecD10056": {
"basic": {
"managementAddress": "grpc://10.41.8.31:28000?device_id=1",
"driver": "stratum-tofino",
"pipeconf": "org.opencord.fabric.tofino.montara_sde_9_0_0"
},

Also, attached complete netcfg.
Expected behavior
Node 0 error messages should not appear on console.

Additional context
ARP resolved and L3 Traffic can pass fine and flows added correctly.

ONOS branch 2.2
onos_netcfg_sde900.txt

commit 089f4ae079fd99d9187f092bd94306e56dfdcb83
Author: pier [email protected]
Date: Thu Apr 25 18:51:51 2019 +0200

Improves link up handling

- trying rehash of the routes for each link up
- redoing reroute only for the routes where rehash fails

Change-Id: I7495277af73d8948300f170fa92cbbfecc338d89

Make ONLP tests testable on CI

Currently they require root rights because of some issue in the ONLP libraries.

bazel build stratum failed

I want build stratum with barefoot. But using bazel-command failed: bazel build //stratum/hal/bin/barefoot:stratum_bf
it seems like the PI dependency does not have the right version.

Please help me.

bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi.pic.o:pi.c:function pi_assign_device: error: undefined reference to '_pi_assign_device'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi.pic.o:pi.c:function pi_update_device_start: error: undefined reference to '_pi_update_device_start'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi.pic.o:pi.c:function pi_update_device_end: error: undefined reference to '_pi_update_device_end'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi.pic.o:pi.c:function pi_remove_device: error: undefined reference to '_pi_remove_device'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi.pic.o:pi.c:function pi_session_init: error: undefined reference to '_pi_session_init'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi.pic.o:pi.c:function pi_session_cleanup: error: undefined reference to '_pi_session_cleanup'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi.pic.o:pi.c:function pi_batch_begin: error: undefined reference to '_pi_batch_begin'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi.pic.o:pi.c:function pi_batch_end: error: undefined reference to '_pi_batch_end'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi.pic.o:pi.c:function pi_destroy: error: undefined reference to '_pi_destroy'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi.pic.o:pi.c:function pi_packetout_send: error: undefined reference to '_pi_packetout_send'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi_act_prof.pic.o:pi_act_prof.c:function pi_act_prof_mbr_create: error: undefined reference to '_pi_act_prof_mbr_create'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi_act_prof.pic.o:pi_act_prof.c:function pi_act_prof_mbr_delete: error: undefined reference to '_pi_act_prof_mbr_delete'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi_act_prof.pic.o:pi_act_prof.c:function pi_act_prof_mbr_modify: error: undefined reference to '_pi_act_prof_mbr_modify'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi_act_prof.pic.o:pi_act_prof.c:function pi_act_prof_grp_create: error: undefined reference to '_pi_act_prof_grp_create'
bazel-out/k8-fastbuild/bin/external/com_github_p4lang_PI_bf/_objs/pi/pi_act_prof.pic.o:pi_act_prof.c:function pi_act_prof_grp_delete: error: undefined reference to '_pi_act_prof_grp_delete'
bazel-out/k8-fastbuild/bin/external```

Add the phaldb service

For debugging purposes it is useful the be able to look into the PhalDb at runtime.

We don't want to expose this service as part of hal.cc, as it's purely for debugging. It could create it's own grpc endpoint under a different port.

Existing code is in PR #43

Check history for replacements of flat_map and flat_set

In several places gtl::flat_map and similar were used when key ordering inside the map was important. During cleanup and conversion from internal google code, some instances were replaced with their unordered counterparts (absl::flat_hash_map, std::unordered_map). This leads to error where code depends on the actual order.

Check the git history for code changes involving these types and make sure they were correct or revert them. This commit in particular: 16c70dd

git log -S "flat_map"

Add support for clearing interface counters

Implement method in gNOI Interface service: https://github.com/openconfig/gnoi/blob/master/interface/interface.proto

Would be useful for TestVectors

Support configuring loopback mode

Bump Bazel version to 2.0.0

The current Bazel master version is 1.0.0 rc5. We would like to upgrade the build system from 0.23.2 to 1.0.0 when it released.

Dependencise which supports Bazel >= 1.0.0

• com_github_grpc_grpc: 1.2.6 supports Bazel 1.0.0
• com_google_protobuf: Can be installed from grpc_deps
• com_google_googleapis: Can be installed from grpc_deps
• cython: Can be installed from grpc_deps
• net_zlib: can be installed from grpc_deps
• com_google_absl: lts_2019_08_08 branch works with Bazel 1.2.1
• com_googlesource_code_cctz: current version works with Bazel 1.2.1 (b4935eef53820cf1643355bb15e013b4167a2867)
• com_github_google_glog: current version works with Bazel 1.2.1 (ba8a9f6952d04d1403b97df24e6836227751454e)
• com_github_gflags_gflags: current version works with Bazel 1.2.1 (28f50e0fed19872e0fd50dd23ce2ee8cd759338e)
• com_google_googletest: current version works with Bazel 1.2.1 (3525e3984282c827c7207245b1d4a47f4eaf3c91)
• com_googlesource_code_re2: current version works with Bazel 1.2.1 (be0e1305d264b2cbe1d35db66b8c5107fc2a727e)
• com_github_opennetworkinglab_fabric_p4test: should be able to support new Bazel version since it only has a single file-group

Will support soon

• io_bazel_rules_python: 0.0.2 supports Bazel 2.0

Build file need to be updated

• com_github_p4lang_p4c: We are using our own build file
• com_github_p4lang_p4runtime: We are using our own build file
• com_github_openconfig_gnmi_proto: We are using our own build file
• com_github_openconfig_gnoi: We are using our own build file
• com_github_openconfig_public: We are using our own build file
• com_github_openconfig_hercules: We are using our own build file
• com_github_yang_models_yang: We are using our own build file
• com_github_systemd_systemd: We are using our own build file
• com_github_opennetworkinglab_sdklt: we are using our own build file

Doesn't support, and there is no plan yet

• judy: build file from PI, not support
• build_stack_rules_proto: Doesn't support 1.0.0
• com_github_p4lang_PI: Doesn't support 1.0.0
• com_github_p4lang_PI_bf: Doesn't support 1.0.0

Can be removed

• boringssl: chromium-stable-with-bazel branch doesn't support it, but can be removed since no one is depend on it

Need to be upgraded

• com_github_nelhage_rules_boost: Need to bump to latest version
• rules_cc: need to bump to latest version
• bazel_latex: need to upgrade to v0.19 (tested with Bazel 2.0.0)
• io_bazel_rules_go: bump to latest release (v0.20.3)
• bazel_gazelle: bump to v0.19.1

Consider using clang toolchain

https://clang.llvm.org/comparison.html
TL;DR: Clang is much faster and uses far less memory than GCC.

Should consider move to clang since Stratum project is getting bigger and CI may crashed due to some resource limit.
Also, clang tool chain provides AddressSanitizer which helps us to debug Stratum when it crashed.
Moreover, the development time can be reduced.

Error executing the stratum on tofino

There was a problem when I executed a script named "start-stratum-container.sh"：

/usr/local/bin/stratum_bf: symbol lookup error: /usr/local/bin/stratum_bf: undefined symbol: onlp_sw_init

My environment is:

$  uname -a
Linux localhost 3.16.56-OpenNetworkLinux #1 SMP Mon May 6 12:54:49 UTC 2019 x86_64 GNU/Linux

Docker image is :

$ docker images
REPOSITORY                  TAG                               IMAGE ID            CREATED             SIZE
stratumproject/stratum-bf   latest-3.16.56-OpenNetworkLinux   82a233636b82        5 weeks ago         159MB

where is the onlp_sw_init ？

More resilient ONLP error handling

The ONLPv2 API sometimes fails to read data from SFP modules. This error is detected in Stratum and usually results in an early function return with an error status. In the case of the BcmChassisManager::TransceiverEventHandler this means that no port is ever enabled and the switch is unusable.

This could either be fixed in ONLP or Stratum.

Example of corrupted SFP DAC:

# onlps sfp inventory
Port  Type    Module          Media   Status  Len    Vendor            Model             S/N
----  ------  --------------  ------  ------  -----  ----------------  ----------------  ----------------
   1
01-16 01:14:35.891041 [sff] sff_info_valid() failed: invalid module type
   2          Unknown                                                  M��+�                         u�+�
   3
   4
   5          Unknown                                W2W               75-Q010-3M        140673900069

How does the stratum project map the switch ports to frr or quagga?

Hi all, I want to know how does the stratum project map the switch ports to frr or quagga, just like a normal interface on the linux? Where is the code?　Thanks.

Fix memory leaks in AdminService

The whole AdminService class is of questionable quality, if it's easier to rewrite it that's ok too.

Command to run: bazel test --config=asan --test_output=errors //stratum/hal/lib/common:admin_service_test

Sample output for reference:

=================================================================
==17681==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 240 byte(s) in 12 object(s) allocated from:
    #0 0x4cf5bd in strdup /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_interceptors.cc:447:3
    #1 0x7f37c0f72989 in stratum::hal::AdminService::ValidatePackageMessage(gnoi::system::Package const&) /proc/self/cwd/stratum/hal/lib/common/admin_service.cc:185:27
    #2 0x7f37c0f7321a in stratum::hal::AdminService::SetPackage(grpc::ServerContext*, grpc::ServerReader<gnoi::system::SetPackageRequest>*, gnoi::system::SetPackageResponse*) /proc/self/cwd/stratum/hal/lib/common/admin_service.cc:215:17
    #3 0x7f37c0d22af6 in grpc::Status std::__invoke_impl<grpc::Status, grpc::Status (gnoi::system::System::Service::* const&)(grpc::ServerContext*, grpc::ServerReader<gnoi::system::SetPackageRequest>*, gnoi::system::SetPackageResponse*), gnoi::system::System::Service*, grpc::ServerContext*, grpc::ServerReader<gnoi::system::SetPackageRequest>*, gnoi::system::SetPackageResponse*>(std::__invoke_memfun_deref, grpc::Status (gnoi::system::System::Service::* const&)(grpc::ServerContext*, grpc::ServerReader<gnoi::system::SetPackageRequest>*, gnoi::system::SetPackageResponse*), gnoi::system::System::Service*&&, grpc::ServerContext*&&, grpc::ServerReader<gnoi::system::SetPackageRequest>*&&, gnoi::system::SetPackageResponse*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227:14
    #4 0x7f37c0d229be in std::result_of<grpc::Status (gnoi::system::System::Service::* const& (gnoi::system::System::Service*&&, grpc::ServerContext*&&, grpc::ServerReader<gnoi::system::SetPackageRequest>*&&, gnoi::system::SetPackageResponse*&&))(grpc::ServerContext*, grpc::ServerReader<gnoi::system::SetPackageRequest>*, gnoi::system::SetPackageResponse*)>::type std::__invoke<grpc::Status (gnoi::system::System::Service::* const&)(grpc::ServerContext*, grpc::ServerReader<gnoi::system::SetPackageRequest>*, gnoi::system::SetPackageResponse*), gnoi::system::System::Service*, grpc::ServerContext*, grpc::ServerReader<gnoi::system::SetPackageRequest>*, gnoi::system::SetPackageResponse*>(grpc::Status (gnoi::system::System::Service::* const&)(grpc::ServerContext*, grpc::ServerReader<gnoi::system::SetPackageRequest>*, gnoi::system::SetPackageResponse*), gnoi::system::System::Service*&&, grpc::ServerContext*&&, grpc::ServerReader<gnoi::system::SetPackageRequest>*&&, gnoi::system::SetPackageResponse*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250:14
    #5 0x7f37c0d228f3 in _ZNKSt12_Mem_fn_baseIMN4gnoi6system6System7ServiceEFN4grpc6StatusEPNS4_13ServerContextEPNS4_12ServerReaderINS1_17SetPackageRequestEEEPNS1_18SetPackageResponseEELb1EEclIJPS3_S7_SB_SD_EEEDTclsr3stdE8__invokedtdefpT6_M_pmfspclsr3stdE7forwardIT_Efp_EEEDpOSJ_ /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604:11
    #6 0x7f37c0d2277e in std::_Function_handler<grpc::Status (gnoi::system::System::Service*, grpc::ServerContext*, grpc::ServerReader<gnoi::system::SetPackageRequest>*, gnoi::system::SetPackageResponse*), std::_Mem_fn<grpc::Status (gnoi::system::System::Service::*)(grpc::ServerContext*, grpc::ServerReader<gnoi::system::SetPackageRequest>*, gnoi::system::SetPackageResponse*)> >::_M_invoke(std::_Any_data const&, gnoi::system::System::Service*&&, grpc::ServerContext*&&, grpc::ServerReader<gnoi::system::SetPackageRequest>*&&, gnoi::system::SetPackageResponse*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1716:9
    #7 0x7f37c0d25f5e in std::function<grpc::Status (gnoi::system::System::Service*, grpc::ServerContext*, grpc::ServerReader<gnoi::system::SetPackageRequest>*, gnoi::system::SetPackageResponse*)>::operator()(gnoi::system::System::Service*, grpc::ServerContext*, grpc::ServerReader<gnoi::system::SetPackageRequest>*, gnoi::system::SetPackageResponse*) const /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127:14
    #8 0x7f37c0d25d53 in grpc::internal::ClientStreamingHandler<gnoi::system::System::Service, gnoi::system::SetPackageRequest, gnoi::system::SetPackageResponse>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()::operator()() const /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/method_handler_impl.h:128:14
    #9 0x7f37c0d234b3 in grpc::Status grpc::internal::CatchingFunctionHandler<grpc::internal::ClientStreamingHandler<gnoi::system::System::Service, gnoi::system::SetPackageRequest, gnoi::system::SetPackageResponse>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()>(grpc::internal::ClientStreamingHandler<gnoi::system::System::Service, gnoi::system::SetPackageRequest, gnoi::system::SetPackageResponse>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()&&) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/method_handler_impl.h:42:12
    #10 0x7f37c0d22f74 in grpc::internal::ClientStreamingHandler<gnoi::system::System::Service, gnoi::system::SetPackageRequest, gnoi::system::SetPackageResponse>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/method_handler_impl.h:127:21
    #11 0x7f37bfe58322 in grpc_impl::Server::SyncRequest::CallData::ContinueRunAfterInterception() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/server/server_cc.cc:479:18
    #12 0x7f37bfe57c79 in grpc_impl::Server::SyncRequest::CallData::Run(std::shared_ptr<grpc_impl::Server::GlobalCallbacks> const&, bool) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/server/server_cc.cc:466:9
    #13 0x7f37bfe6faf1 in grpc::ThreadManager::MainWorkLoop() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/thread_manager/thread_manager.cc:200:9
    #14 0x7f37bfe6f7b1 in grpc::ThreadManager::WorkerThread::Run() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/thread_manager/thread_manager.cc:42:13
    #15 0x7f37bf127b5c in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/thd_posix.cc:114:27
    #16 0x7f37bf1278f8 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/thd_posix.cc:84:25
    #17 0x7f37be0734a3 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3)

Indirect leak of 21813 byte(s) in 3 object(s) allocated from:
    #0 0x4e4943 in malloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7f37bf11e9ce in gpr_malloc /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/alloc.cc:57:7
    #2 0x7f37bf11ea7d in gpr_malloc_aligned /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/alloc.cc:93:13
    #3 0x7f37bf124d9b in grpc_core::Arena::CreateWithAlloc(unsigned long, unsigned long) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/arena.cc:72:12
    #4 0x7f37bf240a9a in grpc_call_create(grpc_call_create_args const*, grpc_call**) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/call.cc:338:7
    #5 0x7f37bf24df51 in grpc_channel_create_call_internal(grpc_channel*, grpc_call*, unsigned int, grpc_completion_queue*, grpc_pollset_set*, grpc_mdelem, grpc_mdelem, long) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/channel.cc:337:3
    #6 0x7f37bf24e825 in grpc_channel_create_registered_call /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/channel.cc:416:21
    #7 0x7f37bfde7bcb in grpc_impl::Channel::CreateCallInternal(grpc::internal::RpcMethod const&, grpc::ClientContext*, grpc_impl::CompletionQueue*, unsigned long) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/channel_cc.cc:117:14
    #8 0x7f37bfde80a0 in grpc_impl::Channel::CreateCall(grpc::internal::RpcMethod const&, grpc::ClientContext*, grpc_impl::CompletionQueue*) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/channel_cc.cc:160:10
    #9 0x7f37c0d009b1 in grpc::internal::BlockingUnaryCallImpl<gnoi::system::RebootRequest, gnoi::system::RebootResponse>::BlockingUnaryCallImpl(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::RebootRequest const&, gnoi::system::RebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:53:24
    #10 0x7f37c0cbbddf in grpc::Status grpc::internal::BlockingUnaryCall<gnoi::system::RebootRequest, gnoi::system::RebootResponse>(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::RebootRequest const&, gnoi::system::RebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:39:10
    #11 0x7f37c0cb5a70 in gnoi::system::System::Stub::Reboot(grpc::ClientContext*, gnoi::system::RebootRequest const&, gnoi::system::RebootResponse*) /proc/self/cwd/bazel-out/k8-fastbuild/genfiles/external/com_github_openconfig_gnoi/gnoi/system/system.grpc.pb.cc:155:10
    #12 0x5161dd in stratum::hal::AdminServiceTest_CancelReboot_Test::TestBody() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:173:34
    #13 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #14 0x7f37bf03a394 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2522:5
    #15 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #16 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #17 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #18 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #19 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #20 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #21 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 21813 byte(s) in 3 object(s) allocated from:
    #0 0x4e4943 in malloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7f37bf11e9ce in gpr_malloc /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/alloc.cc:57:7
    #2 0x7f37bf11ea7d in gpr_malloc_aligned /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/alloc.cc:93:13
    #3 0x7f37bf124d9b in grpc_core::Arena::CreateWithAlloc(unsigned long, unsigned long) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/arena.cc:72:12
    #4 0x7f37bf240a9a in grpc_call_create(grpc_call_create_args const*, grpc_call**) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/call.cc:338:7
    #5 0x7f37bf24df51 in grpc_channel_create_call_internal(grpc_channel*, grpc_call*, unsigned int, grpc_completion_queue*, grpc_pollset_set*, grpc_mdelem, grpc_mdelem, long) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/channel.cc:337:3
    #6 0x7f37bf24e825 in grpc_channel_create_registered_call /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/channel.cc:416:21
    #7 0x7f37bfde7bcb in grpc_impl::Channel::CreateCallInternal(grpc::internal::RpcMethod const&, grpc::ClientContext*, grpc_impl::CompletionQueue*, unsigned long) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/channel_cc.cc:117:14
    #8 0x7f37bfde80a0 in grpc_impl::Channel::CreateCall(grpc::internal::RpcMethod const&, grpc::ClientContext*, grpc_impl::CompletionQueue*) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/channel_cc.cc:160:10
    #9 0x7f37c0d0ee11 in grpc::internal::BlockingUnaryCallImpl<gnoi::system::CancelRebootRequest, gnoi::system::CancelRebootResponse>::BlockingUnaryCallImpl(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:53:24
    #10 0x7f37c0cbddff in grpc::Status grpc::internal::BlockingUnaryCall<gnoi::system::CancelRebootRequest, gnoi::system::CancelRebootResponse>(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:39:10
    #11 0x7f37c0cb6730 in gnoi::system::System::Stub::CancelReboot(grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/bazel-out/k8-fastbuild/genfiles/external/com_github_openconfig_gnoi/gnoi/system/system.grpc.pb.cc:211:10
    #12 0x51639e in stratum::hal::AdminServiceTest_CancelReboot_Test::TestBody() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:180:19
    #13 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #14 0x7f37bf03a394 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2522:5
    #15 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #16 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #17 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #18 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #19 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #20 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #21 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 21621 byte(s) in 3 object(s) allocated from:
    #0 0x4e4943 in malloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7f37bf11e9ce in gpr_malloc /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/alloc.cc:57:7
    #2 0x7f37bf11ea7d in gpr_malloc_aligned /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/alloc.cc:93:13
    #3 0x7f37bf124edb in grpc_core::Arena::AllocZone(unsigned long) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/arena.cc:93:18
    #4 0x7f37bf983045 in grpc_core::ConnectedSubchannel::CreateCall(grpc_core::ConnectedSubchannel::CallArgs const&, grpc_error**) /proc/self/cwd/external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc:124:24
    #5 0x7f37bf94bd1a in grpc_core::(anonymous namespace)::CallData::CreateSubchannelCall(grpc_call_element*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc:3142:40
    #6 0x7f37bf200e8b in exec_ctx_run(grpc_closure*, grpc_error*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/iomgr/exec_ctx.cc:40:3
    #7 0x7f37bf200c9a in grpc_core::ExecCtx::Flush() /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/iomgr/exec_ctx.cc:148:9
    #8 0x7f37bfe1046b in grpc_core::ExecCtx::~ExecCtx() /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/iomgr/exec_ctx.h:121:5
    #9 0x7f37bf24348d in grpc_call_start_batch /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/call.cc:1924:3
    #10 0x7f37c0d0fe1e in grpc::internal::CallOpSet<grpc::internal::CallOpSendInitialMetadata, grpc::internal::CallOpSendMessage, grpc::internal::CallOpRecvInitialMetadata, grpc::internal::CallOpRecvMessage<gnoi::system::CancelRebootResponse>, grpc::internal::CallOpClientSendClose, grpc::internal::CallOpClientRecvStatus>::ContinueFillOpsAfterInterception() /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/call_op_set.h:940:5
    #11 0x7f37c0d0ef29 in grpc::internal::BlockingUnaryCallImpl<gnoi::system::CancelRebootRequest, gnoi::system::CancelRebootResponse>::BlockingUnaryCallImpl(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:69:10
    #12 0x7f37c0cbddff in grpc::Status grpc::internal::BlockingUnaryCall<gnoi::system::CancelRebootRequest, gnoi::system::CancelRebootResponse>(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:39:10
    #13 0x7f37c0cb6730 in gnoi::system::System::Stub::CancelReboot(grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/bazel-out/k8-fastbuild/genfiles/external/com_github_openconfig_gnoi/gnoi/system/system.grpc.pb.cc:211:10
    #14 0x51639e in stratum::hal::AdminServiceTest_CancelReboot_Test::TestBody() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:180:19
    #15 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #16 0x7f37bf03a394 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2522:5
    #17 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #18 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #19 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #20 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #21 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #22 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #23 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 21621 byte(s) in 3 object(s) allocated from:
    #0 0x4e4943 in malloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7f37bf11e9ce in gpr_malloc /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/alloc.cc:57:7
    #2 0x7f37bf11ea7d in gpr_malloc_aligned /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/alloc.cc:93:13
    #3 0x7f37bf124edb in grpc_core::Arena::AllocZone(unsigned long) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/arena.cc:93:18
    #4 0x7f37bf983045 in grpc_core::ConnectedSubchannel::CreateCall(grpc_core::ConnectedSubchannel::CallArgs const&, grpc_error**) /proc/self/cwd/external/com_github_grpc_grpc/src/core/ext/filters/client_channel/subchannel.cc:124:24
    #5 0x7f37bf94bd1a in grpc_core::(anonymous namespace)::CallData::CreateSubchannelCall(grpc_call_element*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc:3142:40
    #6 0x7f37bf200e8b in exec_ctx_run(grpc_closure*, grpc_error*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/iomgr/exec_ctx.cc:40:3
    #7 0x7f37bf200c9a in grpc_core::ExecCtx::Flush() /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/iomgr/exec_ctx.cc:148:9
    #8 0x7f37bf1f2bc2 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1151:34
    #9 0x7f37bf200159 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/iomgr/ev_posix.cc:322:21
    #10 0x7f37bf25370b in cq_pluck(grpc_completion_queue*, void*, gpr_timespec, void*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/completion_queue.cc:1273:9
    #11 0x7f37c0f78424 in grpc_impl::CompletionQueue::Pluck(grpc::internal::CompletionQueueTag*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/completion_queue_impl.h:316:51
    #12 0x7f37c0d00ad5 in grpc::internal::BlockingUnaryCallImpl<gnoi::system::RebootRequest, gnoi::system::RebootResponse>::BlockingUnaryCallImpl(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::RebootRequest const&, gnoi::system::RebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:70:8
    #13 0x7f37c0cbbddf in grpc::Status grpc::internal::BlockingUnaryCall<gnoi::system::RebootRequest, gnoi::system::RebootResponse>(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::RebootRequest const&, gnoi::system::RebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:39:10
    #14 0x7f37c0cb5a70 in gnoi::system::System::Stub::Reboot(grpc::ClientContext*, gnoi::system::RebootRequest const&, gnoi::system::RebootResponse*) /proc/self/cwd/bazel-out/k8-fastbuild/genfiles/external/com_github_openconfig_gnoi/gnoi/system/system.grpc.pb.cc:155:10
    #15 0x5161dd in stratum::hal::AdminServiceTest_CancelReboot_Test::TestBody() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:173:34
    #16 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #17 0x7f37bf03a394 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2522:5
    #18 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #19 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #20 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #21 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #22 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #23 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #24 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 3072 byte(s) in 6 object(s) allocated from:
    #0 0x4e4943 in malloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7f37bda757a7 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x8f7a7)
    #2 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #3 0x7f37bf03a394 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2522:5
    #4 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #5 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #6 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #7 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #8 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #9 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #10 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 1272 byte(s) in 3 object(s) allocated from:
    #0 0x4e4b3a in calloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:155:3
    #1 0x7f37bf11e9fe in gpr_zalloc /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/alloc.cc:68:7
    #2 0x7f37bf250c69 in grpc_completion_queue_create_internal(grpc_cq_completion_type, grpc_cq_polling_type, grpc_experimental_completion_queue_functor*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/completion_queue.cc:516:7
    #3 0x7f37c0cc3e9c in grpc_impl::CompletionQueue::CompletionQueue(grpc_completion_queue_attributes const&) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/completion_queue_impl.h:248:45
    #4 0x7f37c0d0eda7 in grpc::internal::BlockingUnaryCallImpl<gnoi::system::CancelRebootRequest, gnoi::system::CancelRebootResponse>::BlockingUnaryCallImpl(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:50:21
    #5 0x7f37c0cbddff in grpc::Status grpc::internal::BlockingUnaryCall<gnoi::system::CancelRebootRequest, gnoi::system::CancelRebootResponse>(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:39:10
    #6 0x7f37c0cb6730 in gnoi::system::System::Stub::CancelReboot(grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/bazel-out/k8-fastbuild/genfiles/external/com_github_openconfig_gnoi/gnoi/system/system.grpc.pb.cc:211:10
    #7 0x51639e in stratum::hal::AdminServiceTest_CancelReboot_Test::TestBody() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:180:19
    #8 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #9 0x7f37bf03a394 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2522:5
    #10 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #11 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #12 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #13 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #14 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #15 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #16 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 504 byte(s) in 3 object(s) allocated from:
    #0 0x4e4943 in malloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7f37bda757a7 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x8f7a7)
    #2 0x7f37bfefa206 in grpc_impl::(anonymous namespace)::InsecureChannelCredentialsImpl::CreateChannelWithInterceptors(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, grpc_impl::ChannelArguments const&, std::vector<std::unique_ptr<grpc::experimental::ClientInterceptorFactoryInterface, std::default_delete<grpc::experimental::ClientInterceptorFactoryInterface> >, std::allocator<std::unique_ptr<grpc::experimental::ClientInterceptorFactoryInterface, std::default_delete<grpc::experimental::ClientInterceptorFactoryInterface> > > >) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/insecure_credentials.cc:49:12
    #3 0x7f37bfef9fa0 in grpc_impl::(anonymous namespace)::InsecureChannelCredentialsImpl::CreateChannelImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, grpc_impl::ChannelArguments const&) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/insecure_credentials.cc:36:12
    #4 0x7f37bfdf1ad7 in grpc_impl::CreateCustomChannelImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<grpc_impl::ChannelCredentials> const&, grpc_impl::ChannelArguments const&) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/create_channel.cc:42:25
    #5 0x7f37bfdf18a2 in grpc_impl::CreateChannelImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<grpc_impl::ChannelCredentials> const&) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/create_channel.cc:33:10
    #6 0x522a4d in grpc::CreateChannel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<grpc_impl::ChannelCredentials> const&) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/create_channel.h:30:10
    #7 0x52daa4 in stratum::hal::AdminServiceTest::SetUp() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:85:9
    #8 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #9 0x7f37bf03a330 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2517:3
    #10 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #11 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #12 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #13 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #14 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #15 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #16 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 264 byte(s) in 3 object(s) allocated from:
    #0 0x4e4943 in malloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7f37bf11e9ce in gpr_malloc /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/alloc.cc:57:7
    #2 0x7f37bf94c203 in grpc_core::(anonymous namespace)::CallData::QueuedPickCanceller* grpc_core::New<grpc_core::(anonymous namespace)::CallData::QueuedPickCanceller, grpc_call_element*&>(grpc_call_element*&) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/memory.h:46:13
    #3 0x7f37bf94b19b in grpc_core::(anonymous namespace)::CallData::AddCallToQueuedPicksLocked(grpc_call_element*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc:3236:21
    #4 0x7f37bf943d33 in grpc_core::(anonymous namespace)::CallData::StartPickLocked(void*, grpc_error*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc:3394:40
    #5 0x7f37bf1e5fe1 in grpc_combiner_continue_exec_ctx() /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/iomgr/combiner.cc:268:5
    #6 0x7f37bf200c20 in grpc_core::ExecCtx::Flush() /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/iomgr/exec_ctx.cc:151:17
    #7 0x7f37bfe1046b in grpc_core::ExecCtx::~ExecCtx() /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/iomgr/exec_ctx.h:121:5
    #8 0x7f37bf24348d in grpc_call_start_batch /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/call.cc:1924:3
    #9 0x7f37c0d019be in grpc::internal::CallOpSet<grpc::internal::CallOpSendInitialMetadata, grpc::internal::CallOpSendMessage, grpc::internal::CallOpRecvInitialMetadata, grpc::internal::CallOpRecvMessage<gnoi::system::RebootResponse>, grpc::internal::CallOpClientSendClose, grpc::internal::CallOpClientRecvStatus>::ContinueFillOpsAfterInterception() /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/call_op_set.h:940:5
    #10 0x7f37c0d00ac9 in grpc::internal::BlockingUnaryCallImpl<gnoi::system::RebootRequest, gnoi::system::RebootResponse>::BlockingUnaryCallImpl(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::RebootRequest const&, gnoi::system::RebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:69:10
    #11 0x7f37c0cbbddf in grpc::Status grpc::internal::BlockingUnaryCall<gnoi::system::RebootRequest, gnoi::system::RebootResponse>(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::RebootRequest const&, gnoi::system::RebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:39:10
    #12 0x7f37c0cb5a70 in gnoi::system::System::Stub::Reboot(grpc::ClientContext*, gnoi::system::RebootRequest const&, gnoi::system::RebootResponse*) /proc/self/cwd/bazel-out/k8-fastbuild/genfiles/external/com_github_openconfig_gnoi/gnoi/system/system.grpc.pb.cc:155:10
    #13 0x5161dd in stratum::hal::AdminServiceTest_CancelReboot_Test::TestBody() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:173:34
    #14 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #15 0x7f37bf03a394 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2522:5
    #16 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #17 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #18 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #19 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #20 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #21 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #22 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 96 byte(s) in 3 object(s) allocated from:
    #0 0x4e4d62 in realloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:165:3
    #1 0x7f37bf11ea47 in gpr_realloc /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/alloc.cc:83:7
    #2 0x7f37bf2510f2 in grpc_cq_begin_op(grpc_completion_queue*, void*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/completion_queue.cc:664:48
    #3 0x7f37bf2453b3 in call_start_batch(grpc_call*, grpc_op const*, unsigned long, void*, int) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/call.cc:1867:5
    #4 0x7f37bf243481 in grpc_call_start_batch /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/call.cc:1923:11
    #5 0x7f37c0d0fe1e in grpc::internal::CallOpSet<grpc::internal::CallOpSendInitialMetadata, grpc::internal::CallOpSendMessage, grpc::internal::CallOpRecvInitialMetadata, grpc::internal::CallOpRecvMessage<gnoi::system::CancelRebootResponse>, grpc::internal::CallOpClientSendClose, grpc::internal::CallOpClientRecvStatus>::ContinueFillOpsAfterInterception() /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/call_op_set.h:940:5
    #6 0x7f37c0d0ef29 in grpc::internal::BlockingUnaryCallImpl<gnoi::system::CancelRebootRequest, gnoi::system::CancelRebootResponse>::BlockingUnaryCallImpl(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:69:10
    #7 0x7f37c0cbddff in grpc::Status grpc::internal::BlockingUnaryCall<gnoi::system::CancelRebootRequest, gnoi::system::CancelRebootResponse>(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:39:10
    #8 0x7f37c0cb6730 in gnoi::system::System::Stub::CancelReboot(grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/bazel-out/k8-fastbuild/genfiles/external/com_github_openconfig_gnoi/gnoi/system/system.grpc.pb.cc:211:10
    #9 0x51639e in stratum::hal::AdminServiceTest_CancelReboot_Test::TestBody() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:180:19
    #10 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #11 0x7f37bf03a394 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2522:5
    #12 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #13 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #14 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #15 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #16 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #17 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #18 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 72 byte(s) in 3 object(s) allocated from:
    #0 0x4e4943 in malloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7f37bda757a7 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x8f7a7)
    #2 0x7f37bfdf250b in std::__shared_ptr<grpc_impl::Channel, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<grpc_impl::Channel>(grpc_impl::Channel*) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/shared_ptr_base.h:885:24
    #3 0x7f37bfdf2272 in grpc::CreateChannelInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, grpc_channel*, std::vector<std::unique_ptr<grpc::experimental::ClientInterceptorFactoryInterface, std::default_delete<grpc::experimental::ClientInterceptorFactoryInterface> >, std::allocator<std::unique_ptr<grpc::experimental::ClientInterceptorFactoryInterface, std::default_delete<grpc::experimental::ClientInterceptorFactoryInterface> > > >) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/create_channel_internal.cc:32:10
    #4 0x7f37bfefa206 in grpc_impl::(anonymous namespace)::InsecureChannelCredentialsImpl::CreateChannelWithInterceptors(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, grpc_impl::ChannelArguments const&, std::vector<std::unique_ptr<grpc::experimental::ClientInterceptorFactoryInterface, std::default_delete<grpc::experimental::ClientInterceptorFactoryInterface> >, std::allocator<std::unique_ptr<grpc::experimental::ClientInterceptorFactoryInterface, std::default_delete<grpc::experimental::ClientInterceptorFactoryInterface> > > >) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/insecure_credentials.cc:49:12
    #5 0x7f37bfef9fa0 in grpc_impl::(anonymous namespace)::InsecureChannelCredentialsImpl::CreateChannelImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, grpc_impl::ChannelArguments const&) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/insecure_credentials.cc:36:12
    #6 0x7f37bfdf1ad7 in grpc_impl::CreateCustomChannelImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<grpc_impl::ChannelCredentials> const&, grpc_impl::ChannelArguments const&) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/create_channel.cc:42:25
    #7 0x7f37bfdf18a2 in grpc_impl::CreateChannelImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<grpc_impl::ChannelCredentials> const&) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/create_channel.cc:33:10
    #8 0x522a4d in grpc::CreateChannel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<grpc_impl::ChannelCredentials> const&) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/create_channel.h:30:10
    #9 0x52daa4 in stratum::hal::AdminServiceTest::SetUp() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:85:9
    #10 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #11 0x7f37bf03a330 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2517:3
    #12 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #13 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #14 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #15 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #16 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #17 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #18 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 12 byte(s) in 3 object(s) allocated from:
    #0 0x4e4943 in malloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7f37bf122ecd in gpr_mu_init /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/sync_posix.cc:79:40
    #2 0x7f37bfdedacb in grpc::ClientContext::ClientContext() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/client_context.cc:55:16
    #3 0x516322 in stratum::hal::AdminServiceTest_CancelReboot_Test::TestBody() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:177:17
    #4 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #5 0x7f37bf03a394 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2522:5
    #6 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #7 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #8 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #9 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #10 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #11 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #12 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 12 byte(s) in 3 object(s) allocated from:
    #0 0x4e4943 in malloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7f37bf122ecd in gpr_mu_init /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/sync_posix.cc:79:40
    #2 0x7f37bf1f2621 in pollset_init(grpc_pollset*, gpr_mu**) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/iomgr/ev_epollex_linux.cc:801:3
    #3 0x7f37bf250d2a in grpc_completion_queue_create_internal(grpc_cq_completion_type, grpc_cq_polling_type, grpc_experimental_completion_queue_functor*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/surface/completion_queue.cc:525:3
    #4 0x7f37c0cc3e9c in grpc_impl::CompletionQueue::CompletionQueue(grpc_completion_queue_attributes const&) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/completion_queue_impl.h:248:45
    #5 0x7f37c0d0eda7 in grpc::internal::BlockingUnaryCallImpl<gnoi::system::CancelRebootRequest, gnoi::system::CancelRebootResponse>::BlockingUnaryCallImpl(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:50:21
    #6 0x7f37c0cbddff in grpc::Status grpc::internal::BlockingUnaryCall<gnoi::system::CancelRebootRequest, gnoi::system::CancelRebootResponse>(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/impl/codegen/client_unary_call.h:39:10
    #7 0x7f37c0cb6730 in gnoi::system::System::Stub::CancelReboot(grpc::ClientContext*, gnoi::system::CancelRebootRequest const&, gnoi::system::CancelRebootResponse*) /proc/self/cwd/bazel-out/k8-fastbuild/genfiles/external/com_github_openconfig_gnoi/gnoi/system/system.grpc.pb.cc:211:10
    #8 0x51639e in stratum::hal::AdminServiceTest_CancelReboot_Test::TestBody() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:180:19
    #9 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #10 0x7f37bf03a394 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2522:5
    #11 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #12 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #13 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #14 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #15 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #16 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #17 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 12 byte(s) in 3 object(s) allocated from:
    #0 0x4e4943 in malloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7f37bf122ecd in gpr_mu_init /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/sync_posix.cc:79:40
    #2 0x7f37bfdedacb in grpc::ClientContext::ClientContext() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/client_context.cc:55:16
    #3 0x5160b6 in stratum::hal::AdminServiceTest_CancelReboot_Test::TestBody() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:164:40
    #4 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #5 0x7f37bf03a394 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2522:5
    #6 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #7 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #8 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #9 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #10 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #11 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #12 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

Indirect leak of 12 byte(s) in 3 object(s) allocated from:
    #0 0x4e4943 in malloc /build/llvm-toolchain-7-ZvxGT8/llvm-toolchain-7-7.0.1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x7f37bf122ecd in gpr_mu_init /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gpr/sync_posix.cc:79:40
    #2 0x7f37bfde7129 in grpc_impl::Channel::Channel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, grpc_channel*, std::vector<std::unique_ptr<grpc::experimental::ClientInterceptorFactoryInterface, std::default_delete<grpc::experimental::ClientInterceptorFactoryInterface> >, std::allocator<std::unique_ptr<grpc::experimental::ClientInterceptorFactoryInterface, std::default_delete<grpc::experimental::ClientInterceptorFactoryInterface> > > >) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/channel_cc.cc:53:10
    #3 0x7f37bfdf2266 in grpc::CreateChannelInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, grpc_channel*, std::vector<std::unique_ptr<grpc::experimental::ClientInterceptorFactoryInterface, std::default_delete<grpc::experimental::ClientInterceptorFactoryInterface> >, std::allocator<std::unique_ptr<grpc::experimental::ClientInterceptorFactoryInterface, std::default_delete<grpc::experimental::ClientInterceptorFactoryInterface> > > >) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/create_channel_internal.cc:33:11
    #4 0x7f37bfefa206 in grpc_impl::(anonymous namespace)::InsecureChannelCredentialsImpl::CreateChannelWithInterceptors(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, grpc_impl::ChannelArguments const&, std::vector<std::unique_ptr<grpc::experimental::ClientInterceptorFactoryInterface, std::default_delete<grpc::experimental::ClientInterceptorFactoryInterface> >, std::allocator<std::unique_ptr<grpc::experimental::ClientInterceptorFactoryInterface, std::default_delete<grpc::experimental::ClientInterceptorFactoryInterface> > > >) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/insecure_credentials.cc:49:12
    #5 0x7f37bfef9fa0 in grpc_impl::(anonymous namespace)::InsecureChannelCredentialsImpl::CreateChannelImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, grpc_impl::ChannelArguments const&) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/insecure_credentials.cc:36:12
    #6 0x7f37bfdf1ad7 in grpc_impl::CreateCustomChannelImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<grpc_impl::ChannelCredentials> const&, grpc_impl::ChannelArguments const&) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/create_channel.cc:42:25
    #7 0x7f37bfdf18a2 in grpc_impl::CreateChannelImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<grpc_impl::ChannelCredentials> const&) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/client/create_channel.cc:33:10
    #8 0x522a4d in grpc::CreateChannel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<grpc_impl::ChannelCredentials> const&) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/create_channel.h:30:10
    #9 0x52daa4 in stratum::hal::AdminServiceTest::SetUp() /proc/self/cwd/stratum/hal/lib/common/admin_service_test.cc:85:9
    #10 0x7f37bf0581b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #11 0x7f37bf03a330 in testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2517:3
    #12 0x7f37bf03b227 in testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2698:11
    #13 0x7f37bf03bb91 in testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2828:28
    #14 0x7f37bf04cbf2 in testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5285:44
    #15 0x7f37bf05b567 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2483:14
    #16 0x7f37bf04c4a2 in testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:4873:10
    #17 0x7f37bff8f44e in main /proc/self/cwd/stratum/hal/lib/common/test_main.cc:40:16
    #18 0x7f37bd0422e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)

SUMMARY: AddressSanitizer: 92436 byte(s) leaked in 57 allocation(s).
================================================================================

Issue in bmv2 adding ports to node

While trying to instanciate a bmv2 switch using this command:

./bazel-bin/stratum/hal/bin/bmv2/stratum_bmv2 -device_id=1 -chassis_config_file=/stratum/stratum/hal/bin/bmv2/sample_config.proto.txt -forwarding_pipeline_configs_file=/tmp/s1/pipe.txt -persistent_config_dir=/tmp/s1 -initial_pipeline=/stratum/stratum/hal/bin/bmv2/dummy.json -cpu_port=255 -external_stratum_urls=0.0.0.0:50001 -local_stratum_url=localhost:60705 -max_num_controllers_per_node=10 -write_req_log_file=/tmp/s1/write-reqs.txt -logtosyslog=false -logtostderr=true -bmv2_log_level=warn

I am getting following error:

E0108 07:54:15.370785 73 main.cc:133] Starting bmv2 simple_switch and waiting for P4 pipeline
I0108 07:54:15.375005 73 timer_daemon.cc:101] The timer daemon has been started.
I0108 07:54:15.375104 73 hal.cc:139] Setting up HAL in COLDBOOT mode...
I0108 07:54:15.375197 73 config_monitoring_service.cc:116] Pushing the saved chassis config read from /stratum/stratum/hal/bin/bmv2/sample_config.proto.txt...
I0108 07:54:15.379084 73 bmv2_chassis_manager.cc:488] Registered port status callbacks successfully for node 1.
I0108 07:54:15.379130 73 bmv2_chassis_manager.cc:73] Adding port 1 to node 1
[07:54:15.379] [bmv2] [E] [thread 73] Add port operation failed
E0108 07:54:15.379379 73 bmv2_chassis_manager.cc:82] StratumErrorSpace::ERR_INTERNAL: Error when binding port 1 to interface veth0 in node 1.
I0108 07:54:15.379621 73 bmv2_chassis_manager.cc:73] Adding port 2 to node 1
[07:54:15.379] [bmv2] [E] [thread 73] Add port operation failed
E0108 07:54:15.379721 73 bmv2_chassis_manager.cc:82] StratumErrorSpace::ERR_INTERNAL: Error when binding port 2 to interface veth2 in node 1.
E0108 07:54:15.379748 73 bmv2_switch.cc:59] Return Error: bmv2_chassis_manager_->PushChassisConfig(config) failed with StratumErrorSpace::ERR_INTERNAL: Error when binding port 1 to interface veth0 in node 1. Error when binding port 2 to interface veth2 in node 1.
E0108 07:54:15.379767 73 error_buffer.cc:42] (config_monitoring_service.cc:145): Pushing saved chassis config failed: Error when binding port 1 to interface veth0 in node 1. Error when binding port 2 to interface veth2 in node 1.
E0108 07:54:15.379781 73 error_buffer.cc:45] StratumErrorSpace::ERR_INTERNAL: (config_monitoring_service.cc:145): Pushing saved chassis config failed: Error when binding port 1 to interface veth0 in node 1. Error when binding port 2 to interface veth2 in node 1.
E0108 07:54:15.379801 73 hal.cc:147] Return Error: config_monitoring_service_->Setup(FLAGS_warmboot) failed with StratumErrorSpace::ERR_INTERNAL: Error when binding port 1 to interface veth0 in node 1. Error when binding port 2 to interface veth2 in node 1.
E0108 07:54:15.379812 73 main.cc:180] Error when setting up Stratum HAL (but we will continue running): Error when binding port 1 to interface veth0 in node 1. Error when binding port 2 to interface veth2 in node 1.
E0108 07:54:15.380566336 73 server_chttp2.cc:40] {"created":"@1578470055.380535361","description":"Only 1 addresses added out of total 2 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":357,"referenced_errors":[{"created":"@1578470055.380532177","description":"Address family not supported by protocol","errno":97,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":383,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::1]:60705"}]}
E0108 07:54:15.380703 73 hal.cc:234] Stratum external facing services are listening to 0.0.0.0:50001, localhost:60705...

Any help?

Improve p4c / main.p4 instructions

• Link from p4c-fpm README to pipelines
• Create main.p4 output package
• Link to P4Runtime shell to show how to push the pipeline
• Publish p4c-fpm docker image

Check `phal_config_path` flag

I can think of 3 uses:

• Empty flag phal_config_path=""

• Empty file

• Actual configuration file

We should make sure a cases are handled or properly rejected.

We also have a set of config files for various switch, but do not use them in the bf docker image: https://github.com/stratum/stratum/blob/master/stratum/hal/bin/barefoot/docker/Dockerfile#L79

Some logs of faulty behavior:

E0118 18:38:36.830709    34 onlpphal.cc:167] StratumErrorSpace::ERR_INVALID_PARAM: 'configurator != nullptr' is false. card[1]/port[1]: no configurator for this transceiver
E0118 18:38:36.830785    34 onlpphal.cc:56] Return Error: onlpphal_->HandleTransceiverEvent(event) failed with StratumErrorSpace::ERR_INVALID_PARAM: 'configurator != nullptr' is false. card[1]/port[1]: no configurator for this transceiver

Cleanup / rewrite switch_configurator.cc

There are several issues from code style to design. Make a pass over the code base and related files.