Error when Trying to run the Binary about stratum

[pudelkoM]

I think Stratum does not support BSP mode. Normally we disable it (BSP-less mode) and use ONLPv2 for the platform support. These are the onlp_platformi_get() calls you see.

Are you running Stratum on an ONL OS? cat /etc/onl/platform

[Yukinarii]

Hi,
Thanks for the response. Yes, I am running Stratum on an ONL OS:

root@localhost:~# cat /etc/onl/platform
x86-64-accton-wedge100bf-65x-r0

I followed the instructions, copied the json to $BF_SDE_INSTALL/share, and ran Stratum in BSP-less mode with the following command:

sudo LD_LIBRARY_PATH=$BF_SDE_INSTALL/lib \
     ./bazel-bin/stratum/hal/bin/barefoot/stratum_bf \
       --external_stratum_urls=0.0.0.0:28000 \
       --grpc_max_recv_msg_size=256 \
       --bf_sde_install=$BF_SDE_INSTALL \
       --persistent_config_dir=/root/config \
       --forwarding_pipeline_configs_file=/root/config/p4_pipeline.pb.txt \
       --chassis_config_file=/root/config/chassis_config.pb.txt \
       --write_req_log_file=/root/p4_writes.pb.txt \
       --bf-sim=1 --bf_switchd_cfg=stratum/hal/bin/barefoot/tofino_skip_p4_no_bsp.conf

However, I got the same error:

Cannot read termcap database;
using dumb terminal settings.
bf-sde> 01-26 07:09:49.257100 [onlp_platform_defaults] The default implementation of onlp_platformi_get() has been called.
01-26 07:09:49.257161 [onlp_platform_defaults] This can happen for the following reasons, all fatal:
01-26 07:09:49.257185 [onlp_platform_defaults] * The ONLP build configuration is incorrect.
01-26 07:09:49.257204 [onlp_platform_defaults] * The ONLP platform library for this platform does not contain the onlp_platformi_get() symbol.
01-26 07:09:49.257223 [onlp_platform_defaults] * The ONLP platform shared libraries are not setup properly before we executed.
01-26 07:09:49.257242 [onlp_platform_defaults] * The platform cannot continue until this issue is resolved    

I think the ONLP library will be downloaded and built by bazel according to the document.

./root/.cache/bazel/_bazel_root/adebb43308e0307e846fb1a569522202/execroot/com_github_stratum_stratum/bazel-out/k8-fastbuild/bin/stratum/hal/lib/phal/onlp
./root/.cache/bazel/_bazel_root/adebb43308e0307e846fb1a569522202/external/local_onlp_bin/onlp-bin/include/onlp

However, I also noticed that the guide asks ignore the "Installing the reference BSP for the Wedge" Section, while I installed BSP after building the SDE.

Am I missing something or should I rebuild the SDE without the BSP installation to make it correct?

[Yi-Tseng]

Hi,
Can you show me the output of this command on your switch?

ls -al /lib/x86_64-linux-gnu/libonlp*

I think the libonlp-platform is linked to the default one instead of the platform one.

[Yukinarii]

Hi,
Thank you for the response. The output is:

root@localhost:~# ls -al /lib/x86_64-linux-gnu/libonlp*
-rwxr-xr-x 1 root root 304528 May  4  2019 /lib/x86_64-linux-gnu/libonlp-platform-defaults.so
lrwxrwxrwx 1 root root     28 Jan  9 07:12 /lib/x86_64-linux-gnu/libonlp-platform-defaults.so.1 -> libonlp-platform-defaults.so
lrwxrwxrwx 1 root root    100 Jan  9 08:00 /lib/x86_64-linux-gnu/libonlp-platform.so -> /lib/platform-config/x86-64-accton-wedge100bf-65x-r0/onl/lib/libonlp-x86-64-accton-wedge100bf-65x.so
lrwxrwxrwx 1 root root     19 Jan  9 07:12 /lib/x86_64-linux-gnu/libonlp-platform.so.1 -> libonlp-platform.so
-rwxr-xr-x 1 root root 419208 May  4  2019 /lib/x86_64-linux-gnu/libonlp.so
lrwxrwxrwx 1 root root     10 Jan  9 07:12 /lib/x86_64-linux-gnu/libonlp.so.1 -> libonlp.so
lrwxrwxrwx 1 root root     21 Jan  9 13:07 /lib/x86_64-linux-gnu/libonlp-x86-64-accton-wedge100bf-65x.so.1 -> libonlp-platform.so.1

Do I need to change something ? I have no idea on the default one and platform one you mentioned. Is there any available document?

Thanks again!

[Yi-Tseng]

This looks right.
libonlp-platform.so -> /lib/platform-config/x86-64-accton-wedge100bf-65x-r0/onl/lib/libonlp-x86-64-accton-wedge100bf-65x.so

By the way, did you try adding --bf_sim flag?
https://github.com/stratum/stratum/tree/master/stratum/hal/bin/barefoot#running-the-binary-with-bsp-or-tofino-software-model

[Yukinarii]

Yes, I run Stratum with --bf_sim, as the following command:

sudo LD_LIBRARY_PATH=$BF_SDE_INSTALL/lib \
     ./bazel-bin/stratum/hal/bin/barefoot/stratum_bf \
       --external_stratum_urls=0.0.0.0:28000 \
       --grpc_max_recv_msg_size=256 \
       --bf_sde_install=$BF_SDE_INSTALL \
       --persistent_config_dir=/root/config \
       --forwarding_pipeline_configs_file=/root/config/p4_pipeline.pb.txt \
       --chassis_config_file=/root/config/chassis_config.pb.txt \
       --write_req_log_file=/root/p4_writes.pb.txt \
       --bf-sim
       --bf_switchd_cfg=stratum/hal/bin/barefoot/tofino_skip_p4_no_bsp.conf

I have also tried with the BSP mode without the --bf_switch_cfg flag as:

sudo LD_LIBRARY_PATH=$BF_SDE_INSTALL/lib \
     ./bazel-bin/stratum/hal/bin/barefoot/stratum_bf \
       --external_stratum_urls=0.0.0.0:28000 \
       --grpc_max_recv_msg_size=256 \
       --bf_sde_install=$BF_SDE_INSTALL \
       --persistent_config_dir=/root/config \
       --forwarding_pipeline_configs_file=/root/config/p4_pipeline.pb.txt \
       --chassis_config_file=/root/config/chassis_config.pb.txt \
       --write_req_log_file=/root/p4_writes.pb.txt \
       --bf-sim

Thank you! By the way, though I specify the forwarding_pipeline_configs_file as /root/config/p4_pipeline.pb.txt, I do not really put such a file in the corresponding directory since I have no idea how I can get such a text file. Is it the reason that I got this error message?

[pudelkoM]

By the way, though I specify the forwarding_pipeline_configs_file as /root/config/p4_pipeline.pb.txt

This is the file where the pipeline will be stored once you push one. It does not have to exist. The error you see is unrelated to that.

[Yukinarii]

By the way, though I specify the forwarding_pipeline_configs_file as /root/config/p4_pipeline.pb.txt

This is the file where the pipeline will be stored once you push one. It does not have to exist. The error you see is unrelated to that.

Thank you! Then guess the problem is still related to the onlp settings.

[Yi-Tseng]

We are adding a new build profile that disables the ONLP library, will merge soon after we refactor the PHAL components.
Feel free to try this patch and let me know if there are any issues so I can fix it soon.

[Yukinarii]

Hi,
After trying the patch, I got the following error:

Cannot read termcap database;
using dumb terminal settings.
stratum_bf: external/com_github_p4lang_PI_bf_9_1_0/src/pi.c:159: pi_init: Assertion `abi_version == 1 && "PI ABI version mismatch"' failed.
*** Aborted at 1581171345 (unix time) try "date -d @1581171345" if you are using GNU date ***

I found that in here, it is specified that if the implementation is changed, then the PI ABI version should be incremented and I think that the PI downloaded and built by the bazel script is different from the origin. Do I need to change the definition in the source file and try to rebuild one?

Thank you!

[Yi-Tseng]

Hi, Did you use p4studio script provided by Barefoot to install dependencies?
Looks like the PI version you installed in your device is old.

Here is the script we install dependencies:
https://github.com/stratum/stratum/blob/master/stratum/hal/bin/barefoot/docker/Dockerfile#L40

[Yukinarii]

Hi,
Yes, I ran ./p4studio_build.py -up stratum_profile before building Stratum, and built Stratum with bazel build //stratum/hal/bin/barefoot:stratum_bf --define phal_with_onlp=false The patch I built is b5577df.

I have also tried with the up-to-date patch bf86189 and got this error:

ERROR: /root/bf-sde-9.0.0/stratum/stratum/hal/lib/phal/BUILD:227:1: C++ compilation of rule '//stratum/hal/lib/phal:adapter' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 128 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
In file included from ./stratum/glue/status/status_macros.h:139:0,
                 from ./stratum/lib/macros.h:27,
                 from ./stratum/lib/channel/channel_internal.h:23,
                 from ./stratum/lib/channel/channel.h:29,
                 from ./stratum/hal/lib/phal/attribute_database_interface.h:30,
                 from ./stratum/hal/lib/phal/adapter.h:25,
                 from stratum/hal/lib/phal/adapter.cc:16:
./stratum/glue/status/statusor.h: In instantiation of 'util::StatusOr<T>::StatusOr(const T&) [with T = std::unique_ptr<stratum::hal::phal::Query>]':
stratum/hal/lib/phal/adapter.cc:37:10:   required from here
./stratum/glue/status/statusor.h:238:48: error: use of deleted function 'std::unique_ptr<_Tp, _Dp>::unique_ptr(const std::unique_ptr<_Tp, _Dp>&) [with _Tp = stratum::hal::phal::Query; _Dp = std::default_delete<stratum::hal::phal::Query>]'
     : status_(::util::Status::OK), value_(value) {
                                                ^
In file included from /usr/include/c++/4.9/memory:81:0,
                 from ./stratum/hal/lib/phal/adapter.h:21,
                 from stratum/hal/lib/phal/adapter.cc:16:
/usr/include/c++/4.9/bits/unique_ptr.h:356:7: note: declared here
       unique_ptr(const unique_ptr&) = delete;
       ^
Target //stratum/hal/bin/barefoot:stratum_bf failed to build
Use --verbose_failures to see the command lines of failed build steps.

Am I missing something?

Thanks!

[pudelkoM]

Am I missing something?

Yes, our code requires support for DR1579 from the compiler. GCC implements this since version 5, also see this post.

It seems you are using GCC 4.8? Even Debian 9 (stretch, oldstable), which reaches EOL this year, ships with 6.3. Unless there is a dire need, we'll not accommodate ancient setups.

[Yukinarii]

Hi,
Yes, my GCC version 4.9.2. Since I am using ONL-4.14.49, the based OS is Debian 8, which provides only GCC 4.9.2. I have updated and installed GCC 6.3 as your suggestion.

By the way, the previous error on abi_version of PI is because that I am using SDE-9.0.0 and forgot compiling Stratum with flag --define sde_ver=9.0.0.
Now, I run Stratum with

sudo LD_LIBRARY_PATH=$BF_SDE_INSTALL/lib \
     ./bazel-bin/stratum/hal/bin/barefoot/stratum_bf \
       --external_stratum_urls=0.0.0.0:28000 \
       --grpc_max_recv_msg_size=256 \
       --bf_sde_install=$BF_SDE_INSTALL \
       --persistent_config_dir=/root/config \
       --forwarding_pipeline_configs_file=/root/config/p4_pipeline.pb.txt \
       --chassis_config_file=/root/config/chassis_config.pb.txt \
       --write_req_log_file=/root/p4_writes.pb.txt \
       --phal_config_path=/root/config/phal_config.pb.txt

but it again shows the following errors:

bf-sde> E0213 12:23:55.805009 24310 utils.cc:120] StratumErrorSpace::ERR_FILE_NOT_FOUND: /root/config/p4_pipeline.pb.txt not found.
E0213 12:23:55.805665 24310 utils.cc:76] Return Error: ReadFileToString(filename, &text) failed with StratumErrorSpace::ERR_FILE_NOT_FOUND: /root/config/p4_pipeline.pb.txt not found.
E0213 12:23:55.846778 24310 hal.cc:234] Stratum external facing services are listening to 0.0.0.0:28000, localhost:28000...
E0213 12:23:55.852046 24310 hal.cc:376] StratumErrorSpace::ERR_INTERNAL: Failed to check in with procmon: failed to connect to all addresses
E0213 12:23:55.852283 24310 hal.cc:244] Error when checking in with procmon: Failed to check in with procmon: failed to connect to all addresses.

According to @pudelkoM

By the way, though I specify the forwarding_pipeline_configs_file as /root/config/p4_pipeline.pb.txt

This is the file where the pipeline will be stored once you push one. It does not have to exist. The error you see is unrelated to that.

This config file would be pushed by ONOS once the Stratum can connect with ONOS.
However, I have no idea on why Stratum failed to check with procmon, which seems to be a component of Stratum.

Am I missing or misunderstanding something?

Thank you!

[pudelkoM]

Am I missing or misunderstanding something?

If the snipped you posted are all the errors, then you should be good to go. I see this prompt bf-sde> in the output. Can you just press Enter and try a few bf shell commands? Like pm show.

[Yukinarii]

Hi,
Thank you for the response. After entering Enter it went well and I can key some bf shell commands. However, when I try to connect ONOS with Stratum, I got the following error:
2020-02-14 03:20:27.962725 BF_PI ERROR - pi_bf_cb_add_device: error when assigning device through PI

2020-02-14 03:20:27.962816 BF_DVM ERROR - Device add handling failed for dev 0, sts Unknown error (1000), Client pi
2020-02-14 03:20:27.962849 BF_DVM ERROR - Device add failed for dev 0, sts Unknown error (1000)
2020-02-14 03:20:27.962886 BF_PM ERROR - pm_port_info_get_first:191 Unable to get first port info for dev : 0, port_key 0 : map_err : 2
2020-02-14 03:20:27.962915 BF_PM ERROR - pm_port_info_get_first:191 Unable to get first port info for dev : 0, port_key 0 : map_err : 2
ERROR: bf_device_add failed(1000) for dev_id 0
ERROR: bf_switchd_device_add failed(1000) for dev_id 0
E0214 03:20:27.998915 24286 PI-device_mgr.cpp:0] Error in first phase of device update
E0214 03:20:27.999125 24286 bf_switch.cc:84] Return Error: pi_node->PushForwardingPipelineConfig(config) failed with generic::unknown:
E0214 03:20:27.999179 24286 p4_service.cc:377] generic::unknown: Error without message at stratum/hal/lib/common/p4_service.cc:377
E0214 03:20:27.999215 24286 error_buffer.cc:41] (p4_service.cc:418): Failed to set forwarding pipeline config for node 1: Error without message at stratum/hal/lib/common/p4_service.cc:377

Here is the output messages in the ONOS:

2020-02-14T11:20:43,749 | WARN  | grpc-default-executor-0 | P4RuntimeClientImpl              | 227 - org.onosproject.onos-protocols-grpc-ctl - 2.3.0.SNAPSHOT | Error while performing SET-pipeline-config on device:my-tofino-switch: UNKNOWN: Error without message at stratum/hal/lib/common/p4_service.cc:377
2020-02-14T11:20:49,708 | INFO  | onos-pipeconf-watchdog-6 | PipelineConfigClientImpl         | 234 - org.onosproject.onos-protocols-p4runtime-ctl - 2.3.0.SNAPSHOT | Setting pipeline config for device:my-tofino-switch to org.opencord.fabric.tofino.montara_sde_9_0_0...
2020-02-14T11:20:51,026 | WARN  | grpc-default-executor-0 | StreamClientImpl                 | 234 - org.onosproject.onos-protocols-p4runtime-ctl - 2.3.0.SNAPSHOT | Error on StreamChannel RPC for device:my-tofino-switch: UNAVAILABLE: Network closed for unknown reason

I have no idea on this issue because I just follow the guide and everything went well at the beginning until I push the network setting by
make netcfg ONOS_HOST=localhost
My netcfg file:

{
  "devices": {
    "device:my-tofino-switch": {
      "basic": {
        "managementAddress": "grpc://TOFINO_MANAGEMENT_ADDR:28000?device_id=1",
        "driver": "stratum-tofino",
        "pipeconf": "org.opencord.fabric.tofino.montara_sde_9_0_0"
      }
    }
  }
}

Why this error occured?

[Yi-Tseng]

Please replace montara to mavericks in your ONOS network config since you are using Wedge100BF-65x
Also, I think you need to add -bf_switchd_cfg with tofino_skip_p4.conf config.

tofino_skip_p4_no_bsp.conf: No BSP
tofino_skip_p4.conf: with BSP

[Yukinarii]

Hi @Yi-Tseng ,
Thanks for your suggestion. I have followed them and try it again, but I still got the same error message as described in:

Hi,
Thank you for the response. After entering Enter it went well and I can key some bf shell commands. However, when I try to connect ONOS with Stratum, I got the following error:
2020-02-14 03:20:27.962725 BF_PI ERROR - pi_bf_cb_add_device: error when assigning device through PI

2020-02-14 03:20:27.962816 BF_DVM ERROR - Device add handling failed for dev 0, sts Unknown error (1000), Client pi
2020-02-14 03:20:27.962849 BF_DVM ERROR - Device add failed for dev 0, sts Unknown error (1000)
2020-02-14 03:20:27.962886 BF_PM ERROR - pm_port_info_get_first:191 Unable to get first port info for dev : 0, port_key 0 : map_err : 2
2020-02-14 03:20:27.962915 BF_PM ERROR - pm_port_info_get_first:191 Unable to get first port info for dev : 0, port_key 0 : map_err : 2
ERROR: bf_device_add failed(1000) for dev_id 0
ERROR: bf_switchd_device_add failed(1000) for dev_id 0
E0214 03:20:27.998915 24286 PI-device_mgr.cpp:0] Error in first phase of device update
E0214 03:20:27.999125 24286 bf_switch.cc:84] Return Error: pi_node->PushForwardingPipelineConfig(config) failed with generic::unknown:
E0214 03:20:27.999179 24286 p4_service.cc:377] generic::unknown: Error without message at stratum/hal/lib/common/p4_service.cc:377
E0214 03:20:27.999215 24286 error_buffer.cc:41] (p4_service.cc:418): Failed to set forwarding pipeline config for node 1: Error without message at stratum/hal/lib/common/p4_service.cc:377

Here is the output messages in the ONOS:

2020-02-14T11:20:43,749 | WARN  | grpc-default-executor-0 | P4RuntimeClientImpl              | 227 - org.onosproject.onos-protocols-grpc-ctl - 2.3.0.SNAPSHOT | Error while performing SET-pipeline-config on device:my-tofino-switch: UNKNOWN: Error without message at stratum/hal/lib/common/p4_service.cc:377
2020-02-14T11:20:49,708 | INFO  | onos-pipeconf-watchdog-6 | PipelineConfigClientImpl         | 234 - org.onosproject.onos-protocols-p4runtime-ctl - 2.3.0.SNAPSHOT | Setting pipeline config for device:my-tofino-switch to org.opencord.fabric.tofino.montara_sde_9_0_0...
2020-02-14T11:20:51,026 | WARN  | grpc-default-executor-0 | StreamClientImpl                 | 234 - org.onosproject.onos-protocols-p4runtime-ctl - 2.3.0.SNAPSHOT | Error on StreamChannel RPC for device:my-tofino-switch: UNAVAILABLE: Network closed for unknown reason

I have no idea on this issue because I just follow the guide and everything went well at the beginning until I push the network setting by
make netcfg ONOS_HOST=localhost
My netcfg file:

{
  "devices": {
    "device:my-tofino-switch": {
      "basic": {
        "managementAddress": "grpc://TOFINO_MANAGEMENT_ADDR:28000?device_id=1",
        "driver": "stratum-tofino",
        "pipeconf": "org.opencord.fabric.tofino.montara_sde_9_0_0"
      }
    }
  }
}

Why this error occured?

Is it possible that it is an issue caused by my wrong configuration on ONOS? Likes there may be a private driver and I need to download and compile it.

Thank you so much!

[Yi-Tseng]

Hi, can you send me all log messages via email ([email protected])?
I can't reproduce the issue on the switch I have.
Thanks

[Yi-Tseng]

As per email discussion
Set up the huge page should fix this issue
I also update the README file here: #126

This issue can be re-opened if needed