Issues for changing node_id for stratum, how to fix it? about stratum

[xfyan0408]

It seems that it can not change node id for stratum, how to fix it?

[Dscano]

Hi linhil,

The nodeID in the switch should be 0 or 1, I think that is a design choiche is defined in the P4 standard. You can change the deviceID in ONOS changing the string in the netcfg file, more precisely here https://github.com/stratum/fabric-tna/blob/fd3c3f0989ae6c2bb2384bdd1b5b7755168ba886/tofino-netcfg.json#L3

[pudelkoM]

I confirmed that the node ID can be picked at users discretion.

chassis_config.pb.txt

description: "Default Chassis Config for Edgecore Wedge100BF-32x"
chassis {
  platform: PLT_GENERIC_BAREFOOT_TOFINO
  name: "Edgecore Wedge100BF-32x"
}
nodes {
  id: 2
  slot: 1
  index: 1
}
singleton_ports {
  id: 1
  name: "1/3"
  slot: 1
  port: 1
  channel: 4
  speed_bps: 10000000000
  config_params {
    admin_state: ADMIN_STATE_ENABLED
    autoneg: TRI_STATE_FALSE
  }
  node: 2
}

Then connect with device ID 2:
./p4runtime-sh-docker --config=<...> --grpc-addr=10.70.10.94:28000 --device-id=2

Logs:

bf_kdrv_mod found! Unloading first...
loading bf_kdrv_mod...
I20220630 21:45:10.167529 10084 logging.cc:72] Stratum version: not stamped.
I20220630 21:45:10.168002 10084 bf_sde_wrapper.cc:1757] bf_sysfs_fname: /sys/class/bf/bf0/device/dev_add
Install dir: /usr (0x3bd8020)
bf_switchd: system services initialized
bf_switchd: loading conf_file /usr/share/stratum/tofino_skip_p4.conf...
bf_switchd: processing device configuration...
Configuration for dev_id 0
  Family        : Tofino
  pci_sysfs_str : /sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0
  pci_domain    : 0
  pci_bus       : 5
  pci_fn        : 0
  pci_dev       : 0
  pci_int_mode  : 1
  sbus_master_fw: /usr/
  pcie_fw       : /usr/
  serdes_fw     : /usr/
  sds_fw_path   : /usr/share/tofino_sds_fw/avago/firmware
  microp_fw_path:
bf_switchd: processing P4 configuration...
P4 profile for dev_id 0
  p4_name: dummy
    libpd:
    libpdthrift:
    context:
    config:
  Agent[0]: /usr/lib/libpltfm_mgr.so
  diag:
  accton diag:
  non_default_port_ppgs: 0
  SAI default initialize: 1
bf_switchd: library /usr/lib/libpltfm_mgr.so loaded
bf_switchd: agent[0] initialized
Health monitor started
Operational mode set to ASIC
Initialized the device types using platforms infra API
ASIC detected at PCI /sys/class/bf/bf0/device
ASIC pci device id is 16
bf_switchd: drivers initialized
Skipping P4 program load for dev_id 0
Setting core_pll_ctrl0=cd44cbfe
2022-06-30 21:45:18.348978 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:18.349005 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:18.349022 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:18.349035 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:18.349062 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:18.349076 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:18.349089 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:18.349102 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage

bf_switchd: dev_id 0 initialized

bf_switchd: initialized 1 devices
Skip p4 lib init
Skip mav diag lib init
bf_switchd: spawning cli server thread
bf_switchd: spawning driver shell
bf_switchd: server started - listening on port 9999
I20220630 21:45:20.305521 10084 bf_sde_wrapper.cc:1767] switchd started successfully
I20220630 21:45:20.305662 10084 main_bfrt.cc:50] Detected is_sw_model: 0
I20220630 21:45:20.305699 10084 main_bfrt.cc:51] SDE version: 9.7.2
Cannot read termcap database;
using dumb terminal settings.
I20220630 21:45:20.305723 10084 main_bfrt.cc:52] Switch SKU: TOFINO_32D, revision B0, chip_id 0x83058b061255c85
W20220630 21:45:20.306016 10084 credentials_manager.cc:59] No key files provided, using insecure server credentials!
bf-sde> W20220630 21:45:20.307399 10084 credentials_manager.cc:78] No key files provided, using insecure client credentials!
I20220630 21:45:20.308743 10084 timer_daemon.cc:91] The timer daemon has been started.
I20220630 21:45:20.308969 10084 hal.cc:127] Setting up HAL in COLDBOOT mode...
I20220630 21:45:20.309191 10084 config_monitoring_service.cc:94] Pushing the saved chassis config read from ./chassis_config_alt_node_id.pb.txt...
I20220630 21:45:20.324800 10084 bfrt_switch.cc:321] Chassis config verified successfully.
I20220630 21:45:20.326619 10084 phal.cc:94] No phal_config_file specified and no switch configurator found! PHAL will start without any data source backend. You can specify '--define phal_with_tai=true' while building to enable TAI support, or '-enable_onlp' at runtime to enable the ONLP plugin.
I20220630 21:45:20.327490 10084 attribute_database.cc:207] PhalDB service is listening to localhost:28003...
I20220630 21:45:20.327567 10084 bf_chassis_manager.cc:1404] Successfully registered port status notification callback.
I20220630 21:45:20.338596 10084 bf_chassis_manager.cc:111] Added port 1 in node 2 (SDK Port 131).
I20220630 21:45:20.338644 10084 bf_chassis_manager.cc:128] Set autoneg policy TRI_STATE_FALSE for port 1 in node 2 (SDK Port 131).
I20220630 21:45:20.342468 10084 bf_chassis_manager.cc:147] Enabled port 1 in node 2 (SDK Port 131).
I20220630 21:45:20.342634 10084 bfrt_switch.cc:60] Chassis config pushed successfully.
I20220630 21:45:20.342674 10084 gnmi_events.h:709] Handling stratum::hal::ConfigHasBeenPushedEvent
I20220630 21:45:20.342705 10084 gnmi_publisher.h:205] Configuration has changed.
I20220630 21:45:20.345501 10084 p4_service.cc:121] Pushing the saved forwarding pipeline configs read from /etc/stratum/pipeline_cfg.pb.txt...
I20220630 21:45:21.068913 10084 bfrt_switch.cc:277] P4-based forwarding pipeline config verified successfully for node with ID 2.
bf_switchd: starting warm init for dev_id 0 mode 1 serdes_upgrade 0
bf_switchd: agent[0] library unloaded for dev_id 0
bf_switchd: library /usr/lib/libpltfm_mgr.so loaded
bf_switchd: agent[0] initialized
Health monitor started
Skip diag lib deinit
\2022-06-30 21:45:22.098339 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:22.098365 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:22.098381 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:22.098394 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:22.098407 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:22.098419 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:22.098433 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:22.098445 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
I20220630 21:45:24.803139 10084 bf_sde_wrapper.cc:1929] Registered packetio callbacks on device 0.
I20220630 21:45:24.814724 10084 bf_chassis_manager.cc:1089] Set MTU 0 for port 1 in node 2 (SDK Port 131).
I20220630 21:45:24.814771 10084 bf_chassis_manager.cc:1096] Set autoneg policy TRI_STATE_FALSE for port 1 in node 2 (SDK Port 131).
I20220630 21:45:24.814793 10084 bf_chassis_manager.cc:1104] Set loopback mode LOOPBACK_STATE_UNKNOWN for port 1 in node 2 (SDK Port 131).
I20220630 21:45:24.818393 10084 bf_chassis_manager.cc:1113] Enabled port 1 in node 2 (SDK Port 131).
I20220630 21:45:24.818416 10084 bf_chassis_manager.cc:1123] Replayed port 1 in node 2.
I20220630 21:45:24.818538 10084 bf_chassis_manager.cc:1171] Replayed chassis config for node 2.
I20220630 21:45:24.818554 10084 bfrt_switch.cc:78] P4-based forwarding pipeline config pushed successfully to node with ID 2.
E20220630 21:45:24.821306 10084 hal.cc:220] Stratum external facing services are listening to 0.0.0.0:28000, localhost:9559...
I20220630 21:45:26.415447 10325 bf_chassis_manager.cc:1288] State of port 1 in node 2 (SDK port 131): UP.
I20220630 21:45:26.415500 10322 gnmi_events.h:709] Handling stratum::hal::PortOperStateChangedEvent
I20220630 21:45:36.355947 10414 sdn_controller_manager.cc:148] New SDN connection (role_name: <default>, election_id: { high: 1 }, uri: ipv4:10.70.10.200:48878): device_id: 2 election_id { high: 1 }
I20220630 21:45:36.356567 10414 sdn_controller_manager.cc:323] Informing all connections about primary connection change.
I20220630 21:45:36.356878 10414 sdn_controller_manager.cc:199] New primary connection for role <default> with election ID { high: 1 }.
I20220630 21:45:36.356958 10414 p4_service.cc:590] Controller (role_name: <default>, election_id: { high: 1 }, uri: ipv4:10.70.10.200:48878) is connected as MASTER for node (aka device) with ID 2.
I20220630 21:45:36.393503 10567 bfrt_switch.cc:277] P4-based forwarding pipeline config verified successfully for node with ID 2.
bf_switchd: starting warm init for dev_id 0 mode 1 serdes_upgrade 0
bf_switchd: agent[0] library unloaded for dev_id 0
bf_switchd: library /usr/lib/libpltfm_mgr.so loaded
bf_switchd: agent[0] initialized
Health monitor started
/2022-06-30 21:45:37.405411 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:37.405432 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:37.405456 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:37.405468 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:37.405479 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:37.405490 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:37.405502 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
2022-06-30 21:45:37.405513 BF_TM  WARN - Traffic Manager buffer pool 0 pfc limit is higher or equal to pool size. PFC might not trigger based on pool usage
I20220630 21:45:40.116822 10567 bf_sde_wrapper.cc:1929] Registered packetio callbacks on device 0.
I20220630 21:45:40.127763 10567 bf_chassis_manager.cc:1089] Set MTU 0 for port 1 in node 2 (SDK Port 131).
I20220630 21:45:40.127830 10567 bf_chassis_manager.cc:1096] Set autoneg policy TRI_STATE_FALSE for port 1 in node 2 (SDK Port 131).
I20220630 21:45:40.127851 10567 bf_chassis_manager.cc:1104] Set loopback mode LOOPBACK_STATE_UNKNOWN for port 1 in node 2 (SDK Port 131).
I20220630 21:45:40.131418 10567 bf_chassis_manager.cc:1113] Enabled port 1 in node 2 (SDK Port 131).
I20220630 21:45:40.131440 10567 bf_chassis_manager.cc:1123] Replayed port 1 in node 2.
I20220630 21:45:40.131565 10567 bf_chassis_manager.cc:1171] Replayed chassis config for node 2.
I20220630 21:45:40.131584 10567 bfrt_switch.cc:78] P4-based forwarding pipeline config pushed successfully to node with ID 2.
I20220630 21:45:41.727454 10325 bf_chassis_manager.cc:1288] State of port 1 in node 2 (SDK port 131): UP.
I20220630 21:45:41.727514 10322 gnmi_events.h:709] Handling stratum::hal::PortOperStateChangedEvent