omec-project / ngic-rtc Goto Github PK

Add manifests under deploy/k8s folder that will help bring up each of the S/PGW-C/U components in this repo

Attach, detach issues observed randomly in jenkins tests (TC1 and TC2). Issue is reproducible in manual run as well.

Profile: 32K UES, 250 TPS
Setup: ONF jenkins setup
Testcase: Testcase 1 and 2 ( ZMQ + KNI, UDP + STATIC ARP)

Polaris Report:
+----------------------------------------------Summary-----------------------------------------------+
+ 1. Test start time.............................................................04-04-2019 17:43:28 +
+ 2. Test end time...............................................................04-04-2019 17:50:07 +
+ 3. Total time taken....................................................................399 seconds +
+ 4. Average time per test case..........................................................399 seconds +
+ 5. Total number of tests executed................................................................1 +
+ 6. Total number of tests passed..................................................................0 +
+ 7. Total number of tests failed..................................................................1 +
+ 8. Test pass percentage.........................................................................0% +
+--------------------------------------------Passed Tests--------------------------------------------+
+  No Test Passed.                                                                                   +
+--------------------------------------------Error Tests---------------------------------------------+
+--------------------------------------------Failed Tests--------------------------------------------+
+ 1. ./Attach-Detach-wdata.tcl                                                                       +
+     Errors:                                                                                        +
+      1. Attach failed : 739 out of 32000 attempts.                                                 +
+      2. Detach failed : 4 out of 31261 attempts.                                                   +
+----------------------------------------------Results-----------------------------------------------+
+ 1. ./Attach-Detach-wdata.tcl                                                                       +
+     Results:                                                                                       +
+      1. Attach: Attempt: 32000, Success: 31261,  Completion Time (max/min/avg): 4111.92/8.66/77.15 +
+      2. Detach: Attempt: 31261, Success: 31257,  Completion Time (max/min/avg): 240.31/4.81/21.65  +

The SGW-C is sending reversed values of S1U_F-TEID to SGWU

Currently in DP configuration (dp_config.cfg) user needs to specify PCI ID and MAC Address where, there is chance of typo error. Instead, we can use the Port index (DPDK port bind order) and using DPDK API get the MAC address which will make configuration simple.

This was briefly mentioned in #11 but what bots can we add to make the GitHub experience better?

https://probot.github.io/apps/todo/
https://probot.github.io/apps/dep/
https://probot.github.io/apps/commitlint/
https://probot.github.io/apps/ci-reporter/
https://probot.github.io/apps/stale/
https://github.com/apps/toc-me

@omec-project/ngic-maintainers @omec-project/tst

Opening this issue as per comment #59 (comment)

Since the git history is not available, we should at least credit contributors to NGIC codebase, from before release in the README.md or CONTRIBUTING.md.

Certificates are checked in under dp/conf. We should delete this folder and include a helper script to generate the certs like in c3po components.

The CP [SGW-C/PGW-C] is sending reversed values in the fields S11 MME F-TEID IPv4 header.

Capture the bug we discovered internally in the release notes, if it has not been fixed already.

Excerpt:

We tried to execute a couple of commands specific to S1Udev/SGIdev interface configurations with ngic-rtc (KNI mode). It turns that running commands such as changing MTU size (when ngic-rtc is running against il_trafficgen) triggers a crash. This is because the current code is not taking care of the race condition(s) that may arise when MTU size is being changed (rx/tx operations on the NIC are not paused, while ports are being configured).

Close this issue if it has been fixed.

To keep coding style consistent across developers and IDEs we should use clang-format

Example diff-1: aba7e81
Example diff-2: 2583b62

Have clang-format done on the diff in hooks/pre-commit

To help re-use in other build environments, e.g docker build with Dockerfile, CI systems

• Non-interactive
• Source able
• Do not require sudo if already root user - check if $EUID or id -u is 0

https://github.com/omec-project/ngic-rtc/pull/20/files#diff-b56887c211590e2ed739c0c98ab38264R77

-CFLAGS += -I$(SRCDIR)/../dpdk/lib/librte_acl/
+# TODO: This is a hack
+# We should not be looking into private structures
+# Handle this in a patch to DPDK if necessary
+# Check release notes
+CFLAGS += -I$(RTE_SDK)/lib/librte_acl/

This will also cause issues in clang-format see #16

The CP [SGW-C/PGW-C] is sending reversed values in the fields F-TEID IPv4 header.

• Remove hardcode of configuration directory location and replace with --conf-dir option
  • #define METER_PROFILE_FILE "../config/meter_profile.cfg"
  • #define PCC_RULE_FILE "../config/pcc_rules.cfg"
  • #define SDF_RULE_FILE "../config/sdf_rules.cfg"
  • #define ADC_RULE_FILE "../config/adc_rules.cfg"
  • #define IFACE_FILE "../config/interface.cfg"
  • #define SIMU_CP_FILE "../config/simu_cp.cfg"
• Consolidate all config parameters in config files instead of split between scripts/commandline and config to help ease customization. Ideally ./binary --conf-dir /path/to/conf/dir should be the usage

Pull request #32 introduces a patch that enables the data-plane to communicate with the kernel via af_packet sockets. We use netlink sockets in order to capture link events (such as mtu size changes, promiscuous mode changes, etc.), with this new interface. We submitted a commit that uses libmnl-dev library to handle such events. Introducing this library to the dataplane means that one will have to install this library (Ubuntu deb package is already available) and then link it in dp's Makefile. I suggest that we make this update since using this library simplifies the overall code structure.

In future revisions, one can also use the same API (/usr/include/libnl3, libmnl-dev) to capture route event changes that are currently being handled in dp/pipeline/epc_arp.c file. In the long run, the overall code structure of epc_arp.c may improve.

Breaking the S5S8 communication between SGWU and PGWU. Ping requests are not working either between both elements.

Any reason to have CFLAGS += -DNGCORE_SHRINK in config/ng-core_cfg.mk ? Can we move it into dp/Makefile

Need to remove the following parameters from dp_config and run.sh

 --sgw_s5s8gw_ip $SGW_S5S8GW_IP  \
 --sgw_s5s8gw_mask $SGW_S5S8GW_MASK    
 --sgw_s5s8gw_ip $PGW_S5S8GW_IP  \
 --sgw_s5s8gw_mask $PGW_S5S8GW_MASK  

EAL: FATAL: Invalid 'command line' arguments.
EAL: Invalid 'command line' arguments.
EAL: Error - exiting with code: 1
Cause: Error with EAL initialization

When testing with Dockerfile and Ubuntu 18.04 base, DP crashes when CP connects

        0         0         0         0         0   ||         0         0         0         0         0
12: [/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f394ea6688f]]
11: [/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f394ed3d6db]]
10: [ngic_dataplane(eal_thread_loop+0x1f1) [0x5598082fc9f1]]
9: [ngic_dataplane(+0x81b91) [0x559808279b91]]
8: [ngic_dataplane(+0x81bbd) [0x559808279bbd]]
7: [ngic_dataplane(iface_process_ipc_msgs+0xdf) [0x55980828119f]]
6: [ngic_dataplane(+0x781b4) [0x5598082701b4]]
5: [ngic_dataplane(meter_profile_entry_add+0xb9) [0x559808280e29]]
4: [ngic_dataplane(dp_meter_profile_entry_add+0x2a) [0x55980827080a]]
3: [/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7f394e983f20]]
2: [ngic_dataplane(sig_handler+0x38) [0x559808280568]]
1: [ngic_dataplane(rte_dump_stack+0x2e) [0x5598082fecae]]
EAL: Error - exiting with code: 1
  Cause: received SIGSEGV

As noted in #54 (comment) this only appears when compiling with -O3

In our tests with ngic-rtc against ng40, both UL and DL cores are sometimes reporting checksum errors on packet reception:

        0        15         0         0         0   ||         0        15         0         0         0
        0        17         0         0         0   ||         0        17         0         0         0
DP: DL Bad checksum: 50
DP: DL Bad checksum: 48
DP: DL Bad checksum: 52
DP: DL Bad checksum: 48
DP: DL Bad checksum: 48
DP: DL Bad checksum: 48
DP: DL Bad checksum: 170
DP: DL Bad checksum: 172
DP: DL Bad checksum: 168
DP: DL Bad checksum: 168
DP: DL Bad checksum: 168
DP: DL Bad checksum: 174
DP: DL Bad checksum: 168
DP: DL Bad checksum: 168
DP: DL Bad checksum: 168
DP: UL Bad checksum: 24
DP: UL Bad checksum: 24
DP: UL Bad checksum: 24
DP: DL Bad checksum: 168
DP: DL Bad checksum: 174
DP: DL Bad checksum: 54
DP: DL Bad checksum: 48
DP: DL Bad checksum: 50
DP: DL Bad checksum: 174
        0   1128679   1131981   1131975         6   ||         0   3069894   3074366   3074366         0

The code needs to be analyzed to find the root cause of this problem.

The CP [SGW-C/PGW-C] is sending multiple reversed values in the fields F-TEID IPv4 and PDN Address and Prefix (IPv4) to the MME in the Create Session Response back to the MME [NG40]. The MME is rejecting the responses because they don't have the proper values.

pdn-fteid-s11-pcap.zip

1. dev_scripts contains scripts and C code.
2. kni_ifcfg has some extra scripts. This is in addition to install.sh and setenv.sh.
3. There is lib which contains libsponsdn, then there is top-level libgtpv2c, which in turn one would assume would be related to gtpv2c_messages. Similarly gtpv2c* files under cp.
4. Is DP's transition to run-to-completion complete? Can we remove pipeline.
5. Also, should we move dp/dp_rules_verify to top-level test folder?

Today we carry DPDK patch for rearm issue here

Lets see if the original patch to address the issue works with 18.02 and carry that instead
http://patches.dpdk.org/patch/43451/

cc @ajamshed

Today patches/ is flat. We should create sub directories denoting which dependency we are patching so that install scripts can apply all of them at once per dependency.

As we start working on the codebase, nobody including owners should be able to push directly to master. Let us put branch protection in place to make all changes through a pull request.

This file has not stayed up to date with DPDK version common_linuxapp, and by overwriting in install.sh some of the libraries may not get compiled in.

If we still need this file then we update and have absolute minimal additions we want different to support ngic and document the need,

Else get rid of this file altogether.

This is essential in dynamic environments where depending on

• https://github.com/omec-project/ngic-rtc/blob/master/cp/main.c#L134-L138
• https://github.com/omec-project/ngic-rtc/blob/master/interface/interface.c#L989-L996

Have retry logic that would do X tries with Y seconds of sleep before giving up on name resolution and exit

libgtpv2c is built is differently than other components in the repo

• as a shared library
• independent of DPDK build system

Any particular reasoning for this? If there is no technical requirement should we

• build as a static library
• leverage DPDK build system

From SGWU dp logs,

The ret_arp_data IP is reversed.

DP: SGI packet
DP: BEAR_SESS LKUP:DL_KEY ue_addr:16.255.255.253, rid:1
DP: ARP entry not found for 102.1.8.11!!!
DP: Sendto:: ret_arp_data->ip= 102.1.8.11
DP: construct_ether_hdr::arp_queue_unresolved_packet::
return -1; arp_key.ip= 0xB080166

Add Dockerfile and Makefile targets to build S/PGW-C/U images.

Depends-on: #6 to stay free of duplication of installation steps

• Makefiles should not exist in this config directory (PR #18)
• Document within config files or README.md in this folder or docs/ as per #5
  • the key and possible values
  • optional/required
  • which entity/mode requires this file e.g., SPGWU, PGWU, etc.,

• Keep the landing README.md light. Move documentation into appropriate files under docs and provide pointers, with exceptions of contributing, release notes, etc. E.g., 1, 2
• New user quick start
  • build - install.sh
  • run - minimum config files to modify and test without needing a server like https://github.com/ngiccorddemo/cordbuild2017
• Packet walks showing in different configs/modes S/PGW-C/U
  • cores, port, queues
  • key functions/tables and packet manipulations
• Call flow diagram with messages supported and not supported
• Create svg under docs/img instead of ascii art and reference in markdown

make works but not make -j $CPUS for higher CPUS or when not set. Has someone else observed?

These directories are empty and of no use. So delete them. The application should create any directory necessary or take an argument --log-dir for user to specify pre-created directory

• https://github.com/omec-project/ngic-rtc/tree/master/logs
• https://github.com/omec-project/ngic-rtc/tree/master/cp/logs
• https://github.com/omec-project/ngic-rtc/tree/master/dp/logs

Remove hardcode folder name. If need to then define in one place and re-use

dp/main.h

• #define SPGW_S1U_PCAP_FILE "logs/uplink.pcap"
• #define SPGW_SGI_PCAP_FILE "logs/downlink.pcap"
• #define SGW_S1U_PCAP_FILE "logs/sgw_s1u.pcap"
• #define SGW_S5S8_PCAP_FILE "logs/sgw_s5s8.pcap"
• #define PGW_S5S8_PCAP_FILE "logs/pgw_s5s8.pcap"
• #define PGW_SGI_PCAP_FILE "logs/pgw_sgi.pcap"

dp/timer_stats.c

• #define STATS_PATH "./logs/"

Related -
dp/session_cdr.c

• #define SESS_CDR_FILE "/var/log/dpn/sess_cdr.csv"
• https://github.com/omec-project/ngic-rtc/blob/master/dp/session_cdr.c#L39-L45

Need to do change in run.sh for memory parameter.

Instead of this NUMA0_MEMEORY, change the NUMA0_MEMORY.

Environments that do not allow to ship and install kni kernel module should continue to work. Since we rely on the Linux kernel to handle control traffic (ARP, ICMP, etc.,) we should provide an alternative where mirror veth interfaces are leverged using DPDK AF_PACKET vdev to send and receive packets to the kernel.

Instead of user providing S1U IP of DP in the CP commandline arguments, the DP can send a message to CP with this information. This will reduce the number of things to know upfront for CP.

EBI is coming back as ‘0’ in the create session response

s11_1.zip

There are lot of TODOs sprinkled over the code. Lets have a corresponding issue open here. This can be the top-level tracker issue.

• #21

Would this better? Opening an issue here because we don't have a top-level repo purely for this kind of discussion.

/* SPDX-License-Identifier: Apache-2.0
 * Copyright(c) 2019 Intel Corporation
 */

Source:
https://github.com/kata-containers/documentation/blob/master/Licensing-strategy.md#license-for-individual-files
https://doc.dpdk.org/guides/contributing/patches.html#source-license

File ng-core_cfg.mk has been moved to custom-cp/dp.mk in PR #18.
Old name is still referenced in repo:

ngic-rtc {master}$ grep -nr ng-core_cfg.mk .
./dev_scripts/wokni/Makefile_wokni:14:include $(NG_CORE)/config/ng-core_cfg.mk
./dev_scripts/wokni/Makefile_wkni:14:include $(NG_CORE)/config/ng-core_cfg.mk
./INSTALL.MD:216:`ng-core_cfg.mk`. In this case ZMQ push/pull model is used.
./INSTALL.MD:218:ZMQ_COMM flag (ZMQ Push Pull) is enabled by default in config/ng-core_cfg.mk
./INSTALL.MD:257:    `ng-core_cfg.mk`.
./INSTALL.MD:571:  If ZMQ_COMM flag is enabled in the config file ng-core_cfg.mk, then we need to start the
./.git/COMMIT_EDITMSG:3:Removed config/ng-core_cfg.mk
./config/interface.cfg:11:; ng-core_cfg.mk
./config/interface.cfg:46:; ng-core_cfg.mk
./config/interface.cfg:54:; ng-core_cfg.mk

We are currently relying on NIC-based checksum verification in init.c:

Does this cover checksum offloading for vdevs as well (e.g. virtio & af_packet devices)? This needs to be investigated.

Otherwise we need to have a pure SW-based fallback for RX-based checksum verification.

Expanding on question 5 in #12 this folder currently contains some big files and no documentation of its purpose and usage.

Also provide feedback on whether this should be moved to a test or ci folder

In Create Session Response under GTPv2 header Sequence Number is not correct, that's why MME rejects the Create Session Response.

Please find below the analysis when the sequence number is getting skewed:

CP: Before byte conversion:6e5
CP: 1 S11 SGWC recv after byte conversion Seq:e50600
CP: 2 SGWC Send to PGWC Seq:e50600
CP: 3 SGWC recv from PGWC Seq:e50600
CP: 4 Filled in cs_resp Seq:e50600
CP: 5 Before encoding Seq:e50600
CP: 6 After encoding Seq:6e5
CP: 7 Set Resp for MME seq:6e5
CP: 8 SGWC Send Resp to MME seq:6e5

s11-0212-sgwc.zip

Any idea whats happening in this file? The first paragraph seems like BSD-3 and has UC Berkeley name in it while the copyright notice on top is Intel and we have second paragraph with Apache-2.0 license

PGWU failed to launch due to RX/TX ring creation is failed.

RING: Cannot reserve memory
EAL: Error - exiting with code: 1
Cause: Cannot create RX ring 0

The below files currently do not have the SPDX license identifier and/or copyright

$ grep -rL 'SPDX-License-Identifier: Apache-2.0' * \
--exclude=\*.{cfg,pcap,patch,csv,gitignore,txt,pem} --exclude=LICENSE 

config/ng-core_cfg.mk
cp/chk_cpcfg.sh
cp_dp_api/vepc_cp_dp_api.c
dp/chk_dpcfg.sh
dp/gtpu_echo.c
dp/commands.c
dp/pipeline/epc_arp_icmp_api.hlp
dp/perf_timer.h
dp/timer_stats.c
dp/build_ngic.sh
kni_ifcfg/kni-S1Udevcfg.sh
kni_ifcfg/kni-S1Udevdel.sh
kni_ifcfg/kni-SGIdevdel.sh
kni_ifcfg/kni-SGIdevcfg.sh
test/zmq_device/req_streamer_dev.py
test/zmq_device/resp_streamer_dev.py
test/fmea/test_fmea.sh