shenango / caladan

Interference-aware CPU scheduling that enables performance isolation and high CPU utilization for datacenter servers

Shenango

Shenango is a system that enables servers in datacenters to simultaneously provide low tail latency and high CPU efficiency, by rapidly reallocating cores across applications, at timescales as small as every 5 microseconds.

For similar behavior to Shenango but with higher throughput, see Caladan, which builds on Shenango but enables higher throughput by moving the IOKernel off the datapath with directpath.

How to Run Shenango

1. Clone the Shenango repository.

git clone https://github.com/abelay/shenango
cd shenango

2. Setup DPDK and build the IOKernel and Shenango runtime.

./dpdk.sh
./scripts/setup_machine.sh
make clean && make

To enable debugging, build with make DEBUG=1.

3. Install Rust and build a synthetic client-server application.

curl https://sh.rustup.rs -sSf | sh
rustup default nightly

cd apps/synthetic
cargo clean
cargo update
cargo build --release

4. Run the synthetic application with a client and server. The client sends requests to the server, which performs a specified amount of fake work (e.g., computing square roots for 10us), before responding.

On the server:

sudo ./iokerneld
./apps/synthetic/target/release/synthetic 192.168.1.3:5000 --config server.config --mode spawner-server

On the client:

sudo ./iokerneld
./apps/synthetic/target/release/synthetic 192.168.1.3:5000 --config client.config --mode runtime-client

Supported Platforms

This code has been tested most thoroughly on Ubuntu 18.04, with kernel 4.15.0. It has been tested with Intel 82599ES 10 Gbits/s NICs and Mellanox ConnectX-3 Pro 10 Gbits/s NICs. If you use Mellanox NICs, you should install the Mellanox OFED as described in DPDK's documentation. If you use Intel NICs, you should insert the IGB UIO module and bind your NIC interface to it (e.g., using the script ./dpdk/usertools/dpdk-setup.sh).

caladan's Issues

dp_clients: failed to attach proc

Hi Caladan developers,

I am running into some issues when trying to run Caladan.

Environment

• i7-6700 CPU
• Intel 82599ES NIC
• Ubuntu 20.04 with kernel 5.4

Following instructions in Readme.txt. Intel NIC binds to the IGB UIO module.

Run the synthetic application in Step 5 (Server):

/caladan$ sudo ./iokerneld ias nobw
CPU 03| <5> cpu: detected 8 cores, 1 nodes
CPU 03| <5> time: detected 3407 ticks / us
[  0.000604] CPU 06| <5> sched: CPU configuration...
	node 0: [0,4][1,5][2,6][3,7]
[  0.000618] CPU 06| <5> sched: dataplane on 4, control on 0
[  0.026331] CPU 06| <5> control: spawning control thread
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:00:1f.6 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:15b7 net_e1000_em
EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:01:00.1 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: WARNING! Base virtual address hint (0x1100805000 != 0x7f14f0e3b000) not respected!
EAL:    This may cause issues with mapping memory into secondary processes
[  3.138086] CPU 04| <5> dpdk: driver: net_ixgbe port 0 MAC: 00 1b 21 88 d2 94
[  3.138098] CPU 04| <5> main: core 4 running dataplane. [Ctrl+C to quit]

/caladan$ ./apps/synthetic/target/release/synthetic 192.168.2.2:5000 --config server.config --mode spawner-server
CPU 06| <5> cpu: detected 8 cores, 1 nodes
CPU 06| <5> time: detected 3407 ticks / us
[  0.000229] CPU 06| <5> loading configuration from 'server.config'
[  0.000256] CPU 06| <5> process pid: 542683
[  0.013920] CPU 06| <5> net: started network stack
[  0.013935] CPU 06| <5> net: using the following configuration:
[  0.013938] CPU 06| <5>   addr:	192.168.2.2
[  0.013942] CPU 06| <5>   netmask:	255.255.255.0
[  0.013945] CPU 06| <5>   gateway:	192.168.2.1
[  0.013949] CPU 06| <5>   mac:	0E:47:46:C3:F3:AD
[  0.014025] CPU 06| <5> thread: created thread 0
[  0.014070] CPU 06| <5> spawning 4 kthreads
[  0.014147] CPU 03| <5> thread: created thread 1
[  0.014163] CPU 02| <5> thread: created thread 2
[  0.014185] CPU 05| <5> thread: created thread 3
192.168.2.2:5000

Run the synthetic application in Step 5 (Client. dp_clients: failed to attach proc):

~/caladan$ sudo ./iokerneld ias nobw
CPU 02| <5> cpu: detected 8 cores, 1 nodes
CPU 02| <5> time: detected 3408 ticks / us
[  0.000274] CPU 02| <5> sched: CPU configuration...
	node 0: [0,4][1,5][2,6][3,7]
[  0.000287] CPU 02| <5> sched: dataplane on 4, control on 0
[  0.025720] CPU 02| <5> control: spawning control thread
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:00:1f.6 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:15b7 net_e1000_em
EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:01:00.1 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: WARNING! Base virtual address hint (0x1100805000 != 0x7fe473b0a000) not respected!
EAL:    This may cause issues with mapping memory into secondary processes
[  1.980692] CPU 04| <5> dpdk: driver: net_ixgbe port 0 MAC: 00 1b 21 89 6e ac
[  1.980703] CPU 04| <5> main: core 4 running dataplane. [Ctrl+C to quit]
[258.711498] CPU 04| <2> dp_clients: failed to attach proc.

~/caladan$ ./apps/synthetic/target/release/synthetic 192.168.2.2:5000 --config client.config --mode runtime-client
CPU 03| <5> cpu: detected 8 cores, 1 nodes
CPU 03| <5> time: detected 3408 ticks / us
[  0.000281] CPU 03| <5> loading configuration from 'client.config'
[  0.000311] CPU 03| <5> process pid: 755087
[  0.004069] CPU 03| <5> net: started network stack
[  0.004085] CPU 03| <5> net: using the following configuration:
[  0.004088] CPU 03| <5>   addr:	192.168.2.3
[  0.004104] CPU 03| <5>   netmask:	255.255.255.0
[  0.004109] CPU 03| <5>   gateway:	192.168.2.1
[  0.004113] CPU 03| <5>   mac:	BE:E6:DD:BF:9C:EA
[  0.004195] CPU 03| <5> thread: created thread 0
[  0.004240] CPU 03| <5> spawning 4 kthreads
[  0.004314] CPU 06| <5> thread: created thread 1
[  0.004333] CPU 07| <5> thread: created thread 2
[  0.004408] CPU 03| <5> thread: created thread 3

Empty server_pcicfg_uncore in pcm_caladan_init()

Environment

Hardware

• i7-6700 CPU
• Intel 82599ES NIC

OS & kernel

• Ubuntu 18.04 with kernel 4.15
• Ubuntu 20.04 with kernel 5.4

build/config

Debug enabled, other options disabled, following all the instructions mentioned in github readme.

iokerneld

sudo gdb iokerneld
(gdb) r
Starting program: /home/user/caladan/iokerneld
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
CPU 04| <6> entering 'iokernel' init phase
CPU 04| <6> init -> base
CPU 04| <5> cpu: detected 8 cores, 1 nodes
CPU 04| <5> time: detected 3407 ticks / us
CPU 04| <6> init: entering 'early' init
CPU 04| <6> init: entering 'normal' init
CPU 04| <6> init: entering 'late' init
[  0.000301] CPU 04| <6> init -> ksched
[  0.000325] CPU 04| <6> init -> sched
[  0.000329] CPU 04| <5> sched: CPU configuration...
        node 0: [0,4][1,5][2,6][3,7]
[  0.000341] CPU 04| <5> sched: dataplane on 4, control on 0
[  0.000346] CPU 04| <6> init -> simple
[  0.000358] CPU 04| <6> init -> numa
[  0.000365] CPU 04| <6> init -> ias
IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : no
IBRS enabled in the kernel   : no
STIBP enabled in the kernel  : no
[New Thread 0x7ffff7045700 (LWP 134361)]
[New Thread 0x7ffff6844700 (LWP 134362)]
[New Thread 0x7ffff6043700 (LWP 134363)]
terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)

Thread 1 "iokerneld" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7a71859 in __GI_abort () at abort.c:79
#2  0x00007ffff7d2f951 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff7d3b47c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff7d3b4e7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff7d3b799 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff7d323eb in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00005555556b2ec9 in std::vector<std::shared_ptr<pcm::ServerPCICFGUncore>, std::allocator<std::shared_ptr<pcm::ServerPCICFGUncore> > >::_M_range_check (__n=0, this=<optimized out>)
    at /usr/include/c++/9/ext/atomicity.h:69
#8  std::vector<std::shared_ptr<pcm::ServerPCICFGUncore>, std::allocator<std::shared_ptr<pcm::ServerPCICFGUncore> > >::at (__n=0, this=<optimized out>)
    at /usr/include/c++/9/bits/stl_vector.h:1091
#9  pcm_caladan_init (socket=0) at pcm-caladan.cpp:28
#10 0x000055555560050f in ias_bw_init () at iokernel/ias_bw.c:260
#11 0x0000555555601a38 in ias_init () at iokernel/ias.c:555
#12 0x00005555555eb4d2 in run_init_handlers (phase=0x5555556f286b "iokernel", h=0x555555770aa0 <iok_init_handlers>, nr=13) at iokernel/main.c:65
#13 0x00005555555eb968 in main (argc=1, argv=0x7fffffffe6a8) at iokernel/main.c:190
(gdb)

test.cpp

#include "cpucounters.h"
#include <cstdio>
int main(void) {
    printf("1\n");
    auto a = pcm::PCM::getInstance();
    printf("2\n");
    a->server_pcicfg_uncore.at(0); //throws std::out_of_range
    //a->server_pcicfg_uncore[0];
    printf("3\n");
}

Compile this test with g++ test.cpp -I caladan/deps/pcm/ -L caladan/deps/pcm -o test-pcm -lPCM -lpthread.
Run sudo ./test-pcm and here is the output:

1
IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : no
IBRS enabled in the kernel   : no
STIBP enabled in the kernel  : no
2
terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)
Aborted

P999 may not be accurate if Never Send is high.

Hi all,

I find the synthetic in Caladan endures a high Never-Send rate (above 1%) when clients issue requests with a relatively high rate which is close to server’s capacity.
This is especially problematic under a Poisson distribution: when two adjacent requests are generated within a short time window (i.e., a bursty period), the latter one is more likely to be droped due to the Never-Send logic (see code). We have profiled Caladan's client logic, and find that the scheduling often causes the request to be delayed (which has already violated the Poisson distribution) and finally be dropped.

We further designed an experiment to confirm this. We modify the Caladan client with the scheduling policy disabled: specifically, workers are bound to different cores, which can execute send, do_softirq (directpath), handle_timeout, and recv in cycles without yielding.
We equip Caladan server with 4 kthreads and launch 16 client workers (each of which owns a TCP connection) to generate requests w/ a Poisson distribution and vary the request rate (last for 32 seconds). The following table shows the experiment result:

Client Type	Thoughtput(pps)	P50 (us)	P90 (us)	P999 (us)	Never Send
synthetic	0.75M	13.4	23.1	40.0	1.39%
client w/o sched	0.75M	9.809091	18.500909	50.903636	0.000442%
synthetic	0.8M	13.6	22.6	37.9	1.4486%
client w/o sched	0.8M	9.49	16.81	584.52	0.000430%
synthetic	1M	13.6	21.9	38.6	1.6726%
client w/o sched	1M	9.64	17.59	2841.83	0.000694%
synthetic	1.1M	13.5	21.1	55.5	1.7345%
client w/o sched	1.1M	10.6	21.9	5177.75	0.000781%

Can Caladan run on CloudLab m510 machines?

Hi Caladan developers,

I am running into some issues when trying to build and run Caladan on CloudLab m510 machines.

1. README.md says one should use make submodules to build the submodules. I think the correct command is actually build/init_submodules, right?

2. build/shared.mk didn't set MLX4_INC and MLX4_LIBS like it did for mlx5, so I added the following lines:

+# mlx4 build                       
+MLX4_INC = -I$(ROOT_PATH)/rdma-core/build/include
+MLX4_LIBS = -L$(ROOT_PATH)/rdma-core/build/lib/statics/
+MLX4_LIBS += -lmlx4 -libverbs -lnl-3 -lnl-route-3

Similarly, Makefile didn't use MLX4_[INC,LIBS], so I changed that too.

3. Typo in build/config. CONFIG_MLX4 is for ConnectX-3 support, not ConnectX-4.

4. After fixing the problems above, I can build iokerneld on m510 successfully. However, DPDK fails to initialize the port at startup. Here is the error message:

yilongl@rc01:/shome/caladan$ sudo ./iokerneld
CPU 09| <5> cpu: detected 16 cores, 1 nodes
CPU 09| <5> time: detected 1995 ticks / us
[  0.001901] CPU 09| <5> sched: CPU configuration...
        node 0: [0,8][1,9][2,10][3,11][4,12][5,13][6,14][7,15]
[  0.001944] CPU 09| <5> sched: dataplane on 8, control on 0
IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : no
IBRS enabled in the kernel   : no
STIBP enabled in the kernel  : no
Socket 0: 1 memory controllers detected with total number of 4 channels. 0 QPI ports detected. 0 M2M (mesh to memory) blocks detected. 1 Home Agents detected. 0 M3UPI blocks detected.
[  0.101894] CPU 00| <5> control: spawning control thread
EAL: Detected 16 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:09:00.0 on NUMA socket 0
EAL:   probe driver: 15b3:1007 net_mlx4
net_mlx4: 0x561d86587300: cannot attach flow rules (code 95, "Operation not supported"), flow error type 2, cause 0x1073cd200, message: flow rule rejected by device

CloudLab m510 machines are equipped with ConnectX3-Pro NIC so I assume Caladan should be able run on them. Any help would be greatly appreciated.

Potental stack allocation bug

Hi Caladan authors!

I just tried to print out each uthread's stack address and find they are generally 0x7ffff... However, based on the variable
of STACK_BASE_ADDR, stack address should start from 0x2000... Then I find this line actually calls atomic64_fetch_and_and() instead of atomic64_fetch_and_add(). So it seems this is a small typo?

Just want to let you know about this potential bug, although the runtime works perfectly without fixing it!

Best,
Yang

Breakwater running error: couldn't connect to raddr

Hi Calanda, Breakwater developers,
I try to run the Breakwater example application in readme but encounter problems. (Before that I do not encounter the same problem when I run the synthetic application in Calanda readme).
The Breakwater example application does run successfully when I run it the first time. But when I want to run it for many times, (each time, I resetup the machine using sudo ./scripts/setup_machine.sh, then start iokerneld and the application at the client. so does at the server) I can never run it successfully again unless I reboot the machine. The IP-related configuration is right since I can run the synthetic application in Calanda successfully.

Environment

• i7-6700 CPU
• Intel 82599ES NIC
• Ubuntu 20.04 with kernel 5.4

server.config file

host_addr 192.168.2.2
host_netmask 255.255.255.0
host_gateway 192.168.2.1
runtime_kthreads 4
runtime_guaranteed_kthreads 4

Server output:

I omit the output of sudo ./iokerneld ias nobw no_hw_qdel. so does client.

~/caladan/breakwater: sudo ./apps/netbench/netbench breakwater ../server.config server
CPU 02| <5> cpu: detected 8 cores, 1 nodes
CPU 02| <5> time: detected 3407 ticks / us
CPU 02| <6> init: entering 'early' init
CPU 02| <6> init: entering 'normal' init
CPU 02| <6> init: entering 'late' init
[  0.000296] CPU 02| <5> loading configuration from '../server.config'
[  0.006781] CPU 02| <5> process pid: 151042
[  0.006786] CPU 02| <6> entering 'global' init phase
[  0.006788] CPU 02| <6> init -> kthread
[  0.006798] CPU 02| <6> init -> ioqueues
[  0.019990] CPU 02| <6> init -> stack
[  0.020006] CPU 02| <6> init -> sched
[  0.020011] CPU 02| <6> init -> preempt
[  0.020017] CPU 02| <6> init -> smalloc
[  0.020025] CPU 02| <6> init -> net
[  0.020699] CPU 02| <5> net: started network stack
[  0.020702] CPU 02| <5> net: using the following configuration:
[  0.020706] CPU 02| <5>   addr:	192.168.2.2
[  0.020710] CPU 02| <5>   netmask:	255.255.255.0
[  0.020713] CPU 02| <5>   gateway:	192.168.2.1
[  0.020716] CPU 02| <5>   mac:	EA:82:2E:DD:AE:1E
[  0.020720] CPU 02| <6> init -> directpath
[  0.020723] CPU 02| <6> init -> arp
[  0.020734] CPU 02| <6> init -> trans
[  0.020825] CPU 02| <6> init -> storage
[  0.020828] CPU 02| <6> init -> gc
[  0.020854] CPU 02| <5> thread: created thread 0
[  0.020857] CPU 02| <6> init: entering 'thread' init
[  0.020860] CPU 02| <6> entering 'per-thread' init phase
[  0.020863] CPU 02| <6> init -> kthread
[  0.020867] CPU 02| <6> init -> ioqueues
[  0.020870] CPU 02| <6> init -> stack
[  0.020873] CPU 02| <6> init -> timer
[  0.020882] CPU 02| <6> init -> sched
[  0.020922] CPU 02| <6> init -> smalloc
[  0.020925] CPU 02| <6> init -> net
[  0.020928] CPU 02| <6> init -> directpath
[  0.020931] CPU 02| <6> init -> storage
[  0.020933] CPU 02| <5> spawning 4 kthreads
[  0.021004] CPU 01| <5> thread: created thread 1
[  0.021014] CPU 01| <6> init: entering 'thread' init
[  0.021018] CPU 01| <6> entering 'per-thread' init phase
[  0.021030] CPU 01| <6> init -> kthread
[  0.021022] CPU 03| <5> thread: created thread 2
[  0.021042] CPU 03| <6> init: entering 'thread' init
[  0.021046] CPU 03| <6> entering 'per-thread' init phase
[  0.021050] CPU 03| <6> init -> kthread
[  0.021052] CPU 02| <5> thread: created thread 3
[  0.021061] CPU 02| <6> init: entering 'thread' init
[  0.021065] CPU 03| <6> init -> ioqueues
[  0.021054] CPU 01| <6> init -> ioqueues
[  0.021067] CPU 02| <6> entering 'per-thread' init phase
[  0.021075] CPU 01| <6> init -> stack
[  0.021082] CPU 01| <6> init -> timer
[  0.021072] CPU 03| <6> init -> stack
[  0.021092] CPU 01| <6> init -> sched
[  0.021094] CPU 03| <6> init -> timer
[  0.021077] CPU 02| <6> init -> kthread
[  0.021107] CPU 03| <6> init -> sched
[  0.021121] CPU 02| <6> init -> ioqueues
[  0.021126] CPU 02| <6> init -> stack
[  0.021129] CPU 02| <6> init -> timer
[  0.021134] CPU 02| <6> init -> sched
[  0.021161] CPU 01| <6> init -> smalloc
[  0.021165] CPU 01| <6> init -> net
[  0.021167] CPU 01| <6> init -> directpath
[  0.021171] CPU 01| <6> init -> storage
[  0.021181] CPU 03| <6> init -> smalloc
[  0.021184] CPU 03| <6> init -> net
[  0.021187] CPU 03| <6> init -> directpath
[  0.021190] CPU 03| <6> init -> storage
[  0.021192] CPU 02| <6> init -> smalloc
[  0.021200] CPU 02| <6> init -> net
[  0.021203] CPU 02| <6> init -> directpath
[  0.021206] CPU 02| <6> init -> storage
[  0.021237] CPU 03| <6> entering 'late' init phase
[  0.021241] CPU 03| <6> init -> arp
[  0.021246] CPU 03| <6> init -> stat
[  0.021253] CPU 03| <6> init -> tcp
[  0.021257] CPU 03| <6> init -> rcu
[  0.021261] CPU 03| <6> init -> directpath

client.config file

host_addr 192.168.2.3
host_netmask 255.255.255.0
host_gateway 192.168.2.1
runtime_kthreads 4
runtime_spinning_kthreads 4
runtime_guaranteed_kthreads 4
runtime_priority lc

Client output:

/caladan/breakwater$ ./apps/netbench/netbench breakwater ../client.config client 100 192.168.2.2 10 exp 100 0 100000
CPU 06| <5> cpu: detected 8 cores, 1 nodes
CPU 06| <5> time: detected 3408 ticks / us
CPU 06| <6> init: entering 'early' init
CPU 06| <6> init: entering 'normal' init
CPU 06| <6> init: entering 'late' init
[  0.000290] CPU 06| <5> loading configuration from '../client.config'
[  0.000309] CPU 06| <5> process pid: 859656
[  0.000313] CPU 06| <6> entering 'global' init phase
[  0.000316] CPU 06| <6> init -> kthread
[  0.000323] CPU 06| <6> init -> ioqueues
[  0.013486] CPU 06| <6> init -> stack
[  0.013502] CPU 06| <6> init -> sched
[  0.013508] CPU 06| <6> init -> preempt
[  0.013512] CPU 06| <6> init -> smalloc
[  0.013520] CPU 06| <6> init -> net
[  0.014196] CPU 06| <5> net: started network stack
[  0.014200] CPU 06| <5> net: using the following configuration:
[  0.014204] CPU 06| <5>   addr:	192.168.2.3
[  0.014207] CPU 06| <5>   netmask:	255.255.255.0
[  0.014210] CPU 06| <5>   gateway:	192.168.2.1
[  0.014212] CPU 06| <5>   mac:	1E:C3:48:5A:08:5B
[  0.014216] CPU 06| <6> init -> directpath
[  0.014650] CPU 06| <6> init -> arp
[  0.014661] CPU 06| <6> init -> trans
[  0.014745] CPU 06| <6> init -> storage
[  0.014752] CPU 06| <6> init -> gc
[  0.014777] CPU 06| <5> thread: created thread 0
[  0.014780] CPU 06| <6> init: entering 'thread' init
[  0.014782] CPU 06| <6> entering 'per-thread' init phase
[  0.014788] CPU 06| <6> init -> kthread
[  0.014792] CPU 06| <6> init -> ioqueues
[  0.014796] CPU 06| <6> init -> stack
[  0.014798] CPU 06| <6> init -> timer
[  0.014804] CPU 06| <6> init -> sched
[  0.014844] CPU 06| <6> init -> smalloc
[  0.014847] CPU 06| <6> init -> net
[  0.014849] CPU 06| <6> init -> directpath
[  0.014850] CPU 06| <6> init -> storage
[  0.014852] CPU 06| <5> spawning 4 kthreads
[  0.014912] CPU 01| <5> thread: created thread 1
[  0.014921] CPU 01| <6> init: entering 'thread' init
[  0.014925] CPU 01| <6> entering 'per-thread' init phase
[  0.014930] CPU 01| <6> init -> kthread
[  0.014939] CPU 07| <5> thread: created thread 2
[  0.014947] CPU 07| <6> init: entering 'thread' init
[  0.014951] CPU 07| <6> entering 'per-thread' init phase
[  0.014955] CPU 01| <6> init -> ioqueues
[  0.014963] CPU 01| <6> init -> stack
[  0.014967] CPU 01| <6> init -> timer
[  0.014955] CPU 07| <6> init -> kthread
[  0.014957] CPU 06| <5> thread: created thread 3
[  0.014986] CPU 06| <6> init: entering 'thread' init
[  0.014990] CPU 06| <6> entering 'per-thread' init phase
[  0.014994] CPU 06| <6> init -> kthread
[  0.014973] CPU 01| <6> init -> sched
[  0.014994] CPU 07| <6> init -> ioqueues
[  0.015005] CPU 06| <6> init -> ioqueues
[  0.015019] CPU 06| <6> init -> stack
[  0.015024] CPU 06| <6> init -> timer
[  0.015013] CPU 07| <6> init -> stack
[  0.015035] CPU 01| <6> init -> smalloc
[  0.015037] CPU 07| <6> init -> timer
[  0.015040] CPU 01| <6> init -> net
[  0.015048] CPU 07| <6> init -> sched
[  0.015050] CPU 01| <6> init -> directpath
[  0.015059] CPU 01| <6> init -> storage
[  0.015031] CPU 06| <6> init -> sched
[  0.015116] CPU 06| <6> init -> smalloc
[  0.015119] CPU 06| <6> init -> net
[  0.015122] CPU 06| <6> init -> directpath
[  0.015124] CPU 06| <6> init -> storage
[  0.015124] CPU 07| <6> init -> smalloc
[  0.015135] CPU 07| <6> init -> net
[  0.015139] CPU 07| <6> init -> directpath
[  0.015143] CPU 07| <6> init -> storage
[  0.015177] CPU 06| <6> entering 'late' init phase
[  0.015180] CPU 06| <6> init -> arp
[  0.015185] CPU 06| <6> init -> stat
[  0.015191] CPU 06| <6> init -> tcp
[  0.015195] CPU 06| <6> init -> rcu
[  0.015199] CPU 06| <6> init -> directpath
num_threads,offered_load,throughput,goodput,cpu,min,mean,p50,p90,p99,p999,p9999,max,p1_win,mean_win,p99_win,p1_q,mean_q,p99_q,mean_stime,p99_stime,server:rx_pps,server:tx_pps,server:rx_bps,server:tx_bps,server:rx_drops_pps,server:rx_ooo_pps,server:winu_rx_pps,server:winu_tx_pps,server:win_tx_wps,server:req_rx_pps,server:req_drop_rate,server:resp_tx_pps,client:min_tput,client:max_tput,client:winu_rx_pps,client:winu_tx_pps,client:resp_rx_pps,client:req_tx_pps,client:win_expired_wps,client:req_dropped_rps
[  0.015883] CPU 01| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  0.015895] CPU 01| <6> tcp: 0x100000019000 CREATE -> SYN-SENT
[  0.325878] CPU 05| <6> tcp: 0x100000019000 retransmission timeout
[  0.325890] CPU 01| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  0.625914] CPU 05| <6> tcp: 0x100000019000 retransmission timeout
[  0.625924] CPU 01| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  0.925954] CPU 01| <6> tcp: 0x100000019000 retransmission timeout
[  0.925964] CPU 05| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  1.225988] CPU 02| <6> tcp: 0x100000019000 retransmission timeout
[  1.226002] CPU 06| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  1.526025] CPU 01| <6> tcp: 0x100000019000 retransmission timeout
[  1.526033] CPU 06| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  1.826060] CPU 01| <6> tcp: 0x100000019000 retransmission timeout
[  1.826074] CPU 02| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  2.126100] CPU 05| <6> tcp: 0x100000019000 retransmission timeout
[  2.126108] CPU 01| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  2.426134] CPU 01| <6> tcp: 0x100000019000 retransmission timeout
[  2.426141] CPU 05| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  2.726169] CPU 02| <6> tcp: 0x100000019000 retransmission timeout
[  2.726177] CPU 05| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  3.026204] CPU 06| <6> tcp: 0x100000019000 retransmission timeout
[  3.026212] CPU 05| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  3.326239] CPU 02| <6> tcp: 0x100000019000 retransmission timeout
[  3.326248] CPU 05| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  3.626277] CPU 05| <6> tcp: 0x100000019000 retransmission timeout
[  3.626284] CPU 01| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  3.926313] CPU 02| <6> tcp: 0x100000019000 retransmission timeout
[  3.926322] CPU 06| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  4.226350] CPU 02| <6> tcp: 0x100000019000 retransmission timeout
[  4.226359] CPU 06| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  4.526388] CPU 06| <6> tcp: 0x100000019000 retransmission timeout
[  4.526395] CPU 01| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  4.826423] CPU 01| <6> tcp: 0x100000019000 retransmission timeout
[  4.826430] CPU 06| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[  5.016447] CPU 02| <6> tcp: 0x100000019000 SYN-SENT -> CLOSED
[  5.016453] CPU 06| <0> couldn't connect to raddr.
[  5.016458] CPU 06| <5> init: shutting down -> FAILURE

machine stuck at cpuidle_pause_and_lock

I'm trying to deploy Caladan, however my testbed stuck completely at cpuidle_pause_and_lock, and nothing can be done to rescue.
CPU:Intel(R) Xeon(R) Gold 6240M CPU @ 2.60GHz
kernel: Linux-kernel 5.2
System : Ubuntu18.04

cannot send udp packets

I'm trying to complie and run Caladan with an easy udp-send application. The application is modified from apps/bench/netbench_udp.cc by just calling udp_send() in the RunExperiment() function. I run the app in server A, and run tcpdump in the server B. However, server B cannot capture any packet from server A.
Any idea on how to resolve the problem? Thank you for your time and effort.

Environment
CPU: Intel(R) Xeon(R) CPU E5-2620
NIC: Intel XL710
Ubuntu 20.04.1 with kernel 5.15.0
The NIC binds to vfio-pci.
The two NICs in the two servers are connected directly.

More details
I have read the packet from rte_mbuf in tx_burst, the packet is valid.
The eth_rte_tx_burst return a right value.
I write another app with pure dpdk. I construct an identical packet, and send it by the rte_eth_tx_burst. Server B can receive the packet.

The configuration file

host_addr 192.168.40.233
host_netmask 255.255.255.0
host_gateway 192.168.40.1
host_mac 3c:fd:fe:9e:81:98
runtime_kthreads 4
runtime_guaranteed_kthreads 4
runtime_spinning_kthreads 4
runtime_priority lc
runtime_qdelay_us 10

the output

$ sudo ./iokerneld simple noht
CPU 03| <5> cpu: detected 12 cores, 2 nodes
CPU 03| <5> time: detected 2397 ticks / us
[  0.003655] CPU 03| <5> sched: CPU configuration...
        node 0: [0][2][4][6][8][10]
        node 1: [1][3][5][7][9][11]
[  0.003687] CPU 03| <5> sched: dataplane on 2, control on 0

=====  Processor information  =====
Linux arch_perfmon flag  : yes
Hybrid processor         : no
IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : no
Max CPUID level          : 15
IBRS enabled in the kernel   : no
STIBP enabled in the kernel  : no
Socket 0: 2 memory controllers detected with total number of 5 channels. 2 QPI ports detected. 0 M2M (mesh to memory) blocks detected. 1 Home Agents detected. 0 M3UPI blocks detected.
Socket 1: 2 memory controllers detected with total number of 5 channels. 2 QPI ports detected. 0 M2M (mesh to memory) blocks detected. 1 Home Agents detected. 0 M3UPI blocks detected.
[  0.189100] CPU 00| <5> control: spawning control thread
EAL: Detected CPU lcores: 12
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_i40e (8086:1583) device: 0000:06:00.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
[  0.796731] CPU 02| <5> dpdk: driver: net_i40e port 0 MAC: 3c fd fe 9e 81 98
[  0.797045] CPU 02| <5> main: core 2 running dataplane. [Ctrl+C to quit]
[  4.456723] CPU 02| <3> tx: transmitted 1 packets on port 0
[  5.456809] CPU 02| <3> tx: transmitted 1 packets on port 0
[  6.456958] CPU 02| <3> tx: transmitted 1 packets on port 0
[  7.457047] CPU 02| <3> tx: transmitted 1 packets on port 0
...

Dropped support for ConnectX-3

Hi, is there a reason for dropping support for ConnectX-3? e2846c0

Setting CONFIG_MLX4=y in build/config, cause the build error:

build/shared.mk:56: *** mlx4 support is not available currently.  Stop.

I'm not able to run Caladan on m510 Cloudlab machines anymore. m510 machines uses a Dual-port Mellanox ConnectX-3 10 GB NIC (https://docs.cloudlab.us/hardware.html)

Are we now required to use machines with Mellanox ConnectX-4, and set CONFIG_MLX5=y? Or is there a way to build Caladan using Mellanox ConnectX-3?

Thank you!

cannot find macro `llvm_asm`

Hi,

Thank you for your clear instructions in compiling Caladan! I'm trying to compile and run Caladan on the XL170 machines in CloudLab (running Ubuntu 18.04.6 LTS, kernel 4.15.0-191-generic) . All instructions worked out fine until the very last step of building the synthetic client-server application.

Specifically, cargo build --release fails to compile Shenango. Any idea on how to resolve this? Below is the output

I'd greatly appreciate your help!

Best regards,
Hamed

   Compiling nom v3.2.1
   Compiling mersenne_twister v1.1.1
   Compiling cexpr v0.2.3
   Compiling env_logger v0.4.3
   Compiling shenango v0.1.0 (/users/hameds/orig/caladan/bindings/rust)
error: cannot find macro `llvm_asm` in this scope
  --> /users/hameds/orig/caladan/bindings/rust/src/asm.rs:21:14
   |
21 |     unsafe { llvm_asm!("rdtscp" : "={eax}"(a), "={edx}"(d), "={ecx}"(c) : : : "volatile") };
   |              ^^^^^^^^

error: cannot find macro `llvm_asm` in this scope
  --> /users/hameds/orig/caladan/bindings/rust/src/asm.rs:13:14
   |
13 |     unsafe { llvm_asm!("rdtsc" : "={eax}"(a), "={edx}"(d) : : : "volatile" ) };
   |              ^^^^^^^^

error: cannot find macro `llvm_asm` in this scope
 --> /users/hameds/orig/caladan/bindings/rust/src/asm.rs:7:14
  |
7 |     unsafe { llvm_asm!("cpuid" : : : "rax", "rbx", "rcx", "rdx": "volatile") }
  |              ^^^^^^^^

error: cannot find macro `llvm_asm` in this scope
 --> /users/hameds/orig/caladan/bindings/rust/src/asm.rs:3:14
  |
3 |     unsafe { llvm_asm!("pause" :::: "volatile") }
  |              ^^^^^^^^

error: cannot find macro `llvm_asm` in this scope
  --> /users/hameds/orig/caladan/bindings/rust/src/lib.rs:55:9
   |
55 |         llvm_asm!("" ::: "memory" : "volatile");
   |         ^^^^^^^^

error: cannot find macro `llvm_asm` in this scope
  --> /users/hameds/orig/caladan/bindings/rust/src/lib.rs:54:9
   |
54 |         llvm_asm!("addl $$1, %fs:preempt_cnt@tpoff" : : : "memory", "cc" : "volatile");
   |         ^^^^^^^^

error: cannot find macro `llvm_asm` in this scope
  --> /users/hameds/orig/caladan/bindings/rust/src/lib.rs:44:9
   |
44 |         llvm_asm!("subl $$1, %fs:preempt_cnt@tpoff" : : : "memory", "cc" : "volatile");
   |         ^^^^^^^^

error: cannot find macro `llvm_asm` in this scope
  --> /users/hameds/orig/caladan/bindings/rust/src/lib.rs:43:9
   |
43 |         llvm_asm!("" ::: "memory" : "volatile");
   |         ^^^^^^^^

error[E0635]: unknown feature `llvm_asm`
 --> /users/hameds/orig/caladan/bindings/rust/src/lib.rs:4:12
  |
4 | #![feature(llvm_asm)]
  |            ^^^^^^^^

For more information about this error, try `rustc --explain E0635`.
error: could not compile `shenango` due to 9 previous errors