shenango / caladan Goto Github PK
View Code? Open in Web Editor NEWInterference-aware CPU scheduling that enables performance isolation and high CPU utilization for datacenter servers
License: Apache License 2.0
Interference-aware CPU scheduling that enables performance isolation and high CPU utilization for datacenter servers
License: Apache License 2.0
Hi Calanda, Breakwater developers,
I try to run the Breakwater example application in readme but encounter problems. (Before that I do not encounter the same problem when I run the synthetic application in Calanda readme).
The Breakwater example application does run successfully when I run it the first time. But when I want to run it for many times, (each time, I resetup the machine using sudo ./scripts/setup_machine.sh, then start iokerneld and the application at the client. so does at the server) I can never run it successfully again unless I reboot the machine. The IP-related configuration is right since I can run the synthetic application in Calanda successfully.
host_addr 192.168.2.2
host_netmask 255.255.255.0
host_gateway 192.168.2.1
runtime_kthreads 4
runtime_guaranteed_kthreads 4
I omit the output of sudo ./iokerneld ias nobw no_hw_qdel. so does client.
~/caladan/breakwater: sudo ./apps/netbench/netbench breakwater ../server.config server
CPU 02| <5> cpu: detected 8 cores, 1 nodes
CPU 02| <5> time: detected 3407 ticks / us
CPU 02| <6> init: entering 'early' init
CPU 02| <6> init: entering 'normal' init
CPU 02| <6> init: entering 'late' init
[ 0.000296] CPU 02| <5> loading configuration from '../server.config'
[ 0.006781] CPU 02| <5> process pid: 151042
[ 0.006786] CPU 02| <6> entering 'global' init phase
[ 0.006788] CPU 02| <6> init -> kthread
[ 0.006798] CPU 02| <6> init -> ioqueues
[ 0.019990] CPU 02| <6> init -> stack
[ 0.020006] CPU 02| <6> init -> sched
[ 0.020011] CPU 02| <6> init -> preempt
[ 0.020017] CPU 02| <6> init -> smalloc
[ 0.020025] CPU 02| <6> init -> net
[ 0.020699] CPU 02| <5> net: started network stack
[ 0.020702] CPU 02| <5> net: using the following configuration:
[ 0.020706] CPU 02| <5> addr: 192.168.2.2
[ 0.020710] CPU 02| <5> netmask: 255.255.255.0
[ 0.020713] CPU 02| <5> gateway: 192.168.2.1
[ 0.020716] CPU 02| <5> mac: EA:82:2E:DD:AE:1E
[ 0.020720] CPU 02| <6> init -> directpath
[ 0.020723] CPU 02| <6> init -> arp
[ 0.020734] CPU 02| <6> init -> trans
[ 0.020825] CPU 02| <6> init -> storage
[ 0.020828] CPU 02| <6> init -> gc
[ 0.020854] CPU 02| <5> thread: created thread 0
[ 0.020857] CPU 02| <6> init: entering 'thread' init
[ 0.020860] CPU 02| <6> entering 'per-thread' init phase
[ 0.020863] CPU 02| <6> init -> kthread
[ 0.020867] CPU 02| <6> init -> ioqueues
[ 0.020870] CPU 02| <6> init -> stack
[ 0.020873] CPU 02| <6> init -> timer
[ 0.020882] CPU 02| <6> init -> sched
[ 0.020922] CPU 02| <6> init -> smalloc
[ 0.020925] CPU 02| <6> init -> net
[ 0.020928] CPU 02| <6> init -> directpath
[ 0.020931] CPU 02| <6> init -> storage
[ 0.020933] CPU 02| <5> spawning 4 kthreads
[ 0.021004] CPU 01| <5> thread: created thread 1
[ 0.021014] CPU 01| <6> init: entering 'thread' init
[ 0.021018] CPU 01| <6> entering 'per-thread' init phase
[ 0.021030] CPU 01| <6> init -> kthread
[ 0.021022] CPU 03| <5> thread: created thread 2
[ 0.021042] CPU 03| <6> init: entering 'thread' init
[ 0.021046] CPU 03| <6> entering 'per-thread' init phase
[ 0.021050] CPU 03| <6> init -> kthread
[ 0.021052] CPU 02| <5> thread: created thread 3
[ 0.021061] CPU 02| <6> init: entering 'thread' init
[ 0.021065] CPU 03| <6> init -> ioqueues
[ 0.021054] CPU 01| <6> init -> ioqueues
[ 0.021067] CPU 02| <6> entering 'per-thread' init phase
[ 0.021075] CPU 01| <6> init -> stack
[ 0.021082] CPU 01| <6> init -> timer
[ 0.021072] CPU 03| <6> init -> stack
[ 0.021092] CPU 01| <6> init -> sched
[ 0.021094] CPU 03| <6> init -> timer
[ 0.021077] CPU 02| <6> init -> kthread
[ 0.021107] CPU 03| <6> init -> sched
[ 0.021121] CPU 02| <6> init -> ioqueues
[ 0.021126] CPU 02| <6> init -> stack
[ 0.021129] CPU 02| <6> init -> timer
[ 0.021134] CPU 02| <6> init -> sched
[ 0.021161] CPU 01| <6> init -> smalloc
[ 0.021165] CPU 01| <6> init -> net
[ 0.021167] CPU 01| <6> init -> directpath
[ 0.021171] CPU 01| <6> init -> storage
[ 0.021181] CPU 03| <6> init -> smalloc
[ 0.021184] CPU 03| <6> init -> net
[ 0.021187] CPU 03| <6> init -> directpath
[ 0.021190] CPU 03| <6> init -> storage
[ 0.021192] CPU 02| <6> init -> smalloc
[ 0.021200] CPU 02| <6> init -> net
[ 0.021203] CPU 02| <6> init -> directpath
[ 0.021206] CPU 02| <6> init -> storage
[ 0.021237] CPU 03| <6> entering 'late' init phase
[ 0.021241] CPU 03| <6> init -> arp
[ 0.021246] CPU 03| <6> init -> stat
[ 0.021253] CPU 03| <6> init -> tcp
[ 0.021257] CPU 03| <6> init -> rcu
[ 0.021261] CPU 03| <6> init -> directpath
host_addr 192.168.2.3
host_netmask 255.255.255.0
host_gateway 192.168.2.1
runtime_kthreads 4
runtime_spinning_kthreads 4
runtime_guaranteed_kthreads 4
runtime_priority lc
/caladan/breakwater$ ./apps/netbench/netbench breakwater ../client.config client 100 192.168.2.2 10 exp 100 0 100000
CPU 06| <5> cpu: detected 8 cores, 1 nodes
CPU 06| <5> time: detected 3408 ticks / us
CPU 06| <6> init: entering 'early' init
CPU 06| <6> init: entering 'normal' init
CPU 06| <6> init: entering 'late' init
[ 0.000290] CPU 06| <5> loading configuration from '../client.config'
[ 0.000309] CPU 06| <5> process pid: 859656
[ 0.000313] CPU 06| <6> entering 'global' init phase
[ 0.000316] CPU 06| <6> init -> kthread
[ 0.000323] CPU 06| <6> init -> ioqueues
[ 0.013486] CPU 06| <6> init -> stack
[ 0.013502] CPU 06| <6> init -> sched
[ 0.013508] CPU 06| <6> init -> preempt
[ 0.013512] CPU 06| <6> init -> smalloc
[ 0.013520] CPU 06| <6> init -> net
[ 0.014196] CPU 06| <5> net: started network stack
[ 0.014200] CPU 06| <5> net: using the following configuration:
[ 0.014204] CPU 06| <5> addr: 192.168.2.3
[ 0.014207] CPU 06| <5> netmask: 255.255.255.0
[ 0.014210] CPU 06| <5> gateway: 192.168.2.1
[ 0.014212] CPU 06| <5> mac: 1E:C3:48:5A:08:5B
[ 0.014216] CPU 06| <6> init -> directpath
[ 0.014650] CPU 06| <6> init -> arp
[ 0.014661] CPU 06| <6> init -> trans
[ 0.014745] CPU 06| <6> init -> storage
[ 0.014752] CPU 06| <6> init -> gc
[ 0.014777] CPU 06| <5> thread: created thread 0
[ 0.014780] CPU 06| <6> init: entering 'thread' init
[ 0.014782] CPU 06| <6> entering 'per-thread' init phase
[ 0.014788] CPU 06| <6> init -> kthread
[ 0.014792] CPU 06| <6> init -> ioqueues
[ 0.014796] CPU 06| <6> init -> stack
[ 0.014798] CPU 06| <6> init -> timer
[ 0.014804] CPU 06| <6> init -> sched
[ 0.014844] CPU 06| <6> init -> smalloc
[ 0.014847] CPU 06| <6> init -> net
[ 0.014849] CPU 06| <6> init -> directpath
[ 0.014850] CPU 06| <6> init -> storage
[ 0.014852] CPU 06| <5> spawning 4 kthreads
[ 0.014912] CPU 01| <5> thread: created thread 1
[ 0.014921] CPU 01| <6> init: entering 'thread' init
[ 0.014925] CPU 01| <6> entering 'per-thread' init phase
[ 0.014930] CPU 01| <6> init -> kthread
[ 0.014939] CPU 07| <5> thread: created thread 2
[ 0.014947] CPU 07| <6> init: entering 'thread' init
[ 0.014951] CPU 07| <6> entering 'per-thread' init phase
[ 0.014955] CPU 01| <6> init -> ioqueues
[ 0.014963] CPU 01| <6> init -> stack
[ 0.014967] CPU 01| <6> init -> timer
[ 0.014955] CPU 07| <6> init -> kthread
[ 0.014957] CPU 06| <5> thread: created thread 3
[ 0.014986] CPU 06| <6> init: entering 'thread' init
[ 0.014990] CPU 06| <6> entering 'per-thread' init phase
[ 0.014994] CPU 06| <6> init -> kthread
[ 0.014973] CPU 01| <6> init -> sched
[ 0.014994] CPU 07| <6> init -> ioqueues
[ 0.015005] CPU 06| <6> init -> ioqueues
[ 0.015019] CPU 06| <6> init -> stack
[ 0.015024] CPU 06| <6> init -> timer
[ 0.015013] CPU 07| <6> init -> stack
[ 0.015035] CPU 01| <6> init -> smalloc
[ 0.015037] CPU 07| <6> init -> timer
[ 0.015040] CPU 01| <6> init -> net
[ 0.015048] CPU 07| <6> init -> sched
[ 0.015050] CPU 01| <6> init -> directpath
[ 0.015059] CPU 01| <6> init -> storage
[ 0.015031] CPU 06| <6> init -> sched
[ 0.015116] CPU 06| <6> init -> smalloc
[ 0.015119] CPU 06| <6> init -> net
[ 0.015122] CPU 06| <6> init -> directpath
[ 0.015124] CPU 06| <6> init -> storage
[ 0.015124] CPU 07| <6> init -> smalloc
[ 0.015135] CPU 07| <6> init -> net
[ 0.015139] CPU 07| <6> init -> directpath
[ 0.015143] CPU 07| <6> init -> storage
[ 0.015177] CPU 06| <6> entering 'late' init phase
[ 0.015180] CPU 06| <6> init -> arp
[ 0.015185] CPU 06| <6> init -> stat
[ 0.015191] CPU 06| <6> init -> tcp
[ 0.015195] CPU 06| <6> init -> rcu
[ 0.015199] CPU 06| <6> init -> directpath
num_threads,offered_load,throughput,goodput,cpu,min,mean,p50,p90,p99,p999,p9999,max,p1_win,mean_win,p99_win,p1_q,mean_q,p99_q,mean_stime,p99_stime,server:rx_pps,server:tx_pps,server:rx_bps,server:tx_bps,server:rx_drops_pps,server:rx_ooo_pps,server:winu_rx_pps,server:winu_tx_pps,server:win_tx_wps,server:req_rx_pps,server:req_drop_rate,server:resp_tx_pps,client:min_tput,client:max_tput,client:winu_rx_pps,client:winu_tx_pps,client:resp_rx_pps,client:req_tx_pps,client:win_expired_wps,client:req_dropped_rps
[ 0.015883] CPU 01| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 0.015895] CPU 01| <6> tcp: 0x100000019000 CREATE -> SYN-SENT
[ 0.325878] CPU 05| <6> tcp: 0x100000019000 retransmission timeout
[ 0.325890] CPU 01| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 0.625914] CPU 05| <6> tcp: 0x100000019000 retransmission timeout
[ 0.625924] CPU 01| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 0.925954] CPU 01| <6> tcp: 0x100000019000 retransmission timeout
[ 0.925964] CPU 05| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 1.225988] CPU 02| <6> tcp: 0x100000019000 retransmission timeout
[ 1.226002] CPU 06| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 1.526025] CPU 01| <6> tcp: 0x100000019000 retransmission timeout
[ 1.526033] CPU 06| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 1.826060] CPU 01| <6> tcp: 0x100000019000 retransmission timeout
[ 1.826074] CPU 02| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 2.126100] CPU 05| <6> tcp: 0x100000019000 retransmission timeout
[ 2.126108] CPU 01| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 2.426134] CPU 01| <6> tcp: 0x100000019000 retransmission timeout
[ 2.426141] CPU 05| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 2.726169] CPU 02| <6> tcp: 0x100000019000 retransmission timeout
[ 2.726177] CPU 05| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 3.026204] CPU 06| <6> tcp: 0x100000019000 retransmission timeout
[ 3.026212] CPU 05| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 3.326239] CPU 02| <6> tcp: 0x100000019000 retransmission timeout
[ 3.326248] CPU 05| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 3.626277] CPU 05| <6> tcp: 0x100000019000 retransmission timeout
[ 3.626284] CPU 01| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 3.926313] CPU 02| <6> tcp: 0x100000019000 retransmission timeout
[ 3.926322] CPU 06| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 4.226350] CPU 02| <6> tcp: 0x100000019000 retransmission timeout
[ 4.226359] CPU 06| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 4.526388] CPU 06| <6> tcp: 0x100000019000 retransmission timeout
[ 4.526395] CPU 01| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 4.826423] CPU 01| <6> tcp: 0x100000019000 retransmission timeout
[ 4.826430] CPU 06| <6> tcp: 0x100000019000 192.168.2.3:56120 -> 192.168.2.2:8123 FLAGS=SYN SEQ=ISS+0 ACK=IRS+0 WND=64240 LEN=0
[ 5.016447] CPU 02| <6> tcp: 0x100000019000 SYN-SENT -> CLOSED
[ 5.016453] CPU 06| <0> couldn't connect to raddr.
[ 5.016458] CPU 06| <5> init: shutting down -> FAILURE
Hi,
Thank you for your clear instructions in compiling Caladan! I'm trying to compile and run Caladan on the XL170 machines in CloudLab (running Ubuntu 18.04.6 LTS, kernel 4.15.0-191-generic) . All instructions worked out fine until the very last step of building the synthetic client-server application.
Specifically, cargo build --release
fails to compile Shenango. Any idea on how to resolve this? Below is the output
I'd greatly appreciate your help!
Best regards,
Hamed
Compiling nom v3.2.1
Compiling mersenne_twister v1.1.1
Compiling cexpr v0.2.3
Compiling env_logger v0.4.3
Compiling shenango v0.1.0 (/users/hameds/orig/caladan/bindings/rust)
error: cannot find macro `llvm_asm` in this scope
--> /users/hameds/orig/caladan/bindings/rust/src/asm.rs:21:14
|
21 | unsafe { llvm_asm!("rdtscp" : "={eax}"(a), "={edx}"(d), "={ecx}"(c) : : : "volatile") };
| ^^^^^^^^
error: cannot find macro `llvm_asm` in this scope
--> /users/hameds/orig/caladan/bindings/rust/src/asm.rs:13:14
|
13 | unsafe { llvm_asm!("rdtsc" : "={eax}"(a), "={edx}"(d) : : : "volatile" ) };
| ^^^^^^^^
error: cannot find macro `llvm_asm` in this scope
--> /users/hameds/orig/caladan/bindings/rust/src/asm.rs:7:14
|
7 | unsafe { llvm_asm!("cpuid" : : : "rax", "rbx", "rcx", "rdx": "volatile") }
| ^^^^^^^^
error: cannot find macro `llvm_asm` in this scope
--> /users/hameds/orig/caladan/bindings/rust/src/asm.rs:3:14
|
3 | unsafe { llvm_asm!("pause" :::: "volatile") }
| ^^^^^^^^
error: cannot find macro `llvm_asm` in this scope
--> /users/hameds/orig/caladan/bindings/rust/src/lib.rs:55:9
|
55 | llvm_asm!("" ::: "memory" : "volatile");
| ^^^^^^^^
error: cannot find macro `llvm_asm` in this scope
--> /users/hameds/orig/caladan/bindings/rust/src/lib.rs:54:9
|
54 | llvm_asm!("addl $$1, %fs:preempt_cnt@tpoff" : : : "memory", "cc" : "volatile");
| ^^^^^^^^
error: cannot find macro `llvm_asm` in this scope
--> /users/hameds/orig/caladan/bindings/rust/src/lib.rs:44:9
|
44 | llvm_asm!("subl $$1, %fs:preempt_cnt@tpoff" : : : "memory", "cc" : "volatile");
| ^^^^^^^^
error: cannot find macro `llvm_asm` in this scope
--> /users/hameds/orig/caladan/bindings/rust/src/lib.rs:43:9
|
43 | llvm_asm!("" ::: "memory" : "volatile");
| ^^^^^^^^
error[E0635]: unknown feature `llvm_asm`
--> /users/hameds/orig/caladan/bindings/rust/src/lib.rs:4:12
|
4 | #![feature(llvm_asm)]
| ^^^^^^^^
For more information about this error, try `rustc --explain E0635`.
error: could not compile `shenango` due to 9 previous errors
Hi Caladan authors!
I just tried to print out each uthread's stack address and find they are generally 0x7ffff... However, based on the variable
of STACK_BASE_ADDR, stack address should start from 0x2000... Then I find this line actually calls atomic64_fetch_and_and() instead of atomic64_fetch_and_add(). So it seems this is a small typo?
Just want to let you know about this potential bug, although the runtime works perfectly without fixing it!
Best,
Yang
Debug enabled, other options disabled, following all the instructions mentioned in github readme.
sudo gdb iokerneld
(gdb) r
Starting program: /home/user/caladan/iokerneld
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
CPU 04| <6> entering 'iokernel' init phase
CPU 04| <6> init -> base
CPU 04| <5> cpu: detected 8 cores, 1 nodes
CPU 04| <5> time: detected 3407 ticks / us
CPU 04| <6> init: entering 'early' init
CPU 04| <6> init: entering 'normal' init
CPU 04| <6> init: entering 'late' init
[ 0.000301] CPU 04| <6> init -> ksched
[ 0.000325] CPU 04| <6> init -> sched
[ 0.000329] CPU 04| <5> sched: CPU configuration...
node 0: [0,4][1,5][2,6][3,7]
[ 0.000341] CPU 04| <5> sched: dataplane on 4, control on 0
[ 0.000346] CPU 04| <6> init -> simple
[ 0.000358] CPU 04| <6> init -> numa
[ 0.000365] CPU 04| <6> init -> ias
IBRS and IBPB supported : yes
STIBP supported : yes
Spec arch caps supported : no
IBRS enabled in the kernel : no
STIBP enabled in the kernel : no
[New Thread 0x7ffff7045700 (LWP 134361)]
[New Thread 0x7ffff6844700 (LWP 134362)]
[New Thread 0x7ffff6043700 (LWP 134363)]
terminate called after throwing an instance of 'std::out_of_range'
what(): vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)
Thread 1 "iokerneld" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff7a71859 in __GI_abort () at abort.c:79
#2 0x00007ffff7d2f951 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007ffff7d3b47c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff7d3b4e7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff7d3b799 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff7d323eb in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00005555556b2ec9 in std::vector<std::shared_ptr<pcm::ServerPCICFGUncore>, std::allocator<std::shared_ptr<pcm::ServerPCICFGUncore> > >::_M_range_check (__n=0, this=<optimized out>)
at /usr/include/c++/9/ext/atomicity.h:69
#8 std::vector<std::shared_ptr<pcm::ServerPCICFGUncore>, std::allocator<std::shared_ptr<pcm::ServerPCICFGUncore> > >::at (__n=0, this=<optimized out>)
at /usr/include/c++/9/bits/stl_vector.h:1091
#9 pcm_caladan_init (socket=0) at pcm-caladan.cpp:28
#10 0x000055555560050f in ias_bw_init () at iokernel/ias_bw.c:260
#11 0x0000555555601a38 in ias_init () at iokernel/ias.c:555
#12 0x00005555555eb4d2 in run_init_handlers (phase=0x5555556f286b "iokernel", h=0x555555770aa0 <iok_init_handlers>, nr=13) at iokernel/main.c:65
#13 0x00005555555eb968 in main (argc=1, argv=0x7fffffffe6a8) at iokernel/main.c:190
(gdb)
#include "cpucounters.h"
#include <cstdio>
int main(void) {
printf("1\n");
auto a = pcm::PCM::getInstance();
printf("2\n");
a->server_pcicfg_uncore.at(0); //throws std::out_of_range
//a->server_pcicfg_uncore[0];
printf("3\n");
}
Compile this test with g++ test.cpp -I caladan/deps/pcm/ -L caladan/deps/pcm -o test-pcm -lPCM -lpthread
.
Run sudo ./test-pcm
and here is the output:
1
IBRS and IBPB supported : yes
STIBP supported : yes
Spec arch caps supported : no
IBRS enabled in the kernel : no
STIBP enabled in the kernel : no
2
terminate called after throwing an instance of 'std::out_of_range'
what(): vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)
Aborted
Hi, is there a reason for dropping support for ConnectX-3? e2846c0
Setting CONFIG_MLX4=y
in build/config
, cause the build error:
build/shared.mk:56: *** mlx4 support is not available currently. Stop.
I'm not able to run Caladan on m510 Cloudlab machines anymore. m510 machines uses a Dual-port Mellanox ConnectX-3 10 GB NIC (https://docs.cloudlab.us/hardware.html)
Are we now required to use machines with Mellanox ConnectX-4, and set CONFIG_MLX5=y
? Or is there a way to build Caladan using Mellanox ConnectX-3?
Thank you!
I'm trying to complie and run Caladan with an easy udp-send application. The application is modified from apps/bench/netbench_udp.cc by just calling udp_send()
in the RunExperiment()
function. I run the app in server A, and run tcpdump in the server B. However, server B cannot capture any packet from server A.
Any idea on how to resolve the problem? Thank you for your time and effort.
Environment
CPU: Intel(R) Xeon(R) CPU E5-2620
NIC: Intel XL710
Ubuntu 20.04.1 with kernel 5.15.0
The NIC binds to vfio-pci.
The two NICs in the two servers are connected directly.
More details
I have read the packet from rte_mbuf in tx_burst
, the packet is valid.
The eth_rte_tx_burst
return a right value.
I write another app with pure dpdk. I construct an identical packet, and send it by the rte_eth_tx_burst
. Server B can receive the packet.
The configuration file
host_addr 192.168.40.233
host_netmask 255.255.255.0
host_gateway 192.168.40.1
host_mac 3c:fd:fe:9e:81:98
runtime_kthreads 4
runtime_guaranteed_kthreads 4
runtime_spinning_kthreads 4
runtime_priority lc
runtime_qdelay_us 10
the output
$ sudo ./iokerneld simple noht
CPU 03| <5> cpu: detected 12 cores, 2 nodes
CPU 03| <5> time: detected 2397 ticks / us
[ 0.003655] CPU 03| <5> sched: CPU configuration...
node 0: [0][2][4][6][8][10]
node 1: [1][3][5][7][9][11]
[ 0.003687] CPU 03| <5> sched: dataplane on 2, control on 0
===== Processor information =====
Linux arch_perfmon flag : yes
Hybrid processor : no
IBRS and IBPB supported : yes
STIBP supported : yes
Spec arch caps supported : no
Max CPUID level : 15
IBRS enabled in the kernel : no
STIBP enabled in the kernel : no
Socket 0: 2 memory controllers detected with total number of 5 channels. 2 QPI ports detected. 0 M2M (mesh to memory) blocks detected. 1 Home Agents detected. 0 M3UPI blocks detected.
Socket 1: 2 memory controllers detected with total number of 5 channels. 2 QPI ports detected. 0 M2M (mesh to memory) blocks detected. 1 Home Agents detected. 0 M3UPI blocks detected.
[ 0.189100] CPU 00| <5> control: spawning control thread
EAL: Detected CPU lcores: 12
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_i40e (8086:1583) device: 0000:06:00.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
[ 0.796731] CPU 02| <5> dpdk: driver: net_i40e port 0 MAC: 3c fd fe 9e 81 98
[ 0.797045] CPU 02| <5> main: core 2 running dataplane. [Ctrl+C to quit]
[ 4.456723] CPU 02| <3> tx: transmitted 1 packets on port 0
[ 5.456809] CPU 02| <3> tx: transmitted 1 packets on port 0
[ 6.456958] CPU 02| <3> tx: transmitted 1 packets on port 0
[ 7.457047] CPU 02| <3> tx: transmitted 1 packets on port 0
...
Hi Caladan developers,
I am running into some issues when trying to run Caladan.
Following instructions in Readme.txt. Intel NIC binds to the IGB UIO module.
/caladan$ sudo ./iokerneld ias nobw
CPU 03| <5> cpu: detected 8 cores, 1 nodes
CPU 03| <5> time: detected 3407 ticks / us
[ 0.000604] CPU 06| <5> sched: CPU configuration...
node 0: [0,4][1,5][2,6][3,7]
[ 0.000618] CPU 06| <5> sched: dataplane on 4, control on 0
[ 0.026331] CPU 06| <5> control: spawning control thread
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:00:1f.6 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:15b7 net_e1000_em
EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:01:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: WARNING! Base virtual address hint (0x1100805000 != 0x7f14f0e3b000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
[ 3.138086] CPU 04| <5> dpdk: driver: net_ixgbe port 0 MAC: 00 1b 21 88 d2 94
[ 3.138098] CPU 04| <5> main: core 4 running dataplane. [Ctrl+C to quit]
/caladan$ ./apps/synthetic/target/release/synthetic 192.168.2.2:5000 --config server.config --mode spawner-server
CPU 06| <5> cpu: detected 8 cores, 1 nodes
CPU 06| <5> time: detected 3407 ticks / us
[ 0.000229] CPU 06| <5> loading configuration from 'server.config'
[ 0.000256] CPU 06| <5> process pid: 542683
[ 0.013920] CPU 06| <5> net: started network stack
[ 0.013935] CPU 06| <5> net: using the following configuration:
[ 0.013938] CPU 06| <5> addr: 192.168.2.2
[ 0.013942] CPU 06| <5> netmask: 255.255.255.0
[ 0.013945] CPU 06| <5> gateway: 192.168.2.1
[ 0.013949] CPU 06| <5> mac: 0E:47:46:C3:F3:AD
[ 0.014025] CPU 06| <5> thread: created thread 0
[ 0.014070] CPU 06| <5> spawning 4 kthreads
[ 0.014147] CPU 03| <5> thread: created thread 1
[ 0.014163] CPU 02| <5> thread: created thread 2
[ 0.014185] CPU 05| <5> thread: created thread 3
192.168.2.2:5000
~/caladan$ sudo ./iokerneld ias nobw
CPU 02| <5> cpu: detected 8 cores, 1 nodes
CPU 02| <5> time: detected 3408 ticks / us
[ 0.000274] CPU 02| <5> sched: CPU configuration...
node 0: [0,4][1,5][2,6][3,7]
[ 0.000287] CPU 02| <5> sched: dataplane on 4, control on 0
[ 0.025720] CPU 02| <5> control: spawning control thread
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:00:1f.6 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:15b7 net_e1000_em
EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:01:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: WARNING! Base virtual address hint (0x1100805000 != 0x7fe473b0a000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
[ 1.980692] CPU 04| <5> dpdk: driver: net_ixgbe port 0 MAC: 00 1b 21 89 6e ac
[ 1.980703] CPU 04| <5> main: core 4 running dataplane. [Ctrl+C to quit]
[258.711498] CPU 04| <2> dp_clients: failed to attach proc.
~/caladan$ ./apps/synthetic/target/release/synthetic 192.168.2.2:5000 --config client.config --mode runtime-client
CPU 03| <5> cpu: detected 8 cores, 1 nodes
CPU 03| <5> time: detected 3408 ticks / us
[ 0.000281] CPU 03| <5> loading configuration from 'client.config'
[ 0.000311] CPU 03| <5> process pid: 755087
[ 0.004069] CPU 03| <5> net: started network stack
[ 0.004085] CPU 03| <5> net: using the following configuration:
[ 0.004088] CPU 03| <5> addr: 192.168.2.3
[ 0.004104] CPU 03| <5> netmask: 255.255.255.0
[ 0.004109] CPU 03| <5> gateway: 192.168.2.1
[ 0.004113] CPU 03| <5> mac: BE:E6:DD:BF:9C:EA
[ 0.004195] CPU 03| <5> thread: created thread 0
[ 0.004240] CPU 03| <5> spawning 4 kthreads
[ 0.004314] CPU 06| <5> thread: created thread 1
[ 0.004333] CPU 07| <5> thread: created thread 2
[ 0.004408] CPU 03| <5> thread: created thread 3
Hi all,
I find the synthetic
in Caladan
endures a high Never-Send rate (above 1%) when clients issue requests with a relatively high rate which is close to server’s capacity.
This is especially problematic under a Poisson distribution: when two adjacent requests are generated within a short time window (i.e., a bursty period), the latter one is more likely to be droped due to the Never-Send logic (see code). We have profiled Caladan's client logic, and find that the scheduling often causes the request to be delayed (which has already violated the Poisson distribution) and finally be dropped.
We further designed an experiment to confirm this. We modify the Caladan client with the scheduling policy disabled: specifically, workers are bound to different cores, which can execute send
, do_softirq
(directpath), handle_timeout
, and recv
in cycles without yielding.
We equip Caladan server with 4 kthreads and launch 16 client workers (each of which owns a TCP connection) to generate requests w/ a Poisson distribution and vary the request rate (last for 32 seconds). The following table shows the experiment result:
Client Type | Thoughtput(pps) | P50 (us) | P90 (us) | P999 (us) | Never Send |
---|---|---|---|---|---|
synthetic | 0.75M | 13.4 | 23.1 | 40.0 | 1.39% |
client w/o sched | 0.75M | 9.809091 | 18.500909 | 50.903636 | 0.000442% |
synthetic | 0.8M | 13.6 | 22.6 | 37.9 | 1.4486% |
client w/o sched | 0.8M | 9.49 | 16.81 | 584.52 | 0.000430% |
synthetic | 1M | 13.6 | 21.9 | 38.6 | 1.6726% |
client w/o sched | 1M | 9.64 | 17.59 | 2841.83 | 0.000694% |
synthetic | 1.1M | 13.5 | 21.1 | 55.5 | 1.7345% |
client w/o sched | 1.1M | 10.6 | 21.9 | 5177.75 | 0.000781% |
Hi Caladan developers,
I am running into some issues when trying to build and run Caladan on CloudLab m510 machines.
README.md
says one should use make submodules
to build the submodules. I think the correct command is actually build/init_submodules
, right?
build/shared.mk
didn't set MLX4_INC
and MLX4_LIBS
like it did for mlx5, so I added the following lines:
+# mlx4 build +MLX4_INC = -I$(ROOT_PATH)/rdma-core/build/include +MLX4_LIBS = -L$(ROOT_PATH)/rdma-core/build/lib/statics/ +MLX4_LIBS += -lmlx4 -libverbs -lnl-3 -lnl-route-3
Similarly, Makefile
didn't use MLX4_[INC,LIBS]
, so I changed that too.
Typo in build/config
. CONFIG_MLX4
is for ConnectX-3
support, not ConnectX-4
.
After fixing the problems above, I can build iokerneld
on m510 successfully. However, DPDK fails to initialize the port at startup. Here is the error message:
yilongl@rc01:/shome/caladan$ sudo ./iokerneld CPU 09| <5> cpu: detected 16 cores, 1 nodes CPU 09| <5> time: detected 1995 ticks / us [ 0.001901] CPU 09| <5> sched: CPU configuration... node 0: [0,8][1,9][2,10][3,11][4,12][5,13][6,14][7,15] [ 0.001944] CPU 09| <5> sched: dataplane on 8, control on 0 IBRS and IBPB supported : yes STIBP supported : yes Spec arch caps supported : no IBRS enabled in the kernel : no STIBP enabled in the kernel : no Socket 0: 1 memory controllers detected with total number of 4 channels. 0 QPI ports detected. 0 M2M (mesh to memory) blocks detected. 1 Home Agents detected. 0 M3UPI blocks detected. [ 0.101894] CPU 00| <5> control: spawning control thread EAL: Detected 16 lcore(s) EAL: Detected 1 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:09:00.0 on NUMA socket 0 EAL: probe driver: 15b3:1007 net_mlx4 net_mlx4: 0x561d86587300: cannot attach flow rules (code 95, "Operation not supported"), flow error type 2, cause 0x1073cd200, message: flow rule rejected by device
CloudLab m510 machines are equipped with ConnectX3-Pro NIC so I assume Caladan should be able run on them. Any help would be greatly appreciated.
I'm trying to deploy Caladan, however my testbed stuck completely at cpuidle_pause_and_lock, and nothing can be done to rescue.
CPU:Intel(R) Xeon(R) Gold 6240M CPU @ 2.60GHz
kernel: Linux-kernel 5.2
System : Ubuntu18.04
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.