securesystemslab / agamotto

Agamotto: Accelerating Kernel Driver Fuzzing with Lightweight Virtual Machine Checkpoints

Agamotto: Accelerating Kernel Driver Fuzzing with Lightweight Virtual Machine Checkpoints

Prerequisite

• CMake 3.7.2 or higher (cmake -version)
• Go 1.12.3 or higher (go version)
• Python 3

Setup

Download source code

git clone --recursive https://github.com/securesystemslab/agamotto.git
cd agamotto
export AGPATH=$PWD # assumed by commands that follow
./setup.sh

Change the host Linux kernel for custom hypercall support

Build the host Linux kernel with our patch applied, and with CONFIG_KVM_AGAMOTTO=y, and install & reboot it.

Tested environment:

• Ubuntu-hwe-4.18.0-18.19_18.04.1 on AMD EPYC 7601

Download and build Syzkaller

# Get Syzkaller source code
go get -u -d github.com/google/syzkaller
cd $GOPATH/src/github.com/google/syzkaller
git checkout ddc3e85997efdad885e208db6a98bca86e5dd52f

# Apply patch and build
cd $GOPATH/src/github.com/google/syzkaller
patch -p0 <$AGPATH/syzkaller.patch
make

Build project and generate necessary files

# Build project
cd $AGPATH/build
cmake ..
make

Setup QEMU

# Apply patch
cd $AGPATH/qemu
patch -p0 <$AGPATH/qemu.patch

# Build
mkdir $AGPATH/build/qemu
cd $AGPATH/build/qemu
$AGPATH/qemu/configure --prefix=$AGPATH/build/qemu/install --target-list=x86_64-softmmu --with-agamotto=$AGPATH/build/libagamotto --enable-debug
make -j4 install

Setup VM

• Patch and build Linux kernel

  cd $AGPATH/guest/linux/kernel
  patch -p0 <../kernel.patch

  cd $AGPATH/scripts
  ./build-linux-guest.sh all ../guest/linux/kernel/

• Create a Debian image

  cd $AGPATH/scripts
  ./create-debian-image.sh             # Create an image
  ./copy-modules.py all -d stretch.img # Copy necessary files into the image

Start fuzzing

# Generate Syzkaller config files
cd $AGPATH
make -C configs/syzkaller VMCNT=<number of fuzzing instances> -B

# Run Syzkaller USB fuzzing
cd $GOPATH/src/github.com/google/syzkaller
export PATH=$AGPATH/build/qemu/install/bin:$PATH
export LD_LIBRARY_PATH=$AGPATH/build/libagamotto:$LD_LIBRARY_PATH
./bin/syz-manager -config $AGPATH/configs/syzkaller/generated/<CFG_FILE>.cfg

# Run AFL PCI fuzzing
cd $AGPATH/scripts
./create-overlay-image.py rtl8139 -d stretch.img
export PATH=$AGPATH/build/qemu/install/bin:$PATH
export LD_LIBRARY_PATH=$AGPATH/build/libagamotto:$LD_LIBRARY_PATH
./fuzz.py rtl8139 -g linux-prog05 -i seed/ -N <number of fuzzing instances>

Citing our work

@inproceedings{song2020agamotto,
  title =        {{Agamotto}: Accelerating Kernel Driver Fuzzing with
                  Lightweight Virtual Machine Checkpoints},
  author =       {Song, Dokyung and Hetzelt, Felicitas and Kim, Jonghwan and
                  Kang, Brent Byunghoon and Seifert, Jean-Pierre and Franz,
                  Michael},
  booktitle =    {{USENIX} Security Symposium},
  year =         {2020}
}

agamotto's Issues

agamotto login: panic: runtime error: invalid memory address or nil pointer dereference

Hi guys,
Trying to run agamotto on GCE instance.
Compiled and installed the host kernel (exactly the same one that you guys used), obviously with the patch and CONFIG_KVM_AGAMOTTO enabled.
Used the same guest kernel, QEMU and syzkaller, all of them patched just as in your installation tutorial.
We face a repetetive error, that occurs with all of the syzkaller configs.

Ran
roi@agamotto1nested:/opt/gopath/src/github.com/google/syzkaller$ ./bin/syz-manager -config /home/roi/agamotto/configs/syzkaller/generated/snapshot-usb.go7007.cfg -debug -vv 999999 > debugged.txt

1176 2020/08/17 13:06:40 executor already running - using existing in/out/err pipes
1177 2020/08/17 13:06:40 makeCommandWithPipes bin:[/usr/bin/ssh -p 1569 -F /dev/null -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roi/agamotto/scripts/stretch.id_rsa root@localhost /syz-executor.wrapper]
1178 2020/08/17 13:06:40 performing handshake with an already running executor...
1179 periscope: syz-fuzzer receive handshake (magic=0xbadc0ffeebadface)
1180 periscope: paddr=0x12ce5000 for vaddr=0x7f75f8955000 l=0x18
1181 periscope: syz-fuzzer receive handshake (size=24)
1182 periscope: guest agent did not request shutdown 0
1183 [^[[0;32m  OK  ^[[0m] Started /etc/rc.local Compatibility.^M
1184 [^[[0;32m  OK  ^[[0m] Started Serial Getty on ttyS0.^M
1185 [^[[0;32m  OK  ^[[0m] Started Getty on tty1.^M
1186 [^[[0;32m  OK  ^[[0m] Reached target Login Prompts.^M
1187 [^[[0;32m  OK  ^[[0m] Reached target Multi-User System.^M
1188 [^[[0;32m  OK  ^[[0m] Reached target Graphical Interface.^M
1189          Starting Update UTMP about System Runlevel Changes...^M
1190 [^[[0;32m  OK  ^[[0m] Started Update UTMP about System Runlevel Changes.^M
1191 ^M^M
1192 Debian GNU/Linux 9 agamotto ttyS0^M
1193 ^M
1194 agamotto login: panic: runtime error: invalid memory address or nil pointer dereference
1195 [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8276dd]
1196
1197 goroutine 57 [running]:
1198 main.(*Proc).executeRaw(0xc00018ff40, 0xc00039ded8, 0xc000036040, 0x0, 0x0)
1199         /opt/gopath/src/github.com/google/syzkaller/syz-fuzzer/proc.go:349 +0xdd
1200 main.(*Proc).execute(0xc00018ff40, 0xc00039ded8, 0xc000036040, 0x0, 0x0, 0x2)
1201         /opt/gopath/src/github.com/google/syzkaller/syz-fuzzer/proc.go:299 +0x6a
1202 main.(*Proc).loop(0xc00018ff40)
1203         /opt/gopath/src/github.com/google/syzkaller/syz-fuzzer/proc.go:113 +0x3b6
1204 created by main.main
1205         /opt/gopath/src/github.com/google/syzkaller/syz-fuzzer/fuzzer.go:449 +0x166a

attaching debugged.txt

`arch/x86/entry/thunk_64.o: warning: objtool: missing symbol table` error while building linux kernel on Ubuntu 21.10

system info:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 21.10
Release:	21.10
Codename:	impish
$ uname -a
Linux b3ale-OMEN-by-HP-Laptop-16-b0xxx 5.13.0-23-generic #23-Ubuntu SMP Fri Nov 26 11:41:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

the error appeared while building the linux kernel:

$ ./build-linux-guest.sh all ../guest/linux/kernel
~/agamotto/guest/linux/kernel ~/agamotto/scripts
+ for config in ${GUEST_CONFIGS[*]}
+ defconfig=agamotto_aqtion_defconfig
+ '[' -f arch/x86/configs/agamotto_aqtion_defconfig ']'
+ echo Compiling agamotto_aqtion_defconfig...
Compiling agamotto_aqtion_defconfig...
+ make agamotto_aqtion_defconfig O=/home/b3ale/agamotto/scripts/./../build/guest/linux/image/aqtion
make[1]: Entering directory '/home/b3ale/agamotto/build/guest/linux/image/aqtion'
  GEN     Makefile
#
# No change to .config
#
make[1]: Leaving directory '/home/b3ale/agamotto/build/guest/linux/image/aqtion'
+ make -j40 O=/home/b3ale/agamotto/scripts/./../build/guest/linux/image/aqtion
make[1]: Entering directory '/home/b3ale/agamotto/build/guest/linux/image/aqtion'
  GEN     Makefile
  DESCEND  objtool
  CALL    /home/b3ale/agamotto/guest/linux/kernel/scripts/atomic/check-atomics.sh
  CALL    /home/b3ale/agamotto/guest/linux/kernel/scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  AS      arch/x86/entry/thunk_64.o
arch/x86/entry/thunk_64.o: warning: objtool: missing symbol table
make[3]: *** [/home/b3ale/agamotto/guest/linux/kernel/scripts/Makefile.build:348: arch/x86/entry/thunk_64.o] Error 1
make[3]: *** Deleting file 'arch/x86/entry/thunk_64.o'
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [/home/b3ale/agamotto/guest/linux/kernel/scripts/Makefile.build:503: arch/x86/entry] Error 2
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [/home/b3ale/agamotto/guest/linux/kernel/Makefile:1693: arch/x86] Error 2
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/home/b3ale/agamotto/build/guest/linux/image/aqtion'
make: *** [Makefile:179: sub-make] Error 2

is this agamotto's error? or something wrong with my host machine?