KVM-based Virtual Machine Introspection.
This repository is a fork of KVM and contains a set of modifications that enables Virtual Machine Introspection
features.
Fork of KVM with Virtual Machine Introspection patches
License: Other
I wanted to install and use libvmi/libvmi, so I tried to compile the kernel here to enable the kvm introspection feature.
I followed the step in https://kvm-vmi.github.io/kvm-vmi/kvmi-v7/setup.html#kvm. But I had a problem:
➜ kvm git:(master) sudo make -j$(nproc)
CHK include/config/kernel.release
CHK include/generated/uapi/linux/version.h
DESCEND objtool
/home/zzc/Desktop/zzc/kvm-vmi/kvm/tools/objtool/.fixdep.o.cmd:1: *** 缺失分隔符。 停止。
make[3]: *** [Makefile:42:/home/zzc/Desktop/zzc/kvm-vmi/kvm/tools/objtool/fixdep-in.o] 错误 2
make[2]: *** [/home/zzc/Desktop/zzc/kvm-vmi/kvm/tools/build/Makefile.include:4:fixdep] 错误 2
make[1]: *** [Makefile:60:objtool] 错误 2
make: *** [Makefile:1614:tools/objtool] 错误 2
make: *** 正在等待未完成的任务....
It says "Missing separator" in line 1. And I find the file ".fixdep.o.cmd":
# cannot find fixdep (/home/zzc/Desktop/zzc/kvm-vmi/kvm/tools/objtool//fixdep)
# using basic dep data/home/zzc/Desktop/zzc/kvm-vmi/kvm/tools/objtool/fixdep.o: fixdep.c
/usr/include/stdc-predef.h /usr/include/x86_64-linux-gnu/sys/types.h
/usr/include/features.h /usr/include/features-time64.h
/usr/include/x86_64-linux-gnu/bits/wordsize.h
/usr/include/x86_64-linux-gnu/bits/timesize.h
/usr/include/x86_64-linux-gnu/sys/cdefs.h
/usr/include/x86_64-linux-gnu/bits/long-double.h
/usr/include/x86_64-linux-gnu/gnu/stubs.h
/usr/include/x86_64-linux-gnu/gnu/stubs-64.h
/usr/include/x86_64-linux-gnu/bits/types.h
/usr/include/x86_64-linux-gnu/bits/typesizes.h
/usr/include/x86_64-linux-gnu/bits/time64.h
......
I can see no space or tab here. How can I solve this problem? Please help me.
I'm trying to understand how you set up page protection in the guest, so that you can exit on PFs. (E.g., you use this mechanism to track system calls.) I wanted to ask about the EPT view, that is passed to e.g., kvm_slot_page_track_add_page().
My naive thinking is that when you call kvm_slot_page_track_add_page(), you set protection in the EPT of the current guest, so that you later trap when the guest OS wants to access the gfn, so a gfn->pfn translation is needed.
However, I don't see why you need to distinguish between different EPT views? (I.e., isn't the "right" one being used now?) I must be missing something obvious...
EDIT: certainly it is possible to develop a mechanism which would use VMFUNCs for e.g., isolation. Then it may involve multiple EPTs per guest....
Thank you!
I tried to setup the vagrant on an AMD processor, and the self-tests fails:
TASK [run kvm self-tests] ******************************************************
fatal: [kvmi]: FAILED! => changed=true
cmd:
- ./tools/testing/selftests/kvm/x86_64/kvmi_test
delta: '0:00:00.607688'
end: '2021-02-18 15:32:08.759027'
msg: non-zero return code
rc: 254
start: '2021-02-18 15:32:08.151339'
stderr: |-
==== Test Assertion Failure ====
x86_64/kvmi_test.c:829: get_ucall(ctx->vm, ctx->vcpu_id, &uc)
pid=8492 tid=8522 - Success
1 0x0000000000403337: vcpu_worker at kvmi_test.c:828
2 0x00007fe482c65fa2: ?? ??:0
3 0x00007fe482b964ce: ?? ??:0
No guest request
stderr_lines:
- ==== Test Assertion Failure ====
- ' x86_64/kvmi_test.c:829: get_ucall(ctx->vm, ctx->vcpu_id, &uc)'
- ' pid=8492 tid=8522 - Success'
- " 1\t0x0000000000403337: vcpu_worker at kvmi_test.c:828"
- " 2\t0x00007fe482c65fa2: ?? ??:0"
- " 3\t0x00007fe482b964ce: ?? ??:0"
- ' No guest request'
stdout: |-
Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages
Guest physical address width detected: 40
KVMI version: 1
singlestep: 0
vmfunc: 0
eptp: 0
ve: 0
spp: 0
vcpu count: 1
tsc_speed: 0 HZ
get_registers rip 0x40bc0f
cpuid(0, 0) => eax 0x0000000d, ebx 0x68747541, ecx 0x444d4163, edx 0x69746e65
Hypercall event, rip 0x402e5d
Breakpoint event, rip 0x402ee0, len 1
CR4, old 0x220, new 0x40220
Exception event: vector 6, error_code 0x0, cr2 0x0
stdout_lines: <omitted>
Any ideas @adlazar ?
Hello,I do some tests on "sub-page-protection" but when I called "KVM_INIT_SPP" the function return error "-KVM_EOPNOTSUPP " So I don 't know is there need hardware support or need to set the "spp_supported =1" by myself. if need set it and how to set it ?
Hello, Is that x86_emulate_instruction can emulate all instructions? I want to emulate some call instruction,but I don't know how to do it,Do you have any information about this problem,Thanks.
Hi,
Trying to generate Rust
bindings to kvmi/libkvmi.h
results in the following output:
The error is simple, stddef.h
is missing, even though size_t
is used.
@adlazar: would it be possible to add it in the next patch release ?
Also, a rule in the Makefile
to install the library and the headers on the system would be helpful :)
Thanks !
Hi,
I would like to print the available information in the KVMi handshake callback:
static int cb_handshake(
const struct kvmi_qemu2introspector *qemu,
struct kvmi_introspector2qemu *intro,
void *ctx)
{
(void)ctx;
if (!qemu || !intro) {
errprint("Invalid parameters in KVMi handshake callback");
return 1;
}
char str_time[20] = {0};
strftime(str_time, 20, "%Y-%m-%d %H:%M:%S", localtime(&qemu->start_time));
// print name and start time
dbprint(VMI_DEBUG_KVM, "--KVMi handshake - Domain name: %s, Start time: %s\n", qemu->name, str_time);
// print UUID
for (int i = 0; i < 16; i++)
printf("%.2X ", qemu->uuid[i]);
printf("\n");
// print cookie
for (int i = 0; i < 20; i++)
printf("%.2X ", intro->cookie_hash[i]);
printf("\n");
return 0;
}
(Just opening this issue to start brainstorming)
As per yesterdays discussion, the following proposal has been made to streamline the delivery of events from KVM to the userspace component. Rather then implementing a get_event() logic to be called by the userspace component for each vCPU, we should introduce a new KVM exit code to be delivered when vcpu_run returns to QEMU (or any other userspace tool driving the VM, like LKVM).
The KVM exit code is delivered via vcpu->run->exit_reason
when vcpu_run
returns. This run structure is outlined here http://lxr.free-electrons.com/source/include/uapi/linux/kvm.h#L218. In addition to the exit code, we should add a structure here for VMI specific information to be conveyed.
struct {
__u32 type;
__u32 flags;
union {
struct {
__u8 direction;
} nitro;
};
} vmi;
Where type
can be defined as VMI_NITRO_SYSCALL, VMI_NITRO_SYSENTER and other types of events in the future as we add more (EPT events, MTF, CPUID, etc..).
The flags
field can be useful for an introspecting application to return a decision on how to handle an event that was trapped. For example in Xen this is used to trigger altp2m switches, trigger MTF or emulation without having to issue a separate call to the VMM.
This will likely need to be wrapped into a CONFIG setting too so that it is conditionally compiled.
RFC!
It would be convient to add an install
rule in libkvmi's Makefile, to install the library and headers system-wide.
In my Travis scripts, I have to do the following:
https://github.com/Wenzel/libmicrovmi/blob/master/.travis.yml#L31
- sudo apt-get install git
- git clone https://github.com/KVM-VMI/kvm --branch kvmi --depth 1
- cd kvm/tools/kvm/kvmi && make && sudo cp -rv include/ /usr/local && sudo cp libkvmi.so /usr/local/lib
cc @adlazar
Thanks !
Hi,
While using kvmi_set_page_access
on KVMi-v7
, I triggered a 100% reproducible KVM crash:
Apr 24 14:06:57 debian10.localdomain kernel: kvm [1457]: f555864c-5917-454c-83e5-f0ee98452249 Hooking VM
Apr 24 14:07:29 debian10.localdomain kernel: walk_shadow_page_get_mmio_spte: detect reserved bits on spte, addr 0x268fe10, dump hierarchy:
Apr 24 14:07:29 debian10.localdomain kernel: ------ spte 0x1000007abb7007 level 4.
Apr 24 14:07:29 debian10.localdomain kernel: ------ spte 0x10000035a59007 level 3.
Apr 24 14:07:29 debian10.localdomain kernel: ------ spte 0x1000007a88a007 level 2.
Apr 24 14:07:29 debian10.localdomain kernel: ------ spte 0x1000005d48fc76 level 1.
Apr 24 14:07:29 debian10.localdomain kernel: ------------[ cut here ]------------
Apr 24 14:07:29 debian10.localdomain kernel: WARNING: CPU: 0 PID: 1195 at arch/x86/kvm/mmu.c:4373 kvm_mmu_page_fault.cold.160+0x51/0x5f
Apr 24 14:07:29 debian10.localdomain kernel: Modules linked in: rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) fscache(E) xt_CHECKSUM(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_MASQUERADE(E) nf_conntrack_netlink(E) xfrm_user(E) xfrm_algo(E) nft_counter(E) xt_tcpudp(E) nft_chain_nat(E) xt_addrtype(E) nft_compat(E) tun(E) xt_conntrack(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) libcrc32c(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) nf_tables(E) nfnetlink(E) intel_rapl_msr(E) intel_rapl_common(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) aesni_intel(E) crypto_simd(E) cryptd(E) glue_helper(E) cirrus(E) snd_pcm(E) snd_timer(E) joydev(E) drm_kms_helper(E) snd(E) evdev(E) soundcore(E) pcspkr(E) virtio_balloon(E) serio_raw(E) sg(E) drm(E) qemu_fw_cfg(E) button(E) sunrpc(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc32c_generic(E) crc16(E) mbcache(E) jbd2(E) sd_mod(E) ata_generic(E) virtio_net(E) net_failover(E) failover(E) ata_piix(E) uhci_hcd(E)
Apr 24 14:07:29 debian10.localdomain kernel: sym53c8xx(E) ehci_hcd(E) scsi_transport_spi(E) crc32c_intel(E) usbcore(E) psmouse(E) virtio_pci(E) virtio_ring(E) virtio(E) i2c_piix4(E) libata(E) scsi_mod(E)
Apr 24 14:07:29 debian10.localdomain kernel: CPU: 0 PID: 1195 Comm: CPU 1/KVM Tainted: G E 5.4.24-kvmi+ #7
Apr 24 14:07:29 debian10.localdomain kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
Apr 24 14:07:29 debian10.localdomain kernel: RIP: 0010:kvm_mmu_page_fault.cold.160+0x51/0x5f
Apr 24 14:07:29 debian10.localdomain kernel: Code: 8b 04 24 44 8b 4c 24 08 44 89 c2 45 89 cc 44 39 e2 7e 17 8d 5a ff 48 89 ef 48 63 c3 48 8b 74 c4 68 e8 90 10 12 00 89 da eb e4 <0f> 0b b8 ea ff ff ff e9 56 81 ff ff 90 90 0f 90 c0 c3 0f 91 c0 c3
Apr 24 14:07:29 debian10.localdomain kernel: RSP: 0018:ffffa901806cfc50 EFLAGS: 00010246
Apr 24 14:07:29 debian10.localdomain kernel: RAX: 0000000000000025 RBX: 0000000000000000 RCX: 0000000000000006
Apr 24 14:07:29 debian10.localdomain kernel: RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff89337da17660
Apr 24 14:07:29 debian10.localdomain kernel: RBP: ffffffff820adc08 R08: 00000000000001f4 R09: 00000000ad55ad55
Apr 24 14:07:29 debian10.localdomain kernel: R10: 0000000000000000 R11: 0000000000000002 R12: 0000000000000000
Apr 24 14:07:29 debian10.localdomain kernel: R13: 0000000000000008 R14: ffff89337a960000 R15: 0000000000000001
Apr 24 14:07:29 debian10.localdomain kernel: FS: 00007f2ea2ffd700(0000) GS:ffff89337da00000(0000) knlGS:000007fffffae000
Apr 24 14:07:29 debian10.localdomain kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 14:07:29 debian10.localdomain kernel: CR2: fffff900c1ccd000 CR3: 000000007a92a005 CR4: 0000000000362ef0
Apr 24 14:07:29 debian10.localdomain kernel: Call Trace:
Apr 24 14:07:29 debian10.localdomain kernel: vcpu_enter_guest+0x324/0x1450
Apr 24 14:07:29 debian10.localdomain kernel: ? vmx_prepare_switch_to_host+0xd1/0x180
Apr 24 14:07:29 debian10.localdomain kernel: ? tomoyo_init_request_info+0x84/0x90
Apr 24 14:07:29 debian10.localdomain kernel: ? emulator_pio_in_emulated+0x109/0x190
Apr 24 14:07:29 debian10.localdomain kernel: ? vmx_get_cs_db_l_bits+0x17/0x30
Apr 24 14:07:29 debian10.localdomain kernel: ? skip_emulated_instruction+0x91/0xe0
Apr 24 14:07:29 debian10.localdomain kernel: ? kvm_skip_emulated_instruction+0x36/0x50
Apr 24 14:07:29 debian10.localdomain kernel: kvm_arch_vcpu_ioctl_run+0x10e/0x5a0
Apr 24 14:07:29 debian10.localdomain kernel: kvm_vcpu_ioctl+0x22f/0x5e0
Apr 24 14:07:29 debian10.localdomain kernel: do_vfs_ioctl+0xa4/0x630
Apr 24 14:07:29 debian10.localdomain kernel: ksys_ioctl+0x60/0x90
Apr 24 14:07:29 debian10.localdomain kernel: __x64_sys_ioctl+0x16/0x20
Apr 24 14:07:29 debian10.localdomain kernel: do_syscall_64+0x52/0x160
Apr 24 14:07:29 debian10.localdomain kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Apr 24 14:07:29 debian10.localdomain kernel: RIP: 0033:0x7f2eaa94e427
Apr 24 14:07:29 debian10.localdomain kernel: Code: 00 00 90 48 8b 05 69 aa 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 39 aa 0c 00 f7 d8 64 89 01 48
Apr 24 14:07:29 debian10.localdomain kernel: RSP: 002b:00007f2ea2ffc898 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Apr 24 14:07:29 debian10.localdomain kernel: RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f2eaa94e427
Apr 24 14:07:29 debian10.localdomain kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000012
Apr 24 14:07:29 debian10.localdomain kernel: RBP: 0000000000000000 R08: 00005653f52d1590 R09: 063f6abfbafd4ab5
Apr 24 14:07:29 debian10.localdomain kernel: R10: 000000003b9aca00 R11: 0000000000000246 R12: 00005653f73f8960
Apr 24 14:07:29 debian10.localdomain kernel: R13: 00007f2eab348000 R14: 0000000000000000 R15: 00005653f73f8960
Apr 24 14:07:29 debian10.localdomain kernel: ---[ end trace b6dc7a2914b8def6 ]---
KVM patches can cause monitored VMs to freeze.
Unfortunatelly, the steps to reproduce this problem are little convoluted since they relly on my multiple backends fork of nitro. I am sure a more minimal test case can be constructed but I've yet to do so. However, asuming the user has setup Nitro:
nose2 --log-level=debug -v test_linux.TestLinux.test_write
CTRL-C
After terminating the test, the VM should continue execution normally.
I haven't really looked into the possible causes of this, but it seems clear that the problem is in the kernel. I am not sure if the problem is present with Windows guests. This problem has been previosly discussed in the Multiple Backends issue
While implementing VMI_EVENT_RESPONSE_SET_REGISTERS
in Libvmi KVM driver, I realized that it wasn't possible to set
special registers and MSRs.
The API provided by kvmi-v6
allows to get these registers:
kvm/tools/kvm/kvmi/include/kvmi/libkvmi.h
Line 110 in 5205f80
but not to set them:
kvm/tools/kvm/kvmi/include/kvmi/libkvmi.h
Line 112 in 5205f80
We need a Debian package including Nitro's KVM patches in order to help other developers quickly setting up a Nitro capable environment.
Steps:
Using Debian default kernel config:
# aptitude download linux-image-4.10
$ dpkg-deb -x linux-image-4.10-amd64.deb ./
$ cp boot/config-4.10-amd64 /path/to/sources/linux/.config
Hi,I'm interested in introspected VM, I find that you make "kvm_slot_page_track_add_page" to set one slot of write_protect.But I can't find what new processes have you added in "handle_ept_violation" if this slot trigger write_protect. Could you tell me ? By the way I can't find the function implementation code of "trace_kvm_page_faule"、"trace_kvmi_set_gfn_access"...etc. Thank you!
There are some conflict in the file that push to the git. For example in kvm_main.c there is:
<<<<<<< HEAD
nitro_create_vm_hook(kvm);
r = anon_inode_getfd("kvm-vm", &kvm_vm_fops, kvm, O_RDWR | O_CLOEXEC);
if (r < 0)
=======
r = get_unused_fd_flags(O_CLOEXEC);
if (r < 0) {
>>>>>>> linux-vmi
Hi guys,
Can you explain what is the difference between these branches? I want to update my application which was written two years ago. Which branch should I work on?
Best wishes,
0001-Fix-wrong-assignment.patch.zip
Explanation.
In nitro_main.c::nitro_iotcl_attach_vcpus one can see
kvm_for_each_vcpu(r, v, kvm){
nvcpus->ids[r] = v->vcpu_id;
kvm_get_kvm(kvm);
nvcpus->fds[r] = create_vcpu_fd(v);
if(nvcpus->fds[r]<0){
for(i=r;r>=0;i--){
nvcpus->ids[r] = 0; <-------------------------------- This is should be i
nvcpus->fds[i] = 0;
kvm_put_kvm(kvm);
}
goto error_out;
}
}
In the KVMi-v6 API, the ability to emulate new data or new instructions is tied to the reply of a memory event:
Line 251 in 5205f80
struct kvmi_event_pf_reply {
__u64 ctx_addr;
__u32 ctx_size;
__u8 singlestep;
__u8 rep_complete;
__u16 padding;
__u8 ctx_data[256];
};
It might be beneficial to have the same fields in a breakpoint response too.
In fact, upon a breakpoint event, you can do a couple of things:
emulate the original instruction in breakpoint even handler (a.k.a RIP++
), but this is not very reliable, unless you can embed a decompiler and can implement a specific behavior for each instruction.
configure the original instruction to be emulated, via the breakpoint event response.
In LibVMI, this behavior is already implemented in breakpoint-emulate-example.c
:
https://github.com/libvmi/libvmi/blob/master/examples/breakpoint-emulate-example.c#L87
if (data->vaddr == event->interrupt_event.gla) {
// our breakpoint !
printf("We hit our breakpoint on %s, setting emulation buffer to 0x%"PRIx64"\n",
data->symbol, *(uint64_t*)data->emul.data);
// don't reinject
event->interrupt_event.reinject = 0;
// set previous opcode for emulation
event->emul_insn = &data->emul;
// set response to emulate instruction
rsp |= VMI_EVENT_RESPONSE_SET_EMUL_INSN;
}
Amond the benefits of simplifying recoiling on a breakpoint (which is tedius since you need to handle one event handler for the interrupt, and a second for the singlestep event), this solution has the advantage of being race-condition free in a multi-VCPU context.
Also, on a side note, this makes it more difficult to implement VMI_EVENT_RESPONSE_SET_EMUL_READ_DATA
and VMI_EVENT_RESPONSE_SET_EMUL_INSN
, as for kvmi-v6, they should be specific to a kvmi_event_pf
.
So this type of generic process_cb_response()
is not possible:
https://github.com/libvmi/libvmi/blob/master/libvmi/driver/xen/xen_events.c#L604
On Xen, I'm not sure, but LibVMI has an internal structure, with emulation context available for all events:
https://github.com/libvmi/libvmi/blob/master/libvmi/driver/xen/xen_events_private.h#L97
typedef struct vm_event_compat {
uint32_t version;
uint32_t flags;
uint32_t reason;
uint32_t vcpu_id;
uint16_t altp2m_idx;
union {
struct vm_event_mem_access mem_access;
struct vm_event_write_ctrlreg write_ctrlreg;
struct vm_event_mov_to_msr_3 mov_to_msr;
struct vm_event_desc_access_3 desc_access;
struct vm_event_singlestep singlestep;
struct vm_event_debug_6 software_breakpoint;
struct vm_event_debug_6 debug_exception;
struct vm_event_cpuid cpuid;
struct vm_event_interrupt_x86 x86_interrupt;
};
union {
union {
x86_registers_t x86;
arm_registers_t arm;
} regs;
union {
struct vm_event_emul_read_data_4 read;
struct vm_event_emul_insn_data insn;
} emul;
} data;
} vm_event_compat_t;
@tklengyel , can you tell us if an emulation context is theoretically available for all VM events on Xen ?
Thanks.
Hi,
I have tried to compile kvm on Ubuntu 22.04. I'm not sure if it's possible without downgrading a loooot.
I wanted to ask if you have by any chance ported the VMI extensions to 5.15?
Thank you!
Is there any way to know which guest physical memory addresses are valid for KVMI_READ_PHYSICAL and KVMI_WRITE_PHYSICAL?
As far as I can see, KVMI_GET_MAX_GFN gives the largest physical memory address , say max_addr
(max_gfn << 12
), used by the guest. But I found that I could get an error code -22 from KVM when I am reading from some addresses smaller than max_addr
. I suspect that this is because those addresses are not mapped by the KVM.
Currently, KVMI_READ_PHYSICAL returns error code -22 when reading from invalid addresses and the socket will be closed by KVM, which is not convenient when I try to scan the whole memory of the guest.
Given that there is no way to get ranges of valid guest memory address, I think a better way is not to close the socket when reading from invalid addresses.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.