openchannelssd / qemu-nvme Goto Github PK

The LightNVM qemu implementation, based on NVMe

Home Page: http://openchannelssd.readthedocs.org/en/latest/

License: Other

Makefile 0.24% C 90.56% C++ 3.80% Haxe 0.44% Objective-C 0.25% Assembly 0.42% Python 2.70% NSIS 0.01% Shell 1.27% Perl 0.30% GDB 0.01% Emacs Lisp 0.01% GLSL 0.01% SmPL 0.02%

qemu-nvme's Introduction

QEMU Open-Channel SSD 2.0

This repository contains a fork of qemu/qemu with modifications to the NVMe device to allow the device to expose itself as an Open-Channel 2.0 device.

Also included is support for metadata, SGLs, predefined data according to DLFEAT, optional error recovery through the error recovery DULBE-attribute and error injection.

Compiling & Installing

Below is a minimal example of the installation process.

git clone https://github.com/OpenChannelSSD/qemu-nvme.git

cd qemu-nvme
./configure --target-list=x86_64-softmmu --prefix=$HOME/qemu-nvme
make
make install

NOTE Consider using the --enable-trace-backends=log configure option for better debugging.

Configuring the Open-Channel 2.0 SSD device

The device must have a backing file to store its data. An initialized OCSSD backing file must be created using qemu-img:

qemu-img create -f ocssd -o num_grp=2,num_pu=4,num_chk=60 ocssd.img

Besides the geometry options (num_{grp,pu,chk,sec}), qemu-img also supports options related to write characteristics (ws_min, ws_opt and mw_cunits). These options can also be overwritten as parameters to the device. Issue

qemu-img create -f ocssd -o help

to see the full list of supported options.

To add the OCSSD NVMe device, extend the QEMU arguments with something like:

-blockdev ocssd,node-name=nvme01,file.driver=file,file.filename=ocssd.img
-device nvme,drive=nvme01,serial=deadbeef,id=lnvm

To get a complete list of all options supported by the NVMe device, issue

qemu-system-x86_64 -device nvme,help

or look into the source.

There are two QEMU device parameters that change the behavior of the device. The first, learly_reset is enabled by default and allows OPEN chunks to be reset. While the OCSSD 2.0 specification does not allow this most available drives do. The second, lsgl_lbal is disabled by default and governs how the LBAL field should be interpreted if DPTR is an SGL (PSDT is 0x1 or 0x2). By default LBAL will be not be interpreted as an SGL in any case. Enabling this option may be useful for toying around with NVMe over Fabrics.

Chunk State

The emulated device maintains a Chunk Info Log Page on the backing block device. When the device is brought up any state will be restored. The restored chunk states may be overwritten using the lchunkstate parameter. An example chunk state file:

grp=0 pu=0 chk=0 state=OPEN wp=65535 type=W_RAN wi=0
grp=0 pu=0 chk=1 state=OFFLINE wp=65535 type=W_SEQ wi=0
grp=0 pu=0 chk=2 state=CLOSED wp=4096 type=W_SEQ wi=0
grp=0 pu=0 chk=3 state=OPEN wp=2048 type=W_SEQ wi=0

Error Injection

The lresetfail and lwritefail QEMU parameters can be used to do probabilistic error injection. The parameters points to text files.

Write error injection is done per sector.

grp=0 pu=3 chk=0 sec=53 writefail_prob=100

Reset error injection is done per chunk, so exclude the sec parameter.

grp=0 pu=3 chk=5 resetfail_prob=100
grp=0 pu=3 chk=6 resetfail_prob=20

Guest Kernel

You probably want to make sure the following options are enabled in the kernel you are going to use.

CONFIG_BLK_DEV_INTEGRITY=y
CONFIG_HOTPLUG_PCI_PCIE=y
CONFIG_HOTPLUG_PCI_ACPI=y

qemu-nvme's People

Contributors

Stargazers

Watchers

qemu-nvme's Issues

create filesystem fails for me

Hi,

I started to play with this qemu-nvme ocssd simulator. I am able to create the pblk device but when I tried to create a filesystem on top of it, the kernel crashes. What did I do wrong? Thanks,

guest OS: debian 10, with kernel 5.2.0-rc5+, downloaded from github.

command to start qemu:
qemu-system-x86_64 -m 2g -hda mydisk.img -enable-kvm -cpu host -nographic -append "console=ttyS0 root=/dev/sda1" -kernel linux/arch/x86_64/boot/bzImage -blockdev ocssd,node-name=nvme01,file.driver=file,file.filename=ocssd.img -device nvme,drive=nvme01,serial=123456789,id=lnvm

xing@debian:~$ sudo nvme lnvm create -d nvme0n1 -n mydevice -t pblk -b 0 -e 7
[sudo] password for xing:
[   33.349066] pblk mydevice: Default OP insufficient, adjusting OP to 16
[   33.364653] pblk mydevice: luns:8, lines:60, secs:1630208, buf entries:512

xing@debian:~$ sudo fdisk -l
Disk /dev/nvme0n1: 8 GiB, 8589934592 bytes, 2097152 sectors
Disk model: QEMU NVMe OCSSD Ctrl
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/mydevice: 6.2 GiB, 6677331968 bytes, 1630208 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


xing@debian:~$ sudo mkfs -t ext4 /dev/mydevice
mke2fs 1.44.5 (15-Dec-2018)
Discarding device blocks: done
Creating filesystem with 1630208 4k blocks and 408000 inodes
Filesystem UUID: b29146ad-e13c-42cb-a1d0-c1178c6c6f17
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: [   53.873897] WARNING:0
[   53.876577] Modules linked in:
[   53.877434] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.2.0-rc5+ #1
[   53.879151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14
[   53.881732] RIP: 0010:pblk_bio_free_pages+0x63/0x70
[   53.882814] Code: 48 c1 e3 04 49 c1 e5 04 49 8b 46 70 4c 89 e6 48 8b 3c 18 48 85
[   53.886861] RSP: 0018:ffffa90040003dc0 EFLAGS: 00010006
[   53.888006] RAX: 0000000000000008 RBX: 0000000000000003 RCX: 0000000000000005
[   53.889557] RDX: 0000000000000003 RSI: ffff9a3bba26d9c0 RDI: ffff9a3bb9235800
[   53.891112] RBP: ffffa90040003de0 R08: 0000000000000002 R09: 0000000000000000
[   53.892661] R10: ffffa90040003b78 R11: 0000000005a9f7cf R12: ffff9a3bbbe0bbd8
[   53.894210] R13: ffff9a3bb9235a90 R14: ffffa90040e67ce0 R15: ffff9a3bbbe0bb80
[   53.895765] FS:  0000000000000000(0000) GS:ffff9a3bbda00000(0000) knlGS:00000000
[   53.897520] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   53.898785] CR2: 00007f0a5cfd4000 CR3: 000000006ae68005 CR4: 00000000001606f0
[   53.900342] Call Trace:
[   53.900898]  <IRQ>
[   53.901360]  pblk_end_w_bio+0x9b/0xe0
[   53.902178]  pblk_end_io_write+0x14f/0x1d0
[   53.903095]  nvm_end_io+0x3d/0x40
[   53.903836]  nvme_nvm_end_io+0x2e/0x50
[   53.904675]  blk_mq_end_request+0xb3/0x130
[   53.905585]  nvme_complete_rq+0x8d/0x230
[   53.906468]  nvme_pci_complete_rq+0x4c/0xc0
[   53.907398]  blk_done_softirq+0x92/0xc0
[   53.908252]  __do_softirq+0xe4/0x2f3
[   53.909051]  irq_exit+0xb6/0xc0
[   53.909754]  do_IRQ+0x8a/0xe0
[   53.910431]  common_interrupt+0xf/0xf
[   53.911249]  </IRQ>
[   53.911727] RIP: 0010:native_safe_halt+0x12/0x20
[   53.912748] Code: 00 0f 00 2d c2 f3 59 00 f4 5d c3 0f 1f 00 66 2e 0f 1f 84 00 09
[   53.916805] RSP: 0018:ffffffffa9e03e10 EFLAGS: 00000246 ORIG_RAX: fffffffffffff8
[   53.918466] RAX: ffffffffa926aad0 RBX: 0000000000000000 RCX: 0000000000000001
[   53.920012] RDX: ffff9a3bbda2be00 RSI: 0000000c822e9656 RDI: 0000000000000000
[   53.921554] RBP: ffffffffa9e03e10 R08: 000000000001da40 R09: 0000000c95416656
[   53.923107] R10: ffff9a3bbffacca8 R11: 0000000000000000 R12: 0000000000000000
[   53.924661] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   53.926216]  ? __cpuidle_text_start+0x8/0x8
[   53.927150]  default_idle+0x22/0x150
[   53.927942]  arch_cpu_idle+0x15/0x20
[   53.928735]  default_idle_call+0x23/0x30
[   53.929603]  do_idle+0x1cb/0x280
[   53.930320]  cpu_startup_entry+0x1d/0x20
[   53.931198]  rest_init+0xae/0xb0
[   53.931921]  arch_call_rest_init+0xe/0x1b
[   53.932808]  start_kernel+0x501/0x520
[   53.933620]  x86_64_start_reservations+0x24/0x26
[   53.934641]  x86_64_start_kernel+0x74/0x77
[   53.935545]  secondary_startup_64+0xa4/0xb0
[   53.936467] ---[ end trace 6ff2439226ef9969 ]---
[   53.939978] WARNING: CPU: 0 PID: 0 at drivers/lightnvm/pblk-core.c:329 pblk_bio0
[   53.941983] Modules linked in:
[   53.942682] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W         5.2.0-rc51
[   53.944342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14
[   53.946785] RIP: 0010:pblk_bio_free_pages+0x63/0x70
[   53.947851] Code: 48 c1 e3 04 49 c1 e5 04 49 8b 46 70 4c 89 e6 48 8b 3c 18 48 85
[   53.951885] RSP: 0018:ffffa90040003dc0 EFLAGS: 00010006
[   53.953026] RAX: 0000000000000008 RBX: 0000000000000001 RCX: 0000000000000007
[   53.954583] RDX: 0000000000000001 RSI: ffff9a3bba26d9c0 RDI: ffff9a3bb9235800
[   53.956127] RBP: ffffa90040003de0 R08: 0000000000000002 R09: 0000000000000000
[   53.957675] R10: ffffa90040003b78 R11: 000000000000b738 R12: ffff9a3bbbe0bbd8
[   53.959238] R13: ffff9a3bb9235a90 R14: ffffa90040e67d28 R15: ffff9a3bbbe0bb80
[   53.960804] FS:  0000000000000000(0000) GS:ffff9a3bbda00000(0000) knlGS:00000000
[   53.962585] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   53.963850] CR2: 00007f5c4da9b000 CR3: 0000000077d0a002 CR4: 00000000001606f0
[   53.965417] Call Trace:
[   53.965973]  <IRQ>
[   53.966448]  pblk_end_w_bio+0x9b/0xe0
[   53.967271]  pblk_end_io_write+0x14f/0x1d0
[   53.968187]  nvm_end_io+0x3d/0x40
[   53.968934]  nvme_nvm_end_io+0x2e/0x50
[   53.969771]  blk_mq_end_request+0xb3/0x130
[   53.970693]  nvme_complete_rq+0x8d/0x230
[   53.971582]  nvme_pci_complete_rq+0x4c/0xc0
[   53.972517]  blk_done_softirq+0x92/0xc0
[   53.973375]  __do_softirq+0xe4/0x2f3
[   53.974177]  irq_exit+0xb6/0xc0
[   53.974888]  do_IRQ+0x8a/0xe0
[   53.975563]  common_interrupt+0xf/0xf
[   53.976380]  </IRQ>
[   53.976862] RIP: 0010:native_safe_halt+0x12/0x20
[   53.977884] Code: 00 0f 00 2d c2 f3 59 00 f4 5d c3 0f 1f 00 66 2e 0f 1f 84 00 09
[   53.981975] RSP: 0018:ffffffffa9e03e10 EFLAGS: 00000246 ORIG_RAX: fffffffffffff8
[   53.983648] RAX: ffffffffa926aad0 RBX: 0000000000000000 RCX: 0000000000000001
[   53.985226] RDX: ffff9a3bbda2be00 RSI: ffffffffa9e03de0 RDI: 0000000000000000
[   53.986807] RBP: ffffffffa9e03e10 R08: 0000001003f93c5c R09: 0000000000000000
[   53.988387] R10: ffffa900414afd08 R11: 0000000000000000 R12: 0000000000000000
[   53.989962] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   53.991547]  ? __cpuidle_text_start+0x8/0x8
[   53.992483]  default_idle+0x22/0x150
[   53.993287]  arch_cpu_idle+0x15/0x20
[   53.994093]  default_idle_call+0x23/0x30
[   53.994980]  do_idle+0x1cb/0x280
[   53.995715]  cpu_startup_entry+0x1d/0x20
[   53.996589]  rest_init+0xae/0xb0
[   53.997320]  arch_call_rest_init+0xe/0x1b
[   53.998215]  start_kernel+0x501/0x520
[   53.999041]  x86_64_start_reservations+0x24/0x26
[   54.000065]  x86_64_start_kernel+0x74/0x77
[   54.000980]  secondary_startup_64+0xa4/0xb0
[   54.001917] ---[ end trace 6ff2439226ef996a ]---

compile werror for branch ocssd12

qga/commands-posix.c: In function ‘dev_major_minor’:
qga/commands-posix.c:618:13: error: In the GNU C Library, "major" is defined
by <sys/sysmacros.h>. For historical compatibility, it is
currently defined by <sys/types.h> as well, but we plan to
remove this soon. To use "major", include <sys/sysmacros.h>
directly. If you did not intend to use a system-defined macro
"major", you should undefine it after including <sys/types.h>. [-Werror]
*devmajor = major(st.st_rdev);
^~~~~~~~~~~~~~~~~~~~~~~~~~
qga/commands-posix.c:619:13: error: In the GNU C Library, "minor" is defined
by <sys/sysmacros.h>. For historical compatibility, it is
currently defined by <sys/types.h> as well, but we plan to
remove this soon. To use "minor", include <sys/sysmacros.h>
directly. If you did not intend to use a system-defined macro
"minor", you should undefine it after including <sys/types.h>. [-Werror]
*devminor = minor(st.st_rdev);

How do I reload the block drive I created when the VM restarts?

Hello,
I have successfully initialized a PBLK Target in the VM and formatted it to read and write properly. However, I was puzzled that when I closed the virtual machine and opened the virtual machine again with the following command, I could not find the hard disk I created previously. How can I solve this problem? Thanks.

The command I used to restart the virtual machine:
sudo ./qemu-system-x86_64 -m 4G -enable-kvm ubuntu.raw -blockdev ocssd,node-name=nvme01,file.driver=file,file.filename=ocssd.img -device nvme,drive=nvme01,serial=deadbeef,id=lnvm

Suggestions on how to debug when OCSSD device not showing up.

I launched QEMU with following flag:

./bin/qemu-system-x86_64 -hda ./ubuntu-server-ssd.img  -m 2048 -blockdev ocssd,node-name=nvme01,file.driver=file,file.filename=/home/zinechant/qemu-nvme/build/ocssd.img -device nvme,drive=nvme01,serial=deadbeef,id=lnvm

But I could not find open-channel SSD device in the booted VM.

$ sudo nvme lnvm list
Number of devices: 0
Device                     Block manager             Version

But I do find the NVME LightNVM Controller showing on PCIe bus.

00:04.0 Non-Volatile memory controller: CNEX Labs QEMU NVM Express LightNVM Controller

There is a /dev/nvme0 file.

I am not sure if this is actually working properly. But my guess is it isn't. I don't know how to make it work properly.b Any suggestions on how to debug is appretiated!

[Feature] Please port OpenChannelSSD support to newer qemu

Compiling the current qemu-nvme tree on Ubuntu 16.04 yields two issues which break compilation: issue #11 (needs patching) and some deprecation warnings which need ./configure --disable-werror.

A newer qemu might possibly not have those issues. However, block device internals in qemu changed a lot since qemu-nvme was forked from mainline qemu, so porting might not be entirely straightforward.

If I want to simulate a 2TB open channel SSD on qemu-nvme, how larger memory do I need to maintain the speed?

Property '.lchannels', '.ltype' not found

when i types the qemu running command:

sudo $HOME/qemu-nvme/bin/qemu-system-x86_64 -m 4G -enable-kvm ~/qemu-nvme2.0/ubuntu.raw \

-drive file=ocssd_backend.img,if=none,id=myocssd
-device nvme,drive=myocssd,serial=deadbeef,lver=1,ltype=0,lba_index=3,nlbaf=5,lchannels=1,namespaces=1

it shows error Property '.ltype' not found and Property '.lchannels' not found

Qemu window is too small by vnc client

when i start the qemu nvme vnc server, and use vnc client to display it. I found the resolution is too small. Can i display it fullscreen?

g_mem_set_vtable’ is deprecated

I try to make the qemu-nvme on my Ubuntu, but encounter the following error, Could anybody help to fix it.

andy@andy-MS-7758:~/qemu-nvme$ make CC vl.o vl.c: In function ‘main’: vl.c:2785:5: error: ‘g_mem_set_vtable’ is deprecated [-Werror=deprecated-declarations] g_mem_set_vtable(&mem_trace); ^ In file included from /usr/include/glib-2.0/glib/glist.h:32:0, from /usr/include/glib-2.0/glib/ghash.h:33, from /usr/include/glib-2.0/glib.h:50, from vl.c:59: /usr/include/glib-2.0/glib/gmem.h:357:7: note: declared here void g_mem_set_vtable (GMemVTable *vtable); ^ cc1: all warnings being treated as errors /home/andy/qemu-nvme/rules.mak:57: recipe for target 'vl.o' failed make: *** [vl.o] Error 1

qemu-nvme working with libvirt

Hi All
In my LAB i'm currently using libvirt to emulate VMs, i would like to move using qemu-nvme as I'm required to emulate NVME disks on my VMs.
libvirt is able to get XML file with all the required params and transform it to qemu command to create the VM.

does libvirt support also working with qemu-nvme? and how? maybe specific libvirt vertion?
does there is any equivalent tool which can preserve this capability?

thanks in advance
Dror

Latest patches trigger a null-ptr deref on nvme bringup

The latest patches on ocssd2 break on 4.17 with 2.0 LightNVM core support.

The error is between
fa0ea07 small fixes
d24c464 More cleanup and fix up 2.0 support
Cannot bisect since fa0ea07 fails compilation too.

Seems to have something to do with the 2.0 geometry identification in the kernel:
6b304ba63148 lightnvm: add 2.0 geometry identification

Though, without the new qemu patches, 4.17/core with 2.0 support (also pblk) works fine.

Also, there was a necessary patch fixing a bad dword formatting on get log page that has been removed (probably by force push)

lnvm: fix get log dword formatting

hw/block/nvme.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index 53ca966cdd33..427211e24c98 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -1900,8 +1900,8 @@ static uint16_t nvme_get_log(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd cmd)
/ NVMe R1.3 */
numdl = (dw10 >> 16) << 2;
numdu = (dw11 & 0xffff) << 2;

lpol = dw12 << 2;
lpou = dw13 << 2;

lpol = dw12;
lpou = dw13;

len = numdl + numdu;
off = lpol + lpou;

n->namespaces[i].ctrl != n

qemu-nvme/hw/block/nvme.c

Line 2912 in 6f2e658

lnvm_chunk_meta_save(&n->namespaces[i]);

As I tried device_del, ns structure is corrupted already at lnvm_exit() and lnvm_chunk_meta_save() is failing due to the wrong ctrl and chunk meta.

lnvm_rw npages too large(65536). Max:64 supported

I create a 128GB pblkdev using the below configuration:
serial=deadbeef,lver=1,ll2pmode=0,namespaces=16,lnum_lun=8,lnum_pln=4,lpgs_per_plk=256,lsecs_per_pg=4,lsec_size=4086, so each plane has 64 blocks. Because of the OOB, I see that there are 63 blocks. After I use the command:
sudo nvme lnvm create -d nvme0n1 -n mydevice01 -t pblk -b 0 -e 7
sudo mkfs -t ext4 /dev/mydevice01
sudo nvme lnvm create -d nvme0n2 -n mydevice02 -t pblk -b 0 -e 7
sudo mkfs -t ext4 /dev/mydevice02
then the qemu-nvme has the problem:
lnvm_rw npages too large(65536). Max:64 supported

Errors with below instruction for qemu configure

Cannot compile with below instructions. Please remove the ' ' in the readme

./configure' '--python=/usr/bin/python2' '--enable-kvm' '--target-list=x86_64-softmmu' '--enable-linux-aio' '--prefix=/home//qemu-nvme

double free or corruption (!prev) error?

Hello nvme developers,

I am trying to make a custom kernel to do some experiments over ocssd. But before that, I wanted to get some benchmark result on the current kernel that I was using in qemu-nvme(kernel version 4.16)

However, whenever I tried to run fio or ezfio on my qemu-nvme, on host Ubuntu(17.10), this message comes out and halts,
double free or corruption ( !prev) Aborted (core dumped)

This is my option for qemu :
-drive file=blknvme,if=none,id=mydevice -device nvme,drive=mydevice,serial=deadbeef,namespaces=1,lver=1,lmetasize=16,ll2pmode=0,nlbaf=5,lba_index=3,mdts=10,lnum_lun=4,lnum_pln=2,lsec_size=4096,lsecs_per_pg=4,lpgs_per_blk=512,lbbtable=bbtable.qemu,lmetadata=meta.qemu,ldebug=1

Thank you,
HyeongTak Ji

cannot start up the qemu-nvme when I add the qemu arguments

1.fisrt,I have installed the qemu with cdrom ubuntu-16.04.iso
2.but when I use the command to start qemu with the virtual open-channel SSD drive,I failed, the problem like this:

I dont know what's wrong with it

Error when trying to boot kernel 4.12 in qemu-nvme

Installed qemu-nvme and Ubuntu 16.04 in a .img file for the virtual machine. I have checked that using qemu command "qemu-system-x86_64 -hda openChanSSDtest1.img --enable-kvm -m 4G" can boot the machine normally.

However, when I tried the command from http://openchannelssd.readthedocs.io/en/latest/gettingstarted, "qemu-system-x86_64 -m 4G -smp 1 --enable-kvm -hda openChanSSDtest1.img -append "root=/dev/sda" -initrd /boot/initrd.img-4.12.0-rc6 -kernel "/home/baoning/Downloads/linux-4.12-rc6/arch/x86_64/boot/bzImage" -drive file=blknvme,if=none,id=mynvme -device nvme,drive=mynvme,serial=deadbeef,namespaces=1,lver=1,nlbaf=5,lba_index=3,mdts=10 ",
it showed the following image.

However, when I input "ls /dev/sda", there exists the device.

I have searched a lot of relevant questions and qemu documents to debug but got no clue.
Do you think it is a command error, or any inconsistency of the kernel version and the qemu-nvm, or anything else? I use kernel 4.12 because I want to use pblk.

Question when reading erase function of lnvm

Hi. I want to write a function to record the erase times of each block for a .
I focused on /hw/block/nvme.c with commit 6e5dab6.
The function I located is lnvm_erase_async which first calls lnvm_meta_blk_set_erased then erase_io_complete_cb. Is this function a critical path for erasing operation?
Moreover, I am quite confused with the short names of different data granularities especially without comments, could you tell me which variables are used to represent the block ids?
Thank you so much

Is it possible to monitor the utilization of each parallel unit inside OCSSD?

Hello, I'm wondering if it's possible to monitor the utilization of OCSSD's internel parallelism, with or without using pblk?
for example showing something like this:
LUN#1: 100% LUN#2: 30% LUN#3: 80% LUN#4: 2%

thank you so much!

What is your detailed configurations of host machine and virtual machine?

Hi. Since I couldn't get a normal emulated nvme device, I want to fully copy your machine configurations. I believe that there must be some reasons that the device works normal on your device but not on mine. If it is possible, I'd like to download the back file of your virtual machine.
If not, could you provide me the following information:

The OS of your host machine and its release, f. ex., Ubuntu 16.10.
The kernel version of your host machine.
The version of the headers files needed by qemu-nvme, such as x86_64-softmmu, libaio.
What is the version of your qemu-nvme? What is the configuration of the software?
What is the version of the kernel in the virtual machine？ Could you provide the config file of the kernel?
What are the versions of header files needed by the kernel?
I know my question costs much time, especially when some of them seem irrelevant. But I failed too many times and wanted to be careful. Thank you so much for your support.
If you are not available now, could you leave a message so that I can roughly know when I can get the answer.
If you dislike my question, pls tell me where is inappropriate.
Thank you again for all your help!

Does qemu-nvme support multiple channel?

Hi qemu-nvme developer

this my start.sh for qemu


sudo qemu-system-x86_64 -m 4096 -smp 4  -enable-kvm -hda openchannelssd.raw  \
    -drive file=blknvme,if=none,id=blknvme \
    -device nvme,drive=blknvme,serial=deadbeef,namespaces=1,lver=1,nlbaf=5,lba_index=3,mdts=10,lnum_ch=16,lnum_lun=1,lnum_pln=2 \
    -net nic,macaddr=54:52:d5:ab:83:10 -net tap,script=/etc/qemu-ifup   
    #-redir tcp:2223::22
    #-append "root=/dev/sda1" 
    #-kernel "/home/raohui/open_channel_ssd/linux/arch/x86_64/boot/bzImage" \

but the infortions is that:nvme: Only 1 channel is supported at the moment

Does qemu-nvme support multiple channel?

Thanks~
RaoXiaoMan

Failure of installation of the latest qemu-nvme

Hi! Thank you for looking at my question.
I tried to install the latest version of qemu-nvme with commit "6e5dab678c305f22a5f845156b570967910b3484" on my Ubuntu 16.04 with kernel 4.8.0-36-generic.
After make install, nothing special but there is no qemu installed. qemu-img, qemu-system-x86_64 don't work.
The result of make install:
install -d -m 0755 "/home/baoning/qemu-nvme/share/qemu"
install -d -m 0755 "/home/baoning/qemu-nvme/etc/qemu"
install -c -m 0644 /home/baoning/Downloads/qemu-nvme-6e5dab678c305f22a5f845156b570967910b3484/sysconfigs/target/target-x86_64.conf "/home/baoning/qemu-nvme/etc/qemu"
install -d -m 0755 "/home/baoning/qemu-nvme/var"/run
install -d -m 0755 "/home/baoning/qemu-nvme/bin"
install -c -m 0755 qemu-ga qemu-nbd qemu-img qemu-io "/home/baoning/qemu-nvme/bin"
strip "/home/baoning/qemu-nvme/bin/qemu-ga" "/home/baoning/qemu-nvme/bin/qemu-nbd" "/home/baoning/qemu-nvme/bin/qemu-img" "/home/baoning/qemu-nvme/bin/qemu-io"
install -d -m 0755 "/home/baoning/qemu-nvme/libexec"
install -c -m 0755 qemu-bridge-helper "/home/baoning/qemu-nvme/libexec"
strip "/home/baoning/qemu-nvme/libexec/qemu-bridge-helper"
set -e; for x in bios.bin bios-256k.bin sgabios.bin vgabios.bin vgabios-cirrus.bin vgabios-stdvga.bin vgabios-vmware.bin vgabios-qxl.bin acpi-dsdt.aml q35-acpi-dsdt.aml ppc_rom.bin openbios-sparc32 openbios-sparc64 openbios-ppc QEMU,tcx.bin QEMU,cgthree.bin pxe-e1000.rom pxe-eepro100.rom pxe-ne2k_pci.rom pxe-pcnet.rom pxe-rtl8139.rom pxe-virtio.rom efi-e1000.rom efi-eepro100.rom efi-ne2k_pci.rom efi-pcnet.rom efi-rtl8139.rom efi-virtio.rom qemu-icon.bmp qemu_logo_no_text.svg bamboo.dtb petalogix-s3adsp1800.dtb petalogix-ml605.dtb multiboot.bin linuxboot.bin kvmvapic.bin s390-zipl.rom s390-ccw.img spapr-rtas.bin slof.bin palcode-clipper u-boot.e500; do
install -c -m 0644 /home/baoning/Downloads/qemu-nvme-6e5dab678c305f22a5f845156b570967910b3484/pc-bios/$x "/home/baoning/qemu-nvme/share/qemu";
done
install -d -m 0755 "/home/baoning/qemu-nvme/share/qemu/keymaps"
set -e; for x in da en-gb et fr fr-ch is lt modifiers no pt-br sv ar de en-us fi fr-be hr it lv nl pl ru th common de-ch es fo fr-ca hu ja mk nl-be pt sl tr bepo cz; do
install -c -m 0644 /home/baoning/Downloads/qemu-nvme-6e5dab678c305f22a5f845156b570967910b3484/pc-bios/keymaps/$x "/home/baoning/qemu-nvme/share/qemu/keymaps";
done
install -c -m 0644 /home/baoning/Downloads/qemu-nvme-6e5dab678c305f22a5f845156b570967910b3484/trace-events "/home/baoning/qemu-nvme/share/qemu/trace-events"
for d in x86_64-softmmu; do
make --no-print-directory BUILD_DIR=/home/baoning/Downloads/qemu-nvme-6e5dab678c305f22a5f845156b570967910b3484 TARGET_DIR=$d/ -C $d install || exit 1 ;
done
install -d -m 0755 "/home/baoning/qemu-nvme/bin"
install -c -m 0755 qemu-system-x86_64 "/home/baoning/qemu-nvme/bin"
strip "/home/baoning/qemu-nvme/bin/qemu-system-x86_64"
The configuration I use was
./configure --python=/usr/bin/python2 --enable-kvm --target-list=x8
6_64-softmmu --enable-linux-aio --prefix=$HOME/qemu-nvme
The result of configuration is
Disabling libtool due to broken toolchain support
Install prefix /home/baoning/qemu-nvme
BIOS directory /home/baoning/qemu-nvme/share/qemu
binary directory /home/baoning/qemu-nvme/bin
library directory /home/baoning/qemu-nvme/lib
module directory /home/baoning/qemu-nvme/lib/qemu
libexec directory /home/baoning/qemu-nvme/libexec
include directory /home/baoning/qemu-nvme/include
config directory /home/baoning/qemu-nvme/etc
local state directory /home/baoning/qemu-nvme/var
Manual directory /home/baoning/qemu-nvme/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path /home/baoning/Downloads/qemu-nvme-6e5dab678c305f22a5f845156b570967910b3484
C compiler cc
Host C compiler cc
C++ compiler c++
Objective-C compiler cc
ARFLAGS rv
CFLAGS -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -pthread -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -g
QEMU_CFLAGS -I/usr/include/pixman-1 -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong
LDFLAGS -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g
make make
install install
python /usr/bin/python2 -B
smbd /usr/sbin/smbd
module support no
host CPU x86_64
host big endian no
target list x86_64-softmmu
tcg debug enabled no
gprof enabled no
sparse enabled no
strip binaries yes
profiler no
static build no
pixman system
SDL support no
GTK support no
VTE support no
curses support no
curl support no
mingw32 support no
Audio drivers oss
Block whitelist (rw)
Block whitelist (ro)
VirtFS support no
VNC support yes
VNC TLS support no
VNC SASL support no
VNC JPEG support no
VNC PNG support no
VNC WS support no
xen support no
brlapi support no
bluez support no
Documentation no
GUEST_BASE yes
PIE yes
vde support no
netmap support no
Linux AIO support yes
ATTR/XATTR support yes
Install blobs yes
KVM support yes
RDMA support no
TCG interpreter no
fdt support no
preadv support yes
fdatasync yes
madvise yes
posix_madvise yes
sigev_thread_id yes
uuid support no
libcap-ng support no
vhost-net support yes
vhost-scsi support yes
Trace backends nop
spice support no
rbd support no
xfsctl support no
nss used no
libusb no
usb net redir no
GLX support no
libiscsi support no
libnfs support no
build guest agent yes
QGA VSS support no
seccomp support no
coroutine backend ucontext
coroutine pool yes
GlusterFS support no
Archipelago support no
gcov gcov
gcov enabled no
TPM support yes
libssh2 support no
TPM passthrough yes
QOM debugging yes
vhdx no
Quorum no
lzo support no
snappy support no
NUMA host support no

Is README right?

Hello ocssd experts,

I was using qemu-nvme version 1.2 for some reasons, but I decided to try 2.0. So I was following the steps written on readme file, in this repo, and I can't step forward now.
I have installed qemu, and I made backend file(8GB), and vm image file correctly. Then I used these options :
sudo $HOME/qemu-nvme/bin/qemu-system-x86_64 -m 4G -smp 4 -s \ -drive file={path to vm image},id=diskdrive,format=raw,if=none \ -device virtio-blk-pci,drive=diskdrive,scsi=off,config-wce=off,x-data-plane=on \ -drive file={path to ocssd backend file},id=myocssd,format=raw,if=none \ -device nvme,drive=myocssd,serial=deadbeef,lnum_pu=4,lstrict=1,meta=16,mc=3
and the qemu is showing "no bootable device" message before running ubuntu.

If I recall correctly, I used -hda option for using vm image. Is the readme tutorial correct? If it does, why can't I go forward? I'd be glad if anyone let me know about the correct options.

Thank you.

nvme drive: Unable to set logical and physical sector size to 512 bytes

Hi,

I'm testing the qemu-nvme environment for a research project.
I would like to set the nvme controller to work with 512 bytes log and phy sector size.

This is my command line:
./qemu-system-x86_64
-vga std
-device ide-drive,drive=drv0,serial=0x0
-drive format=raw,file=ubuntu-minimal.img,id=drv0,if=none
-device nvme,drive=drv1,serial=0x1,physical_block_size=512,logical_block_size=512
-drive format=qcow2,file=test.qcow,id=drv1,if=none
-m 2048
-smp 2
--enable-kvm

However, what I get in the guest OS:

sudo fdisk -l /dev/nvme0n1
...
...
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (logical/physical): 4096 bytes / 4096 bytes
...

What am I missing?
Thanks for the help

chunk meta is never updated

Hi @MatiasBjorling,

I'm using kernel-ml-4.17.1-1.el7.elrepo.x86_64 on QEMU NVMe (6f2e658 on master).
Issues I observed:

chunk meta is never updated and nvme_exit() is never called.
vector chunk reset's mptr is not updated with chunk descriptors.
completion queue entry for vector commands doesn't have DW0/1 updated with LBA status.

Didn't you see these on your environment?
I think chunk meta update could be changed as like meta data update.
And it's not easy to estimate workaround for other two.

Thanks,
YT

Device doesn't work normally with latest commits

Hi.
I have used the latest qemu-nvme (commit=6e5dab678c305f22a5f845156b570967910b3484) and the latest version of pblk.for-4.13 (commit=e9eaf4b94311ee7517199c31c4704f0fa4cacf3f) from https://github.com/OpenChannelSSD/linux/commits/pblk.for-4.13.

When I run fio with the following command
#sudo fio -filename=/dev/mydevice -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=200M -numjobs=30 -runtime=10 -group_reporting -name=mytest.
the fio task runs normally but the result is a little weird:

Since I set direct=1, the IO requests should bypass page cache and IO buffer so there is no reason that the disk did not receive any request.

I also tried to create ext4 and mount a partition, and copying and reading file on the device is successful.

However, when I tried another fio configuration:
[global]
ioengine=libaio
direct=1
thread=1
runtime=30
size=300M
filename=/dev/mydevice

[write4k-rand]
stonewall
group_reporting
bs=4k
rw=randwrite
numjobs=2
iodepth=4

The result is

When I tried to create file system later, the dmesg showed:

And this time, writing to the mounted folder is not allowed.

The device can't be removed by nvme lnvm remove either.

Is the device working normally? Is my fio configuration wrong? Or any other problem?
Thank you so much!

How to run qemu with NVMe device emulation?

What options do I have to run to emulate a general NVMe device other than an Open-Channel SSD?

Or does it work as a general NVMe device if I do not create a target with Open-Channel SSD emulation?

Confusing definations of blocks/planes/pages

Hi,

While going through the code, I came across something that I found confusing. I hope to get some clarification here.

The comments in this file say that the divisions are made in the following format:
LUNS - BLOCKS - PLANES - PAGES - SECTORS

How do you define a PLANE in this case?

As per the conventional definition of a PLANE, I would expect the division to be as follows
LUNS - PLANES - BLOCKS - PAGES - SECTORS

Pretty sure that that context in which PLANE is used in the code is different. Could you please explain what it is? Would I be correct to assume that it is "minimum number of pages that have to be read/written in one request"? (this is again against the conventional definition of a PAGE and hence the confusion)

Thank you

_set_written: invalid block state(02) on ocssd12

my test program mainly test erase, write read functions by backend ioctl.
and my qemu is ocssd12 branch.
i run my qemu with parameters :

-device nvme,drive=myocssd,serial=deadbeef,lver=1,lba_index=3,nlbaf=5,lnum_ch=8,lnum_lun=4,lnum_pln=2,lsecs_per_pg=4,namespaces=1

and when i ran my test program on my qemu, errors happened:
"nvm_be_ioctl.c ioctl_vio Failed:cmd->vuser.status:0xfc"
"nvm_be_ioctl.c ioctl_vio Failed:cmd->vuser.result:0x2ff"
"nvm_be_ioctl.c ioctl_vio Failed:NVME_NVM_IOCTL_SUBMIT_VIO err:-1, errno : Input outpur error"

besides, on my qemu terminal, it also shows errors:
ppa:ch(1), lun(3) ...
_set_written: Invalid block state(02)
lnvm_rw: set written status failed

qemu-img: ocssd.img: Unknown file format 'ocssd'

Hello,
I'm trying to do a university course project using open-channel ssd. As I don't have a physical ocssd, I'm trying to configure a virtual one using qemu. After configuring and installing required things now I'm getting into this error after running the code
qemu-img create -f ocssd -o num_grp=2,num_pu=4,num_chk=60 ocssd.img
Error is:
qemu-img: ocssd.img: Unknown file format 'ocssd'
As I'm a total newbie, I would appreciate any help or guide on how to deal with this problem.
Thank you

Question: how fast the emulated device can be? or how to improve the latency/through put

Hi,
I am using the scripts like the below to launch my guest, where nvme.img is a raw disk img.

qemu-system-x86064 -m 2G \
                   -cpu SandyBridge \
                   -machine q35 \   
                   -hda ${SYSTEM_IMG} \
                   -net nic \
                   -net user,hostfwd=tcp::2222-:22 \
                   -drive file=${nvme.img},format=raw,if=none,id=drv0 \
                   -device nvme,drive=drv0,serial=foo0,lver=1,lba_index=3,nlbaf=5,namespaces=1,addr=06.0 \
                   -virtfs local,id=mdev,path=/home/lifeng/Workspace,security_model=none,mount_tag=Workspace \
                   -smp 4 \
                   -enable-kvm

In the guest, i use fio and it give me 55us second latency and around 70MB/s for 4kwrite:

sudo fio -filename=/dev/nvme0n1 -direct=1 -rw=write -bs=4k -size=100G -runtime=10 -group_reporting -name=test_write

Is there any approach to make the device behave more like the real device(in the sense of lower latency and higher bandwidth)?

I tried to create nvme.img in tmpfs(ramdisk), but still get the same performance.

Any suggestion is appreciated

Namespace with non-zero metadata size - rw access rejected

Hello OpenChannelSSD,

I've been using your fork of qemu-nvme for simulation of an NVMe device. I've noticed that configuring a device with non-zero metadata size causes subsequent access failures.

I've observed behaviour this whilst running quemu-nvme's NVMe device against the vanilla NVMe driver in Linux ******* 4.10.0-26-generic #30-Ubuntu SMP Tue Jun 27 09:30:12 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux.

My qemu start-up options are:
$ qemu-system-x86_64 -m 2048 -hda ./30GB.img -drive file=./4GB_oc.img,if=none,id=drv0 -device nvme,drive=drv0,serial=foo,id=drv0,nlbaf=4,extended=1,mc=3,meta=8,dpc=0x1f,dps=3 --enable-kvm -smp 2 -net nic -net user

If I attempt to read from /dev/nvme0n1, say using dd, I encounter a device I/O error. It all seems to stem from this check in the qemu's NVMe device layer:

static uint16_t nvme_rw_check_req(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd, NvmeRequest *req, uint64_t slba, uint64_t elba, uint32_t nlb, uint16_t ctrl, uint64_t data_size, uint64_t meta_size) { : : if (meta_size) { nvme_set_error_page(n, req->sq->sqid, cmd->cid, NVME_INVALID_FIELD, offsetof(NvmeRwCmd, control), ctrl, ns->id); return NVME_INVALID_FIELD | NVME_DNR; } : :
Removing the above if (meta_size) check seems to fix things. So far as I can tell, this check was originally inherited from Keith Busch's dev code but there seems to have been some ongoing work within this project's fork. Perhaps someone can confirm that removing this test is a sane thing to do.

Thanks,
Andrew Meir

nvme_blk_submit_io: potential memory management bug

I set the mccap parameter: -device nvme,drive=nvme01,serial=deadbeef,id=lnvm,lmccap=7, but found that it wasn't honored in the double reset sanity check in lnvm_chunk_set_free:

// in hw/block/nvme/lightnvm.c
if (!(params->mccap & LNVM_PARAMS_MCCAP_MULTIPLE_RESETS)) {
        return LNVM_INVALID_RESET | NVME_DNR;
    }

GDB reveals that params->mccap is set correctly in initialization phase, but somewhere in execution it is overwritten to other values, possibly caused by writes to random stack space.

Dr. Memory reports unaddressable read / write at nvme_blk_submit_io, introduced in commit bf40471:

~~Dr.M~~ Error #1: UNADDRESSABLE ACCESS of freed memory: writing 0x00007fc7b109ed30-0x00007fc7b109ed38 8 byte(s)
~~Dr.M~~ # 0 nvme_blk_submit_io.part.21               [hw/block/nvme/nvme.c:1117]
~~Dr.M~~ # 1 nvme_process_sq                          [hw/block/nvme/nvme.c:1091]
~~Dr.M~~ # 2 libpthread.so.0!longjmp_compat           [../sysdeps/x86/nptl/pt-longjmp.c:62]
~~Dr.M~~ # 3 qemu_coroutine_switch                    [util/coroutine-ucontext.c:222]
~~Dr.M~~ # 4 timerlist_run_timers                     [util/qemu-timer.c:574]
~~Dr.M~~ # 5 qemu_clock_run_all_timers                [util/qemu-timer.c:588]
~~Dr.M~~ # 6 main_loop_wait                           [util/main-loop.c:521]
~~Dr.M~~ # 7 main_loop                                [/home/yunchih/research/ocssd/qemu-nvme-debug/vl.c:1923]
~~Dr.M~~ # 8 main                                     [/home/yunchih/research/ocssd/qemu-nvme-debug/vl.c:4584]
~~Dr.M~~ Note: @0:00:05.278 in thread 26172
~~Dr.M~~ Note: 0x00007fc7b109ed30-0x00007fc7b109ed38 overlaps memory 0x00007fc7b109ed10-0x00007fc7b109eda0 that was freed here:
~~Dr.M~~ Note: # 0 replace_free                             [/home/travis/build/DynamoRIO/drmemory/common/alloc_replace.c:2707]
~~Dr.M~~ Note: # 1 nvme_rw_cb                               [hw/block/nvme/nvme.c:1023]
~~Dr.M~~ Note: # 2 dma_blk_cb                               [/home/yunchih/research/ocssd/qemu-nvme-debug/dma-helpers.c:147]
~~Dr.M~~ Note: # 3 dma_blk_io                               [/home/yunchih/research/ocssd/qemu-nvme-debug/dma-helpers.c:258]
~~Dr.M~~ Note: # 4 dma_blk_read                             [/home/yunchih/research/ocssd/qemu-nvme-debug/dma-helpers.c:276]
~~Dr.M~~ Note: # 5 nvme_blk_submit_io.part.21               [hw/block/nvme/nvme.c:1118]
~~Dr.M~~ Note: instruction: mov    %rax -> 0x20(%rbx)
~~Dr.M~~
~~Dr.M~~ Error #2: UNADDRESSABLE ACCESS of freed memory: reading 0x00007fc7b109ed90-0x00007fc7b109ed98 8 byte(s)
~~Dr.M~~ # 0 nvme_blk_submit_io.part.21               [hw/block/nvme/nvme.c:1096]
~~Dr.M~~ # 1 nvme_process_sq                          [hw/block/nvme/nvme.c:1091]
~~Dr.M~~ # 2 libpthread.so.0!longjmp_compat           [../sysdeps/x86/nptl/pt-longjmp.c:62]
~~Dr.M~~ # 3 qemu_coroutine_switch                    [util/coroutine-ucontext.c:222]
~~Dr.M~~ # 4 timerlist_run_timers                     [util/qemu-timer.c:574]
~~Dr.M~~ # 5 qemu_clock_run_all_timers                [util/qemu-timer.c:588]
~~Dr.M~~ # 6 main_loop_wait                           [util/main-loop.c:521]
~~Dr.M~~ # 7 main_loop                                [/home/yunchih/research/ocssd/qemu-nvme-debug/vl.c:1923]
~~Dr.M~~ # 8 main                                     [/home/yunchih/research/ocssd/qemu-nvme-debug/vl.c:4584]
~~Dr.M~~ Note: @0:00:05.285 in thread 26172
~~Dr.M~~ Note: 0x00007fc7b109ed90-0x00007fc7b109ed98 overlaps memory 0x00007fc7b109ed10-0x00007fc7b109eda0 that was freed here:
~~Dr.M~~ Note: # 0 replace_free                             [/home/travis/build/DynamoRIO/drmemory/common/alloc_replace.c:2707]
~~Dr.M~~ Note: # 1 nvme_rw_cb                               [hw/block/nvme/nvme.c:1023]
~~Dr.M~~ Note: # 2 dma_blk_cb                               [/home/yunchih/research/ocssd/qemu-nvme-debug/dma-helpers.c:147]
~~Dr.M~~ Note: # 3 dma_blk_io                               [/home/yunchih/research/ocssd/qemu-nvme-debug/dma-helpers.c:258]
~~Dr.M~~ Note: # 4 dma_blk_read                             [/home/yunchih/research/ocssd/qemu-nvme-debug/dma-helpers.c:276]
~~Dr.M~~ Note: # 5 nvme_blk_submit_io.part.21               [hw/block/nvme/nvme.c:1118]
~~Dr.M~~ Note: instruction: mov    0x80(%rbx) -> %rbx
~~Dr.M~~
~~Dr.M~~ Error #3: UNADDRESSABLE ACCESS: reading 0xf1fdf1fdf1fdf150-0xf1fdf1fdf1fdf158 8 byte(s)
~~Dr.M~~ # 0 dma_acct_start                           [/home/yunchih/research/ocssd/qemu-nvme-debug/dma-helpers.c:334]
~~Dr.M~~ # 1 nvme_blk_submit_io.part.21               [hw/block/nvme/nvme.c:1110]
~~Dr.M~~ # 2 nvme_process_sq                          [hw/block/nvme/nvme.c:1091]
~~Dr.M~~ # 3 libpthread.so.0!longjmp_compat           [../sysdeps/x86/nptl/pt-longjmp.c:62]
~~Dr.M~~ # 4 qemu_coroutine_switch                    [util/coroutine-ucontext.c:222]
~~Dr.M~~ # 5 timerlist_run_timers                     [util/qemu-timer.c:574]
~~Dr.M~~ # 6 qemu_clock_run_all_timers                [util/qemu-timer.c:588]
~~Dr.M~~ # 7 main_loop_wait                           [util/main-loop.c:521]
~~Dr.M~~ # 8 main_loop                                [/home/yunchih/research/ocssd/qemu-nvme-debug/vl.c:1923]
~~Dr.M~~ # 9 main                                     [/home/yunchih/research/ocssd/qemu-nvme-debug/vl.c:4584]

It seems that the blk_req structure is not allocated; I tried adding g_malloc0 or nvme_blk_req_new but Dr. Memory continued to report the same errors.
I tried GCC 10's static analyzer, but it didn't report issue on that file.

Update:
It seems that nvme_blk_req_destroy(blk_req) in nvme_rw_cb is the cause, nvme_rw_cb is callback to the dma read in nvme_blk_submit_io. Valgrind & Dr. Memory no longer reported unaddressable access at nvme_blk_submit_io after commenting out the deallocation; unfortunately, inadvertent modification of params->mccap persists.

QEMU will not boot

Hello,
I am currently using the following options to boot my VM.

sudo $HOME/qemu-nvme/bin/qemu-system-x86_64 -m 4G -smp 4 -s \
-vnc :4 \
-drive file={path to vm image},id=diskdrive,format=raw,if=none \
-device virtio-blk-pci,drive=diskdrive,scsi=off,config-wce=off,x-data-plane=on \
-drive file={path to ocssd backend file},id=myocssd,format=raw,if=none \
-device nvme,drive=myocssd,serial=deadbeef,lnum_pu=4,lstrict=1,meta=16,mc=3

This is what it looks like when I execute the command.

Are there any more options I need to add to get this to work?

Thank you.

making qemu-nvme fails with perl regex error

When making qemu-nvme, the following error happens.

Unescaped left brace in regex is illegal here in regex; marked by <-- HERE in m/^\@strong{ <-- HERE (.*)}$/ at /home/qemu-nvme/scripts/texi2pod.pl line 320. Makefile:470: recipe for target 'qemu.1' failed make: *** [qemu.1] Error 255

OS: Ubuntu 18.04.1 LTS
Perl version: V5.26.1

I think the problem is a literal { in a regular expression is deprecated and causes a syntax error in perl version 5.26

Command Arbitration not supported

Each SQ uses its own QEMU timer, meaning that each SQ is processed independently, most likely in a separate thread. So, command arbitration techniques like weighted round robin don't work as expected.
Is there a reason for this implementation choice or is this a bug?

MLC Flash Support in QEMU

Hi,

I wish to evaluate LightNVM with MLC Flash.

But, setting lfmtype device flag to any value other than 0 (SLC), QEMU returns following error:
nvme: Only SLC Flash is supported at the moment

Is there any plans to support MLC Flash in qemu-nvme anytime soon?

Thank You

No NVMe device created in Ubuntu-16.04

Hi,

Not sure if this project is still active but have a question about the usage. After compiling the code I start a VM like so as per the instructions:

./qemu-nvme/x86_64-softmmu/qemu-system-x86_64 -nographic -no-reboot -m 512M -smp cpus=1 --enable-kvm -kernel ./bzImage -drive file=./rootfs-target.img,if=ide,format=raw -blockdev ocssd,node-name=nvme01,file.driver=file,file.filename=ocssd.img -device nvme,drive=nvme01,serial=deadbeef,id=lnvm -append "console=ttyS0 root=/dev/sda rw panic=1 earlyprintk=serial,ttyS0,115200"

However, no NVMe device gets created:

root@target:~# fdisk -l
Disk /dev/sda: 3.7 GiB, 3914907648 bytes, 7646304 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

The recommended kernel parameters are enabled. Am I missing something?

The kernel related config:

$ grep NVM linux-5.4.2/.config
# NVME Support
CONFIG_NVME_CORE=m
CONFIG_BLK_DEV_NVME=m
CONFIG_NVME_MULTIPATH=y
CONFIG_NVME_FABRICS=m
CONFIG_NVME_RDMA=m
CONFIG_NVME_FC=m
CONFIG_NVME_TCP=m
CONFIG_NVME_TARGET=m
CONFIG_NVME_TARGET_LOOP=m
CONFIG_NVME_TARGET_RDMA=m
CONFIG_NVME_TARGET_FC=m
# CONFIG_NVME_TARGET_FCLOOP is not set
CONFIG_NVME_TARGET_TCP=m
# end of NVME Support
CONFIG_NVM=y
CONFIG_NVM_PBLK=m
# CONFIG_NVM_PBLK_DEBUG is not set
# CONFIG_BATTERY_BQ27XXX_DT_UPDATES_NVM is not set
CONFIG_RTC_NVMEM=y
CONFIG_NVMEM=y
CONFIG_NVMEM_SYSFS=y

Thanks

lnvm init error: <_set_written: Invalid block state(00)> on qemu

Hi,

I'm currently working on qemu to run pblk at pblk.20 and pblk.for-4.13-fixes, and there are some problems.

First of all, I tried to run pblk.20 code at the qemu-nvme, but when I run lnvm init -d nvme0n1 -m gennvm,
qemu shows an error like this.

_set_written: Invalid block state(00)
lnvm_rw: set written status failed
ppa: ch(0), lun(0), blk(0), pg(0), pl(0), sec(0)

So, I just thought that pblk.20 is not supported by qemu-nvme.
But, are there any solution for this?

Secondly, thus, I tried to run pblk.for-4.13-fixes on qemu too.
In this case, I successfully init the device with lnvm init.

But, after that, I got same error I asked before at linux repository of OpenChannelSSD.
(https://github.com/OpenChannelSSD/linux/issues/37)

Do you have any solution for this?

or, can I get linux version and qemu option for successful running at current state?

Thanks,
Hyukjoong

nvme device not found

Hi everyone,
I am trying to run the qemu-nvme, but facing configuration problems. Following are my configuration settings:
sudo x86_64-softmmu/qemu-system-x86_64 -hda /home/umar/qemu-nvme/ocssd_backend.img --enable-kvm -initrd /boot/initrd.img-4.16.0+ -m 16G -smp 12 -append "root=/dev/sda1"-kernel "/home/umar/linux/arch/x86_64/boot/bzImage" -drive file=/home/umar/blknvme,if=none,id=mynvme -device nvme,drive=mynvme,serial=deadbeef,namespaces=1,lmetasize=16,mdts=10,ldebug=0,lstrict=1,nlbaf=5,lba_index=3 -curses

Which results in BusyBox error i.e "ALERT! /dev/sda1 does not exist. Dropping to a shell!"

Similarly, i tried another recommended configuration by javigon from previous posts, which is as follows:
sudo ../qemu-nvme/x86_64-softmmu/qemu-system-x86_64 -m 16G -smp 12 -s -drive file=/home/home/umar/qemu-nvme/ocssd_backend.img,id=diskdrive,format=raw,if=none -device virtio-blk-pci,drive=diskdrive,scsi=off,config-wce=off,x-data-plane=on -drive file=/home/umar/blknvme,id=extradrive,format=raw,if=none -device nvme,drive=mynvme,serial=deadbeef,namespaces=1,lver=1,lmetasize=16,ll2pmode=0,nlbaf=5,lba_index=3,mdts=10,lnum_lun=4,lnum_pln=2,lsec_size=4096,lsecs_per_pg=4
-device ide-hd,drive=extradrive

Which pops-up with can error "can't find value 'mynvme' ".

I have also tried other configurations, but none of them works fine. Please suggest me, what is missing in the configuration.

Thank you

Qemu Development Environment kernel caused the QEMU virtual machine to crash

I have problems that I can start QEMU successfully by the command
./qemu-system-x86_64 -m 8192 -enable-kvm ubuntu.img -blockdev ocssd,node-name=nvme01,file.driver=file,file.filename=ocssd.img -device nvme,drive=nvme01,serial=deadbeef,id=lnvm -vnc :2 -net user,hostfwd=tcp::2222-:22 -net nic -kernel /home/kathy/linux/arch/x86_64/boot/bzImage -append root=/dev/sda1

The problem is that it always caused the QEMU virtual machine to crash
Only the mouse can move, but it cannot work,the entire window screen crashed

I have always modify the linux config file in physical PC

How to count the P/E cycles of an virtual OCSSD?

Hello, I'm surprised the so few issues. LigntNVM is a great tool.
Recently, I'd like to build an open-channel SSD environment to simulate my FTL. And I want to know if Virtual OCSSD can support P/E tracking or evaluating performance？

Fix compilation of "make check" with glib 2.46

From 5d80fa7ee8331bc947c08615c46609d198e15c81 Mon Sep 17 00:00:00 2001
From: Carl-Daniel Hailfinger <[email protected]>
Date: Wed, 5 Jul 2017 01:24:50 +0200
Subject: [PATCH] Fix compilation with glib 2.46

---
 tests/test-string-output-visitor.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/tests/test-string-output-visitor.c b/tests/test-string-output-visitor.c
index 101fb27..e047194 100644
--- a/tests/test-string-output-visitor.c
+++ b/tests/test-string-output-visitor.c
@@ -248,39 +248,39 @@ int main(int argc, char **argv)
 
     output_visitor_test_add("/string-visitor/output/int",
                             &out_visitor_data, test_visitor_out_int, false);
-    output_visitor_test_add("/string-visitor/output/int",
+    output_visitor_test_add("/string-visitor/output/inttrue",
                             &out_visitor_data, test_visitor_out_int, true);
     output_visitor_test_add("/string-visitor/output/bool",
                             &out_visitor_data, test_visitor_out_bool, false);
-    output_visitor_test_add("/string-visitor/output/bool",
+    output_visitor_test_add("/string-visitor/output/booltrue",
                             &out_visitor_data, test_visitor_out_bool, true);
     output_visitor_test_add("/string-visitor/output/number",
                             &out_visitor_data, test_visitor_out_number, false);
-    output_visitor_test_add("/string-visitor/output/number",
+    output_visitor_test_add("/string-visitor/output/numbertrue",
                             &out_visitor_data, test_visitor_out_number, true);
     output_visitor_test_add("/string-visitor/output/string",
                             &out_visitor_data, test_visitor_out_string, false);
-    output_visitor_test_add("/string-visitor/output/string",
+    output_visitor_test_add("/string-visitor/output/stringtrue",
                             &out_visitor_data, test_visitor_out_string, true);
     output_visitor_test_add("/string-visitor/output/no-string",
                             &out_visitor_data, test_visitor_out_no_string,
                             false);
-    output_visitor_test_add("/string-visitor/output/no-string",
+    output_visitor_test_add("/string-visitor/output/no-stringtrue",
                             &out_visitor_data, test_visitor_out_no_string,
                             true);
     output_visitor_test_add("/string-visitor/output/enum",
                             &out_visitor_data, test_visitor_out_enum, false);
-    output_visitor_test_add("/string-visitor/output/enum",
+    output_visitor_test_add("/string-visitor/output/enumtrue",
                             &out_visitor_data, test_visitor_out_enum, true);
     output_visitor_test_add("/string-visitor/output/enum-errors",
                             &out_visitor_data, test_visitor_out_enum_errors,
                             false);
-    output_visitor_test_add("/string-visitor/output/enum-errors",
+    output_visitor_test_add("/string-visitor/output/enum-errorstrue",
                             &out_visitor_data, test_visitor_out_enum_errors,
                             true);
     output_visitor_test_add("/string-visitor/output/intList",
                             &out_visitor_data, test_visitor_out_intList, false);
-    output_visitor_test_add("/string-visitor/output/intList",
+    output_visitor_test_add("/string-visitor/output/intListtrue",
                             &out_visitor_data, test_visitor_out_intList, true);
 
     g_test_run();
-- 
2.7.4

Unexpected slow speed of simulated nvme device

When I tried to use "dd" command or "cp" command to copy a file to a block device exposed by pblk on nvme device, the speed is extremely slow.

I'am not sure this is caused by my wrong configuration of nvme device, wrong command running on the device or some kind of shortcomings of the simulated device.

Here is my configuration of nvme device.
nvme -device nvme,drive=mynvme,serial=deadbeef,namespaces=1,lver=1,lmetasize=16,ll2pmode=0,nlbaf=5,lba_index=3,mdts=10,lnum_lun=4,lnum_pln=2,lsec_size=4096,lsecs_per_pg=4,lpgs_per_blk=512,lbbtable=bbtable.qemu,lmetadata=meta.qemu,ldebug=1

Here is how I expose the device.
sudo nvme lnvm init -d nvme0n1
sudo nvme lnvm create -d nvme0n1 --lun-begin=0 --lun-end=3 -n mydevice -t pblk

Here are my dd and cp commands:
dd if=/dev/zero | pv | sudo dd of=/dev/mydevice bs=1M count=144

dd if=/dev/zero of=test.file bs=1M count=100
cp test.file /dev/mydevice
pv test.file > sudo /dev/mydevice

Thank you so much for looking at my question.
Look forward to your reply.

make Command Showing errors

Hi,
I have Ubuntu 16.04 installed on Oracle VM VirtualBox and am unable to install qemu-nvme. After configuring the file and running the make command, I get the following error :

  CC      hw/block/nvme/lightnvm.o
hw/block/nvme/lightnvm.c: In function ‘lnvm_trace_rw’:
hw/block/nvme/lightnvm.c:33:49: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
         bufp += lnvm_lba_str(bufp, n, req->ns, ((uint64_t *) req->slba)[i]);
                                                 ^
hw/block/nvme/lightnvm.c: In function ‘lnvm_blk_setup’:
hw/block/nvme/lightnvm.c:170:34: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
                 blk_req->slba = ((uint64_t *) req->slba)[i];
                                  ^
hw/block/nvme/lightnvm.c:179:56: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
             if (!blk_req || (i > 0 && last_lba + 1 != ((uint64_t *) req->slba)[
                                                        ^
hw/block/nvme/lightnvm.c:180:63: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
                 uint64_t soffset = n->dialect.blk_idx(n, ns, ((uint64_t *) req-
                                                               ^
hw/block/nvme/lightnvm.c:188:34: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
                 blk_req->slba = ((uint64_t *) req->slba)[i];
                                  ^
hw/block/nvme/lightnvm.c: In function ‘lnvm_inject_write_err’:
hw/block/nvme/lightnvm.c:217:29: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
             uint64_t lba = ((uint64_t *) req->slba)[i];
                             ^
hw/block/nvme/lightnvm.c:228:28: error: passing argument 1 of ‘bitmap_set’ from incompatible pointer type [-Werror=incompatible-pointer-types]
                 bitmap_set(&req->cqe.res64, i, 1);
                            ^
In file included from /home/harsh/Desktop/qemu-nvme/include/hw/qdev-core.h:5:0,
                 from /home/harsh/Desktop/qemu-nvme/include/qom/cpu.h:23,
                 from ./trace/control-internal.h:13,
                 from ./trace/control.h:270,
                 from hw/block/nvme/trace.h:7,
                 from hw/block/nvme/lightnvm.c:5:
/home/harsh/Desktop/qemu-nvme/include/qemu/bitmap.h:231:6: note: expected ‘long unsigned int *’ but argument is of type ‘uint64_t * {aka long long unsigned int *}’
 void bitmap_set(unsigned long *map, long i, long len);
      ^
hw/block/nvme/lightnvm.c: In function ‘lnvm_post_cqe’:
hw/block/nvme/lightnvm.c:259:20: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
             g_free((void *) req->slba);
                    ^
hw/block/nvme/lightnvm.c: In function ‘lnvm_rw_check_write_req’:
hw/block/nvme/lightnvm.c:331:21: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
     uint64_t lba = ((uint64_t *) req->slba)[0];
                     ^
hw/block/nvme/lightnvm.c:338:30: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
         uint64_t next_lba = ((uint64_t *) req->slba)[i];
                              ^
hw/block/nvme/lightnvm.c:349:20: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
             lba = ((uint64_t *) req->slba)[i];
                    ^
hw/block/nvme/lightnvm.c: In function ‘lnvm_rw_check_read_req’:
hw/block/nvme/lightnvm.c:422:63: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
         uint16_t err = lnvm_rw_check_chunk_read(n, cmd, req, ((uint64_t *) req-
                                                               ^
hw/block/nvme/lightnvm.c: In function ‘lnvm_rw’:
hw/block/nvme/lightnvm.c:882:17: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
     req->slba = (uint64_t) g_malloc_n(nlb, sizeof(uint64_t));
                 ^
hw/block/nvme/lightnvm.c:896:40: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
             err = nvme_dma_read_sgl(n, (uint8_t *) req->slba, len, sgl, req);
                                        ^
hw/block/nvme/lightnvm.c:909:37: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
             nvme_addr_read(n, lbal, (void *) req->slba, len);
                                     ^
hw/block/nvme/lightnvm.c:912:10: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
         ((uint64_t *) req->slba)[0] = lbal;
          ^
hw/block/nvme/lightnvm.c: In function ‘lnvm_erase’:
hw/block/nvme/lightnvm.c:967:17: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
     req->slba = (uint64_t) g_malloc_n(nlb, sizeof(uint64_t));
                 ^
hw/block/nvme/lightnvm.c:970:33: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
         nvme_addr_read(n, lbal, (void *) req->slba, nlb * sizeof(void *));
                                 ^
hw/block/nvme/lightnvm.c:972:10: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
         ((uint64_t *) req->slba)[0] = lbal;
          ^
hw/block/nvme/lightnvm.c:979:61: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
         if (NULL == (cs = lnvm_chunk_get_state(n, req->ns, ((uint64_t *) req->s
                                                             ^
hw/block/nvme/lightnvm.c:983:52: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
         int err = lnvm_chunk_set_free(n, req->ns, ((uint64_t *) req->slba)[i], 
                                                    ^
hw/block/nvme/lightnvm.c:992:53: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
         sectr_idx = n->dialect.blk_idx(n, req->ns, ((uint64_t *) req->slba)[i])
                                                     ^
cc1: all warnings being treated as errors
/home/harsh/Desktop/qemu-nvme/rules.mak:69: recipe for target 'hw/block/nvme/lightnvm.o' failed
make: *** [hw/block/nvme/lightnvm.o] Error 1 ```


Is there a bug in the code or am I doing something wrong?

qemu-nvme in Docker

Hi, I set up a repository that packages this qemu-nvme environment in Docker container. People who want to have a quick try of the LightNVM or run SPDK applications on top of this emulation env, I think this tool is a good starting point.

https://github.com/ljishen/nvme-env/tree/master/docker/qemu-nvme

Any comments and suggestions will be greatly appreciated.

After I create a pblk device, it failed when I create file system on it.

mkfs -t ext4 /dev/pblkdev
it is always running without stoping.

make check fails

$ make check
[...]
  CC    tests/vhost-user-test.o
  LINK  tests/vhost-user-test
  CC    tests/qom-test.o
  LINK  tests/qom-test
GTESTER check-qtest-x86_64
blkdebug: Suspended request 'A'
blkdebug: Resuming request 'A'
main-loop: WARNING: I/O thread spun for 1000 iterations
main-loop: WARNING: I/O thread spun for 1000 iterations
Broken pipe
[vmxnet3][WR][vmxnet3_peer_has_vnet_hdr]: Peer has no virtio extension. Task offloads will be emulated.
/home/kubuntu/qemu-experimental/tests/Makefile:398: recipe for target 'check-qtest-x86_64' failed
make: *** [check-qtest-x86_64] Error 1

The broken pipe is actually what triggers the test failure. The vmxnet3 message is harmless.

bisection result:
First bad commit ff646a5
Author: Keith Busch [email protected] 2014-10-23 20:02:44
Committer: Keith Busch [email protected] 2014-10-23 20:02:44
Parent: 4bee77b (NVMe: Updates for google extension)
Parent: e40830a (Merge remote-tracking branch 'remotes/mdroth/tags/qga-pull-2014-10-22-tag' into staging)
Merge remote-tracking branch 'upstream/master'

Conflicts:
    hw/block/nvme.c
    hw/block/nvme.h

As part of this, I am removing all meta-data support. It's not well
supported in any immediatly available operating system, so the code is
not well tested.

Signed-off-by: Keith Busch <[email protected]>