nvmecompliance / dnvme Goto Github PK

Hi,

I am validating NVMe drive. For validation of that Drive, I have written the Host Test Cases(on Ubuntu 14.04) which calls/uses the DNVMe Low APIs.

Issue Found: Phase-bit toggle issue in IO queues

I have written the test case to validate the Phase-bit toggle condition. For this I have set the IOSQ and IOCQ q_size to 5. As NVMe specs expect it as 0's based value. There are six Entries to process by the Drive. But Host sends 5 commands only. And it expects wrap-around of the Queue at Fifth command only.

On further debug, I found following observations with dnvme Driver:

1. NVME_IOCTL_CREATE_ADMN_Q expects the 1 based for value for the No. Of Elements in the Admin Queues. But when it issues the Write to AQA register, it sends (No. Of Elements - 1) value. Confirmed on the PCIe Analyzer trace and Reading AQA Register in the FW.

For Example, if No. of elements value = 100, the ioctl(file_desc,NVME_IOCTL_CREATE_ADMN_Q,&aq_data ) writes the value 99 to the AQA register for that particular Q. This is what expected by the NVMe Drive. With Admin Queues, Q-Full and Phase toggle test case passes correctly.

2.But in case of NVME_IOCTL_PREPARE_SQ_CREATION &
NVME_IOCTL_PREPARE_CQ_CREATION, driver sends the 1 based value as it is (without decrementing it by 1). This actually conflicts with NVMe Spec. Because of this, On Q_Full condition the driver fails and phase-bit toggling in completion never happens. For this I have to do changes in my FW. i.e. I have to decrement the IO Q size by 1. Then Everything work fines.

Note: This issue can be replicated very easily, by writing the test case for Q-Full condition or to verify Phase-bit toggling. This Issue is with both IO queues only.
Regards,
Pratik

What changes do we need to run 128bit command using dnvme driver?
Little bit of insight will help alot.

dnvme ignores the command ID passed in the ASQ.
This is due to the following line in dnvme_ioctls.c, driver_send_64b()

nvme_gen_cmd->command_id = user_data->unique_id;

Comment out this line, and the correct command id will be returned in the ACQ

There is a bug in dnvme_irq.c at line 417. Here is the fix, thanks!

    if (irq_new->num_irqs > (1<<((mc_val & MSI_MME) >> 4))) {
        LOG_ERR("IRQs = %d exceed MSI MME = %d", irq_new->num_irqs,
            (1<<((mc_val & MSI_MME) >> 4)));
        /* does not support the requested irq's*/
        return -EINVAL;
    }

I compile the source and try the following steps of replacing the exisiting dnvme.ko with the newly generated one:

1. sudo rmmod dnvme
2. sudo make install
3. sudo modprobe

This sequence makes my /dev/dnvme0X00 (X is the bus number) disappear. However when I modprobe the old driver, everything becomes normal. Does this point to an issue with the generated dnvme.ko or am I doing it the wrong way?

Thanks

dnvme_queue.c
/* nvme_ctrlrdy_capto - This function is used for checking if the controller
is ready to process commands after CC.EN is set to 1. This will wait a
min of CAP.TO seconds before failing_/
int nvme_ctrlrdy_capto(struct nvme_device *pnvme_dev)
{
u64 timer_delay; /_ Timer delay read from CAP.TO register */
u64 ini, end;

/* Read in the value of CAP.TO */
timer_delay = ((readl(&pnvme_dev->private_dev.ctrlr_regs->cap)
    >> NVME_TO_SHIFT_MASK) & 0xF);               

// BUG:dnvme considering only 4 bits for CAP.TO but it should be 8
bits(NVMe Spec 1.0C or 1.0E Page 34).
// Code should be changed to

/* Read in the value of CAP.TO */
timer_delay = ((readl(&pnvme_dev->private_dev.ctrlr_regs->cap)
    >> NVME_TO_SHIFT_MASK) & 0xFF);

The dnevme driver compilation works fine but the driver fails to load.

kernel: 3.10.0-862.el7.x86_64

log snippet:
Nov 16 03:45:43 allnvmepl2 kernel: Call Trace:
Nov 16 03:45:43 allnvmepl2 kernel: [] dump_stack+0x19/0x1b
Nov 16 03:45:43 allnvmepl2 kernel: [] __warn+0xd8/0x100
Nov 16 03:45:43 allnvmepl2 kernel: [] warn_slowpath_fmt+0x5f/0x80
Nov 16 03:45:43 allnvmepl2 kernel: [] ? sysfs_warn_dup+0x6c/0x80
Nov 16 03:45:43 allnvmepl2 kernel: [] kobject_add_internal+0x274/0x330
Nov 16 03:45:43 allnvmepl2 kernel: [] ? kobject_set_name+0x4e/0x70
Nov 16 03:45:43 allnvmepl2 kernel: [] kset_register+0x46/0x70
Nov 16 03:45:43 allnvmepl2 kernel: [] __class_register+0xe0/0x250
Nov 16 03:45:43 allnvmepl2 kernel: [] __class_create+0x51/0x90
Nov 16 03:45:43 allnvmepl2 kernel: [] ? 0xffffffffc0b77fff
Nov 16 03:45:43 allnvmepl2 kernel: [] dnvme_init+0xa3/0x1000 [dnvme]
Nov 16 03:45:43 allnvmepl2 kernel: [] do_one_initcall+0xba/0x240
Nov 16 03:45:43 allnvmepl2 kernel: [] load_module+0x272c/0x2bc0
Nov 16 03:45:43 allnvmepl2 kernel: [] ? ddebug_proc_write+0xf0/0xf0
Nov 16 03:45:43 allnvmepl2 kernel: [] ? copy_module_from_fd.isra.43+0x53/0x150
Nov 16 03:45:43 allnvmepl2 kernel: [] SyS_finit_module+0xa6/0xd0
Nov 16 03:45:43 allnvmepl2 kernel: [] system_call_fastpath+0x1c/0x21
Nov 16 03:45:43 allnvmepl2 kernel: ---[ end trace 970d8c01bd61c12c ]---
Nov 16 03:45:43 allnvmepl2 kernel: dnvme-err:/opt/dnvme-master/sysdnvme.c:122: NVMe class creation failed