pynvme / pynvme

builds your own tests.

pynvme: builds your own tests.

Pynvme is an user-space NVMe test driver with Python API. It is an open, fast, and extensible solution for SSD developers and test engineers to build their own tests intuitively.

Features

1. access PCI configuration space
2. access NVMe registers in BAR space
3. send any NVMe admin/IO commands
4. support callback functions for NVMe commands
5. support MSI/MSIx interrupts
6. transparent checksum verification on every LBA
7. generates IO workload of high performance and low latency
8. support multiple namespaces
9. support multiple tests on different controllers
10. integrate with pytest
11. integrate with VSCode to display cmdlog in GUI
12. support NVMe over TCP targets
13. HMB: Host Memory Buffer (PyNVMe3 only)
14. ZNS: Zoned Namespace (PyNVMe3 only)
15. TCG: security features (PyNVMe3 only)
16. Enterprise and Data Center SSD features (PyNVMe3 only)
17. extensive test scripts (PyNVMe3 only)

Links

Repositories

• GitHub: https://github.com/pynvme/pynvme
• Mirror: https://gitee.com/pynvme/pynvme

Documents

• Web: https://pynvme.readthedocs.io/
• PDF: https://pynvme.readthedocs.io/_/downloads/en/latest/pdf/

Presentation

• Introduction: pynvme builds your own tests.
• SPDK PRC Summit 2019, Beijing. Why SSD Developers Need Pynvme and Why Pynvme Needs SPDK. pdf, video.
• SDC2020. Pynvme: an Open, Fast and Extensible NVMe SSD Test Tool. pdf, video.

Contact

• Email: [email protected]

pynvme's Issues

Suggest controller reset option

Hi crane

I read spdk 19.10 release note

Spdk 19.10 support "no shutdown notification reset"

Could you support this feature in pynvme option?

Thank you

vscode plugin not work with some devices

Hi

Some enterprise essd have problem in visual studio code plug in queue window

These device print some hex code at device active (nvme0=d.Contoller(...)) unlike normal device
(After MSIX address print)
Device operation no problem. But not show at queue pair status window

How can i see all log at this time?

Thank you

Performance Gauge IOPS count issue

Hi Crane

Performance Gauge IOPS have some issue

Performance Gause IOPS count value 1/10 smaller than ioworker result

ex. ioworker io_count_read : 125964, but Performance Gauge value is 12K

thank you

power cycle with Quarch power module

@pytest.mark.parametrize("repeat", range(100))
def test_quarch(nvme0n1, subsystem, repeat):
    with nvme0n1.ioworker(io_size=8, time=10):
	time.sleep(5)
        subsystem.poweroff()

    subsystem.poweron()
    subsystem.reset()
    with nvme0n1.ioworker(io_size=8, time=5):
	pass

make test report error “failed to create the controller”

During pynvme installation, when I tried to execute "make test", error message displays as following:
-------------------------------- live log setup --------------------------------
[2019-08-01 12:47:15.155] INFO pciaddr(19): running tests on DUT 04:00.0
EAL: Cannot write command to PCI config space!
EAL: Cannot set up bus mastering!
EAL: Driver cannot attach the device (0000:04:00.0)
created ctrlr: 0
ERROR

==================================== ERRORS ====================================
_____________________ ERROR at setup of test_create_device _____________________

pciaddr = '04:00.0'

@pytest.fixture(scope="session")
def nvme0(pciaddr):

  ret = d.Controller(pciaddr.encode('utf-8'))

conftest.py:38:

---

driver_wrap.pyx:821: in nvme.Controller.cinit
self._create()

---

raise NvmeEnumerateError(f"fail to create the controller")
E nvme.NvmeEnumerateError: fail to create the controller

driver_wrap.pyx:836: NvmeEnumerateError
------------------------------ Captured log setup ------------------------------
INFO root:conftest.py:19 running tests on DUT 04:00.0
=========================== short test summary info ============================
ERROR driver_test.py::test_create_device - nvme.NvmeEnumerateError: fail to c...
SKIPPED [1] driver_test.py:47: nvme over tcp
====================== 1 skipped, 1 error in 0.14 seconds ======================
Makefile:100: recipe for target 'pytest' failed
cat test.log | grep "434 passed, 2 skipped" || exit -1
/bin/sh: 1: exit: Illegal number: -1
Makefile:103: recipe for target 'test' failed
make: *** [test] Error 2

Could you help to figure out this issue for me?
The whole log is
temp.txt
If you have any concern, please contact me.
Looking forward to you response, thanks.

Issue about SHN during power cycle

Hi crane

i found some issue about Shutdown Notify during power cycle

In power cycle test, SHN is set issue when power is turned on.

i changed spdk source in nvme_ctrlr.c line 116

opts -> no_shn_notification = false -> true

after change this. no more show SHN set during power on

plz check this

thank you

Some Question and Some Issue about reset

Hi crane

I have some question about multiple device test and some issue about reset

first, issue about reset

pcie reset and subsystem reset is not working correctly in my company pc(intel skylake) and server (dell r740xd) and my home pc (amd ryzen system)

all system report error in nvme_ctrlr.c (line 702 and line 728) during pcie reset

seems like systems can not read register.

after i comment out these lines, pcie reset and subsystem working correctly

second, question about multiple device test

Currently, one script can run a different workload on each ssd, but if one problem occurs a one ssd, all of them should stop

Is there a way to run the test(different script) on each ssd individually?

thank you

spdk 19.10

Controller.reset() when ioworkers are active

Hi.
Thank you make for superior nvme test tool
I test nvme reset during ioworker using this tool on intel desktop and dell server
But two system show different result.
Desktop issue segment fault after abort pending io but dell server continue test.
I want to know that available reset test during iowoker test
Version 1.3, 1.4 same result
Thank you

How many ioworker can run at the same time iowoker ?

Hi crane

I test pynvme in arm system now

Complile and running , works good!

But can not run over 32 iowoker

Is there a limit to iowoker that can be run at the same time?

Thank you

Question about iowoker performance at qdepth 2

Hi Crane.

Question about iowoker performance

i see iowoker performance data in presentation file

presendation show 4k qd=1,2 performance higher than fio

and i test 512b,4k qd=2 performance at iowoker, fio

but low qdepth performance lower than fio

What parts can you check?

i use fedora 32 and default kernel

OSError: [Errno 24] Too manay open files issue

Hi Crane

I use pynvme on kvm using passthrough
(It seems to be different depending on the type of server system, but it seems to work well on the host Ubuntu 18.04 LTS, Guest Cento7 combination.but have issue msix_isset not work, i will open this issue later)

Currently, all power cycle tests are performed before ioworker close, and close after power on. But when I test this many times, python gives too many file open error. Is there any way to handle a lot of open files during testing?

thank you

Request IOworker API supports different data pattern specify by user

Generally, I expected ioworker API could support different data pattern, which allowed set by user.

Below data patterns suggest to support but not limit to:

1. All zeros;
2. All ones;
3. User defined value; (ie, set 0x5A, buffer pattern is repeating 5A);
4. Random pattern;
5. Different entropy (%);
   and so on.

Thanks a lot!

issue about ioworker close in multi iowoker

Hi crane

i use multi iowoker for extreme stress

but somtimes some ioworker can not close. (detailed no going next step. seems like halt)

All Iowoker process ended at debug process window but not process close ioworker.

i see UserWarning aboit semaphore_tracker in process debug console in vscode

Are there any additional points I should consider to manage iowokers?

sorry too many question ^^;

ERROR nvme_ctrlr.c(2489) nvme_ctrlr_process_init: Initialization timed out in state 3

你好，我想请问下我在中断运行一个脚本的时候，手动中断后，再次运行其他脚本，就会出现error:
INFO driver.c(498) probe_cb: Attaching to NVMe Controller at 0000:01:00.0
[2020-03-02 11:19:07.377150] INFO intr_mgt.c(174) msix_intc_init: vector_num 32
[2020-03-02 11:19:57.234873] ERROR nvme_ctrlr.c(2489) nvme_ctrlr_process_init: Initialization timed out in state 3
python3: nvme_ctrlr.c:2489: nvme_ctrlr_process_init: Assertion `0' failed.
Fatal Python error: Aborted
我想请问下这个可能是什么原因导致的呢，我看百度上说的是SPDK模式的盘已经有进程绑定了，当前进程无法attach 到SPDK模式的盘。kill调所有使用SPDK模式盘的进程，重新拉起程序。但是我看没有这个进程在运行啊？

using shutdown_notify during ioworker

hi crane

i see subsystem.shutdown_notify() cant stop iowoker

if i send shutdown_notify during iowoker then occur csts.shst timeout

how about putting ioworker terminate function in shutdown_notify?

i try to modify driver_wrap.pyx add cleanup_driver in shutdown_notify

but sometimes(multi device testing) failed with ioworker_one_cb

thank you

Setup: alloc_pages_on_heap DPDK IOVA issue

Hello!

I'm trying to setup pynvme on Ubuntu 18.04 -- kernel 5.6.1.

After following both the automated install script, and the manual setup guidelines (separate times) I am repeatedly running into an issue after running ./identify_nvme SPDK test.

The error log is as follows:
EAL: alloc_pages_on_heap(): couldn't allocate memory due to IOVA exceeding limits of current DMA mask

EAL: alloc_pages_on_heap(): Please try initializing EAL with --iova-mode=pa parameter

error allocating rte services array

EAL: FATAL: rte_service_init() failed

EAL: rte_service_init() failed

Failed to initialize DPDK

Unable to initialize SPDK env

Please advise on how to debug the issue.

ioworker issue about sq tail over sq header

hi crane

i tested two ioworker with reset

nvme0n1=d.Namespace(nvme0,1,102410241024)
nvme0n1.verify_enable()
for loop in range(1000):
a=nvme0n1.ioworker(io_size=1,read_percentage=0,lba_random=True,time=20,qdepth=1023).start()
b=nvme0n1.ioworker(io_size=1,read_percentage=100,lba_random=True,time=20,qdepth=1023).start()
time.sleep(5)
nvme0.reset()
a.close()
b.close()

sorry i can not use github editor correctly

test is two qd1023 ioworker run and reset

but pynvme failed with sq tail over sq header message

this error occur aer

thank you

How can use backgorund operation?

Hi crane

I test some workload with trim(background)

I used Process function in multiprocessing module

But not work correctly.
(Maybe not working correctly multiprocess module)
How can i use background operation(ex. Python function) except ioworker?

Thank you

massive deploy with multiple drives and test runtimes

I suggest test mode for Individual test for each device

Some nvme test systems use a separate driver for each device

It would be nice if pynvme had a way to run different tests individually for each device.

Can I use pynvme for ubuntu 16.04 or 18.04?

Can I use pynvme for ubuntu 16.04 or 18.04?

NVMe spec 1.4

add new features and commands

Some questions about v1.8 candidate 1

Hi crane

I test ver 1.8 Candidate 1

I found add delay 2s to define _drive_cleanup

This delay effect to sudden power loss test.

Delay will give ssd time to process the accumulated data before powering down.

I see this case (fail at v1.7 , pass at v1.8 , fail at v1.8 delete delay)

And sometimes, after the ioworker terminates, the test does not proceed and stops.

It is not known exactly what the situation is, but it is a symptom not seen in 1.7.

Be careful not to get corona virus infection

Thank you

How to test multiple device?

Hi

I tried test multiple device using multi linux account

but only 1 pynvme operate available

how can i test multiple device at onece

i want to test mass devices

nvme.NvmeEnumerateError: fail to create the controller after subsystem.power_cycle()

Hi cranechu,
I am interested about nvme test, so I clone the latest pynvme to learn.
When I test about power cycle test, I got an error as title
Below is a detail about log.

=================================== FAILURES ===================================
___________________________ test_powercycle_by_sleep ___________________________

subsystem = <nvme.Subsystem object at 0x7fffe818c438>

    def test_powercycle_by_sleep(subsystem):
        # sleep system for 10 seconds, to make DUT power off and on
>       subsystem.power_cycle()

scripts/test_utilities.py:23: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
driver_wrap.pyx:382: in nvme.Subsystem.power_cycle
    ???
driver_wrap.pyx:638: in nvme.Controller._reinit
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   nvme.NvmeEnumerateError: fail to create the controller

driver_wrap.pyx:664: NvmeEnumerateError
------------------------------ Captured log setup ------------------------------
INFO     root:conftest.py:55 running tests on DUT 0000:0e:00.0
INFO     root:conftest.py:68 setup random seed: 0xd6c32b5b
------------------------------ Captured log call -------------------------------
INFO     root:test_utilities.py:23 power off nvme device for 10 seconds
INFO     root:test_utilities.py:23 power is back
---------------------------- Captured log teardown -----------------------------
INFO     root:conftest.py:70 test duration: 42.462 sec
=========================== short test summary info ============================
FAILED scripts/test_utilities.py::test_powercycle_by_sleep - nvme.NvmeEnumera...
============================== 1 failed in 42.56s ==============================
make[1]: Leaving directory `/home/sean/pynvme'

How I can fix it? Very thank you~~

issue about zns test

hi crane

i tested zns extension use qemu zns emulation driver

but not correctly work on zns driver

could you tell me which environment you tested?

issue when using in a virtual machine(qemu,vfio passthrough)

Hi crane

I am testing how to use pynvme with passthrough right now.

IOMMU+VFIO-PCI Passthrough is working correctly

But there are things that seem to be memory related issues.

@pytest.mark.parametrize("bs", [1,2,4,8,16,32,64,128,256])
@pytest.mark.parametrize("rw_ratio", [0,10,20,30,40,50,60,70,80,90,100])
def test_variable_bs_ratio(nvme0, nvme0n1,bs,rw_ratio):
    ioworks=nvme0n1.ioworker(io_size=bs, read_percentage=rw_ratio, time=600, lba_random=True,qdepth=1023).start().close()
    logging.info("%d, %d"%(ioworks.io_count_read, ioworks.io_count_write)
    logging.info("test complete")'

this script working good at real pc environment. but not in qemu guest

segmentation fault at test progress about 10~30%

the point at which a fault occurs is slightly different for each guest os, but it all happens.

i tested guest os centos 7, fedora 31/32 , ubuntu 19.10

are there any suspected problems in the virtual environment?

thank you

sync pynvme to SPDK v19.07

and DPDK v19.05

ioworker's parameter: io align and io stride

Hi crane

I have some issue about multiple iowoker

If i set multi iowoker to different io align and lba start. All ioworker have same workload

Iowoker a : io size :1 , io align :4 , start lba 0
Iowoker b: io size :1 , io align :4 , start lba 1
Iowoker c : io size :1 , io align :4 , start lba 2
Iowoker d : io size :1 , io align :4 , start lba 3

I expect

Iowoker a : 0,4,8,c
Iowoker b : 1,5,9,d
Iowoker c : 2,6,a,e
Iowoker d : 3,7,b,f

But all iowoker 0,4,8,c....

Please check this issue

Need expand size to pynvme vscode plugin

Hi crane

i have test pcie gen4 device

gen4 device bandwith over 4GB/s

but vscode plug in's performance gauge max number is 3600MB/s

So, the needle turns once and the number comes out as low as 4000

It looks like we need to change the maximum.

And are there any plans to support multi-devices in plugin? Will be very useful

Thank you

Qpair function problem at cc.en transition

Hi

Sorry~ i have too many question

i tested qpair create / delete(del qpair) => CC.EN transition (1->0->1)

but seems like qpair function still remaining in memory

qpair create (qpair=d.Qpair(nvme0,xx)) command to run SQ1,CQ1 create -> SQ1,CQ1 delete -> SQ1,CQ1 Create at 2nd loop

3rd loop more duplicate. SQ1,CQ1 create -> SQ1,CQ1 delete -> SQ1,CQ1 create -> SQ1,CQ1 delete -> SQ1,CQ1 create -> SQ1,CQ1 delete

but insted of using cc.en transition. use reset function no this issue.

Can pynvme support MSI interrupts mode?

Hi:
I am trying to use pynvme to test our nvme ssd .
when i use command :nvme.Controller(b'01:00.0’）,there is an assertion error :
driver.c: 637:probe_cb: INFO:Attaching to NVMe Controller at 0000:01:00.0
python3: driver.c:517: intc_find_msix: Assertion 'next offset !=0' failed.
Aborted (core dumped)

Our Nvme SSD suppurts MSI but does not support MSI-X .
Can pynvme support MSI interrupts mode ?

Thanks.

nvme over tcp

Question about d.crc32_unlock_all in driver_cleanup

Hi Crane

Significant delays are found when testing power cycles on servers with large amounts of memory.

After checking the log in driver_wrap, d.crc32_unlock_all has a delay of about 20 seconds.

On desktop pc testing the same nvme device, only a delay of 1 to 2 seconds occurs, but it takes a long time on a server with a large amount of memory.

So, I am skipping d.crc32_unlock_all right now, and what is d.crc32_unlock_all doing?

pynvme usage

1. I noticed that we can get admin command response by hook an callback funtion, but I saw the parameters are dw0 and status, how could I get other dw values in completion tree?
2. Could I abort an command with specified CID? abort implemented in pynvme is calling which library?
3. How could I parse buffer? smart_log.data(3, 1), what does 3 and 1 indicates?

dynamic qpair count

support qpair count as many as DUT

Admin queue pending failed after 39d54bf

Hi crane

Applied patch v1.9.1 (commit 93d54bf)

But iowoker failed with admin queue pending , process not found

Single iowoker no problem

However, if reset or power off occurs while running more than 10 iowokers, the above problem occurs.

No problem before

Thank you

SGL

Any plans to support SGLs?

Request API to set/get command timeout timer

Usually, we have some commands need specify command max execution time(ie, NVM_Fomat command maybe cost about 40s). So request this API to set command TO timer to meet requirement.

I also need get command TO timer API to get current command TO timer.

Thanks a lot!

corruption of ioworker test with 16T SSD

Hi Crane

I wonder if nvme capacity can affect ioworker stability.

i test 16T nvme drive on dell xeon sever(384GB). about simply bellow code

---

nvme0=d.Controller(b'0a:00.0')
nvme1=d.Controller(b'0b:00.0')

nvme0n1=d.Namespace(nvme0)
nvme1n1=d.Namespace(nvme1)

sub0=d.Subsystem(nvme0)
sub1=d.Subsystem(nvme1)

for loop in range(9999):
iowokers_a=nvme0n1.iowoker(io_size=random.randrange(1,64), read_percentage=0, qdepth=1023,time=30).start()
iowokers_b=nvme1n1.iowoker(io_size=random.randrange(1,64), read_percentage=100, qdepth=1023,time=30).start()

time.sleep(random.rangrange(5,15))

sub0.power_cycle()
sub1.power_cycle()
ioworkers_a.close()
ioworkers_b.close()

---

(I replaced the quanch of the power cycle code with labjack. works great)

When I tested two 1T nvme with the above code, no problem occurred(about 1000 cycle) but when I test two 16T nvme, I often get a segment fault or python abort in the python multi thread module

Do I need another fix to test large nvme?

thank you

multiple namespaces support

trace checksum table for different namespaces.

test case: 2 namespaces in 2 controllers.

Assertion error attaching to NVMe Controller

I am hitting assertion error in driver.c when trying to run scripts/demo_test.py. Here is the output:

[root@localhost pynvme]# python3 -m pytest scripts/bw_test.py::test_trim_basic --pciaddr=01:00.0 -s
Starting SPDK v19.01 / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: pynvme -c 0x8 -m 512 --base-virtaddr=0x200000000000 --file-prefix=spdk0 --proc-type=auto ]
EAL: Detected 12 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Multi-process socket /var/run/dpdk/spdk0/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
============================================================================================= test session starts =============================================================================================
platform linux -- Python 3.6.3, pytest-4.6.2, py-1.8.0, pluggy-0.12.0 -- /opt/rh/rh-python36/root/usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/ssdfw/brwat/pynvme, inifile: pytest.ini
collected 1 item

scripts/bw_test.py::test_trim_basic
----------------------------------------------------------------------------------------------- live log setup ------------------------------------------------------------------------------------------------
[2019-06-10 11:24:41.054] INFO pciaddr(19): running tests on DUT 01:00.0
driver.c: 637:probe_cb: *INFO*: Attaching to NVMe Controller at 0000:01:00.0
python3: driver.c:539: intc_init: Assertion `(table_offset&0x7) == 0' failed.
Aborted

It looks like I am failing attaching to the NVMe controller because the lower 3 bits of table_offset are not zero (table_offset = 4). I have confirmed I have the correct BFD.

Any ideas?

Note: I am trying to get things running on CentOS 7

Subsystem Reset issue ERROR nvme_ctrlr.c

Hi crane

I tested pynvme at 2 platform (1 desktop, 1 server)

but both system can not operate subsystem reset

pynvme occur error below
--desktop log--
ERROR nvme_ctrlr.c(728) nvme_ctrlr_shutdown: get_csys() failed
python3: nvme_ctrl.c:728: nvme_ctrlr_shutdown: Assertion 'false' failed
--server log--
INFO Driver.c(498) probe_cb: Attaching to NVMe Controller at 0000:xx:xx.x
ERROR nvme_pcie.c(488) nvme_pcie_ctrlr_map_cmb: get registers failed
python3: nvme_pcie.c:488: nvme_pcie_ctrlr_map_cmb: Assertion 'false' failed.

any sample can not subsystem reset.

i use master brance

move to CentOS8

use CentOS8 as recommended OS

Availble use ptype, pvalue in iowoker read?

Hi

I tested write(using ioworker) -> sanitize -> read(using ioworker) sequence

But data mismatch failure occurred during a read sequence
(seems like verify function not recognize sanitize operation)

Then i tried read iowoker using ptype=0, pvalue=0 for sanitized area data

But still failed verified data because read data 0x00

How can i resolve this situation?

Thank you

full NVMe spec support

• IO sq and cq
• prp and prp list

Reset and power cycle with qpair remaining

Hi Crane. some suggestion about process qpair

When the iowoker is running, the power cycle and reset are working well, so I'm doing a good test.

However, there is a problem that the test is stuck in the pending state at the moment when the qpair remains, and it does not proceed.

It is also necessary to forcibly discard the qpair during a reset and power cycle with the qpair remaining.

Many abnormal behaviors do not always appear to be accompanied by delete qpair operations.

thank you

fused operations

NVMe Command return status variable contains Phase Tag value

File a bug as a record for NVme CQE status variable issue.

When I get NVMe command return status field value, found status is always wrong, after confirm with Crane, CQE "status" variable contains Phase Tag indeed in pyNVMe code.

So, need workaround for high level function to ((status >> 1) & 0x7fff) for only getting Status Field Value.

Thanks a lot!

issue about EAL errors on dual-socket NUMA platform

hi crane

sometimes i see below erros at close ioworkers

EAL: failed to send to (/var/run/dpdk/spdk0/mp_socket_21381_a0284f9ff2a) due to No such file or directory
EAL: Fail to send request /var/run/dpdk/spdk0/mp_socket_21831_a0284f97ff2a:mp_malloc_sync
EAL: Fail to handle messages: mp_malloc_request
EAL: Wrong request ID
EAL: Request timed out
EAL: failed to send to (/var/run/dpdk/spdk0/mp_socket) due to Bad file descriptor
EAL: recvmsg failed, Bad file descriptor

i use test_read_multiple_devices_50hr in marathon_test.py

System is 2 xeon CPU, 80core, 384GB, target nvme device is 8TB 4ea

not happen always this issue. but this issue stop the test in vscode

can multi processor cause problem?