stonet-research / zns-tools Goto Github PK

There is still a memory error, and sometimes retrieving extents does not work and returns an error. Command run is from zns-contracts-bench zone-size workload

user@stosys:~/src/zns-tools$ sudo ./src/zns.segmap -d /mnt/f2fs
zns.segmap: malloc.c:2617: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.
Aborted

user@stosys:~/src/zns-tools$ sudo ./src/zns.segmap -d /mnt/f2fs/gc.0.0
Error: [main:740] retrieving extents for /mnt/f2fs/gc.0.0

For very large devices the dimensions are wrong on the heatmap.
Also, the differentiation between non existent values (where the heatmap has no respective zones because we the heatmp is always dimension^2 and number of zones may not be) and 0 values (e.g., 0 zone resets issued to a zone). With lower values it also becomes challenging to differentiate 0 and 1 values (these should have totally different colors).

On long running (large files) runs it may be beneficial to see how far we are. Can possibly show some progress based on the total number of extents (if we can get this with fcntl before starting) or accumulate the size of extents retrieved over the file size of the file that is being mapped (or of all files being mapped), and show this in a progress indicator.

Runs with many files can take very long. We can fix this by retrieving extents in parallel.
To avoid messes with parallel extent retrieval I would simply do each file in parallel and combine the retrieved extents in the end.
Other possible enhancements we can make to speed the processes up?

For such operations long options, where there will be a lot of output, it may be beneficial to always go to a file with the log unless specified otherwise?

Running zns.fiemap gives Floating exception.

How to reproduce:

sudo mkfs.f2fs -f -m -c /dev/nullb0 /dev/nullb1
sudo mount -t f2fs /dev/nullb1 /mnt/f2fs
sudo ./src/zns.fiemap -f /mnt/f2fs/db0/CURRENT

With RocksDB load:

 sudo ./db_bench --db=/mnt/f2fs/db0 --wal_dir=/mnt/f2fs/wal0 --benchmarks=overwrite,stats --use_direct_io_for_flush_and_compaction -value_size=1024 --key_size=64 --num=1000000 --target_file_size_base=$((1024*64)) --max_bytes_for_level_base=$((1024 * 1024 * 2))  --compression_type=none --threads=1 --max_bytes_for_level_multiplier=4 --num_levels=4

Can automate detection of ZNS by reading the superlock and identifying the devices. As is done in fsck.f2fs tool in init_sb_info().

To the segmap report we can add counters or a sort of summary on the segment/segment range on what was there. Similar to the hole summaries for fiemap. Number of files, number of extents, average sizes, etc.

Would be nice to have JSON output for analysis.

If the zone size == zone cap, then extents can map across zones (since they are still contiguous) but in the zones we show we have the ending lba of the extent past the zone going into the next zone

Currently all visualisations are in PDF format. Ideally this should be explorable interactively.
For example, like used in the Google trace format, so that it can be loaded in the browser.

The extents don't need to be sorted, just iterated through once then can directly be assigned to zones (i.e., some data structure for the zone mapping and its contents) and then dump the data structure as the report (no longer on the fly report printing).

Will also make json dumping the data a lot simpler, just dump the data structure.