After removing a default setup for CONFIG_ZLIB_PATH from config.mk and wrapper.c, we executed dlopen with "" as parameter when just doing make install and not the rpmbuild, which has an overwrite for that setup variable. Since dlopen in its infinite wisdom returned a pointer other things were failing. I reintroduced the default for the config variable again, until we have a better idea how to correctly get this working.

I think it is best if we keep the dlopen/dlsym approach to comply with Fedoras Packaging rules. We should add proper dependencies into our spec files and ensure that the load path to the zlib.so is properly defined when building the packages, such that we avoid recursive loading and problems associated with this.

As per Debian Policy, binaries shipped in standard $PATH should not end with extensions. E.g. it should /usr/bin/tool, rather than /usr/bin/tool.sh

Furthermore shipping:
bin/zlib_mt_perf
bin/zlib_mt_perf.sh

is confusing. Please rename ".sh" scripts to be installed without an extension, and have a unique name. And possibly choose a better location to ship them at, e.g. unding /usr/lib, or possibly even /usr/share/doc/genwqe/.

Fixes required for:
bin/zlib_mt_perf.sh
bin/zlib_test_gz.sh

Currently ZLIB_INFLATE/DEFLATE_IMPL is set to 0x00 as default for our hardware accelerated zlib. This will cause software zlib to be used first. I would like to use 0x41 as default, the dictionary performance enhancement and hardware usage out of the box. And I would like to set ZLIB_CARD to -1 as default allowing the library to use all available cards. Just like a normal user would probably do it.

That would make the need of these environment variables obsolete in the standard use case. And whoever decides to use our libz.so should be really aware that this is a hardware accelerated version. What do you think?

We include cxl.h for old installations where there was no cxl.h in the kernel headers yet.
Those systems where, to make it even more complicated, in the lab without internet access.
So just downloading cxl.h from the net, as done in libcxl did not work either.

Since ppc64le distibutions nowadays have a copy of this required header file, we can remove this old circumvention.

By the way, if the file was in the kernel headers, the build would take that and not the shadow copy which I remove here.

I received the following contribution from Scott, who had obviously issues with the udev configuration. We had similar things already before, and I think it is time now to update our User's guide to preserve this info helping to setup the cxl device such normal users can access it.

I personally prefere a setup without the cxl group and just have 0666 and root as owner. Let's see.

Frank,

Your doc is a bit vague and sparse concerning udev rules, at least for ubuntu 16.04. In 16.04 the ref'ed dir is empty e.g.

root@smnorm:/etc/udev/rules.d# cd /etc/udev/rules.d/
root@smnorm:/etc/udev/rules.d# ls
root@smnorm:/etc/udev/rules.d#

Michael H used some of his instruction from the CAPI flash GT card to get non-root user to work, below is what he did. And that fixed the non-root user issue (not the path issues)

UDEV Rules / Device Permissions for /dev/cxl

Access to the CAPI device is restricted to members of the 'cxl' group. This configuration is controlled by udev rules, and triggered on reboot. To reconfigure this:

Ensure the 'cxl' group exists. Create it if it doesn't exist. 

useradd --system --user-group cxl

Create /lib/udev/rules.d/80-cxl.rules as root: 

sudo touch /lib/udev/rules.d/80-cxl.rules

Place the following in the 80-cxl.rules file: 

SUBSYSTEM=="cxl", ATTRS{mode}=="dedicated_process", GROUP="cxl", MODE="0660"
SUBSYSTEM=="cxl", ATTRS{mode}=="afu_directed", KERNEL=="afu[0-9]*.[0-9]m", OWNER="cxl", GROUP="cxl", MODE="0660"
SUBSYSTEM=="cxl", ATTRS{mode}=="afu_directed", KERNEL=="afu[0-9].[0-9]*s", OWNER="cxl", GROUP="cxl", MODE="0660"

Add your user to the "cxl" group, then log out / log in: 

adduser ubuntu cxl

Important

You must log out / log in to pick up new group membership for the currently-logged in user!

Retrigger udev to apply the correct permissions: 

udevadm trigger -s cxl

Scott Carroll

Inflate gave back wrong data in those cases. We cerated 228e2a7 to fix this. Use this new version if you depend on deflateCopy().

Software only:
haver@tul2eth3:~/genomics$ time -p ZLIB_INFLATE_IMPL=0x00 ZLIB_DEFLATE_IMPL=0x00 LD_PRELOAD=/opt/genwqe/lib/genwqe/libz.so:/opt/genwqe/lib/libcxl.so samtools sort -@41 -T NA1.bam.tmp -O bam -o NA1.sorted.bam NA1.bam
real 13.75
user 106.33
sys 1.31

GenWQE:
haver@tul2eth3:~/genomics$ time -p ZLIB_INFLATE_IMPL=0x61 ZLIB_DEFLATE_IMPL=0x61 ZLIB_ACCELERATOR=GENWQE LD_PRELOAD=/opt/genwqe/lib/genwqe/libz.so:/opt/genwqe/lib/libcxl.so samtools sort -@41 -T NA1.bam.tmp -O bam -o NA1.sorted.bam NA1.bam
real 10.97
user 16.43
sys 22.90

New CAPI lab version:
haver@tul2eth3:~/genomics$ time -p ZLIB_INFLATE_IMPL=0x61 ZLIB_DEFLATE_IMPL=0x61 ZLIB_ACCELERATOR=CAPI LD_PRELOAD=/opt/genwqe/lib/genwqe/libz.so:/opt/genwqe/lib/libcxl.so samtools sort -@41 -T NA1.bam.tmp -O bam -o NA1.sorted.bam NA1.bam
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
00018764.0001878a deflate.c:487: Error: outp_returned (0) invalid
hardware.c:651: Error: not all input absorbed!
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
00018764.0001877b deflate.c:487: Error: outp_returned (0) invalid
hardware.c:651: Error: not all input absorbed!
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
00018764.00018769 deflate.c:487: Error: outp_returned (0) invalid
hardware.c:651: Error: not all input absorbed!
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
00018764.00018766 deflate.c:487: Error: outp_returned (0) invalid
hardware.c:651: Error: not all input absorbed!
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
[zedc_deflate] What a pitty optimization did not work
00018764.00018764 deflate.c:754: Error: deflate failed rc=-402
DDCB returned (RETC=000 ATTN=0002 PROGR=1) ERR
hardware.c:651: Error: not all input absorbed!
00018764.00018780 deflate.c:754: Error: deflate failed rc=-402
DDCB returned (RETC=000 ATTN=0002 PROGR=1) ERR
hardware.c:651: Error: not all input absorbed!
00018764.0001877f deflate.c:754: Error: deflate failed rc=-402
DDCB returned (RETC=000 ATTN=0002 PROGR=1) ERR
hardware.c:651: Error: not all input absorbed!
^C
real 25.67
user 6.26
sys 2.40

Something seems not working right in our CAPI version. We need to investigate and fix this.

I sometimes get those messages. Eberhard seems to have a fix for this.

Vaibhav tried to cross compile our code. He failed badly:

vajain21@vajain21  ~/source/genwqe-user   master  make CROSS=powerpc64le-unknown-linux-gnu-
make[1]: Entering directory '/home/vajain21/source/genwqe-user/lib'
[CC] libzHW.o
[CC] libzHW.o
[CC] inflate.o
[CC] inflate.o
[CC] deflate.o
[CC] deflate.o
[LD] __libzHW.o
powerpc64le-unknown-linux-gnu-ld: unrecognised emulation mode: elf_x86_64
Supported emulations: elf64lppc elf32lppc elf32lppclinux elf32lppcsim
Makefile:77: recipe for target '__libzHW.o' failed
make[1]: *** [__libzHW.o] Error 1
make[1]: Leaving directory '/home/vajain21/source/genwqe-user/lib'
Makefile:97: recipe for target 'lib' failed
make: *** [lib] Error 1

I tried to repro this and failed even before:

[haverkam@oc7383187364 genwqe-user]$ make CROSS=/usr/local/gcc-4.8.0-nolibc/powerpc64-linux/bin/powerpc64-linux-
make[1]: Entering directory `/home/haverkam/GenWQE/genwqe-git/genwqe-user/lib'
[CC] libzHW.o
In file included from libzHW.c:23:0:
/usr/local/gcc-4.8.0-nolibc/powerpc64-linux/lib/gcc/powerpc64-linux/4.8.0/include/stdint.h:9:26: fatal error: stdint.h: No such file or directory

include_next <stdint.h>

                      ^

compilation terminated.
make[1]: *** [libzHW.o] Fehler 1
make[1]: Leaving directory`/home/haverkam/GenWQE/genwqe-git/genwqe-user/lib'
make: *** [lib] Fehler 1

IBM Tuleta systems can support up to 4 CAPI cards. genwqe_maint currently just supports 2.

Our library has an optimization to use the users own output buffer for inflate if it is exceeding the libraries internal output buffer. To use this efficiently the output buffer size needs to fulfill to criteria:

1. Be larger than the 128KiB lib internal buffer
2. Be as large as the compression ratio is compared to the input buffer

I saw good values with 128KiB input and 512KiB output buffer size. So I suggest to change the default buffer sizes for genwqe_gunzip according to that.

I am currently packaging genwqe-user to be uploaded to Debian.
In order to be able to upload a more recent version of it (without shipping cxl.h, with some resource leaks fixed, symlinks fixed etc), a new release tarball is needed.
The version available at IBM PPA has the fixed symlinks, but not the rest.

Would it be possible to have a new source tarball released, please? Thanks.

We saw cases where we might have detected a Z_STREAM_END condition falsely. The circumvention was put in place to support cases where the calling code offered exactly the amount of output bytes needed to fill the entire output buffer.

It expected furthermore Z_STREAM_END as return code to complete the action. Since our accelerator hardware stops processing once no output bytes are available, it needed to be called a 2nd time with some output bytes, but it would not fill those, but instead just return the occurrence of BFINAL in the data-stream. This 2nd call to hardware was not optimal because of:

1. The code mentioned above was not prepared to call the library a 2nd time, it expected the activity to be finished with 1 call only.
2. Calling the hardware added overhead which will impact performance when small data buffers are used, which was the case in this scenario.

The idea of the circumvention is to check via software if a DEFLATE block with BFINAL is in the remaining input data. The tricky thing is to cope with potentially buffered input data in the stream. The start of empty/last DEFLATE blocks can be hidden in the scratch area.

CFLAGS from environment are overridden and not honored by the makefiles. E.g. -O2 -fstack-protector-strong are not used, which are the default on Ubuntu.

If there is no GenWQE card available and the code is therefore required to do a software fallback, it fails when inflateSetDictionary is involved. We fall to sw, and when the data is big enough we try reenabling hw, but that does not work, so we fall back to sw again and on the way we mess up the state, since even with sw the total_in does not advance, even though 6 bytes got absorbed. That leads to the following bad situation:

### [0xfff879cc0c8] inflateInit2_: w=0xfff90a62510 windowBits=15 version=1.2.8/1.2.8 stream_size=112 impl=1 
hhh [0xfff879cc0c8] h_inflateInit2_: card_type=0 card_no=-1 zlib_obuf_total=131072
### [0xfff879cc0c8] __inflateInit2_: fallback to software (rc=-2)
### [0xfff879cc0c8] inflateInit2_: w=0xfff90a62510 windowBits=15 version=1.2.8/1.2.8 stream_size=112 impl=0
### [0xfff879cc0c8]    w=0xfff90a62510 strm->state=0xfff8dd02010
### [0xfff879cc0c8] inflate: avail_in=40000 bigger or equal 16384 switching to hardware mode!
### [0xfff879cc0c8] inflateGetDictionary: dictionary=0xfff879c3f60 dictLength=0xfff879cbf60 rc=0
### [0xfff879cc0c8] inflateInit2_: w=0xfff90a62510 windowBits=15 version=1.2.8/1.2.8 stream_size=112 impl=1
hhh [0xfff879cc0c8] h_inflateInit2_: card_type=0 card_no=-1 zlib_obuf_total=131072
### [0xfff879cc0c8] __inflateInit2_: fallback to software (rc=-2)
### [0xfff879cc0c8] inflateInit2_: w=0xfff90a62510 windowBits=15 version=1.2.8/1.2.8 stream_size=112 impl=0
### [0xfff879cc0c8]    w=0xfff90a62510 strm->state=0xfff8dd04010
### [0xfff879cc0c8] inflate:   flush=0 Z_NO_FLUSH next_in=0xfff909a0010 avail_in=40000 next_out=0xfff909ac010 avail_out=40000 total_in=0 total_out=0 crc/adler=00000001
### adler32(len=0)
### [0xfff879cc0c8]            flush=0 Z_NO_FLUSH next_in=0xfff909a0016 avail_in=39994 next_out=0xfff909ac010 avail_out=40000 total_in=0 total_out=0 crc/adler=08410215 rc=2 Z_NEED_DICT
### adler32(len=0)
### adler32(len=6)
### [0xfff879cc0c8] inflateSetDictionary: dictionary=0x2751bd58 dictLength=6 adler32=08410215 rc=0
### [0xfff879cc0c8] inflate: avail_in=39994 bigger or equal 16384 switching to hardware mode!
### [0xfff879cc0c8] inflateGetDictionary: dictionary=0xfff879c3f60 dictLength=0xfff879cbf60 rc=0
### [0xfff879cc0c8] inflateInit2_: w=0xfff90a62510 windowBits=15 version=1.2.8/1.2.8 stream_size=112 impl=1
hhh [0xfff879cc0c8] h_inflateInit2_: card_type=0 card_no=-1 zlib_obuf_total=131072
### [0xfff879cc0c8] __inflateInit2_: fallback to software (rc=-2)
### [0xfff879cc0c8] inflateInit2_: w=0xfff90a62510 windowBits=15 version=1.2.8/1.2.8 stream_size=112 impl=0
### [0xfff879cc0c8]    w=0xfff90a62510 strm->state=0xfff8dd06010
### [0xfff879cc0c8] inflateSetDictionary: dictionary=0xfff879c3f60 dictLength=6 adler32=08410215 rc=-2
### [0xfff879cc0c8] inflateEnd w=0xfff90a62510 rc=0
ERROR: inflate with dict error: -2ERROR: Test termination

We discovered that recently. Need to investigate why this happens. What I saw so far is the is_zedc() sometimes falsely reports not being a compression accelerator.

The GenWQE Linux device driver was unable to get a large enough memory area for its sglists.
Error report and fix is to be found here:

https://lkml.org/lkml/2015/9/29/681

This occurred only on System z yet. Once the code is in mainstream Linux, we can close this. For the Linux distros it will probably take a little longer to pick that up.

libzADC should be installed with a soname, into normal %{_libdir} location, with a soname.
Probably a matching soname, of the zlib library that is being mimicked.

genwqe_gzip
genwqe_gunzip
zlib_mt_perf

may not statically link libzADC.

We have currently undefined CONFIG_CIRCUMVENTION_FOR_Z_STREAM_END in lib/hardware.c. This impacts code which relies on the software zlib feature to return Z_STREAM_END once there is no ouput data anymore, e.g. our MongoDB experiment.

Software zlib parses input data containing empty DEFLATE blocks until either new data is produced or a BFINAL bit is seen. The current GenWQE hardware ends processing of input data once there is no output buffer space anymore. This causes that the library returns Z_OK with the intend to get more output space, before it finally returns Z_STREAM_END on a 2nd call to inflate but without producing any more data.

We disabled the circumvention, because we had some internal test-cases failing. We are working on a solution for this. Anyone requiring the circumvention to be active, consider re-enabling it for the sake of the experiment. Under normal circumstances it will work well.

===> Tested genwqe_gzip with a 108GB file:

[root@p8le03 home]# du -h test_file_harsha
108G test_file_harsha

[root@p8le03 home]# genwqe_gzip test_file_harsha

[root@p8le03 home]# genwqe_gzip test_file_harsha.gz -l
compressed uncompressed ratio uncompressed_name
2583984192 -620756992 29.67% test_file_harsha

[root@p8le03 home]# du -h test_file_harsha.gz
2.5G test_file_harsha.gz

ll foo

-rwxr-xr-x 1 root root 95349752 Sep 25 13:24 foo

src/genwqe-user/tools/genwqe_gzip foo

src/genwqe-user/tools/genwqe_gunzip foo.gz

ll foo

-rw-r--r-- 1 root root 95349752 Sep 25 13:24 foo

Eberhard, I think changing the url for the libcxl submodule to ssh instead of https is on the long run a problem. I used a fresh box and needed to go into .gitsubmodules and .git/config to fix this up.
Once you are back, let us discuss if we can change this back to https, such that non-authorized users can easily download our code.

haver@arlab174:~$ git clone --recursive https://github.com/ibm-genwqe/genwqe-user
Cloning into 'genwqe-user'...
remote: Counting objects: 1272, done.
remote: Total 1272 (delta 0), reused 0 (delta 0), pack-reused 1272
Receiving objects: 100% (1272/1272), 492.32 KiB | 0 bytes/s, done.
Resolving deltas: 100% (871/871), done.
Checking connectivity... done.
Submodule 'ext/libcxl' (ssh://[email protected]/ibm-capi/libcxl) registered for path 'ext/libcxl'
Cloning into 'ext/libcxl'...
The authenticity of host 'github.com (192.30.252.129)' can't be established.
RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? y
Please type 'yes' or 'no': yes
Warning: Permanently added 'github.com,192.30.252.129' (RSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Clone of 'ssh://[email protected]/ibm-capi/libcxl' into submodule path 'ext/libcxl' failed

RETC=102 is taken as confirmation that everything was fine. The enhance time echo took is not noticed anymore. We are working on a fix to see that we still loose interrupts in certain scenarios.

libzADC is underlinked, missing -ldl and -pthread

This is usually the case when there are unspecified but necessary dependencies between targets in the Makefile.

root@tul2eth3:~/genwqe-user# make -j
make[1]: Entering directory '/root/genwqe-user/lib'
make[1]: Entering directory '/root/genwqe-user/tools'
    [CFG]   zlib-1.2.8
make[2]: Entering directory '/root/genwqe-user/lib'
make[2]: Entering directory '/root/genwqe-user/lib'
make[2]: Entering directory '/root/genwqe-user/lib'
    [CC]    genwqe_gunzip.o
    [INFO]  Using ../../libcxl/libcxl.a.
    [CC]    libzHW.o
    [CC]    deflate.o
    [CC]    inflate.o
    [CC]    libcard.o
    [CC]    software.o
    [CC]    libddcb.o
    [CC]    ddcb_card.o
    [CC]    hardware.o
    [CC]    wrapper.o
    [CC]    ddcb_capi.o
    [INFO]  Using ../../libcxl/libcxl.a.
    [CC]    libzHW.o
    [CC]    inflate.o
    [CC]    deflate.o
    [INFO]  Using ../../libcxl/libcxl.a.
    [CC]    libcard.o
    [CC]    libddcb.o
    [CC]    libzHW.o
    [CC]    ddcb_card.o
    [INFO]  Using ../../libcxl/libcxl.a.
    [CC]    ddcb_capi.o
    [CC]    inflate.o
    [CC]    libzHW.o
    [CC]    inflate.o
    [CC]    wrapper.o
    [CC]    deflate.o
    [CC]    deflate.o
    [CC]    libcard.o
    [CC]    hardware.o
    [CC]    libcard.o
    [CC]    software.o
    [CC]    libddcb.o
    [CC]    ddcb_card.o
    [CC]    libddcb.o
    [CC]    ddcb_capi.o
    [CC]    ddcb_card.o
    [CC]    wrapper.o
    [CC]    wrapper.o
    [CC]    ddcb_capi.o
    [CC]    software.o
    [CC]    hardware.o
    [CC]    hardware.o
    [CC]    software.o
    [CC]    libddcb.o
    [CC]    libddcb.o
    [CC]    ddcb_card.o
    [CC]    ddcb_card.o
    [CC]    ddcb_card.o
    [CC]    ddcb_card.o
    [CC]    ddcb_capi.o
    [CC]    inflate.o
    [CC]    hardware.o
    [CC]    hardware.o
    [LD]    __libDDCB.o
    [CC]    libDDCB-4.0.6-3-g665b.so
    [CC]    libddcb.o
    [CC]    libddcb.o
    [CC]    libcard.o
    [LD]    __libcard.o
    [CC]    libcard-4.0.6-3-g665b.so
    [CC]    libzHW.o
    [CC]    libzHW.o
    [CC]    libzHW.o
    [CC]    genwqe_gunzip
gcc: error: ../lib/libzADC.a: No such file or directory
gcc: error: ../lib/libzHW.a: No such file or directory
Makefile:60: recipe for target 'genwqe_gunzip' failed
make[1]: *** [genwqe_gunzip] Error 1
make[1]: *** Waiting for unfinished jobs....
    [CC]    libzHW.o
    [CC]    deflate.o
    [CC]    deflate.o
    [CC]    deflate.o
    [CC]    deflate.o
    [LD]    __libzHW.o
    [CC]    libzHW-4.0.6-3-g665b.so
    [CC]    inflate.o
    [LD]    __libzHW.o
    [CC]    libzHW-4.0.6-3-g665b.so
    [CC]    ddcb_capi.o
    [LD]    __libDDCB.o
    [CC]    inflate.o
    [CC]    libDDCB-4.0.6-3-g665b.so
    [CC]    inflate.o
    [CC]    ddcb_capi.o
    [CC]    ddcb_capi.o
    [CC]    hardware.o
    [LD]    __libzHW.o
    [LD]    __libzHW.o
    [CC]    libzHW-4.0.6-3-g665b.so
    [CC]    libzHW-4.0.6-3-g665b.so
    [CC]    libcard.o
    [LD]    __libDDCB.o
    [LD]    __libDDCB.o
    [CC]    libDDCB-4.0.6-3-g665b.so
    [CC]    hardware.o
    [CC]    libDDCB-4.0.6-3-g665b.so
    [CC]    wrapper.o
    [LD]    __libcard.o
    [CC]    wrapper.o
    [CC]    libcard-4.0.6-3-g665b.so
    [CC]    wrapper.o
    [CC]    wrapper.o
    [CC]    software.o
    [CC]    software.o
    [LD]    __libzADC.o
software.o: file not recognized: File truncated
Makefile:116: recipe for target '__libzADC.o' failed
make[2]: *** [__libzADC.o] Error 1
make[2]: Leaving directory '/root/genwqe-user/lib'
Makefile:53: recipe for target '../lib/libDDCB.a' failed
make[1]: *** [../lib/libDDCB.a] Error 2
    [CC]    libcard.o
    [CC]    libcard.o
    [CC]    software.o
    [LD]    __libzADC.o
    [LD]    __libcard.o
    [LD]    __libcard.o
    [CC]    libcard-4.0.6-3-g665b.so
    [CC]    libcard-4.0.6-3-g665b.so
    [LD]    __libzADC.o
    [CC]    libzADC-4.0.6-3-g665b.so
__libzADC.o: file not recognized: File format not recognized
    [CC]    libzADC-4.0.6-3-g665b.so
collect2: error: ld returned 1 exit status
Makefile:125: recipe for target 'libzADC-4.0.6-3-g665b.so' failed
make[2]: *** [libzADC-4.0.6-3-g665b.so] Error 1
make[2]: Leaving directory '/root/genwqe-user/lib'
Makefile:53: recipe for target '../lib/libzHW.a' failed
make[1]: *** [../lib/libzHW.a] Error 2
make[2]: Leaving directory '/root/genwqe-user/lib'
make[1]: Leaving directory '/root/genwqe-user/tools'
Makefile:84: recipe for target 'tools' failed
make: *** [tools] Error 1
make: *** Waiting for unfinished jobs....
    [CC]    software.o
    [LD]    __libzADC.o
    [CC]    libzADC-4.0.6-3-g665b.so
make[1]: Leaving directory '/root/genwqe-user/lib'

We recently introduced building libcxl prior building the main code when doing rpm/deb packages. This has caused non-functional code because code was build with inconsistent settings.

Gabriel, you hardcoded the version number in the spec file. Is there a better way to solve that somehow?
I think I will adjust it manually for now, until we have a better idea.
The other thing is that I discovered that we do not need Eberhards genwqe_maint application, which is CAPI/System p specific. And the rpmbuild likes to install that and does not find it. I will fix that too somehow.

Makefile seems to be modified.
Try using make, and run again,
The code will be build once more.
Please fix.

We need a way to kickstart the genwqe_maint daemon whenever a new card is connected. a uevent triggering the code is a nice approach but genwqe_maint requires the card number so that needs to be handled.

Maybe a wrapper code in the daemon that keeps listening to uevents and starts the health check code when a new card is attached to the system?

Is there any information about the network ports on the EJ1B? I have installed the device and tested the software, all seems to be functional. Are the network ports on the EJ1B functional? If so where can I find documentation for configuration.

There was one corner case which was not considered correctly in the original fix de1be2a Avoid overwriting bytes in scratch buffer. Finally this commit 658862e Improve: Avoid overwriting bytes in scratch buffer (part2) should have fixed the issue entirely.

Maybe here: -%{_libdir}/genwqe/ ??

The new test case for the gz* functions uses functionality, which is not available in libz-1.2.7 as it is currently used for RHEL7.2. Adding some tweaks is needed to avoid breaking the compile. We can drop to the 32-bit offset versions of the critical functions, with the expected degradation in functionality.

We measured that the possible deflate bandwidth was not as expected when we used Z_FULL_FLUSH. We created the following patch to fix this:
7b32d99

ZLIB_FLAG_OMIT_LAST_DICT = 0x40

At this point in time we require that the ZLIB_FLAG_OMIT_LAST_DICT in the ZLIB_DEFLATE_IMPL environment variable must be set to activate this feature. So use this with:

export ZLIB_DEFLATE_IMPL=0x41

Thanks to Thilo for figuring this out.

We got the following suggestion from the Fedora team:
Including 3 spec files in one tarball in not necessary. It is much better to use a single spec file with subpackages.

NOTE: This does not apply for the GenWQE/PCIe variant of the code. It only affects the CAPI lab version.

haver@tul2eth3:/genwqe-user$ ZLIB_DEFLATE_IMPL=1 ZLIB_ACCELERATOR=CAPI /opt/genwqe/bin/zlib_mt_perf -i1MiB -o1MiB -D -f /tmp/test_data.bin -c1 -t16
DEFLATE Statistics
thread ; TID ; err ; #defl ; bytes ; time ; throughput ; checksum ; in/out
16 ; all ; ; 528 ; 536870912 ; 306467 msec ; 1751.802 MiB/sec ; 82a47161 ; 1024/1024 KiB
haver@tul2eth3:/genwqe-user$ ZLIB_DEFLATE_IMPL=1 ZLIB_ACCELERATOR=CAPI /opt/genwqe/bin/zlib_mt_perf -i1MiB -o1MiB -D -f /tmp/test_data.bin -c1 -t17
0000b525.0000b52d deflate.c:487: Error: outp_returned (0) invalid
hardware.c:651: Error: not all input absorbed!
zlib_mt_perf: zlib_mt_perf.c:259: defl: Assertion `ret != (-2)' failed.
Aborted
haver@tul2eth3:~/genwqe-user$

Circumvention until this is fixed: Use less than 17 threads only.

Since there is not yet a final fix for the problem, I created a flag such that we can disable it for those users which do not rely on it and who have problems with it:
d3aa752

csv2bin binary name is too generic. It looks as it it's one of the 2 tools that converts generic one format to another (e.g. ps2ascii ps2pdf ps2pdf13 ps2pdfwr ps2ps ps2txt ps2epsi ps2pdf12 ps2pdf14 ps2pk ps2ps2, etc.).

Is this actually a csv2vpd tool? Or it is specifically a genwqe-csv2vpd tool (only for genwqe cards)?

Please rename csv2bin to a more descriptive name.

Michael Neuling made me aware that there are user-space typedefs for endian related data structures.
See here:

grep -r be64 /usr/include/ | grep type
/usr/include/linux/types.h:typedef __u64 __bitwise __be64;
/usr/include/linux/types.h:#define __aligned_be64 __be64 __attribute((aligned(8)))
haver@tul2eth3:~$

We could consider using them to improve the code.

root@PC-Gzip:~/haver/genwqe-user# genwqe_test_gz -ACAPI
genwqe_test_gz: ERROR: gzip and gunzip tools not found in /usr/bin/genwqe.
Make sure the GenWQELibZ RPM Package is installed and gzip and gunzip
are executable for root in /usr/bin/genwqe.

Eberhart and I reviewed the code and found that we seem not properly handling EINT return codes in that file, especially when sitting in sem_wait(). We are not sure how often this really might happen, but fixing this is probably a good idea.

The direct loading did not work with the current deb packages. Consider removing the overwrite of that variable from the package build and use the new trick in config.mk to figure it out.

https://wiki.debian.org/ReproducibleBuilds/TimestampsFromCPPMacros

There is __DATE__ macro used in the code, which means it's impossible to reproduce the package build (ie. independently build the package again, and get identical binaries out with identical checksum hash).

Please consider removing __DATE__ macro, and instead e.g. use a full Version number, possibly with e.g. packaging version number that is with Debian Revision / RPM Revision&tag instead. That would still uniquely identify the build, but can be re-built without changing the output.

When trying to use CAPI card 1 it fails currently badly. My multicard support therefore fails too.
Test-command:
genwqe-user/tools/zlib_mt_perf.sh -A CAPI -C1 -t ../32MiB.bin

Via dmesg we get repeatedly the following messages:
...
[ 945.813825] [c000003ffff97a10] [c000000000129a68] generic_handle_irq+0x58/0x90
[ 945.813864] [c000003ffff97f90] [c000000000024840] call_do_irq+0x14/0x24
[ 945.813939] e8678110 4800d615 e8410018 <0fe00000> 3d020000 eba88118 e93d0000
[ 945.813946] ---[ end trace 8bd095fa3e51190e ]---
[ 945.813965] CXL SLICE ERROR interrupt 456
[ 945.813990] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.19.0-33-generic #38-Ubuntu
[ 945.813991] task: c000000001408a40 ti: c000003ffff94000 task.ti: c000000001468000
[ 945.814017] NIP [d00000001cb21bf4] cxl_slice_irq_err+0x264/0x290 [cxl]
[ 945.814037] [c000003ffff97a10] [c000000000129a68] generic_handle_irq+0x58/0x90
[ 945.814123] [c00000000146ba80] [c0000000000110a8] do_IRQ+0x98/0x140
[ 945.814151] [c00000000146bf90] [c000000000008c6c] start_here_common+0x20/0xa8
[ 945.814195] ------------[ cut here ]------------
[ 945.814216] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.19.0-33-generic #38-Ubuntu
[ 945.814220] REGS: c000003ffff975d0 TRAP: 0700 Tainted: G W (3.19.0-33-generic)
[ 945.814278] NIP [d00000001cb21bf4] cxl_slice_irq_err+0x264/0x290 [cxl]
945.814288
[ 945.814313] [c000003ffff97e40] [c0000000000b67bc] __do_softirq+0x1ac/0x3d0
[ 945.814375] [c00000000146be00] [c00000000010b5fc] cpu_startup_entry+0xdc/0x4e0
[ 945.814405] cxl afu0.0: PSL_FIR_SLICE_An: 0x0000000000000000
[ 945.814433] Modules linked in:
[ 945.814466] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.19.0-33-generic #38-Ubuntu
[ 945.814471] MSR: 9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28000444 XER: 20000000
[ 945.814520] NIP [d00000001cb21bf4] cxl_slice_irq_err+0x264/0x290 [cxl]
[ 945.814537] [c000003ffff978f0] [c00000000012a860] handle_irq_event_percpu+0x90/0x2b0
[ 945.814543] [c000003ffff97a10] [c000000000129a68] generic_handle_irq+0x58/0x90
[ 945.814563] [c00000000146ba80] [c0000000000110a8] do_IRQ+0x98/0x140
[ 945.814599] [c00000000146bf00] [c000000000dc3ee8] start_kernel+0x528/0x544
[ 945.814612] [c00000000146bf90] [c000000000008c6c] start_here_common+0x20/0xa8
[ 945.814649] cxl afu0.0: CXL_PSL_AFU_DEBUG_An: 0x0000000000000000
[ 945.814681] task: c000000001408a40 ti: c000003ffff94000 task.ti: c000000001468000
[ 945.814734] NIP [d00000001cb21bf4] cxl_slice_irq_err+0x264/0x290 [cxl]
[ 945.814737] LR [d00000001cb21bf0] cxl_slice_irq_err+0x260/0x290 [cxl]
[ 945.814740] Call Trace:
[ 945.814755] [c000003ffff97a70] [c000000000011114] do_IRQ+0x104/0x140
[ 945.814807] [c000003ffff97f90] [c000000000024840] call_do_irq+0x14/0x24
[ 945.814831] [c00000000146bee0] [c00000000000bdb4] rest_init+0xa4/0xc0
[ 945.814833] [c00000000146bf00] [c000000000dc3ee8] start_kernel+0x528/0x544
[ 945.814875] cxl afu0.0: PSL_FIR_SLICE_An: 0x0000000000000000
[ 945.814921] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.19.0-33-generic #38-Ubuntu
[ 945.814942] CFAR: c000000000002160 SOFTE: 0
GPR00: d00000001cb21bf0 c000003ffff97850 d00000001cb3def8 000000000000001d
GPR04: c000000ff8808400 c000000ff8818b28 0000000070757272 0000000002b8f51b
GPR08: c000000000eaa600 0000000000000000 0000000ff7960000 c000000000c98e58
GPR12: 0000000000000000 c00000000fb80000 0000000000000000 0000000000000000
GPR16: c000003ffff94000 c000000000eb9e80 c0000000014a2200 0000000ff96a0000
GPR20: 0000000000000000 0000000000000001 0000000000000002 00000000000001c8
GPR24: c000001ff22dbc00 0000000000000000 0000000000000000 0000000000000000
GPR28: c00000000146bb40 00000000000001c8 00000000000001c8 c000001ff42ba800
[ 945.815138] systemd-journald[1080]: /dev/kmsg buffer overrun, some messages lost.
[ 945.815156] CFAR: c000000000a46484 SOFTE: 0
GPR00: d00000001cb21bf0 c000003ffff97850 d00000001cb3def8 000000000000001d
GPR04: c000000ff8808400 c000000ff8818b28 0000000070757272 0000000002b8f548
GPR08: c000000000eaa600 0000000000000000 0000000ff7960000 c000000000c98e58
GPR12: 0000000000000000 c00000000fb80000 0000000000000000 0000000000000000
GPR16: c000003ffff94000 c000000000eb9e80 c0000000014a2200 0000000ff96a0000
GPR20: 0000000000000000 0000000000000001 0000000000000002 00000000000001c8
GPR24: c000001ff22dbc00 0000000000000000 0000000000000000 0000000000000000
GPR28: c00000000146bb40 00000000000001c8 00000000000001c8 c000001ff42ba800
[ 945.815354] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.19.0-33-generic #38-Ubuntu
[ 945.815385] NIP [d00000001cb21bf4] cxl_slice_irq_err+0x264/0x290 [cxl]
[ 945.815388] LR [d00000001cb21bf0] cxl_slice_irq_err+0x260/0x290 [cxl]
[ 945.815394] [c000003ffff978f0] [c00000000012a860] handle_irq_event_percpu+0x90/0x2b0

In addition to the already available documentation SuSE recommended to add a README file which contains information with links to the existing documentation. In particular how a user can identify a good use-case, and what to do if the performance is not as expected. The README should also contain a reference to the document describing which functions are supported and which not.

We have had a problem in the include and library search path setup preventing the alternate libz in /opt/genwqe/lib|include working out of the box.

Environment: SLES 12.1
Hardware: z13

We are using GENWQE built from GitHub, but the workload mongod hangs when loading it as follows.

zBAOLX1:~# LD_LIBRARY_PATH=/usr/lib/genwqe/:$LD_LIBRARY_PATH ZLIB_DEFLATE_IMPL=1 ZLIB_TRACE=0x0f mongod --storageEngine wiredTiger --wiredTigerCollectionBlockCompressor zlib --dbpath /sys/fs/cgroup/zEDC_output/mongodb

_init: BUILD= ZLIB_TRACE=f ZLIB_INFLATE_IMPL=1 ZLIB_DEFLATE_IMPL=1 ZLIB_INFLATE_THRESHOLD=16384

sss Try loading software zlib "/opt/genwqe/lib/libz.so"
sss Try loading system software zlib "libz.so"
sss loading [zlibVersion]
sss Using dlopened libz.so
sss ZLIB_VERSION 1.2.8 (header version)

But the version 3.0.25 Frank sent me last summer works well with zEDC. Can we get latest stable GENWQE binary? Thanks!

Best Regards,
Qin Yue Chen
Mainframe Performance Analyst, VPA Developer

When the ddcb execution request ioctl failed, we did not trigger health checking. This caused that we might still have failing cards in our list of available cards. And that might cause unsuccessful retries on those, causing plenty messages in the kernel about failing dma translations and so forth.

The following change triggers updating the card handle list:
41e2651

The current version of our code attempts to download zlib and link statically against it. Let us try to avoid this to be more compliant to Fedoras packaging guidelines listed below:

[1] https://fedoraproject.org/wiki/Packaging:Guidelines#Networking_Support (Build time network access)
[2] https://fedoraproject.org/wiki/Packaging:Guidelines#Bundling_and_Duplication_of_system_libraries
[3] https://fedoraproject.org/wiki/Packaging:Guidelines#Statically_Linking_Executable