Reproducing PIL-SIMD about c1-action-recognition-tsn-trn-tsm

[willprice]

I have reproduced the environment described in the README without issue on a Ubuntu 18.04 machine. libjpeg shouldn't be installed (via the jpeg conda package) as the idea is to compile Pillow-SIMD against libjpeg-turbo (which is installed in the first step of the recipe). By using the conda packaged compiler we minimise some of the interactions with the host system. I have installed this environment in 4 different clusters without issue.

Can you first run

$ conda env export > conda-env.yml

and attach conda-env.yml to your response.

Then can you run

$ conda install -y -c conda-forge libjpeg-turbo gxx_linux-64
$ conda uninstall -y --force jpeg libtiff
$ pip uninstall -y pillow
$ export CXX=x86_64-conda-linux-gnu-g++
$ export CC=x86_64-conda-linux-gnu-gcc
$ CFLAGS="${CFLAGS} -mavx2" pip install --upgrade --no-cache-dir --force-reinstall --no-binary :all: --compile pillow-simd

recording the output of each step, this will help me help you debug the issue.

[willprice]

There seem to be a lot of compiler related env variables that I wouldn't expect to be set (e.g. CPP, CPPFLAGS, CXX, CXX_FOR_BUILD, GCC*, CMAKE_PREFIX_PATH, CMAKE_ARGS, CFLAGS, AR... the list goes on).

Try to compile in the cleanest env you can without all these vars. I only have a few CONDA_* env vars

[willprice]

BROWSER=chromium
_CE_CONDA=
_CE_M=
CONDA_DEFAULT_ENV=epic-ar-test
CONDA_EXE=/opt/anaconda3/bin/conda
CONDA_PREFIX=/home/willprice/.conda/envs/epic-ar-test
CONDA_PROMPT_MODIFIER=(epic-ar-test)
CONDA_PYTHON_EXE=/opt/anaconda3/bin/python
CONDA_SHLVL=1
EDITOR=vim
GOPATH=/home/willprice/code/go
HADOOP_PORT_JHS=8001
HADOOP_PORT_RM=8000
HOME=/home/willprice
JAVA_HOME=/usr/lib/jvm/default
LANG=en_US.UTF-8
LD_LIBRARY_PATH=/home/willprice/usr/lib64:
LESS_TERMCAP_mb=
LESS_TERMCAP_md=
LESS_TERMCAP_me=
LESS_TERMCAP_se=
LESS_TERMCAP_so=
LESS_TERMCAP_ue=
LESS_TERMCAP_us=
LOGNAME=willprice
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
MAIL=/var/mail/willprice
MANPATH=/usr/local/man:/usr/share/man
NVM_DIR=/home/willprice/.nvm
OLDPWD=/home/willprice
OMF_CONFIG=/home/willprice/.config/omf
OMF_PATH=/home/willprice/.local/share/omf
PATH=/home/willprice/.conda/envs/epic-ar-test/bin:/home/willprice/.pyenv/libexec/pyenv:/home/willprice/.pyenv/libexec:/home/willprice/.pyenv/shims:/home/willprice/.pyenv/bin:/opt/anaconda3/condabin:/home/willprice/.yarn/bin:/home/willprice/.cargo/bin:/home/willprice/.pyenv/bin:/home/willprice/.cargo/bin:/home/willprice/.local/bin:/home/willprice/.fzf/bin:/home/willprice/usr/bin:/home/willprice/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/default/bin
PWD=/home/willprice
PYENV_ROOT=/home/willprice/.pyenv
PYENV_SHELL=fish
PYTHONPATH=/home/willprice/ml/lib
SHELL=/bin/zsh
SHLVL=1
SSH_CLIENT=128.59.15.44 58114 22
SSH_CONNECTION=128.59.15.44 58114 128.59.8.159 22
SSH_TTY=/dev/pts/37
TERMINAL=urxvt
TERM=xterm-256color
THEFUCK_OVERRIDDEN_ALIASES=g,p
USER=willprice
VISUAL=vim
XDG_RUNTIME_DIR=/run/user/18275
XDG_SESSION_ID=9649
ZOTERO=/home/willprice/.zotero/zotero/ujsfxlfn.default/zotero/storage/

[willprice]

I've updated the instructions to take care of some irritating behaviour of conda that would trample files in the libjpeg-turbo package which might resolve your issue (i too had jpeg not found--conda remove the jpeglib.h header files from libjpeg-turbo after removing the jpeg lib)

[willprice]

What's the issue you are having?

[escorciav]

There was a compilation error at this step

CFLAGS="${CFLAGS} -mavx2" pip install --upgrade --no-cache-dir --force-reinstall --no-binary :all: --compile pillow-simd

reporting jpeg not found.

I also tried reinstalling jpeg and libtiff, but it didn't work.

TBH, I'm curious if one has done in another machine. Sometimes we have libraries in our system, and binaries get linked without knowing. You (or your team) have used docker, so you know what I'm talking about.

[escorciav]

Thanks 4 the quick reply. I'm truly sorry for the delay. I have been rushing with other stuff.

The YAML files and LOG are here

Tested in a different machine with Ubuntu 16.04

Shall I try docker?

[willprice]

Could you also give me the output of env when you have your conda environment activated? Be careful that this doesn't contain any sensitive information (e.g. aws keys etc), drop out any of that stuff.

[escorciav]

Sorry for the delay.

env is here

TBH, I plan to use the code with SIMD in the meantime. Yet, it would be great to see if it offers a speedup :)

[escorciav]

Interesting. I have no clue how those were set 😅

I try to keep my bashrc clean, and limit exports to this. That said, I moved to Ubuntu this year, and I have been installing whatever gets the job done.

I haven't had time to sit down and keep it lean as I used to have my fedora box 😄

[escorciav]

Can you pls share the env of a fresh shell after activating your environment for PIL-SIMD?

[escorciav]

Thanks. I forgot to reply to the email before.

I will report back as soon as I test it :). Likely when it's data loading is itching me, or I have so time. If someone else can test it, pls do!

[escorciav]

May I ask which ratio of improvement in speed-up you observed on your end?
Please mention the kind of storage that you have; HDD, SSD, SSD-NVMe.

[willprice]

It's at least 2x as fast, I use NVMe ssds, otherwise reading files dominates the loading time

…

On Sat, 27 Feb 2021, 00:42 Victor Escorcia Castillo, < ***@***.***> wrote: May I ask which ratio of improvement in speed-up you observed on your end? Please mention the kind of storage that you have (HDD, SSD, SSD-NVMe) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHL4PKQ646HGWLYKN2H47DTBA5YZANCNFSM4UPTAQ7Q> .