Standalone Module impurity predictor and FastFilter.py throwing error about askcos

[thomasstruble]

@sparklytopaz will you post all of the versions of packages in your environment? As Max said the impurity predictor does not work with a single version of tensorflow.

1. The fast filter uses tf==2.x
2. The FW predictor uses tf==1.x

The models cannot be interchanged between versions so you cannot use the impurity predictor in one single environment. You will have to use Max's suggestion from above.

[mliu49]

HI @sparklytopaz, thanks for posting this issue. Unfortunately, there is not an easy solution at the moment. The error is caused by a difference in tensorflow versions used by the forward predictor (TF v1) and the fast filter (TF v2), so it will occur as long as both models are being loaded in the same process. However, the impurity predictor does work when running the ASKCOS website because the models are loaded by different workers.

If you need to run the impurity predictor as a python package, you could try running a tensorflow serving instance for the fast filter model and using it as the inspector:

docker run -d -p 8501:8501 askcos/fast-filter:1.0

from askcos_site.askcos_celery.treebuilder.tb_c_worker import FastFilterAPIModel
hostname = 'localhost'  # change as appropriate
inspector = FastFilterAPIModel(hostname, 'fast_filter').predict

[atharvapurdue]

@mliu49 Thanks !

[atharvapurdue]

Hi @mliu49
In fastfilter.py, the model loading path is set from the global config file.

Why is the 'trained_model_path' used instead of 'model_path' for fast filter?

Also trying to run fast_filter.py as standalone with model_path gives this error

 ValueError: Could not find matching function to call loaded from the SavedModel. Got:
      Positional arguments (3 total):
        * (<tf.Tensor 'inputs:0' shape=(None, 2048) dtype=float32>, <tf.Tensor 'inputs_1:0' shape=(None, 2048) dtype=float32>)
        * False
        * None
      Keyword arguments: {}
    
    Expected these arguments to match one of the following 4 option(s):
    
    Option 1:
      Positional arguments (3 total):
        * [TensorSpec(shape=(None, 2048), dtype=tf.float32, name='inputs/0'), TensorSpec(shape=(None, 2048), dtype=tf.float32, name='inputs/1')]
        * True
        * None
      Keyword arguments: {}
    
    Option 2:
      Positional arguments (3 total):
        * [TensorSpec(shape=(None, 2048), dtype=tf.float32, name='input_1'), TensorSpec(shape=(None, 2048), dtype=tf.float32, name='input_2')]
        * True
        * None
      Keyword arguments: {}
    
    Option 3:
      Positional arguments (3 total):
        * [TensorSpec(shape=(None, 2048), dtype=tf.float32, name='input_1'), TensorSpec(shape=(None, 2048), dtype=tf.float32, name='input_2')]
        * False
        * None
      Keyword arguments: {}
    
    Option 4:
      Positional arguments (3 total):
        * [TensorSpec(shape=(None, 2048), dtype=tf.float32, name='inputs/0'), TensorSpec(shape=(None, 2048), dtype=tf.float32, name='inputs/1')]
        * False
        * None
      Keyword arguments: {}

And trying to run it using trained_model_path gives this error

line 125, in <module>
    ff.load(model_path=gc.FAST_FILTER_MODEL['trained_model_path']) 
KeyError: 'trained_model_path'

[atharvapurdue]

What I am trying to do is run impurity predictor as a python package from template_free.py
@mliu49 Any suggestions for this modification are welcomed!
Thanks!

[atharvapurdue]

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
absl-py                   0.12.0                   pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
binutils_impl_linux-64    2.31.1               h6176602_1  
binutils_linux-64         2.31.1               h6176602_9    conda-forge
boost                     1.74.0           py39h5472131_3    conda-forge
boost-cpp                 1.74.0               hc6e9bd1_2    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2021.1.19            h06a4308_1  
cachetools                4.2.1                    pypi_0    pypi
cairo                     1.16.0            h6cf1ce9_1008    conda-forge
certifi                   2020.12.5        py39h06a4308_0  
chardet                   4.0.0                    pypi_0    pypi
cmake                     3.14.0               h52cb24c_0  
cycler                    0.10.0                     py_2    conda-forge
eigen                     3.3.7                hfd86e86_0  
expat                     2.3.0                h2531618_2  
flatbuffers               1.12                     pypi_0    pypi
fontconfig                2.13.1            hba837de_1004    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
gast                      0.4.0                    pypi_0    pypi
gcc_impl_linux-64         7.3.0                habb00fd_1  
gcc_linux-64              7.3.0                h553295d_9    conda-forge
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
google-auth               1.28.0                   pypi_0    pypi
google-auth-oauthlib      0.4.4                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
greenlet                  1.0.0            py39he80948d_0    conda-forge
grpcio                    1.34.1                   pypi_0    pypi
gxx_impl_linux-64         7.3.0                hdf63c60_1  
gxx_linux-64              7.3.0                h553295d_9    conda-forge
h5py                      3.1.0                    pypi_0    pypi
icu                       68.1                 h58526e2_0    conda-forge
idna                      2.10                     pypi_0    pypi
jpeg                      9d                   h36c2ea0_0    conda-forge
keras-nightly             2.5.0.dev2021032900          pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.3.1            py39h1a9c180_1    conda-forge
krb5                      1.18.2               h173b8e3_0  
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.33.1               h53a641e_7    conda-forge
libblas                   3.9.0                8_openblas    conda-forge
libcblas                  3.9.0                8_openblas    conda-forge
libcurl                   7.71.1               h20c2e04_1  
libedit                   3.1.20210216         h27cfd23_1  
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 9.3.0               h2828fa1_18    conda-forge
libgfortran-ng            9.3.0               hff62375_18    conda-forge
libgfortran5              9.3.0               hff62375_18    conda-forge
libglib                   2.68.0               h3e27bee_2    conda-forge
libgomp                   9.3.0               h2828fa1_18    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0                8_openblas    conda-forge
libopenblas               0.3.12          pthreads_h4812303_1    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libssh2                   1.9.0                h1ba5d50_1  
libstdcxx-ng              9.3.0               h6de172a_18    conda-forge
libtiff                   4.2.0                hdc55705_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libwebp-base              1.2.0                h7f98852_2    conda-forge
libxcb                    1.13              h7f98852_1003    conda-forge
libxml2                   2.9.10               h72842e0_3    conda-forge
lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
markdown                  3.3.4                    pypi_0    pypi
matplotlib-base           3.4.1            py39h2fa2bec_0    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
numpy                     1.19.5                   pypi_0    pypi
oauthlib                  3.1.0                    pypi_0    pypi
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openssl                   1.1.1k               h27cfd23_0  
opt-einsum                3.3.0                    pypi_0    pypi
pandas                    1.2.3            py39hde0f152_0    conda-forge
pcre                      8.44                 he1b5a44_0    conda-forge
pillow                    8.2.0            py39he98fc37_0  
pip                       21.0.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pkg-config                0.29.2               h1bed415_8  
protobuf                  3.15.7                   pypi_0    pypi
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycairo                   1.20.0           py39h08627d8_1    conda-forge
pymongo                   3.11.3           py39h2531618_0  
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
python                    3.9.2           hffdb5ce_0_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.9                      1_cp39    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
rdkit                     2021.03.1        py39hccf6a74_0    conda-forge
readline                  8.0                  he28a2e2_2    conda-forge
reportlab                 3.5.66           py39he59360d_0    conda-forge
requests                  2.25.1                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rhash                     1.4.1                h3c74f83_1  
rsa                       4.7.2                    pypi_0    pypi
scipy                     1.6.2                    pypi_0    pypi
setuptools                49.6.0           py39hf3d152e_3    conda-forge
six                       1.15.0             pyh9f0ad1d_0    conda-forge
sqlalchemy                1.4.5            py39h3811e60_0    conda-forge
sqlite                    3.35.4               h74cdb3f_0    conda-forge
tensorboard               2.4.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
tensorflow                2.5.0rc0                 pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
tf-estimator-nightly      2.5.0.dev2021032501          pypi_0    pypi
tk                        8.6.10               h21135ba_1    conda-forge
tornado                   6.1              py39h3811e60_1    conda-forge
tqdm                      4.60.0                   pypi_0    pypi
typing-extensions         3.7.4.3                  pypi_0    pypi
tzdata                    2021a                he74cb21_0    conda-forge
urllib3                   1.26.4                   pypi_0    pypi
werkzeug                  1.0.1                    pypi_0    pypi
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
wrapt                     1.12.1                   pypi_0    pypi
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.0                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
zstd                      1.4.9                ha95c52a_0    conda-forge

[thomasstruble]

Try using the versions in the requirements.txt file. There tensorflow==2.0.0 you will have to downgrade your python as well because I do not think tensorflow 2.0.0 is compatible with python 3.9. I have not problems running it with those versions. You are correct that you will have to change saved_model_path to model_path when loading the fast filter.
eg:

ff = FastFilterScorer()
ff.load(model_path=gc.FAST_FILTER_MODEL['model_path'])

[thomasstruble]

Even easier would be to simply use the preconfigured docker container. You would not have to mess with any versions. either pull the docker container docker pull askcos/askcos or build based on the instructions in the README

[atharvapurdue]

Thanks @thomasstruble !!
Yes the TF version was the issue for running fast filter independently. (It is now running properly)
Is the score given by fast filter (fast filter scorer - evaluate_reaction_score) and the score from inspector = FastFilterAPIModel(hostname, 'fast_filter').predict the same?
where the hostname is of the hosted docker container

docker run -d -p 8501:8501 askcos/fast-filter:1.0

[mliu49]

Yes, the FastFilterAPIModel(hostname, 'fast_filter').predict method returns the same score as the FastFilterScorer().evaluate_reaction_score method, and both use the same version of the fast filter model.