(fetchenv2) snelders@dop263:~/testfetch/cand$ ls *h5 -l
-rw-r--r-- 1 snelders 9000 359255 nov 15 17:00 cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_13.92710.h5
-rw-r--r-- 1 snelders 9000 359751 nov 15 17:00 cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_16.81280.h5
-rw-r--r-- 1 snelders 9000 357159 nov 15 17:00 cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_18.66470.h5
predict.py -m h -g -1 -c .
2021-11-16 08:55:37.059175: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-11-16 08:55:40,352 - get_model -fetch.utils - INFO - Getting model h
2021-11-16 08:55:47.978135: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-11-16 08:55:47.978571: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-11-16 08:55:47.978598: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-11-16 08:55:47.978633: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (dop263): /proc/driver/nvidia/version does not exist
2021-11-16 08:55:47.978866: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-11-16 08:55:47.983296: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
Traceback (most recent call last):
File "/export/astron/snelders/anaconda3/envs/fetchenv2/bin/predict.py", line 4, in <module>
__import__('pkg_resources').run_script('fetch==0.2.0', 'predict.py')
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/pkg_resources/__init__.py", line 651, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1448, in run_script
exec(code, namespace, namespace)
File "/misc/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/fetch-0.2.0-py3.9.egg/EGG-INFO/scripts/predict.py", line 79, in <module>
model = get_model(args.model)
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/fetch-0.2.0-py3.9.egg/fetch/utils.py", line 92, in get_model
model = model_from_yaml(y.read())
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/saving/model_config.py", line 105, in model_from_yaml
return deserialize(config, custom_objects=custom_objects)
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/layers/serialization.py", line 173, in deserialize
return generic_utils.deserialize_keras_object(
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in deserialize_keras_object
return cls.from_config(
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py", line 2261, in from_config
return functional.Functional.from_config(
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 668, in from_config
input_tensors, output_tensors, created_layers = reconstruct_from_config(
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1275, in reconstruct_from_config
process_layer(layer_data)
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1257, in process_layer
layer = deserialize_layer(layer_data, custom_objects=custom_objects)
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/layers/serialization.py", line 173, in deserialize
return generic_utils.deserialize_keras_object(
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in deserialize_keras_object
return cls.from_config(
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py", line 2261, in from_config
return functional.Functional.from_config(
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 668, in from_config
input_tensors, output_tensors, created_layers = reconstruct_from_config(
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1275, in reconstruct_from_config
process_layer(layer_data)
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1257, in process_layer
layer = deserialize_layer(layer_data, custom_objects=custom_objects)
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/layers/serialization.py", line 173, in deserialize
return generic_utils.deserialize_keras_object(
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in deserialize_keras_object
return cls.from_config(
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/layers/core.py", line 1019, in from_config
function = cls._parse_function_from_config(
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/layers/core.py", line 1071, in _parse_function_from_config
function = generic_utils.func_load(
File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 457, in func_load
code = marshal.loads(raw_code)
ValueError: bad marshal data (unknown type code)
predict.py -m a -g -1 -c .
`2021-11-16 08:53:16.533682: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-11-16 08:53:21,303 - get_model -fetch.utils - INFO - Getting model a
2021-11-16 08:53:24.693684: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-11-16 08:53:24.695914: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-11-16 08:53:24.695944: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-11-16 08:53:24.695976: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (dop263): /proc/driver/nvidia/version does not exist
2021-11-16 08:53:24.696268: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-11-16 08:53:24.700161: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py:1905: UserWarning: `Model.predict_generator` is deprecated and will be removed in a future version. Please use `Model.predict`, which supports generators.
warnings.warn('`Model.predict_generator` is deprecated and '
2021-11-16 08:53:31.605966: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-11-16 08:53:31.607097: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2400190000 Hz
1/1 [==============================] - 5s 5s/step
(fetchenv2) snelders@dop263:~/testfetch/cand$ cat results_a.csv
,candidate,probability,label
0,./cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_13.92710.h5,1.0,1.0
1,./cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_16.81280.h5,1.0,1.0
2,./cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_18.66470.h5,1.0,1.0
I expected that all models would work or that none of the models would work.
Info about environment:
conda list
When I run all models I get the same message for models A, B, C, D, E, F, G, I, K. With these models the results_?.csv is also created correctly.
Models H and J produce the same error message and no results_?.csv is made.
Any ideas why this happens? For now I am just skipping models H and J, but I am curious why the other models would work and these wouldn't.