INFO:tensorflow:run without loading checkpoint
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into ./result/model_WIDE_AND_DEEP_1647592077/model.ckpt.
INFO:tensorflow:Create incremental timer, incremental_save:False, incremental_save_secs:None
2022-03-18 16:28:14.639138: E tensorflow/core/framework/op_segment.cc:54] Create kernel failed: Invalid argument: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
ERROR:tensorflow:Prefetching was cancelled unexpectedly:
Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
2022-03-18 16:28:14.783644: E tensorflow/core/framework/op_segment.cc:54] Create kernel failed: Invalid argument: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
Exception in thread PrefetchThread-PrefetchRunner-0:
Traceback (most recent call last):
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/prefetch_runner.py", line 236, in run
run_fetch(*feed)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1287, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
ERROR:tensorflow:Prefetching was cancelled unexpectedly:
Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
2022-03-18 16:28:14.871604: E tensorflow/core/framework/op_segment.cc:54] Create kernel failed: Invalid argument: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
Exception in thread PrefetchThread-PrefetchRunner-2:
Traceback (most recent call last):
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/prefetch_runner.py", line 236, in run
run_fetch(*feed)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1287, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
ERROR:tensorflow:Prefetching was cancelled unexpectedly:
Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
2022-03-18 16:28:14.975041: E tensorflow/core/framework/op_segment.cc:54] Create kernel failed: Invalid argument: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
Exception in thread PrefetchThread-PrefetchRunner-1:
Traceback (most recent call last):
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/prefetch_runner.py", line 236, in run
run_fetch(*feed)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1287, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
ERROR:tensorflow:Prefetching was cancelled unexpectedly:
Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
2022-03-18 16:28:15.079552: E tensorflow/core/framework/op_segment.cc:54] Create kernel failed: Invalid argument: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
Exception in thread PrefetchThread-PrefetchRunner-5:
Traceback (most recent call last):
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/prefetch_runner.py", line 236, in run
run_fetch(*feed)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1287, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
ERROR:tensorflow:Prefetching was cancelled unexpectedly:
Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
2022-03-18 16:28:15.183314: E tensorflow/core/framework/op_segment.cc:54] Create kernel failed: Invalid argument: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
Exception in thread PrefetchThread-PrefetchRunner-6:
Traceback (most recent call last):
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/prefetch_runner.py", line 236, in run
run_fetch(*feed)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1287, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
ERROR:tensorflow:Prefetching was cancelled unexpectedly:
Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
2022-03-18 16:28:15.288156: E tensorflow/core/framework/op_segment.cc:54] Create kernel failed: Invalid argument: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
Exception in thread PrefetchThread-PrefetchRunner-3:
Traceback (most recent call last):
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/prefetch_runner.py", line 236, in run
run_fetch(*feed)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1287, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
ERROR:tensorflow:Prefetching was cancelled unexpectedly:
Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
2022-03-18 16:28:15.391508: E tensorflow/core/framework/op_segment.cc:54] Create kernel failed: Invalid argument: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
Exception in thread PrefetchThread-PrefetchRunner-4:
Traceback (most recent call last):
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/prefetch_runner.py", line 236, in run
run_fetch(*feed)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1287, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
ERROR:tensorflow:Prefetching was cancelled unexpectedly:
Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
Exception in thread PrefetchThread-PrefetchRunner-7:
Traceback (most recent call last):
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/prefetch_runner.py", line 236, in run
run_fetch(*feed)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1287, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]
INFO:tensorflow:loss = 0.6654865, steps = 1
INFO:tensorflow:Saving checkpoints for 1 into ./result/model_WIDE_AND_DEEP_1647592077/model.ckpt.
Using TensorFlow version 1.15.5
Checking dataset...
Numbers of training dataset is 8000000
Numbers of test dataset is 2000000
The training steps is 15625
The testing steps is 3907
Saving model checkpoints to ./result/model_WIDE_AND_DEEP_1647592077
Enable smart staged feature of DeepRec.
Traceback (most recent call last):
File "train_rebuild.py", line 673, in <module>
main()
File "train_rebuild.py", line 495, in main
checkpoint_dir, tf_config, server)
File "train_rebuild.py", line 375, in train
sess.run([model.loss, model.train_op])
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 911, in __exit__
self._close_internal(exception_type)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 949, in _close_internal
self._sess.close()
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1216, in close
self._sess.close()
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1384, in close
ignore_live_threads=True)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/six.py", line 718, in reraise
raise value.with_traceback(tb)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/prefetch_runner.py", line 236, in run
run_fetch(*feed)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1287, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Length for attr 'dtypes' of 0 must be at least minimum 1
; NodeDef: {{node prefetch_2/DataBufferPut}}; Op<name=DataBufferPut; signature=record: -> ; attr=container:string,default=""; attr=dtypes:list(type),min=1; attr=shared_name:string,default=""; attr=shared_capacity:int,default=1,min=1; attr=timeout_millis:int,default=1000,min=1; is_stateful=true>
[[prefetch_2/DataBufferPut]]