If you’ve found a bug, or have a feature request, then please create an issue with the following information:
- Have I written custom code (as opposed to running examples on an unmodified clone of the repository) : No
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04) : 18.04
- TensorFlow installed from (our builds, or upstream TensorFlow) : pip
- TensorFlow version (use command below) : 1.13.1
- Python version : 3.6
- Bazel version (if compiling from source) :
- GCC/Compiler version (if compiling from source) :
- CUDA/cuDNN version : 10 / 7
- GPU model and memory : Tesla V100 , 16GB
- Exact command to reproduce : ./bin/run-ldc93s1.sh
WARNING:tensorflow:From /home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:358: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/contrib/rnn/python/ops/lstm_ops.py:696: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 1334, in _do_call
return fn(*args)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key layer_1/bias not found in checkpoint
[[{{node save/RestoreV2}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 1276, in restore
{self.saver_def.filename_tensor_name: save_path})
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 929, in run
run_metadata_ptr)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 1152, in _run
feed_dict_tensor, options, run_metadata)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 1328, in _do_run
run_metadata)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/client/session.py”, line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key layer_1/bias not found in checkpoint
[[node save/RestoreV2 (defined at DeepSpeech.py:418) ]]
Caused by op ‘save/RestoreV2’, defined at:
File “DeepSpeech.py”, line 829, in
tf.app.run(main)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/platform/app.py”, line 125, in run
_sys.exit(main(argv))
File “DeepSpeech.py”, line 813, in main
train()
File “DeepSpeech.py”, line 418, in train
checkpoint_saver = tf.train.Saver(max_to_keep=FLAGS.max_to_keep)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 832, in init
self.build()
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 844, in build
self._build(self._filename, build_save=True, build_restore=True)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 881, in _build
build_save=build_save, build_restore=build_restore)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 513, in _build_internal
restore_sequentially, reshape)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 332, in _AddRestoreOps
restore_sequentially)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 580, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py”, line 1572, in restore_v2
name=name)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py”, line 788, in _apply_op_helper
op_def=op_def)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py”, line 507, in new_func
return func(*args, **kwargs)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/framework/ops.py”, line 3300, in create_op
op_def=op_def)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/framework/ops.py”, line 1801, in init
self._traceback = tf_stack.extract_stack()
NotFoundError (see above for traceback): Key layer_1/bias not found in checkpoint
[[node save/RestoreV2 (defined at DeepSpeech.py:418) ]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 1286, in restore
names_to_keys = object_graph_key_mapping(save_path)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 1591, in object_graph_key_mapping
checkpointable.OBJECT_GRAPH_PROTO_KEY)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py”, line 370, in get_tensor
status)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/framework/errors_impl.py”, line 528, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “DeepSpeech.py”, line 829, in
tf.app.run(main)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/platform/app.py”, line 125, in run
_sys.exit(main(argv))
File “DeepSpeech.py”, line 813, in main
train()
File “DeepSpeech.py”, line 436, in train
loaded = try_loading(session, checkpoint_saver, checkpoint_filename, ‘most recent’)
File “DeepSpeech.py”, line 353, in try_loading
saver.restore(session, checkpoint_path)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 1292, in restore
err, “a Variable name or other graph key that is missing”)
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Key layer_1/bias not found in checkpoint
[[node save/RestoreV2 (defined at DeepSpeech.py:418) ]]
Caused by op ‘save/RestoreV2’, defined at:
File “DeepSpeech.py”, line 829, in
tf.app.run(main)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/platform/app.py”, line 125, in run
_sys.exit(main(argv))
File “DeepSpeech.py”, line 813, in main
train()
File “DeepSpeech.py”, line 418, in train
checkpoint_saver = tf.train.Saver(max_to_keep=FLAGS.max_to_keep)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 832, in init
self.build()
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 844, in build
self._build(self._filename, build_save=True, build_restore=True)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 881, in _build
build_save=build_save, build_restore=build_restore)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 513, in _build_internal
restore_sequentially, reshape)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 332, in _AddRestoreOps
restore_sequentially)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/training/saver.py”, line 580, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py”, line 1572, in restore_v2
name=name)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py”, line 788, in _apply_op_helper
op_def=op_def)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py”, line 507, in new_func
return func(*args, **kwargs)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/framework/ops.py”, line 3300, in create_op
op_def=op_def)
File “/home/ubuntu/NewDeepSpeech/mypython/lib/python3.7/site-packages/tensorflow/python/framework/ops.py”, line 1801, in init
self._traceback = tf_stack.extract_stack()
NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Key layer_1/bias not found in checkpoint
[[node save/RestoreV2 (defined at DeepSpeech.py:418) ]]