Error after loading the exported output_graph.pb

Hello,

I finetune the deep speech model using deepspeech-0.5.1 checkpoints. After fine-tuning. I used the below command to convert my checkpoints into output_grpah.pb file.

python DeepSpeech.py 
--export_dir ~/workspace/chib/model/DeepSpeech/newModels/newModel1/ 
--notrain 
--notest 
--checkpoint_dir /home/aml/.local/share/deepspeech/checkpoints/

The directory structure of the checkpoints is --checkpoint_dir is

drwxr-s— 2 aml aml 4096 Jul 16 10:57 ./
drwxr-s— 4 aml aml 4096 Jul 13 18:32 …/
-rw-r----- 1 aml aml 566747524 Jul 14 22:23 best_dev-125436.data-00000-of-00001
-rw-r----- 1 aml aml 1399 Jul 14 22:23 best_dev-125436.index
-rw-r----- 1 aml aml 2305921 Jul 14 22:23 best_dev-125436.meta
-rw-r----- 1 aml aml 566747524 Jul 15 14:59 best_dev-156795.data-00000-of-00001
-rw-r----- 1 aml aml 1399 Jul 15 14:59 best_dev-156795.index
-rw-r----- 1 aml aml 2305921 Jul 15 14:59 best_dev-156795.meta
-rw-r----- 1 aml aml 179 Jul 15 14:59 best_dev_checkpoint
-rw-r----- 1 aml aml 529 Jul 16 10:57 checkpoint
-rw-r----- 1 aml aml 566747524 Jul 14 21:28 train-123570.data-00000-of-00001
-rw-r----- 1 aml aml 1399 Jul 14 21:28 train-123570.index
-rw-r----- 1 aml aml 2305915 Jul 14 21:28 train-123570.meta
-rw-r----- 1 aml aml 566747524 Jul 14 21:38 train-124124.data-00000-of-00001
-rw-r----- 1 aml aml 1399 Jul 14 21:38 train-124124.index
-rw-r----- 1 aml aml 2305915 Jul 14 21:38 train-124124.meta
-rw-r----- 1 aml aml 566747524 Jul 14 21:48 train-124656.data-00000-of-00001
-rw-r----- 1 aml aml 1399 Jul 14 21:48 train-124656.index
-rw-r----- 1 aml aml 2305915 Jul 14 21:48 train-124656.meta
-rw-r----- 1 aml aml 566747524 Jul 14 21:58 train-125159.data-00000-of-00001
-rw-r----- 1 aml aml 1399 Jul 14 21:58 train-125159.index
-rw-r----- 1 aml aml 2305915 Jul 14 21:58 train-125159.meta
-rw-r----- 1 aml aml 566747524 Jul 14 22:04 train-125436.data-00000-of-00001
-rw-r----- 1 aml aml 1399 Jul 14 22:04 train-125436.index
-rw-r----- 1 aml aml 2305915 Jul 14 22:04 train-125436.meta
-rw-r----- 1 aml aml 566747524 Jul 16 10:21 train-249017.data-00000-of-00001
-rw-r----- 1 aml aml 1399 Jul 16 10:21 train-249017.index
-rw-r----- 1 aml aml 2305915 Jul 16 10:21 train-249017.meta
-rw-r----- 1 aml aml 566747524 Jul 16 10:31 train-249572.data-00000-of-00001
-rw-r----- 1 aml aml 1399 Jul 16 10:31 train-249572.index
-rw-r----- 1 aml aml 2305915 Jul 16 10:31 train-249572.meta
-rw-r----- 1 aml aml 566747524 Jul 16 10:41 train-250104.data-00000-of-00001
-rw-r----- 1 aml aml 1399 Jul 16 10:41 train-250104.index
-rw-r----- 1 aml aml 2305915 Jul 16 10:41 train-250104.meta
-rw-r----- 1 aml aml 566747524 Jul 16 10:51 train-250607.data-00000-of-00001
-rw-r----- 1 aml aml 1399 Jul 16 10:51 train-250607.index
-rw-r----- 1 aml aml 2305915 Jul 16 10:51 train-250607.meta
-rw-r----- 1 aml aml 566747524 Jul 16 10:57 train-250872.data-00000-of-00001
-rw-r----- 1 aml aml 1399 Jul 16 10:57 train-250872.index
-rw-r----- 1 aml aml 2305915 Jul 16 10:57 train-250872.meta

The output of my exporting command is

I Exporting the model...
WARNING:tensorflow:From /home/aml/anaconda3/envs/deepspeech-gpu/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /home/aml/anaconda3/envs/deepspeech-gpu/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/lstm_ops.py:696: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/aml/anaconda3/envs/deepspeech-gpu/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
WARNING:tensorflow:From /home/aml/anaconda3/envs/deepspeech-gpu/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:232: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
WARNING:tensorflow:From /home/aml/anaconda3/envs/deepspeech-gpu/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:245: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
I Models exported at /home/aml/workspace/chib/model/DeepSpeech/newModels/newModel1/

Now, I want to run the instance on the trained model using the below command

deepspeech --model ~/workspace/chib/model/DeepSpeech/newModels/newModel1/output_graph.pb
--alphabet ~/workspace/chib/model/DeepSpeech/deepspeech-0.5.1-models/alphabet.txt 
--lm ~/workspace/chib/model/DeepSpeech/deepspeech-0.5.1-models/lm.binary 
--trie ~/workspace/chib/model/DeepSpeech/deepspeech-0.5.1-models/trie 
--audio data/smoke_test/LDC93S1.wav

I am getting a below error

Loading model from file /home/aml/workspace/chib/model/DeepSpeech/newModels/newModel1/output_graph.pb
TensorFlow: v1.13.1-10-g3e0cc53
DeepSpeech: v0.5.1-0-g4b29b78
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2019-07-16 13:34:04.138600: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-16 13:34:04.357931: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel ('op: "UnwrapDatasetVariant" device_type: "CPU"') for unknown op: UnwrapDatasetVariant
2019-07-16 13:34:04.357974: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel ('op: "WrapDatasetVariant" device_type: "GPU" host_memory_arg: "input_handle" host_memory_arg: "output_handle"') for unknown op: WrapDatasetVariant
2019-07-16 13:34:04.357983: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel ('op: "WrapDatasetVariant" device_type: "CPU"') for unknown op: WrapDatasetVariant
2019-07-16 13:34:04.358090: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel ('op: "UnwrapDatasetVariant" device_type: "GPU" host_memory_arg: "input_handle" host_memory_arg: "output_handle"') for unknown op: UnwrapDatasetVariant
Loaded model in 0.222s.
Loading language model from files /home/aml/workspace/chib/model/DeepSpeech/deepspeech-0.5.1-models/lm.binary /home/aml/workspace/chib/model/DeepSpeech/deepspeech-0.5.1-models/trie
Loaded language model in 0.43s.
Running inference.
Error running session: Not found: PruneForTargets: Some target nodes not found: initialize_state 
Segmentation fault

Can you please help me out with this error Error running session: Not found: PruneForTargets: Some target nodes not found: initialize_state

Thanks in Advance …!!

What version of DeepSpeech.py did you used to perform export ?

I am using the 0.5.1 Version

you’re using the wrong version. i have to use 0.6.0a0[because i am on 0.6.0] for the models i trained and 0.5.0 for the pretrained ones.

i have the same error when i use 0.5.0 with the models i’ve trained.

That’s for the checkpoints and the libraries, but your error is consistent with your git checkout for DeepSpeech.py not on v0.5.1

I tried to use the this one , still getting the same error

Can you please tell how to check that git checkout for DeepSpeech.py is onv0.5.1?

Like, if you did git checkout v0.5.1 ?

I did git checkout v0.5.1 got below output

Note: checking out 'v0.5.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 4b29b78... Bump version to 0.5.1

And after this I did git status , got this output

# HEAD detached at v0.5.1
nothing to commit, working directory clean

Now, it is fine ? I can start with exporting the model ?

It seems so, yes, you should be able to do something now.