How to load downloaded checkpoint from 0.6.0

dimanshu.jatav · April 21, 2020, 8:26am

im using this command to load the check point
./DeepSpeech.py --train_files /home/dimanshu/mydatadeepspeech/youtube-course-1/train.csv --dev_files /home/dimanshu/mydatadeepspeech/youtube-course-1/dev.csv --alphabet_config_path /home/dimanshu/alphabet.txt --load_checkpoint_dir /home/dimanshu/deepspeech-0.6.1-models/output_graph.pbmm --save_checkpoint_dir /home/dimanshu/checkpoint/ --learning_rate 0.0001 --epochs 44 --export_dir /home/dimanshu/best_path/ --drop_source_layers 1 --train_batch_size 64 --train_cudnn true

but it is showing this output
I Could not find best validating checkpoint.
I Could not find most recent checkpoint.
I Initializing all variables.

lissyx · April 21, 2020, 8:36am

Have you read and understood what you are doing ?

You are mixing things here,

those flags are for current master, not 0.6.1
output_graph.pbmm is NOT a checkpoint directory, please download and extract 0.6.1 released checkpoint from the releases page

dimanshu.jatav · April 21, 2020, 12:59pm

thank you i solved that by downloading the checkpoint but now there is another problem . i’m training the model again with new data set and loading this checkpoint.

./DeepSpeech.py --train_files /home/dimanshu/mydatadeepspeech/youtube-course-1/train.csv  --alphabet_config_path /home/dimanshu/deepspeech-0.6.1-checkpoint/alphabet.txt --load
_checkpoint_dir ../deepspeech-0.6.1-checkpoint --train_cudnn true --epochs 5 --learning_rate 0.0001 --export_dir /home/dimanshu/best_path/  --save_checkpoint_dir ../checkpoint/

after 5 epochs it is giving error

I Loading variable from checkpoint: layer_5/weights/Adam
I Loading variable from checkpoint: layer_5/weights/Adam_1
I Loading variable from checkpoint: layer_6/bias
Traceback (most recent call last):
  File "./DeepSpeech.py", line 12, in <module>
    ds_train.run_script()
  File "/home/dimanshu/DeepSpeech/training/deepspeech_training/train.py", line 940, in run_script
    absl.app.run(main)
  File "/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/home/dimanshu/DeepSpeech/training/deepspeech_training/train.py", line 912, in main
    train()
  File "/home/dimanshu/DeepSpeech/training/deepspeech_training/train.py", line 511, in train
    load_or_init_graph_for_training(session)
  File "/home/dimanshu/DeepSpeech/training/deepspeech_training/util/checkpoints.py", line 132, in load_or_init_graph_for_training
    _load_or_init_impl(session, methods, allow_drop_layers=True)
  File "/home/dimanshu/DeepSpeech/training/deepspeech_training/util/checkpoints.py", line 97, in _load_or_init_impl
    return _load_checkpoint(session, ckpt_path, allow_drop_layers)
  File "/home/dimanshu/DeepSpeech/training/deepspeech_training/util/checkpoints.py", line 70, in _load_checkpoint
    v.load(ckpt.get_tensor(v.op.name), session=session)
  File "/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py", line 1033, in load
    session.run(self.initializer, {self.initializer.inputs[1]: value})
  File "/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1156, in _run
    (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (29,) for Tensor 'layer_6/bias/Initializer/zeros:0', which has shape '(86,)'

lissyx · April 21, 2020, 1:01pm

Feels like alphabet problem?

Please read documentation?

dimanshu.jatav · April 21, 2020, 1:05pm

do i have to change both alphabet.txt ? one which is made by me and second one which is available in released model ?
i have to make them common ?

lissyx · April 21, 2020, 1:06pm

How can I know ? It depends on what you want to achieve, I can’t do divination and know what you want …

lissyx · April 21, 2020, 1:08pm

And again, please read the documentation, the flags you are passing are NOT for 0.6.1: https://github.com/mozilla/DeepSpeech/blob/v0.6.1/util/flags.py

I have already told you that earlier.

dimanshu.jatav · April 21, 2020, 1:46pm

i read the documentation and tried utils/check_characters.py with command line arguments to create alphabet.txt
python util/check_characters.py -csv /home/dimanshu/mydatadeepspeech/youtube-course-1/train.csv -alpha
but whenever i run this it will give me different output .

lissyx · April 21, 2020, 2:05pm

If you continue to refuse to share more details than that, there’s nothing we can do.

lissyx · April 21, 2020, 2:06pm

I have asked you questions, if you don’t care to answer to them I won’t be able to help either.

dimanshu.jatav · April 21, 2020, 3:10pm

@lissyx i’m having some english dataset and i want to train this over existing checkpoint (deepspeech-0.6.1-checkpoint) but i don’t know how to edit alphabet.txt .
there are three alphabet.txt one is available in deespeech-mode/data/alphabet.txt , second one is available in checkpoint folder ( /deepspeech-0.6.1-checkpoint/alphabet.txt ) and third one is mine . i don’t know which one is to edit . my dataset is having english characters( lowed and smaller both) and special characters like ( + ,- …) i tried with
python util/check_characters.py -csv /home/dimanshu/mydatadeepspeech/youtube-course-1/train.csv -alpha
and i pasted the result to my alphabet.txt and when i started the training it is giving me the same shape error. i know its related to alphabet.txt then i again run python util/check_characters.py -csv /home/dimanshu/mydatadeepspeech/youtube-course-1/train.csv -alpha
and i noticed that everytime i run this it is giving me different output and it is saying to edit data/alphabet.txt but i want to check my alphabet.txt with my dataset.
sorry i’m taking too much time of yours i’m really sorry @lissyx

lissyx · April 21, 2020, 3:14pm

https://deepspeech.readthedocs.io/en/v0.6.1/TRAINING.html#continuing-training-from-a-release-model

dimanshu.jatav · April 21, 2020, 3:47pm

what about the alphabet.txt sir ?

lissyx · April 21, 2020, 3:54pm

Before transfer learning is available (current master), you cannot continue training with a different alphabet file.

dimanshu.jatav · April 21, 2020, 4:13pm

i used this

python3 DeepSpeech.py --n_hidden 2048 --checkpoint_dir ../deepspeech-0.6.1-checkpoint --epochs 3 --train_files my-train.csv --dev_files my-dev.csv --test_files my_dev.csv --learning_rate 0.0001 --train_cudnn true

now it is giving me this output

I Loading best validating checkpoint from ../deepspeech-0.6.1-checkpoint/best_dev-233784
I Loading variable from checkpoint: beta1_power
I Loading variable from checkpoint: beta2_power
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel/Adam
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel/Adam_1
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/bias/Adam
I Loading variable from checkpoint: layer_1/bias/Adam_1
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_1/weights/Adam
I Loading variable from checkpoint: layer_1/weights/Adam_1
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/bias/Adam
I Loading variable from checkpoint: layer_2/bias/Adam_1
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_2/weights/Adam
I Loading variable from checkpoint: layer_2/weights/Adam_1
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/bias/Adam
I Loading variable from checkpoint: layer_3/bias/Adam_1
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_3/weights/Adam
I Loading variable from checkpoint: layer_3/weights/Adam_1
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/bias/Adam
I Loading variable from checkpoint: layer_5/bias/Adam_1
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_5/weights/Adam
I Loading variable from checkpoint: layer_5/weights/Adam_1
I Loading variable from checkpoint: layer_6/bias
Traceback (most recent call last):
  File "DeepSpeech.py", line 12, in <module>
    ds_train.run_script()
  File "/home/dimanshu/DeepSpeech/training/deepspeech_training/train.py", line 940, in run_script
    absl.app.run(main)
  File "/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/home/dimanshu/DeepSpeech/training/deepspeech_training/train.py", line 912, in main
    train()
  File "/home/dimanshu/DeepSpeech/training/deepspeech_training/train.py", line 511, in train
    load_or_init_graph_for_training(session)
  File "/home/dimanshu/DeepSpeech/training/deepspeech_training/util/checkpoints.py", line 132, in load_or_init_graph_for_training
    _load_or_init_impl(session, methods, allow_drop_layers=True)
  File "/home/dimanshu/DeepSpeech/training/deepspeech_training/util/checkpoints.py", line 97, in _load_or_init_impl
    return _load_checkpoint(session, ckpt_path, allow_drop_layers)
  File "/home/dimanshu/DeepSpeech/training/deepspeech_training/util/checkpoints.py", line 70, in _load_checkpoint
    v.load(ckpt.get_tensor(v.op.name), session=session)
  File "/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py", line 1033, in load
    session.run(self.initializer, {self.initializer.inputs[1]: value})
  File "/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1156, in _run
    (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (29,) for Tensor 'layer_6/bias/Initializer/zeros:0', which has shape '(89,)'

lissyx · April 21, 2020, 4:24pm

This is still mismatched alphabet.

lissyx · April 21, 2020, 4:26pm

It would be great if you did an effort, I’ve told you about transfer learning, that it is on master and that you should read the docs: https://deepspeech.readthedocs.io/en/master/TRAINING.html#transfer-learning-new-alphabet

and yes, we don’t have yet 0.7 models and checkpoints.

dimanshu.jatav · April 22, 2020, 6:12am

i read that doc @lissyx
and i run this
** python3 DeepSpeech.py --drop_source_layers 1 --alphabet_config_path /home/dimanshu/alpha.txt --save_checkpoint_dir …/checkpoint/ --load_checkpoint_dir …/deepspeech-0.6.1-checkpoint --train_files my-train.csv --dev_files my-dev.csv --test_files my_dev.csv --train_cudnn true**

output is this:-
I Saved new best validating model with loss 134.440708 to: …/checkpoint/best_dev-264124
Epoch 74 | Training | Elapsed Time: 0:00:56 | Steps: 410 | Loss: 113.645727
Epoch 74 | Validation | Elapsed Time: 0:00:00 | Steps: 10 | Loss: 133.315536 | Dataset: my-dev.csv
I Saved new best validating model with loss 133.315536 to: …/checkpoint/best_dev-264534
I FINISHED optimization in 1:13:08.006625
I Loading best validating checkpoint from …/deepspeech-0.6.1-checkpoint/best_dev-233784
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
Traceback (most recent call last):
File “DeepSpeech.py”, line 12, in
ds_train.run_script()
File “/home/dimanshu/DeepSpeech/training/deepspeech_training/train.py”, line 940, in run_script
absl.app.run(main)
File “/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/absl/app.py”, line 299, in run
_run_main(main, args)
File “/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/absl/app.py”, line 250, in _run_main
sys.exit(main(argv))
File “/home/dimanshu/DeepSpeech/training/deepspeech_training/train.py”, line 916, in main
test()
File “/home/dimanshu/DeepSpeech/training/deepspeech_training/train.py”, line 642, in test
samples = evaluate(FLAGS.test_files.split(’,’), create_model)
File “/home/dimanshu/DeepSpeech/training/deepspeech_training/evaluate.py”, line 85, in evaluate
load_graph_for_evaluation(session)
File “/home/dimanshu/DeepSpeech/training/deepspeech_training/util/checkpoints.py”, line 146, in load_graph_for_evaluation
_load_or_init_impl(session, methods, allow_drop_layers=False)
File “/home/dimanshu/DeepSpeech/training/deepspeech_training/util/checkpoints.py”, line 97, in _load_or_init_impl
return _load_checkpoint(session, ckpt_path, allow_drop_layers)
File “/home/dimanshu/DeepSpeech/training/deepspeech_training/util/checkpoints.py”, line 70, in _load_checkpoint
v.load(ckpt.get_tensor(v.op.name), session=session)
File “/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py”, line 324, in new_func
return func(*args, **kwargs)
File “/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py”, line 1033, in load
session.run(self.initializer, {self.initializer.inputs[1]: value})
File “/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py”, line 956, in run
run_metadata_ptr)
File “/home/dimanshu/DeepSpeech/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py”, line 1156, in _run
(np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (29,) for Tensor ‘layer_6/bias/Initializer/zeros:0’, which has shape ‘(87,)’

reuben · April 22, 2020, 1:24pm

This is the same problem as your other issue. Because you’re using different --load_checkpoint_dir and --save_checkpoint_dir, it can cause weird behavior. Separate the training and testing phase, and use --load_checkpoint_dir for any steps that load a checkpoint (testing, exporting, one shot inference, etc).

dimanshu.jatav · April 22, 2020, 1:48pm

ok @reuben i’ll try .