Input: Continuing training the deepspeech 0.9.3. model checkpoints on Russian alphabet + Russian train/test .wav files
OS: Ubuntu 20.04
Tensorflow: tensorflow:1.15.2-gpu-py3
Python: 3.6.9.
Running the main script through bash file with following flags:
python3 DeepSpeech.py
–epochs 30
–drop_source_layers 1
–train_batch_size 32
–test_batch_size 16
–train_cudnn False
–dropout_rate 0.4
–learning_rate 0.000001
–alphabet_config_path /data/DeepSpeech/data/alphabet.txt
–save_checkpoint_dir /data/DeepSpeech/save_check/
–load_checkpoint_dir /data/DeepSpeech/save_check
–train_files /data/DeepSpeech/data/tt.csv
–test_files /data/DeepSpeech/data/test_withoutsizes.csv
–export_dir /data/DeepSpeech/export_dir/ \
According to the documentation, setting --drop_source_layers 1 to 5 helps to avoid the incompatible geometry - alphabet error.
Citing:
"All dropped layers will be reinitialized, and (crucially) the output layer will be defined to match your supplied target alphabet."©
However, when exporting the model for inference, it looks as if the Russian alphabet cannot be used to continue training from English deepspeech 0.9.3. checkpoints.
Would this be a bug?
Or is it impossible to continue training with Russian files, and I need to initialize the weights randomly?
Thank you for your time.
Error logs:
I FINISHED optimization in 2:05:52.762198
I Loading best validating checkpoint from /data/DeepSpeech/load_check/deepspeech-0.9.3-checkpoint/best_dev-1466475
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
Traceback (most recent call last):
File “DeepSpeech.py”, line 12, in
ds_train.run_script()
File “/data/DeepSpeech/training/deepspeech_training/train.py”, line 982, in run_script
absl.app.run(main)
File “/usr/local/lib/python3.6/dist-packages/absl/app.py”, line 299, in run
_run_main(main, args)
File “/usr/local/lib/python3.6/dist-packages/absl/app.py”, line 250, in _run_main
sys.exit(main(argv))
File “/data/DeepSpeech/training/deepspeech_training/train.py”, line 958, in main
test()
File “/data/DeepSpeech/training/deepspeech_training/train.py”, line 682, in test
samples = evaluate(FLAGS.test_files.split(’,’), create_model)
File “/data/DeepSpeech/training/deepspeech_training/evaluate.py”, line 87, in evaluate
load_graph_for_evaluation(session)
File “/data/DeepSpeech/training/deepspeech_training/util/checkpoints.py”, line 151, in load_graph_for_evaluation
_load_or_init_impl(session, methods, allow_drop_layers=False)
File “/data/DeepSpeech/training/deepspeech_training/util/checkpoints.py”, line 98, in _load_or_init_impl
return _load_checkpoint(session, ckpt_path, allow_drop_layers, allow_lr_init=allow_lr_init)
File “/data/DeepSpeech/training/deepspeech_training/util/checkpoints.py”, line 71, in _load_checkpoint
v.load(ckpt.get_tensor(v.op.name), session=session)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py”, line 324, in new_func
return func(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 1033, in load
session.run(self.initializer, {self.initializer.inputs[1]: value})
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 956, in run
run_metadata_ptr)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1156, in _run
(np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (29,) for Tensor ‘layer_6/bias/Initializer/zeros:0’, which has shape ‘(35,)’
P.S. Added the same info to https://github.com/mozilla/DeepSpeech/issues/3665;
Will remove immediately in one of the two sources, if someone could hint on where the query should go.
Thank you in advance for your time!