Clarification about exported model

Hello,
I’m fairly new to DeepSpeech, and am looking for some clarification about the model that is exported after fine-tuning my model. I’ve read the documentation pretty extensively and searched this forum, but haven’t found the answer I’ve been looking for (but if I missed it somewhere let me know!).

Info if needed (although probably not super relevant to my question):
Training or just running inference: Training
DeepSpeech branch/version: 0.7.4
OS Platform and Distribution: Ubuntu 18.04
Python version: 3.6.9
TensorFlow version: 1.15.2 (GPU)
CUDA/cuDNN version: CUDA 10.0/cuDNN 7.6.5

I am fine-tuning from the v0.7.4 checkpoint using this command:

DeepSpeech.py --n_hidden 2048 --load_checkpoint_dir release_checkpoint --save_checkpoint_dir checkpoint_0 --epochs 3 --train_files train.csv --dev_files dev.csv --test_files test.csv --learning_rate 0.0001 --export_dir adapted_model --alphabet_config_path alphabet.txt --scorer_path deepspeech-0.7.4-models.scorer --load_cudnn True

where release_checkpoint contains the v0.7.4 checkpoints. Everything is working great, but once the training has run and it’s time for the model to be exported, I get this as part of the output:

I Exporting the model...
I Loading best validating checkpoint from release_checkpoint/best_dev-732522

If I’m understanding this output correctly, it means that the exported model being created is based on the checkpoint in release_checkpoint, which contains the v0.7.4 checkpoint. So basically, the exported model will just be a recreation of the v0.7.4 model, not the model trained on my data for the 3 additional epochs.

If this is the case, then does that mean if I want the exported model to be the one trained on my data, I have to use the same directory for loading and saving checkpoints with the --checkpoint flag, instead of specifying two different locations?

Hopefully this question makes sense, I wasn’t sure exactly how to word it :sweat_smile: . Let me know if you need any more information. Any help is greatly appreciated!

That’s right.

I’m unsure, but without the training log we can’t verify if your training actually has produced a new best_dev checkpoint.

export() goes through load_graph_for_evaluation that allows you to specify FLAGS.load_evaluate to force something else than best, but it will load the checkpoint via _checkpoint_path_or_none() which relies on FLAGS.load_checkpoint_dir.

So in your case, maybe it might be easier to:

  • use your command line without the --export_dir to load from release and save to a different directory
  • issue a second call, with only --load_checkpoint_dir pointing to the checkpoint_0 directory and this time, --export_dir (you can here skip all --train_files --dev_files --test_files etc.

Thanks for getting back to me! It is producing new best_dev checkpoints, and they do show up in my checkpoint_0 directory. Here is the training log from a run with --epochs=1 (FYI: the transcripts are not actually asterisks, but the project I’m working on uses personal audio data that I am not allowed to share online so I’ve removed them):

W WARNING: You specified different values for --load_checkpoint_dir and --save_checkpoint_dir, but you are running training and testing in a single invocation. The testing step will respect --load_checkpoint_dir, and thus WILL NOT TEST THE CHECKPOINT CREATED BY THE TRAINING STEP. Train and test in two separate invocations, specifying the correct --load_checkpoint_dir in both cases, or use the same location for loading and saving.
I Loading best validating checkpoint from release_checkpoint/best_dev-732522
I Loading variable from checkpoint: beta1_power
I Loading variable from checkpoint: beta2_power
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel/Adam
I Loading variable from checkpoint: cudnn_lstm/opaque_kernel/Adam_1
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/bias/Adam
I Loading variable from checkpoint: layer_1/bias/Adam_1
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_1/weights/Adam
I Loading variable from checkpoint: layer_1/weights/Adam_1
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/bias/Adam
I Loading variable from checkpoint: layer_2/bias/Adam_1
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_2/weights/Adam
I Loading variable from checkpoint: layer_2/weights/Adam_1
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/bias/Adam
I Loading variable from checkpoint: layer_3/bias/Adam_1
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_3/weights/Adam
I Loading variable from checkpoint: layer_3/weights/Adam_1
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/bias/Adam
I Loading variable from checkpoint: layer_5/bias/Adam_1
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_5/weights/Adam
I Loading variable from checkpoint: layer_5/weights/Adam_1
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/bias/Adam
I Loading variable from checkpoint: layer_6/bias/Adam_1
I Loading variable from checkpoint: layer_6/weights
I Loading variable from checkpoint: layer_6/weights/Adam
I Loading variable from checkpoint: layer_6/weights/Adam_1
I Initializing variable: learning_rate
I STARTING Optimization
Epoch 0 |   Training | Elapsed Time: 0:32:27 | Steps: 10030 | Loss: 107.625153                                                                                                                             
Epoch 0 | Validation | Elapsed Time: 0:03:58 | Steps: 2865 | Loss: 88.195158 | Dataset: dev.csv                                                                                                     
I Saved new best validating model with loss 88.195158 to: checkpoint_0/best_dev-742552
--------------------------------------------------------------------------------
I FINISHED optimization in 0:36:26.581329
I Loading best validating checkpoint from release_checkpoint/best_dev-732522
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/weights
Testing model on test.csv
Test epoch | Steps: 1434 | Elapsed Time: 0:07:36                                                                                                                                                           
Test on test.csv - WER: 0.722950, CER: 0.542535, loss: 91.392250
--------------------------------------------------------------------------------
Best WER: 
--------------------------------------------------------------------------------
WER: 0.000000, CER: 0.038462, loss: 24.549252
 - wav: file:///home/data/audio_03969.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
WER: 0.000000, CER: 0.047619, loss: 14.027812
 - wav: file:///home/data/audio_05854.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
WER: 0.000000, CER: 0.050000, loss: 12.023066
 - wav: file:///home/data/audio_07141.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
WER: 0.000000, CER: 0.333333, loss: 11.255897
 - wav: file:///home/data/audio_09208.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
WER: 0.000000, CER: 0.083333, loss: 10.824797
 - wav: file:///home/data/audio_01818.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
Median WER: 
--------------------------------------------------------------------------------
WER: 0.850000, CER: 0.535354, loss: 193.085495
 - wav: file:///home/data/audio_01541.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
WER: 0.850000, CER: 0.500000, loss: 181.437256
 - wav: file:///home/data/audio_13058.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
WER: 0.850000, CER: 0.541667, loss: 173.570297
 - wav: file:///home/data/audio_08271.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
WER: 0.852941, CER: 0.511905, loss: 357.852325
 - wav: file:///home/data/audio_00097.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
WER: 0.857143, CER: 0.669565, loss: 285.302826
 - wav: file:///home/data/audio_04991.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
Worst WER: 
--------------------------------------------------------------------------------
WER: 1.500000, CER: 0.750000, loss: 46.166569
 - wav: file:///home/data/audio_05179.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
WER: 1.500000, CER: 1.600000, loss: 22.395161
 - wav: file:///home/data/audio_02716.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
WER: 2.000000, CER: 1.300000, loss: 69.320763
 - wav: file:///home/data/audio_03881.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
WER: 2.000000, CER: 1.200000, loss: 34.534180
 - wav: file:///home/data/audio_10661.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
WER: 2.000000, CER: 1.750000, loss: 23.410473
 - wav: file:///home/data/audio_00386.wav
 - src: ***************
 - res: ***************
--------------------------------------------------------------------------------
I Exporting the model...
I Loading best validating checkpoint from release_checkpoint/best_dev-732522
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/weights
I Models exported at models_adapted
I Model metadata file saved to models_adapted/author_model_0.0.1.md. Before submitting the exported model for publishing make sure all information in the metadata file is correct, and complete the URL fields.

Thank you for the explanation! It’s much more clear to me now.

This sounds like the way to go for me. Thanks again! :slightly_smiling_face: