I’m fairly new to DeepSpeech, and am looking for some clarification about the model that is exported after fine-tuning my model. I’ve read the documentation pretty extensively and searched this forum, but haven’t found the answer I’ve been looking for (but if I missed it somewhere let me know!).
Info if needed (although probably not super relevant to my question):
Training or just running inference: Training
DeepSpeech branch/version: 0.7.4
OS Platform and Distribution: Ubuntu 18.04
Python version: 3.6.9
TensorFlow version: 1.15.2 (GPU)
CUDA/cuDNN version: CUDA 10.0/cuDNN 7.6.5
I am fine-tuning from the v0.7.4 checkpoint using this command:
DeepSpeech.py --n_hidden 2048 --load_checkpoint_dir release_checkpoint --save_checkpoint_dir checkpoint_0 --epochs 3 --train_files train.csv --dev_files dev.csv --test_files test.csv --learning_rate 0.0001 --export_dir adapted_model --alphabet_config_path alphabet.txt --scorer_path deepspeech-0.7.4-models.scorer --load_cudnn True
where release_checkpoint contains the v0.7.4 checkpoints. Everything is working great, but once the training has run and it’s time for the model to be exported, I get this as part of the output:
I Exporting the model... I Loading best validating checkpoint from release_checkpoint/best_dev-732522
If I’m understanding this output correctly, it means that the exported model being created is based on the checkpoint in release_checkpoint, which contains the v0.7.4 checkpoint. So basically, the exported model will just be a recreation of the v0.7.4 model, not the model trained on my data for the 3 additional epochs.
If this is the case, then does that mean if I want the exported model to be the one trained on my data, I have to use the same directory for loading and saving checkpoints with the --checkpoint flag, instead of specifying two different locations?
Hopefully this question makes sense, I wasn’t sure exactly how to word it . Let me know if you need any more information. Any help is greatly appreciated!