Questions regarding training arguments

First: a big thanks to the poeple making this possible! great project

Second: I’m trying to figure out on how to train my own model trough a .sh file to overfit a phrase i’ve recorded myself
DeepSpeech version: 0.6.1 (github version on 24/01/2020)
Python 3.7.6
OS: Ubuntu 18.04
Tensorflow: 1.14.0
(using anaconda venv)
My current .sh file:

#!/bin/bash
cd DeepSpeech
python -u DeepSpeech.py \
--train_files /home/plato/Documents/project/train_deep_model/train.csv \
--test_files /home/plato/Documents/project/train_deep_model/train.csv  \
--train_batch_size 1 \
--test_batch_size 1 \
--n_hidden 150 \
--epochs 100 \
--load init \
--checkpoint_dir /home/plato/Documents/project/train_deep_model/checkpoints/ \
--export_dir /home/plato/Documents/project/train_deep_model/ \
"$@"

Now when i try to run this, the model trains for the 150 epochs like it should, but when it’s supposed to save I get the following error:
Could not load checkpoint from /home/plato/Documents/project/train_deep_model/checkpoints/

and the following warning too:

`WARNING:tensorflow:From /home/plato/anaconda3/envs/train_test_env/lib/python3.7/site-packages/tensorflow/python/training/saver.py:960: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.

`
Any idea why this is?

And when I leave the ‘- -load init’ out of my .sh file it works like it should :slight_smile:
But it doesnt start from re-initialized weights :frowning:

Here I have a few other questions regarding the arguments:

  1. Is there a way to keep track of the loss on every epoch on the test (and maybe validation) files trough a .txt file? (or should I go into the source code and make something for myself?)
  2. Is there a way to choose the name of the output_graph.pb file before it gets saved on my computer?
  3. is there a way to disable the WER report at the end of the training?
  4. using the export_language argument, where can I see which language the model is in?

You should look at bin/run-ldc93s1.sh

I think because of --load init

Just disable progressbar and pipe to a file

No, but it’s easy to rename afterwards …

Yes, don’t pass --test_files

I don’t understand that one. Read the code, export_language just burn some metadata in the resulting graph model.

1 Like