env:
O-System Ubuntu 20.04
Graphic card RTX3090
run the DeepSpeech.py in container, which is built by the image 20.11-tf1-py3 from nvidia/tensorflow.
command:
python3 -u DeepSpeech.py
–train_files /workspace/de/clips/train.csv
–test_files /workspace/de/clips/test.csv
–dev_files /workspace/de/clips/dev.csv
–export_dir /workspace/DeepSpeech/data/model
–train_batch_size 150
–dev_batch_size 150
–test_batch_size 100
–epochs 1
–n_hidden 1024
–learning_rate=0.0001
–dropout_rate 0.2
–export_file_name output_616
–export_author_id sun
–export_model_name 616
–export_model_version 1
–summary_dir /workspace/DeepSpeech/data/model
–es_epochs 10
–es_min_delta 0.1
–early_stop True
–reduce_lr_on_plateau True
–plateau_epochs 3
–plateau_reduction 0.01
final logs:
I Exporting the model…
I Loading best validating checkpoint from /root/.local/share/deepspeech/checkpoints/best_dev-67983
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/weights
I Models exported at /workspace/DeepSpeech/data/model
I Model metadata file saved to /workspace/DeepSpeech/data/model/sun_616_1.md. Before submitting the exported model for publishing make sure all information in the metadata file is correct, and complete the URL fields.
there is no error or warning in the logs. After the testing I only get the pb and md file but without summaries for tensorboard. And I touch the default path, there are only the oldest summaries which were exported after the first training 3 days ago. No latest summaries in the default path.
Did I use the flag wrong? How can I get the latest summaries?
Thx a lot.