Lerning process ends after start

Archives DeepSpeech
,
1

Hellow. I’m newbe in deepspeech. But i followed training own model. But when i run command like that:

python3 DeepSpeech.py –train_files $HOME/tmp/ru_records/clips/train.csv –dev_files $HOME/tmp/ru_records/clips/dev.csv –test_files $HOME/tmp/ru_records/clips/test.csv  --alphabet_config_path=$HOME/tmp/ru_records/alphabet.txt --learning_rate 0.0001 –export_dir $HOME/tmp/exported

process starts for a couple of seconds and ends with no any error or message.
I took mozilla’s voice dataset with about 1G data and try to train LM on current deepspech version. I do it on my virtual linux machine with 4CPUx4RAM just to see if that im doing all in right way.
Tell me please where i can find any logs or what i do wrong?

2

Could you please share the current output you get ?

3

Also, python DeepSpeech.py --helpfull will document your --log_level …

4

There is no output at all. I’ll try params --helpfull and --log_level and give you results.

5

set --log_level 0

result

2020-04-13 19:05:34.012807: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-04-13 19:05:34.017747: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2304005000 Hz
2020-04-13 19:05:34.017965: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55d6ccc79a70 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-04-13 19:05:34.017987: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
6

at last, i got more logs:

2020-04-13 19:55:15.246938: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-04-13 19:55:15.250327: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2304005000 Hz
2020-04-13 19:55:15.250656: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55abf5555bb0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-04-13 19:55:15.250716: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
I Exporting the model...
WARNING:tensorflow:From /home/max/src/DeepSpeech/training/deepspeech_training/train.py:671: The name tf.nn.rnn_cell.LSTMStateTuple is deprecated. Please use tf.compat.v1.nn.rnn_cell.LSTMStateTuple instead.

W0413 19:55:15.262728 140247797956992 module_wrapper.py:139] From /home/max/src/DeepSpeech/training/deepspeech_training/train.py:671: The name tf.nn.rnn_cell.LSTMStateTuple is deprecated. Please use tf.compat.v1.nn.rnn_cell.LSTMStateTuple instead.

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W0413 19:55:15.303478 140247797956992 lazy_loader.py:50]
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /home/max/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow_core/contrib/rnn/python/ops/lstm_ops.py:597: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
W0413 19:55:15.307159 140247797956992 deprecation.py:323] From /home/max/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow_core/contrib/rnn/python/ops/lstm_ops.py:597: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
WARNING:tensorflow:From /home/max/src/DeepSpeech/training/deepspeech_training/train.py:770: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

W0413 19:55:15.362227 140247797956992 module_wrapper.py:139] From /home/max/src/DeepSpeech/training/deepspeech_training/train.py:770: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

I Could not find best validating checkpoint.
I Could not find most recent checkpoint.
E All initialization methods failed (['best', 'last']).
7

It directly tried to export? My money on your poor copy paste that broke double dash into one long dash. That’s why it is important you use proper code formatting when sharing code, console commands and output. Triple check the way you pass arguments: ‘–’, while you shared a long unique dash…

1 Like
8

Sorry for that. I was not carefull when copied the same code from somewhere else. Passed dash was the reason of stucking in my case. Now learning is going on. Sorry and thanks for time you spent on me and money you bet.

1 Like