My Training is getting auto-killed just after model training starts

Archives DeepSpeech
1

When ever I am trying to run a deepspeech.py command for traing with my own dataset, the training is starting but after 10 minutes, the entire process is getting autokilled.

My Code :

(deepspeech1) ritish@ritish-VirtualBox:~/DeepSpeech/DeepSpeech-0.6.1$ python DeepSpeech.py --train_files ./20140421/clips/train.csv --dev_files ./20140421/clips/dev.csv --test_files ./20140421/clips/test.csv --beam_width 200 --export_dir ./20140421/models/ --learning_rate 0.001 --lm_alpha 0.4 --lm_beta 1.5 --train_batch_size 3  --cudnn_checkpoint ./20140421/checkpoints_pretrained/deepspeech-0.6.1-checkpoint --epochs 3

My Output:

WARNING:tensorflow:From /home/ritish/.local/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py:347: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W0430 14:53:33.025328 139667009120064 deprecation.py:323] From /home/ritish/.local/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py:347: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
WARNING:tensorflow:From /home/ritish/.local/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py:348: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W0430 14:53:33.025740 139667009120064 deprecation.py:323] From /home/ritish/.local/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py:348: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
WARNING:tensorflow:From /home/ritish/.local/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py:350: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W0430 14:53:33.025959 139667009120064 deprecation.py:323] From /home/ritish/.local/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py:350: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W0430 14:53:33.445173 139667009120064 lazy_loader.py:50] 
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /home/ritish/.local/lib/python3.7/site-packages/tensorflow_core/contrib/rnn/python/ops/lstm_ops.py:597: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
W0430 14:53:33.447514 139667009120064 deprecation.py:323] From /home/ritish/.local/lib/python3.7/site-packages/tensorflow_core/contrib/rnn/python/ops/lstm_ops.py:597: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
WARNING:tensorflow:From DeepSpeech.py:236: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0430 14:53:33.621286 139667009120064 deprecation.py:323] From DeepSpeech.py:236: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
I Converting CuDNN RNN checkpoint from ./20140421/checkpoints_pretrained/deepspeech-0.6.1-checkpoint
WARNING:tensorflow:From DeepSpeech.py:534: Variable.load (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Prefer Variable.assign which has equivalent behavior in 2.X.
W0430 14:53:34.969209 139667009120064 deprecation.py:323] From DeepSpeech.py:534: Variable.load (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Prefer Variable.assign which has equivalent behavior in 2.X.
I Initializing missing Adam moment tensors.
I STARTING Optimization
Epoch 0 |   Training | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000                                                                                Killed

Please Help

2

There’s nothing indicating an error here, we can’t help you.

But you are training in a VM, do you have GPU ? enough RAM ?
Any system log ? dmesg output ?

3

Training on a “fast” system starts after max. 1 minute. If it is 10 minutes for you, your training will be really slow … change sth to get results :slight_smile:

I read about killed processes before, do have limitations on your setup about how long a process can run? Search this forum for it and ask your admin.