Il y a encore un comportement d’entraînement inadapté, que j’avais résolu l’autre fois : le loss augmente. J’ai peut d’avoir un petit bug qui traîne dans l’importation des données.
+ python -u DeepSpeech.py --alphabet_config_path /mnt/models/alphabet.txt --lm_binary_path /mnt/lm/lm.binary --lm_trie_path /mnt/lm/trie --feature_cache /mnt/sources/feature_cache --train_files /mnt/extracted/data/lingualibre/lingua_libre_Q21-fra-French_train.csv,/mnt/extracted/data/trainingspeech/ts_2019-04-11_fr_FR_train.csv,/mnt/extracted/data/cv-fr/clips/train.csv --dev_files /mnt/extracted/data/lingualibre/lingua_libre_Q21-fra-French_dev.csv,/mnt/extracted/data/trainingspeech/ts_2019-04-11_fr_FR_dev.csv,/mnt/extracted/data/cv-fr/clips/dev.csv --test_files /mnt/extracted/data/lingualibre/lingua_libre_Q21-fra-French_test.csv,/mnt/extracted/data/trainingspeech/ts_2019-04-11_fr_FR_test.csv,/mnt/extracted/data/cv-fr/clips/test.csv --train_batch_size 68 --dev_batch_size 68 --test_batch_size 68 --n_hidden 2048 --epochs 100 --learning_rate 0.000001 --dropout_rate 0.30 --lm_alpha 0.0 --lm_beta 0.0 --noearly_stop --display_step 0 --validation_step 1 --checkpoint_step 1 --checkpoint_dir /mnt/checkpoints/ --export_dir /mnt/models/ --export_language fra
WARNING:tensorflow:From /home/trainer/ds-train-fr/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:429: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, use
tf.py_function, which takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means `tf.py_function`s can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
WARNING:tensorflow:From /home/trainer/ds-train-fr/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:358: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /home/trainer/ds-train-fr/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/lstm_ops.py:696: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/trainer/ds-train-fr/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
I Restored variables from most recent checkpoint at /mnt/checkpoints/best_dev-585277, step 585277
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:15:33 | Steps: 1120 | Loss: 198.901285
Epoch 0 | Validation | Elapsed Time: 0:00:02 | Steps: 31 | Loss: 34.221434 | Dataset: /mnt/extracted/data/lingualibre/lingua_libre_Q21-fra-French_dev.csv
Epoch 0 | Validation | Elapsed Time: 0:00:29 | Steps: 91 | Loss: 242.853168 | Dataset: /mnt/extracted/data/trainingspeech/ts_2019-04-11_fr_FR_dev.csv
Epoch 0 | Validation | Elapsed Time: 0:00:14 | Steps: 65 | Loss: 139.231356 | Dataset: /mnt/extracted/data/cv-fr/clips/dev.csv
I Saved new best validating model with loss 172.248882 to: /mnt/checkpoints/best_dev-586397
Epoch 1 | Training | Elapsed Time: 0:15:39 | Steps: 1120 | Loss: 168.445158
Epoch 1 | Validation | Elapsed Time: 0:00:02 | Steps: 31 | Loss: 35.868865 | Dataset: /mnt/extracted/data/lingualibre/lingua_libre_Q21-fra-French_dev.csv
Epoch 1 | Validation | Elapsed Time: 0:00:29 | Steps: 91 | Loss: 252.253984 | Dataset: /mnt/extracted/data/trainingspeech/ts_2019-04-11_fr_FR_dev.csv
Epoch 1 | Validation | Elapsed Time: 0:00:14 | Steps: 65 | Loss: 144.695296 | Dataset: /mnt/extracted/data/cv-fr/clips/dev.csv
Epoch 2 | Training | Elapsed Time: 0:15:35 | Steps: 1120 | Loss: 158.498774
WARNING:tensorflow:From /home/trainer/ds-train-fr/lib/python3.6/site-packages/tensorflow/python/training/saver.py:966: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to delete files with this prefix.
Epoch 2 | Validation | Elapsed Time: 0:00:02 | Steps: 31 | Loss: 37.718021 | Dataset: /mnt/extracted/data/lingualibre/lingua_libre_Q21-fra-French_dev.csv
Epoch 2 | Validation | Elapsed Time: 0:00:29 | Steps: 91 | Loss: 259.248127 | Dataset: /mnt/extracted/data/trainingspeech/ts_2019-04-11_fr_FR_dev.csv
Epoch 2 | Validation | Elapsed Time: 0:00:14 | Steps: 65 | Loss: 149.385020 | Dataset: /mnt/extracted/data/cv-fr/clips/dev.csv
Epoch 3 | Training | Elapsed Time: 0:15:39 | Steps: 1120 | Loss: 144.689475
Epoch 3 | Validation | Elapsed Time: 0:00:02 | Steps: 31 | Loss: 40.592877 | Dataset: /mnt/extracted/data/lingualibre/lingua_libre_Q21-fra-French_dev.csv
Epoch 3 | Validation | Elapsed Time: 0:00:29 | Steps: 91 | Loss: 277.970247 | Dataset: /mnt/extracted/data/trainingspeech/ts_2019-04-11_fr_FR_dev.csv
Epoch 3 | Validation | Elapsed Time: 0:00:14 | Steps: 65 | Loss: 160.652092 | Dataset: /mnt/extracted/data/cv-fr/clips/dev.csv
Epoch 4 | Training | Elapsed Time: 0:15:43 | Steps: 1120 | Loss: 128.832886
Epoch 4 | Validation | Elapsed Time: 0:00:02 | Steps: 31 | Loss: 44.560288 | Dataset: /mnt/extracted/data/lingualibre/lingua_libre_Q21-fra-French_dev.csv
Epoch 4 | Validation | Elapsed Time: 0:00:29 | Steps: 91 | Loss: 308.391034 | Dataset: /mnt/extracted/data/trainingspeech/ts_2019-04-11_fr_FR_dev.csv
Epoch 4 | Validation | Elapsed Time: 0:00:14 | Steps: 65 | Loss: 177.152804 | Dataset: /mnt/extracted/data/cv-fr/clips/dev.csv
Epoch 5 | Training | Elapsed Time: 0:15:36 | Steps: 1120 | Loss: 114.762158 ^[[A
Epoch 5 | Validation | Elapsed Time: 0:00:02 | Steps: 31 | Loss: 49.119470 | Dataset: /mnt/extracted/data/lingualibre/lingua_libre_Q21-fra-French_dev.csv
Epoch 5 | Validation | Elapsed Time: 0:00:29 | Steps: 91 | Loss: 343.108495 | Dataset: /mnt/extracted/data/trainingspeech/ts_2019-04-11_fr_FR_dev.csv
Epoch 5 | Validation | Elapsed Time: 0:00:14 | Steps: 65 | Loss: 193.982464 | Dataset: /mnt/extracted/data/cv-fr/clips/dev.csv