Problem while trainning: [scorer.cpp:77] FATAL: "(access(filename, 4)) == (0)" check failed. Invalid language model path

DeepSpeech
#1

Hi. I am trying to training langgue model from my voice (about 25 words). When i run:
./DeepSpeech.py --checkpoint_dir ~/checkpoints --log_dir ~/logs --epochs 1 --alphabet_config_path data/alphabet.txt --train_files ~/clips/train.csv --dev_files ~/clips/dev.csv --test_files ~/clips/test.csv
Then error:WARNING:tensorflow:From /home/tuan/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It’s easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.

W1119 18:22:12.715300 140444139034432 deprecation.py:323] From /home/tuan/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It’s easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.

WARNING:tensorflow:From /home/tuan/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_types(iterator).
W1119 18:22:12.802873 140444139034432 deprecation.py:323] From /home/tuan/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_types(iterator).
WARNING:tensorflow:From /home/tuan/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_shapes(iterator).
W1119 18:22:12.803174 140444139034432 deprecation.py:323] From /home/tuan/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_shapes(iterator).
WARNING:tensorflow:From /home/tuan/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_classes(iterator).
W1119 18:22:12.803371 140444139034432 deprecation.py:323] From /home/tuan/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_classes(iterator).
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

W1119 18:22:13.776006 140444139034432 lazy_loader.py:50]
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

WARNING:tensorflow:From /home/tuan/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W1119 18:22:13.778615 140444139034432 deprecation.py:506] From /home/tuan/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:Entity <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7fbb78101828>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7fbb78101828>>: AttributeError: module ‘gast’ has no attribute ‘Num’
W1119 18:22:13.817533 140444139034432 ag_logging.py:145] Entity <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7fbb78101828>> could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting <bound method LSTMBlockWrapper.call of <tensorflow.contrib.rnn.python.ops.lstm_ops.LSTMBlockFusedCell object at 0x7fbb78101828>>: AttributeError: module ‘gast’ has no attribute ‘Num’
WARNING:tensorflow:From ./DeepSpeech.py:234: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W1119 18:22:13.899004 140444139034432 deprecation.py:323] From ./DeepSpeech.py:234: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/tuan/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W1119 18:22:14.611679 140444139034432 deprecation.py:323] From /home/tuan/tmp/deepspeech-train-venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from /home/tuan/checkpoints/train-120
I1119 18:22:14.612939 140444139034432 saver.py:1280] Restoring parameters from /home/tuan/checkpoints/train-120
I Restored variables from most recent checkpoint at /home/tuan/checkpoints/train-120, step 120
I STARTING Optimization
Epoch 0 | Training | Elapsed Time: 0:43:18 | Steps: 30 | Loss: 75.908880
Epoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | DatasEpoch 0 | Validation | Elapsed Time: 0:00:03 | Steps: 1 | Loss: 63.777626 | DataEpoch 0 | Validation | Elapsed Time: 0:00:07 | Steps: 2 | Loss: 60.889185 | DataEpoch 0 | Validation | Elapsed Time: 0:00:11 | Steps: 3 | Loss: 57.227089 | DataEpoch 0 | Validation | Elapsed Time: 0:00:16 | Steps: 4 | Loss: 65.569147 | DataEpoch 0 | Validation | Elapsed Time: 0:00:21 | Steps: 5 | Loss: 70.167294 | DataEpoch 0 | Validation | Elapsed Time: 0:00:26 | Steps: 6 | Loss: 69.781456 | DataEpoch 0 | Validation | Elapsed Time: 0:00:31 | Steps: 7 | Loss: 74.682470 | DataEpoch 0 | Validation | Elapsed Time: 0:00:36 | Steps: 8 | Loss: 74.664686 | DataEpoch 0 | Validation | Elapsed Time: 0:00:36 | Steps: 8 | Loss: 74.664686 | Dataset: /home/tuan/clips/dev.csv
I Saved new best validating model with loss 74.664686 to: /home/tuan/checkpoints/best_dev-150
I FINISHED optimization in 0:44:04.502778
[scorer.cpp:77] FATAL: “(access(filename, 4)) == (0)” check failed. Invalid language model path

( I trained and have checkpoints earlier so i want to set low epoch)
Do you know the problems with me?
(Beside, i install git lfs ( i do: git lfs install success but when i run git clone https://github.com/mozilla/DeepSpeech , it download but i have to use Ctrl + C to write new command)

#2

@20164356 Your log is barely readable, please use proper code formatting.

#3

So basically, your setup is not right.

Isn’t the error pretty clear ? You gave an invalid language model path.

#4

Thanks for repling me soon. When i creat my LM, i don’t know how to make .tsv file so i creat .wav file and .csv file “like” another LM i’ve tested, maybe the transcripts i wrote is wrong (my language- vietnamese using latin alphabet and it’s reading like writing so i expect it works). So can you help me creat .tsv file if i need creat .mp3 file or (better) creat true .csv file with my .wav files?
Below is my train.csv files:wav_filename,wav_filesize,transcript
tuan_9muoi0.wav,388124,chín mười không
tuan_012.wav,370484,không một hai
tuan_12thang3.wav,551604,mười hai tháng ba
tuan_163.wav,317564,một sáu ba
tuan_0194.wav,458684,không một chín bốn
tuan_257.wav,458684,hai năm bảy
tuan_321.wav,370484,ba hai một
tuan_345.wav,493964,ba bốn năm
tuan_654.wav,441044,sáu năm bốn
tuan_678.wav,405764,sáu bảy tám
tuan_840.wav,370484,tám bốn không
tuan_987.wav,388124,chín tám bảy
tuan_995.wav,299924,chín chín lăm
tuan_1995.wav,529244,một chín chín lăm
tuan_2013.wav,352844,hai không một ba
tuan_2016.wav,405764,hai không một sáu
tuan_4278.wav,370484,bốn hai bảy tám
tuan_4356.wav,458684,bốn ba năm sáu
tuan_hotensinhvien.wav,458684,họ tên sinh viên
tuan_hovaten.wav,352844,họ và tên
tuan_lamvanam.wav,352844,lăm và năm
tuan_maianhtuan.wav,423404,mai anh tuấn
tuan_maituananh.wav,405764,mai tuấn anh
tuan_masosinhvien.wav,423404,mã số sinh viên
tuan_masotenho.wav,476324,mã số tên họ
tuan_muoilammuoihai.wav,458684,mười lăm mười hai
tuan_namvalam.wav,388124,năm và lăm
tuan_ngaythangnamsinh.wav,529244,ngày tháng năm sinh
tuan_sinhngaythangnam.wav,493964,sinh ngày tháng năm
tuan_thang3ngay10.wav,370484,tháng ba ngày mười

#5

That’s unrelated to your error. Creating language model does not require any TSV file, training DeepSpeech model does not either, so I don’t understand what you are referring to.

Please, make some effort, share better context, share your error using proper code formatting. We can’t help if you don’t do that.

#6

I got this error also and I don’t understand why the language model path is invalid.

#7

Hi. I don’t know what exactly your problem but with me, i install git large file and use git lfs clone to download Deepspeech instead of git clone. The Deepspeech files is much more bigger and Deepspeech.py run. I hope this maybe helpful

#8

Hi, I get this same error as well. I’m just trying to follow the “train your own model instructions” and I don’t know how I’m supposed to give the path to the language model. I can’t find anything about it in the tutorial and I don’t see anything about it using --helpfull. Any help would be greatly appreciated. Thanks!

#9

did you download Deepspeech with git large file( using git lfs clone instead of git clone)?

#10

Thanks this seems to be the solution. I was following the tutorial too closely which showed just using a regular git clone rather than git lfs clone. Basically I was doing a git lfs init followed by a regular clone which was wrong.

Thanks!

#11

We document to install git-lfs so that when you perform your git clone of deepspeech, it does automatically perform the git-lfs one. Please send issues / patches to improve the documentation if you can.