Error when trying to decode: terminate called after throwing an instance of 'std::length_error'

DeepSpeech
#1 
native_client/deepspeech --model /home/xyz/DeepSpeech/data/exprt_dir/output_graph.pb --scorer /home/xyz/DeepSpeech/data/lm/lm.scorer --audio data/test5.wav --beam_width 9000 > data/decoding.txt
TensorFlow: v2.2.0-15-g518c1d0
DeepSpeech: v0.9.0-alpha.3-0-g78ae08c
Warning: reading entire model file into memory. Transform model file into an  mmapped graph to reduce heap usage.
2020-09-22 12:15:00.293543: I tensorflow/core/platform/cpu_feature_guard.cc:143]  Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
terminate called after throwing an instance of 'std::length_error'
what():  vector::_M_default_append
Abandon (core dumped)

Hi, I am trying to decode a wav file after training a model from scratch, i finished the training and the testing phase without errors and I get the WER & CER and Loss values.

NB: I’ve already done the decoding with this command with no errors but it appeared when I tested with a new model.

#2

You did train with v0.9.0-alpha.3-0-g78ae08c?

Try without lm and beamwidth and pipe, just the model.

Listen to the wav or take one from the training material as that did work before.

#3 
native_client/deepspeech --model /home/aymen/Abir/DeepSpeech/data/exprt_dir/output_graph.pb --audio data/decoding_online/test5.wav
TensorFlow: v2.2.0-15-g518c1d0
DeepSpeech: v0.9.0-alpha.3-0-g78ae08c
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2020-09-22 13:56:17.511266: I tensorflow/core/platform/cpu_feature_guard.cc:143]       Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
terminate called after throwing an instance of 'std::length_error'
what():  vector::_M_default_append
Abandon (core dumped)

Thanks for your reply,
Yes I trained with v0.9.0-alpha.3-0-g78ae08c
I tried the command without the scorer, pipe and beam and with a file from the train dataset but the error still persists.

#4

Hm, I thought we are still on TF 1.15? Did you use TF 2 for training? Did you change the file? Ideally use a released version for training, no alpha.

And give more info on the problem:

#5 
(deepspeech-venv) (base) root@xyz:/home/xyz/DeepSpeech# pip  list
Package              Version      Location                            
-------------------- ------------ ------------------------------------
deepspeech-training  0.9.0a3      /home/xyz/DeepSpeech/training
ds-ctcdecoder        0.9.0a3      
tensorboard          1.15.0       
tensorflow-estimator 1.15.1       
tensorflow-gpu       1.15.2

Yes I tried with a file from the train dataset.
Sorry if I asked dumb questions, but this is the versions of packages i have (running pip list shows more than these packages but i just kept the important ones) , i installed tensorflow 1.15, I don’t know why it is mentioned in the log that I use “TensorFlow: v2.2.0-15-g518c1d0”

#6

Maybe you didn’t turn on the virtual environment? Check the TF version, that should be it.

#7 
(deepspeech-venv) (base) root@xyz:/home/xyz/DeepSpeech# python3 -c 'import tensorflow as tf; print(tf.__version__)'
1.15.2

The virtual env is activated, name is deepspeech-venv.

#8

How did you come by the native client, within the virtual environment?

#9

I followed the documentation to decode a wav file, I just run deepspeech under the native_client directory obviously in the virtual env.
What bothers me is that I didn’t have this error before.

#10

Maybe install everything again in a different directory? Something seems to be corrupted.

#11

I’m going to reinstall everything, thanks for your help.

#12

What is “before” ? You request for support and you don’t even care to share basic context on your system.

#13

Don’t work as root, please.

#14

I mean that i trained my own model from scratch with the same configuration & params and tested it without errors

#15

@Bira I’m frustrated that you are cross-posting on discourse and github, while not taking the time to share properly your context and answers the questions I have asked. I’ve closed the github issue, but I will not try to help you until you answer ALL of the questions I asked there and here, and you properly describe your setup etc. Our time is not unlimited.

#16

@lissyx (Apologizes for my late answers but I am trying to read the documentation and understand as I go along so that I can provide you a correct answer.)

System ram : 16GB
Did you tried gpu binaries: No, I used a non gpu binaries
I don’t use Tflite model, i use an “output_graph.pb” to decode.

I trained my own model on a ubuntu 18.04 machine, under a venv, with almost 30 hours of acoustic data, with a customized alphabet and a language model (i generate my own scorer file)

#17

How much free?

You still have not explained whether those are our shipped prebuilt or if you rebuilt.

You still have not replied whether you repro the error on our shipped model files.

#18

I would rather prefer that you want and reply to everything instead of being constantly forced to re-ask you over and over. We’re at the 18th exchange, and I still cannot figure out if there’s a bug in the code or if you are doign somethign wrong. My time is not unlimited, I’m not your personal paid support.

#19 
free -m
total       utilisé      libre     partagé tamp/cache   disponible
Mem:          15853        4061         265         284       11527       11173

I don’t rebuilt binaries, i used prebuilt ones and i don’t understand your last question, you mean that if i evr test with your available english model?

#20

So you have only <4GB free, and you are loading a non mmap file. It’s 100% possible you are just running out of memory …