Segmentation fault during transciption - DS 0.4.1/Spanish

I installed deepspeech 0.4.1, as instructed by the Spanish deepspeech github repo, on a RedHat 7 server with 64GB RAM.

I am getting a “Segmentation Fault" error on both shorty after it starts processing.

Here is an example output:

-bash-4.2$ deepspeech --model models/output_graph.pb --alphabet models/alphabet.txt --lm models/lm.binary --trie models/trie --audio file.wav
Loading model from file models/output_graph.pb
TensorFlow: v1.12.0-10-ge232881
DeepSpeech: v0.4.1-0-g0e40db6
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2020-01-22 09:21:45.252368: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: FMA
Loaded model in 0.365s.
Loading language model from files models/lm.binary models/trie
Loaded language model in 0.181s.
Running inference.

After this line comes the dreaded “Segmentation Fault

Any idea what causes it or how to solve it?

Thanks.

Without more information, it’s not actionable, and 0.4.1 is an old release now, we can’t really do anything.

I’m discovering this repo with your link …

What more info can I add? I do not see any log file or any output to screen.
Also, I am using 0.4.1 since the Spanish model was trained using 0.4.1. I don’t think I can use the latest version, unless I retrain the Spanish model using that version, can I?

You’re right.

A gdb stack at first, but you might need to rebuild with debug info …

@amnonkhen I don’t even reproduce your segfault:

$ ./deepspeech --model output_graph.pb --alphabet alphabet.txt --audio ../test-alex.en.wav -t
TensorFlow: v1.12.0-10-ge232881
DeepSpeech: v0.4.1-0-g0e40db6
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2020-02-03 13:12:21.783527: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
 su saz e ser lo del rojello
cpu_time_overall=1.54403
$ ./deepspeech --model output_graph.pb --alphabet alphabet.txt --lm lm.binary --trie trie --audio ../test-alex.en.wav -t
TensorFlow: v1.12.0-10-ge232881
DeepSpeech: v0.4.1-0-g0e40db6
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2020-02-03 13:12:37.165594: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
se ha de ser lo del rollo
cpu_time_overall=1.19909

Running on Ubuntu 19.10.

Thanks @lissyx, for the efforts.
The differences between us are:

  • I am using RedHat 7 while you are using Ubuntu 19
  • It appears that I am using a far larger file than you.
    Do you think these might be related?
    Thanks

Without at least a gdb stack, we can only waste our times. IMHO given the amount of data this model was trained, a much efficient use of your time would be to just retrain on current codebase.

108 hours of data, with CUDNN and mixed precision enabled, that’s going to be pretty fast (depends on your hw, though).

You might get inspiration from commonvoice-fr/DeepSpeech/Dockerfile.train at master · common-voice/commonvoice-fr · GitHub (currently for 0.6.1, I’m working to update for current 0.7.0a0).

How much larger ? What ressources do you have on that system ?

Are you using the C++ binaries, or some bindings ?