Error when training model

karthikeyank · 2019-01-09 08:36:26 UTC

the output is…

(deep4.0) userk@demosys:~/DeepSpeechPro/native_client4.0$ deepspeech --model models/output_graph.pbmm --alphabet models/alphabet.txt --lm models/lm.binary --trie models/trie --audio /mnt/c/users/karthikeyan/Downloads/chunk15.wav
Loading model from file models/output_graph.pbmm
TensorFlow: v1.12.0-10-ge232881
DeepSpeech: v0.4.0-0-g48ad711
2019-01-09 14:01:51.683099: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Loaded model in 0.27s.
Loading language model from files models/lm.binary models/trie
Loaded language model in 0.419s.
Running inference.
i can do and picture of minutes for eating for the saintly in bloom no deterrent
Inference took 81.658s for 6.060s audio file.

Please have a look…

kdavis · 2019-01-09 08:58:09 UTC

@karthikeyank I was asking @lissyx as I wanted to see if the process is indeterministic which it should not be.

karthikeyank · 2019-01-09 08:59:09 UTC

ohh… am sorry… I missed to read you mentioned lissyx…

kdavis · 2019-01-09 09:01:02 UTC

No problem. I just want a “second opinion” on the nondeterministic result, if indeed it is nondeterministic.

lissyx · 2019-01-09 09:02:29 UTC

alex@portable-alex:~/tmp/deepspeech$ ./tc-linux/deepspeech --model models/output_graph.pbmm --alphabet models/alphabet.txt -t --audio chunk15.wav 
TensorFlow: v1.12.0-10-ge232881
DeepSpeech: v0.4.0-0-g48ad711
2019-01-09 10:01:00.860650: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
i con dag ond pe tucople of minetes fel e gading for tree sod an tlition plovram nodidpersubrok te fint you
cpu_time_overall=4.45048
alex@portable-alex:~/tmp/deepspeech$ ./tc-linux/deepspeech --model models/output_graph.pbmm --alphabet models/alphabet.txt -t --audio chunk15.wav 
TensorFlow: v1.12.0-10-ge232881
DeepSpeech: v0.4.0-0-g48ad711
2019-01-09 10:01:09.841426: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
i con dag ond pe tucople of minetes fel e gading for tree sod an tlition plovram nodidpersubrok te fint you
cpu_time_overall=4.23568
alex@portable-alex:~/tmp/deepspeech$ ./tc-linux/deepspeech --model models/output_graph.pbmm --alphabet models/alphabet.txt -t --audio chunk15.wav --lm models/lm.binary --trie models/trie 
TensorFlow: v1.12.0-10-ge232881
DeepSpeech: v0.4.0-0-g48ad711
2019-01-09 10:01:26.518440: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
i can do and picture of minutes for eating for the saintly in bloom no deterrent
cpu_time_overall=4.08630
alex@portable-alex:~/tmp/deepspeech$ ./tc-linux/deepspeech --model models/output_graph.pbmm --alphabet models/alphabet.txt -t --audio chunk15.wav --lm models/lm.binary --trie models/trie 
TensorFlow: v1.12.0-10-ge232881
DeepSpeech: v0.4.0-0-g48ad711
2019-01-09 10:01:30.556242: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
i can do and picture of minutes for eating for the saintly in bloom no deterrent
cpu_time_overall=3.87949
alex@portable-alex:~/tmp/deepspeech$ ./tc-linux/deepspeech --model models/output_graph.pbmm --alphabet models/alphabet.txt -t --audio chunk15.wav --lm models/lm.binary --trie models/trie 
TensorFlow: v1.12.0-10-ge232881
DeepSpeech: v0.4.0-0-g48ad711
2019-01-09 10:01:34.473983: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
i can do and picture of minutes for eating for the saintly in bloom no deterrent
cpu_time_overall=3.74882

lissyx · 2019-01-09 09:03:36 UTC

alex@portable-alex:~/tmp/deepspeech$ md5sum models/*
7423967be98caae08a348a3c674ce20d  models/alphabet.txt
5f762eecdc4c4cc2068dc1a84ec57873  models/lm.binary
a27b6d0e0b3dfd1ee6174443694ced05  models/output_graph.pb
a3cafcb87fcf09d38ce58cbc41e5c681  models/output_graph.pbmm
fe25baa1de505592b79fdb95432d7ad0  models/output_graph.rounded.pb
1b2f760aca3e9db0d334c33751ac94ce  models/output_graph.rounded.pbmm
182f72835a19800a3564b5da75ffc526  models/trie

kdavis · 2019-01-09 11:21:37 UTC

Funny. Or not so funny. I’m the odd man out. My transcription doesn’t match.

Mine was

i can dig and people of minutes for a god in perfecting latin poem notepaper

but you both got

i can do and picture of minutes for eating for the saintly in bloom no deterrent

Out of curiosity what platforms are you on?

lissyx · 2019-01-09 11:23:09 UTC

AMD64 Linux/Ubuntu 18.10. Do you get the same result without LM loaded ?

karthikeyank · 2019-01-09 11:23:47 UTC

I’m on Windows Subsystem for Linux – Ubuntu 16.04

kdavis · 2019-01-09 13:06:32 UTC

I’m on OSX 10.13.6. Let me try without LM

karthikeyank · 2019-01-17 09:50:06 UTC

Hi @lissyx @kdavis… Is It possible to perform inference on DeepSpeech 0.4.1 model with a batch of 100 audio files at a time in a loop…

lissyx · 2019-01-17 10:22:57 UTC

If you write the code yes …

karthikeyank · 2019-01-17 10:25:56 UTC

I already tried this…
but the models are loaded on each iteration… Is there any way that I can avoid loading models each on iterations…

lissyx · 2019-01-17 11:12:43 UTC

That’s because you wrote the code in a way that loads the model at each iteration …

karthikeyank · 2019-01-17 11:15:05 UTC

Am Sorry… Am unaware how to do that… can you please suggest me any ways to achieve it…

lissyx · 2019-01-17 13:42:04 UTC

I’m pretty sure we already explained you what you need to do …