Error when training model


(karthikeyan k) #43

sir @muruganrajenthirean, are you saying the released model itself has enough knowledge, so no need of fine tuning. or after 3 epochs only it will reflect the proper knowledge.


(karthikeyan k) #44

actually i tried to infer phone call samples using the v0.3.0 models, but i could not get good results, thats why am fine tuning it on the same phone call samples. but when i finished one epoch and try inferencing I thought it would perform with the existing knowledge and added curernt data knowledge. but it seems its a fresh model with only the current data knowledge…


(Murugan R) #45

@karthikeyank sir. the released model itself has enough knowledge
it is working very well. but if you need only a particular purpose, like financial, education, call center then you follow finetuning concept (your own dataset) or alternative if you have more data then you will build own model.


(karthikeyan k) #46

sir @muruganrajenthirean, My assumption is
fine tuning a model will add the currently training data knowledge to the already existing knowledge. so that we can make the model more accurate.
is that correct. or
fine tuning a model will only have the knowledge of the currently training data and we will get a fresh model without the existing v0.3.0 knowledge. …


(Lissyx) #47

@josh_meyer is the expert for fine-tuning and transfer-learning, and the behavior of the network might not be as trivial as you can think in those case :slight_smile:


(karthikeyan k) #48

hello @josh_meyer, which of the following assumptions are correct with respect to fine tuning v0.3.0 model either from checkpoint or frozen graph,

  1. fine tuning a model will add the currently training data knowledge to the already existing knowledge. so that we can make the model more accurate .

or

  1. fine tuning a model will only have the knowledge of the currently training data and we will get a fresh model without the existing v0.3.0 knowledge .

(Lissyx) #49

I just explained to you I’m not the expert on that.


(karthikeyan k) #50

ohh okay I thought you might know it… well I edit that post…


#51

Hi @karthikeyank!

It depends on what you mean by “retains original knowledge.”

Fine tuning the model to a new accent (or even language) in some sense retains some of the knowledge that was learned in the original model… whatever “knowledge” was useful in the new dataset.

However, if you use a new alphabet in fine-tuning, then you will lose the ability to recognize any letters that exist in the original dataset but not the new dataset.

Does that answer your question?


(karthikeyan k) #52

thank you for your answer @josh_meyer

the term fine tuning the model means improving the capability of the model to produce more accurate results right. So when we improve the model, its should add the new knowledge to the existing knowledge and improving its capability right…
please clarify…

  1. fine tuning the model will produce a new model for the current dataset/language.
    ( or )
  2. fine tuning the model will produce a more accurate model with the current dataset/language knowledge added to the older model knowledge.

(karthikeyan k) #53

Hi @lissyx @muruganrajenthirean… The model training has completed for three epochs and model succesfully exported. but when i tried to inference it produces empty responses… can you please have a look into it…

$ deepspeech --model 
/home/userk/DeepSpeechPro/tuned_model/models/output_graph.pb -- 
audio /mnt/c/users/karthikeyan/downloads/chunk9008.wav --alphabet 
/home/userk/DeepSpeechPro/native_client/models/alphabet.txt --lm 
/home/userk/DeepSpeechPro/native_client/mod
els/lm.binary --trie /home/userk/DeepSpeechPro/native_client/models/trie
Loading model from file 
/home/userk/DeepSpeechPro/tuned_model/models/output_graph.pb
TensorFlow: v1.11.0-9-g97d851f
DeepSpeech: v0.3.0-0-gef6b5bd
Warning: reading entire model file into memory. Transform model file into            
an mmapped graph to reduce heap usage.
2018-12-31 10:46:53.525642: I         
tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports     
instructions that this TensorFlow binary was not compiled to use: AVX2     
FMA
Loaded model in 0.485s.
Loading language model from files 
/home/userk/DeepSpeechPro/native_client/models/lm.binary         
/home/userk/DeepSpeechPro/native_client/models/trie
Loaded language model in 3.09s.
Running inference.

Inference took 9.064s for 2.564s audio file.

thank you…!!


(Murugan R) #54

i think @karthikeyank sir model doesn’t learn properly. please train on more epoches and reduce validation loss. :slightly_smiling_face:


(karthikeyan k) #55

hai @lissyx @muruganrajenthirean… I tried inferencing a model released by reuben. I installed all the packages mentioned in the requirement.txt .
But when making inference it throws this error, Do you have any idea about it…

deepspeech --model         
/mnt/c/users/karthikeyan/downloads/output_graph.pbmm --audio     
/mnt/c/users/karthikeyan/downloads/chunk15.wav --alphabet 
models/alphabet.txt --lm models/lm.binary --trie models/trie
Loading model from file 
/mnt/c/users/karthikeyan/downloads/output_graph.pbmm
TensorFlow: v1.11.0-9-g97d851f
DeepSpeech: v0.3.0-0-gef6b5bd
2019-01-07 15:42:35.813162: I 
tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports 
instructions that this TensorFlow binary was not compiled to use: AVX2 
FMA
Invalid argument: No OpKernel was registered to support Op 'Softmax' 
with these attrs.  Registered devices: [CPU], Registered kernels:
<no registered kernels>

     [[{{node Softmax}} = Softmax[T=DT_FLOAT](raw_logits)]]
Traceback (most recent call last):
File "/home/userk/.local/bin/deepspeech", line 11, in <module>
sys.exit(main())
File "/home/userk/.local/lib/python3.5/site-        
packages/deepspeech/client.py", line 81, in main
ds = Model(args.model, N_FEATURES, N_CONTEXT, args.alphabet, 
BEAM_WIDTH)
File "/home/userk/.local/lib/python3.5/site- 
packages/deepspeech/__init__.py", line 14, in __init__
raise RuntimeError("CreateModel failed with error code {}".format(status))
RuntimeError: CreateModel failed with error code 3

Thank you, :slight_smile:


(Lissyx) #56

Looks like you are trying to run post 0.3.0 models, with Softmax layer, with a 0.3.0 binary that does not have Softmax code :).

Try 0.4.0-alpha.3 ?


(karthikeyan k) #57

the DeepSpeech model is 0.2.0 but the language model and trie files are from 0.3.0. I tried without the language model and trie file, same error persists.


(karthikeyan k) #58

does the 0.4.0-alpha.3 contains pretrained models, I am just trying inferencing with 0.2.0 model just because people say its better than 0.3.0… that’s why…


(karthikeyan k) #59

And trying transcribing audio files which are converted using SoX and processed using Audacity, will this affect the final outcome… ( In a github issue @lissyx you recommended to use ffmpeg)…


(Lissyx) #60

That’s no 0.2.0, that’s 0.3.0 with softmax. So please, try to use the proper combination of models and binaries. We are working on releasing updated 0.4.0 models.


(karthikeyan k) #61

okay… thank you… :slight_smile:


(karthikeyan k) #62

Hi @lissyx, I downloaded the DeepSpeech 0.4.0 model and made inference , I’m getting different results when compared to 0.3.0 model… The results are here…

Actual/Expected Transcription :::::::

i can i can speak to you couple of minutes for regarding for free solar installation program not interested brother thank you

DeepSpeech Inference - - -

DeepSpeech 0.3.0 ::::::

~$ deepspeech --model models/output_graph.pbmm --audio /mnt/c/users/karthikeyan/downloads/chunk15.wav --alphabet models/alphabet.txt --lm models/lm.binary --trie models/trie

result :
i can dig and speak to a couple of moneteforegardingfortreesolanstlationplovlam no dinpersubertecank

DeepSpeech 0.4.0 ::::::

~$ deepspeech --model models/output_graph.pbmm --audio /mnt/c/users/karthikeyan/downloads/chunk15.wav --alphabet models/alphabet.txt --lm models/lm.binary --trie models/trie

result :
i can degeneration lithospermoides

Did I missed any parameter or any other things… which model you would suggest the best model for inference…

Thank You, :slight_smile: