Using Deep Speech

DeepSpeech
#21

Hello, what are the training data sets that went into the model that is available at https://github.com/mozilla/DeepSpeech/releases?

Will you release a fully trained NN?
#22

LibriSpeech[1], Fisher[2,3,4,5], and Switchboard[6]

2 Likes
#23

Hi,
I was wondering on execution time:

Inference took 3.607s for 1.393s audio file.

Is this normal exec time, since I have seen some examples online that 30s was needed for 28sec of audio file.
Also I know few examples where usual time is half of duration of audio.
I haven’t tried GPU powered deepspeech since my hardware+OS is in fight with Nvidia atm.

Thanks,
mirko

#24

Unfortunately, as to whether this is “normal” is all hardware dependent.

#25

Thanks for reply :slightly_smiling_face:
So if I prepare a more powerful GPU box, I should expect much better results.
The only reason is that some of the proprietary software brag about 1/0.5 ratio of duration/transcription …

#26

A 1/0.5 ratio should be achievable on a GeForce GTX 1070 or above for clips a few sec long.

#27

Great, thanks for info kdavis :slight_smile:
Regards,
mirko

#28

How i can use the pre trained model?

#29

The README of the current v.0.1.1 release describes usage.

#30

Hi! I got same error, did yu fixed it?

#31

anybody can fix? :roll_eyes::roll_eyes::roll_eyes::frowning:

#32

Please avoid hijacking threads, properly document your error and your setup, otherwise, nobody can help you.

2 Likes
#33

based on snapshot by @yesterdays, python 2.7.5 is used. use python 3.5 instead. i am using python 3.5 and it worked fine.

#34

Hi All,
I am trying to train and use a model for English from scratch on version 0.5.1. My aim to train two models, one with and without a language model. Request your help on several fronts please. Sorry this is long but trying be as detailed as possible; and also, being new to Linux and data-science I may be stating some very obvious things.
Thank you in advance for your help.
Regards,
Rohit

A) Background:

A1) Using Ubuntu 18.04LTS, no GPU, 32GB ram.

  • Downloaded Mozilla Common Voice Corpus (English) around mid-June 2019.

  • Took the validated.tsv file, did some basic transcript validation and pruned dataset to 629731 entries.

  • Selected first 10k entries and split using ratio of 70:20:10 as train:dev:test and created csv files.

  • MP3s converted to wav files (16kHz, mono, 16bit), length less than 10 seconds.
    Setup Anaconda environment with Deepspeech v0.5.1.

  • Cloned github v0.5.1 code.

  • Issued command in the Deepspeech folder, which seems to be required to create the generate_trie executable and other required setup:
    python util/taskcluster.py --target .

  • Installed the CTC-decoder from the link obtained from command:
    python util/taskcluster.py --decoder

  • Next created vocabulary file with only the transcripts.

  • No changes in any of the flags and other default parameters.

A2) Language model related:

  • Used KenLM. Downloaded from git repo and compiled. Commands to create 4-gram version:

  • vocabulary file to arpa:

./lmplz -o 4 --text /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/vocabDir/vocabulary-Set3First10k.txt --arpa /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/vocabDir/vocabulary-Set3First10k_4gram.arpa

  • arpa to lm_binary file:

./build_binary /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/vocabDir/vocabulary-Set3First10k_4gram.arpa /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/lm/lm4gram/vocabulary-Set3First10k_4gram.klm

  • used the generate_trie to make the trie file

/home/rohit/dpspCODE/v051/DeepSpeech/generate_trie /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/alphabetDir/alphabet-Set3First10k.txt /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/lm/lm4gram/vocabulary-Set3First10k_4gram.klm /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/trie/trie4gram/set3First10k_4gram.trie

  • Note the trie file was made successfully and later used to start training.

A3) Commands to start model training (training in progress still):

A3a) Model without language model:

python3 -u DeepSpeech.py
–train_files /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/csvFiles/train.csv
–dev_files /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/csvFiles/dev.csv
–test_files /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/csvFiles/test.csv
–train_batch_size 1
–dev_batch_size 1
–test_batch_size 1
–n_hidden 2048
–epoch 20
–dropout_rate 0.15
–learning_rate 0.0001
–export_dir /home/rohit/dpspTraining/models/v051/model5-validFirst10k-noLM/savedModel
–checkpoint_dir /home/rohit/dpspTraining/models/v051/model5-validFirst10k-noLM/checkpointDir
–alphabet_config_path /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/alphabetDir/alphabet-Set3First10k.txt
“$@”

A3b) Model with Language model:

python3 -u DeepSpeech.py
–train_files /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/csvFiles/train.csv
–dev_files /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/csvFiles/dev.csv
–test_files /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/csvFiles/test.csv
–train_batch_size 1
–dev_batch_size 1
–test_batch_size 1
–n_hidden 2048
–epoch 20
–dropout_rate 0.15
–learning_rate 0.0001
–export_dir /home/rohit/dpspTraining/models/v051/model6-validFirst10k-yesLM-4gram/savedModel
–checkpoint_dir /home/rohit/dpspTraining/models/v051/model6-validFirst10k-yesLM-4gram/checkpointDir
–decoder_library_path /home/rohit/dpspCODE/v051/DeepSpeech/native_client/libctc_decoder_with_kenlm.so
–alphabet_config_path /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/alphabetDir/alphabet-Set3First10k.txt
–lm_binary_path /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/lm/lm4gram/vocabulary-Set3First10k_4gram.klm
–lm_trie_path /home/rohit/dpspTraining/data/wavFiles/commVoiceSet3-10kTotal/trie/trie4gram/set3First10k_4gram.trie
“$@”

B) My questions:

B1) When using a language model either for training or inference, do I HAVE to specify the lm_binary parameter AND the corresponding trie file? Can using only the trie work?

B2) Irrespective of whether a language model was used while training the model (binaryFile and trie together), later when the model is used for inference, can I choose to use OR not use a language model? Can a different language model be used later or only the one used for training? Are there things to note while choosing an alternative model? E.g. training using a 3-gram model but using a 4-gram model during inference? etc…

B3) Suppose my model is already built by training on a vocabulary file, arpa, trie and lm_binary built from only 10k data points. Say I create a new vocabulary called BigVocabulary.file from a larger corpus than the one used for training.

E.g. the entire 629731 data points in validated.tsv file; use bigger vocabulary to create the .arpa, lmBinary and trie files. I ensure that the valid characters are exactly the same by comparing the alphabet files. Then on the model trained with smaller vocabulary, can I use BigVocabulary.binary.file and BigVocabulary.trie while doing inference using the command?

I already created a model with only first 1000 files and inference is poor but works.
Command:

deepspeech
–model /home/rohit/dpspTraining/models/v051/model8-validFirst1k-yesLM-4gram/savedModel/output_graph.pb
–alphabet /home/rohit/dpspTraining/data/wavFiles/commVoiceSet5-1kTotal/alphabetDir/alphabet-Set5First1050.txt
–lm /home/rohit/dpspTraining/data/wavFiles/commVoiceSet5-1kTotal/lm/lm4gram/vocabulary-Set5First1050_4gram.klm
–trie /home/rohit/dpspTraining/data/wavFiles/commVoiceSet5-1kTotal/trie/trie4gram/Set5First1050_4gram.trie
–audio /home/rohit/dpspTraining/data/wavFiles/wav33/test/File28.

Console output:

(dpsp5v051basic) rohit@DE-W-0246802:~/dpspCODE/v051/DeepSpeech$ deepspeech \

–model /home/rohit/dpspTraining/models/v051/model8-validFirst1k-yesLM-4gram/savedModel/output_graph.pb
–alphabet /home/rohit/dpspTraining/data/wavFiles/commVoiceSet5-1kTotal/alphabetDir/alphabet-Set5First1050.txt
–lm /home/rohit/dpspTraining/data/wavFiles/commVoiceSet5-1kTotal/lm/lm4gram/vocabulary-Set5First1050_4gram.klm
–trie /home/rohit/dpspTraining/data/wavFiles/commVoiceSet5-1kTotal/trie/trie4gram/Set5First1050_4gram.trie
–audio /home/rohit/dpspTraining/data/wavFiles/wav33/test/File28.wav
Loading model from file /home/rohit/dpspTraining/models/v051/model8-validFirst1k-yesLM-4gram/savedModel/output_graph.pb
TensorFlow: v1.13.1-10-g3e0cc53
DeepSpeech: v0.5.1-0-g4b29b78
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2019-08-01 16:11:02.155443: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-08-01 16:11:02.179690: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel (‘op: “UnwrapDatasetVariant” device_type: “CPU”’) for unknown op: UnwrapDatasetVariant
2019-08-01 16:11:02.179740: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel (‘op: “WrapDatasetVariant” device_type: “GPU” host_memory_arg: “input_handle” host_memory_arg: “output_handle”’) for unknown op: WrapDatasetVariant
2019-08-01 16:11:02.179756: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel (‘op: “WrapDatasetVariant” device_type: “CPU”’) for unknown op: WrapDatasetVariant
2019-08-01 16:11:02.179891: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel (‘op: “UnwrapDatasetVariant” device_type: “GPU” host_memory_arg: “input_handle” host_memory_arg: “output_handle”’) for unknown op: UnwrapDatasetVariant
Loaded model in 0.0283s.
Loading language model from files /home/rohit/dpspTraining/data/wavFiles/commVoiceSet5-1kTotal/lm/lm4gram/vocabulary-Set5First1050_4gram.klm /home/rohit/dpspTraining/data/wavFiles/commVoiceSet5-1kTotal/trie/trie4gram/Set5First1050_4gram.trie
Loaded language model in 0.068s.
Running inference.
a on a in a is the
Inference took 0.449s for 3.041s audio file.

But if i use the BigVocabulary.trie and lmBinary files then I get an error saying “Error: Trie file version mismatch (4 instead of expected 3). Update your trie file.” but it still seems to load the language model. So did Deepspeech actually pick it up and apply it correctly? How do I fix this error?

Command:

deepspeech
–model /home/rohit/dpspTraining/models/v051/model8-validFirst1k-yesLM-4gram/savedModel/output_graph.pb
–alphabet /home/rohit/dpspTraining/data/wavFiles/commVoiceSet5-1kTotal/alphabetDir/alphabet-Set5First1050.txt
–lm /home/rohit/dpspTraining/data/wavFiles/testVocabAllValidated/lm/lm4gram/vocabulary-allValidated_o4gram.klm
–trie /home/rohit/dpspTraining/data/wavFiles/testVocabAllValidated/trie/trie4gram/allValidated_o4gram.trie
–audio /home/rohit/dpspTraining/data/wavFiles/wav33/test/File28.wav

Console output:

(dpsp5v051basic) rohit@DE-W-0246802:~/dpspCODE/v051/DeepSpeech$ deepspeech \

–model /home/rohit/dpspTraining/models/v051/model8-validFirst1k-yesLM-4gram/savedModel/output_graph.pb
–alphabet /home/rohit/dpspTraining/data/wavFiles/commVoiceSet5-1kTotal/alphabetDir/alphabet-Set5First1050.txt
–lm /home/rohit/dpspTraining/data/wavFiles/testVocabAllValidated/lm/lm4gram/vocabulary-allValidated_o4gram.klm
–trie /home/rohit/dpspTraining/data/wavFiles/testVocabAllValidated/trie/trie4gram/allValidated_o4gram.trie
–audio /home/rohit/dpspTraining/data/wavFiles/wav33/test/File28.wav
Loading model from file /home/rohit/dpspTraining/models/v051/model8-validFirst1k-yesLM-4gram/savedModel/output_graph.pb
TensorFlow: v1.13.1-10-g3e0cc53
DeepSpeech: v0.5.1-0-g4b29b78
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2019-08-01 16:11:58.305524: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-08-01 16:11:58.322902: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel (‘op: “UnwrapDatasetVariant” device_type: “CPU”’) for unknown op: UnwrapDatasetVariant
2019-08-01 16:11:58.322945: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel (‘op: “WrapDatasetVariant” device_type: “GPU” host_memory_arg: “input_handle” host_memory_arg: “output_handle”’) for unknown op: WrapDatasetVariant
2019-08-01 16:11:58.322956: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel (‘op: “WrapDatasetVariant” device_type: “CPU”’) for unknown op: WrapDatasetVariant
2019-08-01 16:11:58.323063: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel (‘op: “UnwrapDatasetVariant” device_type: “GPU” host_memory_arg: “input_handle” host_memory_arg: “output_handle”’) for unknown op: UnwrapDatasetVariant
Loaded model in 0.0199s.
Loading language model from files /home/rohit/dpspTraining/data/wavFiles/testVocabAllValidated/lm/lm4gram/vocabulary-allValidated_o4gram.klm /home/rohit/dpspTraining/data/wavFiles/testVocabAllValidated/trie/trie4gram/allValidated_o4gram.trie
Error: Trie file version mismatch (4 instead of expected 3). Update your trie file.
Loaded language model in 0.00368s.
Running inference.
an on o tn o as te tee
Inference took 1.893s for 3.041s audio file.

Thank you for your time.

#35

I think this should be in a new thread.

However, to the best of my knowledge, the language model doesn’t affect training or validation, it is just used in the decoding step while testing the trained model. So, the is no mean of training two different models.

#36

Thank you for your inputs @SamahZaro . As suggested, I will try and move this to a new thread too.

#37

As suggested @SamahZaro , I moved the question to Language Model during training effect and @lissyx saw it and was kind enough to give some inputs. Thanks again.