Train model but actual prediction is too poor

DeepSpeech
#1

Details:
I have less number of data-set around 1666
for training = 1249
dev = 396
test = 19

Command use for train model:

   python DeepSpeech.py \
                    --train_files all_augumented_samples/train.csv \
                   --dev_files all_augumented_samples/dev.csv \
                   --test_files all_augumented_samples/test.csv \
                   --epoch 13 \
                   --train_batch_size 10 \
                   --dev_batch_size 10 \
                   --test_batch_size 10 \
                   --learning_rate 0.00001 \
                   --export_dir export2/ \
                   --checkpoint_dir cpoint2/ \

after training model testing result are:

Test on all_augumented_samples
WER: 0.340000, CER: 0.070127, loss: 8.018482

I think results are good.
after saving model when i was check actual prediction is too much poor as compare to testing result why?

I was train model without batch size and with batch size but not success.
any suggestion please provide me.
Please help me

#2

Most likely, you need far more data.

1 Like
#3

how much minimum is there any restriction ?

command and batch are right?

#4

You could try lowering n_hidden down as well; you still would want more data.

Using approximately 2500 (about 3 hours total) utterances, I was never able to get a model to work with any level of accuracy.

1 Like
#5

Thanks @baconator for quick reply.
I have one question if loss is less as well as validation loss also then why actual prediction result too much poor ?

Is it totally depend on dataset?

#6

Yes, very dependent upon the data.

1 Like
#7

Thanks @baconator I am trying with more data set

1 Like
#8

WER: 0.340000 mean 34% right sir?

#9

Yes, 34% Word Error Rate, but your CER is pretty much good (7% Character Error Rate). Try using a Language Model to fix your model’s grammar errors. But, overall, try to have more test samples. Try first something like: 70% of your total data is for train, 15% is for dev and 15% is for test.

1 Like
#10

I was train model on around 2515 samples till Word Error Rate is not decrease.

@victornoriega7 as you said Try using a Language Model to fix your model’s grammar errors. can you please tell me in details?

#11

Use KenLM to build your own language model. Simply put, if you just want to recognize 10 words, only those should be in there and DeepSpeech will only find them. So by limiting or extending the language model you control what words will be recognized.

And you will probably need more training material.

1 Like
#12

@othiele I want to recognize around 300 command is it possible using KenLM

#13

Yes it is possible. For that to happen, you have to create a .txt file with those 300 commands in differents lines.
Then, you clone the KenLM from github.com/kpu/kenlm and build it. After that, there’s a executable bin/lmplz: the output will be a .arpa file. In your case I will recommend to run something like this:

bin/lmplz -o 1 < text.txt >text.arpa

the -o argument provides the order of the ngram. In most cases should be bigger, but if you just use commands, it should be just unigrams.

After that, transform your .arpa to a .binary. You can find how to do this in the same repo.

1 Like
#14

As you said I was follow steps:
1 vocab.txt [commands in separate line]
2 clone and build it
3 create arpa as well as lm.binary file

       bin/./lmplz --text vocab.txt --arpa words.arpa --o 3
       bin/./build_binary -T -s words.arpa  lm.binary

After that how to use it ?
now i have dataset and above those file.I am confuse how to use those file ?

#15

Try to change --o 3 with --o 1, just a recommendation.
So, you already have lm.binary from your vocab.txt right? I assume your vocab.txt is the file with your 300 commands and not your alphabet.

That lm.binary file is the one you provide as an argument as of version 0.6.1 of DeepSpeech. Something like this:

python DeepSpeech.py --test_files <your_test_file> --lm_binary_path <your_lm.binary_path>

and other parameters like the path of your saved model.

The last thing you need to build in 0.6.1 is your trie. For that you have to run a executable file called generate_trie, which you can compile if you follow this:

Remember to checkout to the tag 0.6.1 to use languages models like this.

1 Like
#16

Try to change --o 3 with --o 1 but getting error so I use --o 3

Yes right vocab contain 300 commands

I am using Deepspeech v0.5.1

              python DeepSpeech.py \
                --lm_binary_path bi_file/lm.binary \
                --train_files all_augumented_samples/train.csv \
                --dev_files all_augumented_samples/dev.csv \
                --test_files all_augumented_samples/test.csv \
                --epoch 13 \
                --train_batch_size 10 \
                --dev_batch_size 10 \
                --test_batch_size 10 \
                --learning_rate 0.0001 \
                --export_dir export2/ \

is it right sir?

It possible using DeepSpeech v0.5.1?

#17

Yes, you only need, as I said, the trie, is another file that you need to generate by yourself from the lm.binary that you created. Then, given the size of you data, I would recommend you to train with something like -n_hidden 1024 --dropout_rate 0.3 --learning_rate 0.0001.

Respect with the order of your lm, I didn’t know that you can’t use order 1. Did you try with --o 2 ?

Remember that is very important to create your own trie for your own language model binary.

#18

Thanks so much sir for help me

I was successfully generated trie .

Thanks once again

#19

@victornoriega7 after train using LM still word error rate not decrease

now i was train model using DeepSpeech v0.6.1

command:

python DeepSpeech.py \
--train_files 18_2_2020_CSV/Train/train.csv \
--dev_files 18_2_2020_CSV/Dev/dev.csv \
--test_files 18_2_2020_CSV/Test/test.csv \
--epochs 30 \
--learning_rate 0.00001 \
--export_dir lm_export/ \
--n_hidden 1024 \
--checkpoint_dir lm_checkpoint/ \
--lm_binary_path new_native_client/18_02_lm.binary \
--lm_trie_path new_native_client/trie \
--automatic_mixed_precision=True \

After Result are:
Test on 18_2_2020_CSV/Test/test.csv - WER: 0.993631, CER: 0.941896, loss: 31.854940

Please me please in this problem ? please

TUTORIAL : How I trained a specific french model to control my robot
#20

How much data do you have?