Training Vietnamese model

No no, I just want to know is my data too bad? Why all the results output ares the same.

i think you need to change the batchsize and learning rate ,your mode not convergence

I am working on Urdu language and facing the same issue.
Please help. If somebody find a solution of this problem.
My data is almost 100 hours.

@lissyx @kdavis Kindly help.

What does “same issue” mean in this context?

That my trained model gives me only one word for all files. Either I checked it on train files or test files.

When I trained on a very small corpus, almost 6 files, it learned only one word, though that word doesn’t belong to any set of data. and it gives the same word if I tried to decode either on train file or test file.

and When I tried on almost 100 hours, it doesn’t give any single word in the results.
Epochs: 20, with early stop = TRUE
Learning rate: 0.0001

Why is it so ?

What happens if you run run-ldc93s1.sh as follows:

(.virtualenv) kdavis-19htdh:DeepSpeech kdavis$ ./bin/run-ldc93s1.sh


WER: 1.000000, CER: 48.000000, loss: 27.773754

  • src: “she had your dark suit in greasy wash water all year”
  • res: “edted”
    This is the result of ldc93s1.

I’m not sure how but your install is very broken. This is basically a “smoke test” that every one of our PR’s has to pass.

If I were you, I’d check everything out from scratch and follow the README instructions again.

1 Like

Are we suppose to train the model while activating the virtual environment ?

It’s recommended that you use a virtual environment.

I followed the github guideline throughout and working on 0.4.1 master.
I also got the exported model with best validation loss.

Installation:
Linux: 16.04 LTS
CUDA 9.0
CUDNN= 7.1.3
Python 3.6.3
DeepSpeech 0.4.1 master
requirements.txt installed. but Tensorflow-gpu ==1.12.0
Installed LFS from the link given on github
Bazel 0.5.1
Downloaded and checked the pre-trained model from Common voice utterances and results are almost 99%
install CTC
Data prepared in CSV format.
Build the native client
have language model prepared from KenLM, generate trie file.

There is no error I got in training and got the .pb model in the end and also results gave me WER though it doesn’t have anything in decoded output.

Please let me know where I am doing wrong,
Thank you for all the help.

There are many steps here and a problem can creep in anywhere.

To help in debugging can you supply the final training log?

log.zip (106.9 KB)

Here it is.

Don’t know if it’s the encoding, but the log looks like line noise.

the data is in utf-8 and I believe, DeepSpeech support it. Isn’t ?
If I trained on 5-6 files, I got one word decode, but if I trained on a big corpus of 100 hours, nothing decoded.
Thank you so much.

When I open unzip log.zip and open events.out.tfevents.1557923145.Hafsa it looks like this…

I asked for the log, what’s printed out as training runs.