Training Chinese model

jackhuang · April 20, 2018, 5:06am

I use the datasets which include:
“RASC863-G2――六大方言地方普通话语音语料库-朗读部分（粗标库）” ,
“RASC863——四大方言普通话语音语料库——朗读部分”,
“CASIA南方口音语音库”，“CASIA北方口音语音库" and “THCHS30”.
And there are about 3000 speakers in my training data.
You can learn the introduction of these datasets from:
http://www.chineseldc.org/
and
http://www.openslr.org/18/
2. When I change the learning rate to 0.0001, the WER decreases a lot.

jageshmaharjan · May 10, 2018, 8:04am

Hi @jackhuang, have you trained with “http://www.openslr.org/18” dataset.
I got this error;

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/home/maybe/anaconda3/envs/asr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/maybe/anaconda3/envs/asr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/data/jugs/asr/DeepSpeech/util/feeding.py", line 146, in _populate_batch_queue
    source = audiofile_to_input_vector(wav_file, self._model_feeder.numcep, self._model_feeder.numcontext)
  File "/data/jugs/asr/DeepSpeech/util/audio.py", line 66, in audiofile_to_input_vector
    fs, audio = wav.read(audio_filename)
  File "/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/scipy/io/wavfile.py", line 233, in read
    fid = open(filename, 'rb')
TypeError: integer argument expected, got float

However, all wave file seems mono, with 16-bit.
I checked with sicpy.io.wave ;
int16
[ 20 -467 -825 … -81 -141 -233]
int16
[-161 -151 -151 … 44 126 187]
int16
[-292 -291 -261 … -123 -71 -51]
int16
[-124 -120 -106 … 8 38 29]

and also with sox command, seems to be all ok.

Input File     : 'train/C8_749.wav'
Channels       : 1
Sample Rate    : 16000
Precision      : 16-bit
Duration       : 00:00:07.88 = 126000 samples ~ 590.625 CDDA sectors
File Size      : 252k
Bit Rate       : 256k
Sample Encoding: 16-bit Signed Integer PCM

Any, preprocessing require for those wave files ?

jageshmaharjan · May 11, 2018, 4:29am

Hi @jackhuang, As you mention earlier, the transcript is divided by space on each characters. can you tell me what alphabets did u use to generate trie file. And, I believe your vocabulary contains the Chinese characters but not the pinyin, to train your language model. Also, can you tell me what ‘n’ did u use in n-gram model.

jackhuang · May 16, 2018, 3:32am

The alphabets contains about 6800 characters. The alphabets is generated by the corpus (which contains 10 million 600 thousand sentences,322 million 540 thousand characters from the Internet). It seems that the characters is widely used.
Yes, the vocabulary only contains Chinese characters.
I have used 3-gram and 4-gram. It seems that 4-gram is better.

jackhuang · May 16, 2018, 3:42am

I have used the “THCHS-30” data set. The audio data can be used without any transformation.
I used the 0.1.1 version(Tensorflow 1.4.0, python 2.7). I guess you need to modify the code which is in charge of reading audio.

jageshmaharjan · May 16, 2018, 3:50am

Thanks @jackhuang . So, the alphabets gonna be a huge. Thanks again.

jackhuang · May 16, 2018, 3:56am

Recently, I have used the geometry of 1024. The training process is faster(convergence needs less epoches), but the model is over fitting(The WER decrease on some test sets, and increase on some other test sets).
I wish I can do my job better and share my results at first. Thanks for your kindness. I have got a lot of help from the project DeepSpeech, thank you very much!

jackhuang · May 16, 2018, 4:00am

Yes, so the decoding process is slow, I set the beam width to 100. And the speed of decoding is about 4x (using Nvidia GTX 1080Ti). It seems that the warp-ctc can solve that problem, but I haven’t used it successfully. Maybe you can try it.

kdavis · May 18, 2018, 12:39pm

I’d guess that a width of 1024 is insufficient for Chinese.

For intuition on widths see the results here[1] for English, which is an easier case. In particular the “issue1241 (LSTM BRNN Width)” tab

gjl1682002 · September 25, 2018, 10:26am

请教一下： loss 停在 230或207左右收敛不下去的，后来怎么解决的啊？

azureapertus01 · October 19, 2018, 7:56am

Hi, @jackhuang your alphabets are really huge. It may cause a lot of time for testing.
In my case, I’m using around 100 hours data as training data set, and it take around 280secs for 10 secs. audio.
Did you do any optimization? Otherwise, It’s really a problem for me.

myrainbowandsky · March 6, 2019, 9:34am

Traceback (most recent call last):
File ./DeepSpeech-0.4.1/util/text.py", line 37, in label_from_string
return self._str_to_label[string]
KeyError: ‘我’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “./DeepSpeech.py”, line 941, in
tf.app.run(main)
File “/home/xu/.local/lib/python3.5/site-packages/tensorflow/python/platform/app.py”, line 125, in run
_sys.exit(main(argv))
File “./DeepSpeech.py”, line 893, in main
train()
File “./DeepSpeech.py”, line 388, in train
hdf5_cache_path=FLAGS.train_cached_features_path)
File “/home/xu/DeepSpeech-0.4.1/util/preprocess.py”, line 69, in preprocess
out_data = pmap(step_fn, source_data.iterrows())
File “/home/xu/DeepSpeech-0.4.1/util/preprocess.py”, line 13, in pmap
results = pool.map(fun, iterable)
File “/usr/lib/python3.5/multiprocessing/pool.py”, line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File “/usr/lib/python3.5/multiprocessing/pool.py”, line 608, in get
raise self._value
File “/usr/lib/python3.5/multiprocessing/pool.py”, line 119, in worker
result = (True, func(*args, **kwds))
File “/usr/lib/python3.5/multiprocessing/pool.py”, line 44, in mapstar
return list(map(*args))
File “/home/xu/DeepSpeech-0.4.1/util/preprocess.py”, line 23, in process_single_file
transcript = text_to_char_array(file.transcript, alphabet)
File “/home/xu/DeepSpeech-0.4.1/util/text.py”, line 68, in text_to_char_array
return np.asarray([alphabet.label_from_string© for c in original])
File “/home/xu/DeepSpeech-0.4.1/util/text.py”, line 68, in
return np.asarray([alphabet.label_from_string© for c in original])
File “/home/xu/DeepSpeech-0.4.1/util/text.py”, line 48, in label_from_string
).with_traceback(e. traceback )
File “/home/xu/DeepSpeech-0.4.1/util/text.py”, line 37, in label_from_string
return self._str_to_label[string]
KeyError: '\n ERROR: You have characters in your transcripts\n which do not occur in your data/alphabet.txt\n file. Please verify that your alphabet.txt\n contains all neccessary characters. Use\n util/check_characters.py to see what characters are in\n your train / dev / test transcripts.\n

The trained csv is in Chinese. I have double checked the words. They are all included. Both alphabet.txt and the csv are coded in utf-8. What happened?

I have checked KeyError in self._str_to_label[string] of DeepSpeech/util/text.py when training own model

but failed.

kdavis · March 6, 2019, 10:52am

@myrainbowandsky In future could you please not double, triple, quadruple… post the same problem.

huangtianyu · March 22, 2019, 1:40am

hi,I trained “data_thcs30”.It can run and exported model file correctly,but it output nothing when i use the model to infer. I use the following command:

deepspeech --model /tmp/DeepSpeech/data_thchs30/model_201903201911/output_graph.pbmm
–alphabet /tmp/dataset/data_thchs30/csv/alphabet.txt
–lm /tmp/data_thchs30/csv/lm.binary
–trie /tmp/data_thchs30/csv/trie
–audio /tmp/data_thchs30/test/D8_999.wav

691290164 · March 22, 2019, 7:51am

Hi Jack,

I am building a RPG AR Game. We need a special lady voice for our main role Miss U. Could u help us on the project ?

Thank you,
Harry Chen
Wetcat: chenyd00
Hp: +86 18050283030

hm876556280 · March 27, 2019, 1:45am

deepspeech --model /tmp/DeepSpeech/data_thchs30/model_201903201911/output_graph.pbmm
–alphabet /tmp/dataset/data_thchs30/csv/alphabet.txt
–lm /tmp/data_thchs30/csv/lm.binary
–trie /tmp/data_thchs30/csv/trie
–audio /tmp/data_thchs30/test/D8_999.wav

@huangtianyu 您好，我想问一下，您运行这段代码的时候没有报错吗？
您是用的哪个版本呢？
我运行这段代码的时候：
Invalid argument: No OpKernel was registered to support Op ‘Slice’ with these attrs. Registered devices: [CPU,GPU], Registered kernels:

     [[{{node Slice}} = Slice[Index=DT_INT32, T=DT_INT32](Shape_1, Slice/begin, Slice/size)]]

raul.tang.lc · April 4, 2019, 1:10pm

I was successfully run the model but got the result:
Test - WER: 1.000000, CER: 5.000000, loss: 0.973997

WER: 1.000000, CER: 5.000000, loss: 0.973997

src: "您好在吗 "
res: “”

With the space separated for each character got the same:

Test - WER: 1.000000, CER: 8.000000, loss: 0.967288

WER: 1.000000, CER: 8.000000, loss: 0.967288

src: "您好在吗 "
res: “”

Anything I have missed?

raul.tang.lc · April 21, 2019, 11:23am

@huangtianyu

I think my problem was same with you, using data_thcs30 data set, but infer got nothing, do you get this resolved?

allen23777 · April 27, 2019, 8:02pm

I am working on a prototype now based on the thchs30 data set now and running into similar issue here.

I am using
–n_hidden 100
–learning_rate 0.0001
and batch size as 3 to train the model.

Have you figured out how to solve the problem?

Hung_Phung · April 22, 2021, 1:37am

@allen23777 @raul.tang.lc
Have you solved the problem, can you tell me? Please

Training Chinese model

I was successfully run the model but got the result: Test - WER: 1.000000, CER: 5.000000, loss: 0.973997

Test - WER: 1.000000, CER: 8.000000, loss: 0.967288

I was successfully run the model but got the result:
Test - WER: 1.000000, CER: 5.000000, loss: 0.973997