Generating own scorer file

Archives DeepSpeech
#1

Hi. here im goin to create a scorer file but it failed with message:

9 unique words read from vocabulary file.
Doesn’t look like a character based model.
Error: Can’t parse scorer file, invalid header. Try updating your scorer file.

Цитата

I cound find any related links to get help. Please help me. Thank you

#2

I could not find any steps to repro in your message. Hard to check what you did.

#3

This is my first step

 %cd /content/kenlm/build/bin
 !./lmplz --order 5 --memory 50% --temp_prefix 15 --text /content/deepspeech/uzbek/dictionary.txt --arpa  /content/deepspeech/uzbek/dictionary.arpa  --discount_fallback --prune 0 0 1
 !./build_binary -a 255 -s -q 8 trie   /content/deepspeech/uzbek/dictionary.arpa  /content/deepspeech/uzbek/lm.binary

This is a second one. below error appears after that

%cd /content/deepspeech/       
!python ./data/lm/generate_package.py --alphabet /content/deepspeech/uzbek/alphabet.txt --lm /content/deepspeech/uzbek/lm.binary --vocab /content/deepspeech/uzbek/dictionary.txt  --default_alpha 0.75 --default_beta 1.85 --package /content/deepspeech/uzbek/uzbek.scorer
#4

What’s the content of that file ?

#5

@Akmal_Nodirov Also, what exact commit are you on ? What’s pip list | grep ds_ctcdecoder ?

#6

the content is inside the file:
asslomu aleykum do’stim bu men ismim Akmal Ozodbek Shahzod

#7

what do you mean by exact commit ? ihave cloned last version of deepspeech. its 0.7 or upper than that. the last one

#8

What’s your HEAD at.

#9

I don’t see the -v we document in data/lm/generate_lm.py for that call and that generate_scorer.py --help advises you to ensure.

#10

image
image1123×91 11.4 KB
i think its this. Because i have cloned it approximately 9 hours ago

#11

I have added -v:
!./build_binary -a 255 -s -q 8 -v trie /content/deepspeech/uzbek/dictionary.arpa /content/deepspeech/uzbek/lm.binary

but appears this error:
./build_binary: invalid option – ‘v’
Usage: ./build_binary [-u log10_unknown_probability] [-s] [-i] [-w mmap|after] [-p probing_multiplier]

#12

or could you provide me a correct format of generating file, if you could all steps please. thank you

#13

I had to, sorry :smiley:

5 Likes
#14

Looks like you are not using proper version

#15

@Akmal_Nodirov Try and rebuild build_binary and others from KenLM master?

#16

Or maybe try with released v0.6.1 (if it is enough) and build lm.binary and trie files. Maybe this workflow will work smoother for you.

#17

Version of deepspech ? or some other thing ? . Is it possible to add dictionary to a newer versions of deepspeech ? this is my version : 0.7.0-alpha.2

#18

Version of KenLM. It seems we need polishing on this part of the project :confused:

#19

Просто удаляешь kenlm и скачиваешь kenlm из гитхаб https://github.com/kpu/kenlm
сработает.
Just delete kenlm and download kenlm from githab https://github.com/kpu/kenlm.
it’ll work.

#20

Update of kenlm does not help. I had the same issue and apparently what was changed with respect to v0.6.1 is that you need to provide -v argument to build_binary. The error Error: Can’t parse scorer file, invalid header. Try updating your scorer file. is not quite helpful here.

Now I get only Doesn't look like a character based model, but the package creation succeeds :smiley: