Deep Speech v0.4.1 Released

(kdavis) #1

We just released Deep Speech 0.4.1!

(karthikeyan k) #2

In the Known Issues tab of DeepSpeech 0.4.0 release you mentioned

Incorrect model was uploaded to release which will be fixed in 0.4.1…

is this (0.4.1) release has the correct model. will it produce good results than deepspeech 0.3.0…

and will it solve the issues discussed here

(kdavis) #3

This is the link to 0.4.1 and as stated in the 0.4.0 release the correct model is uploaded in the 0.4.1 release.

As to 0.4.1. vs 0.3.0 results, see the release notes for both for a comparison.

(Carlos Fonseca) #4

Hi @kdavis, I just did a WER test for the Windows client, here’s the result:

Estable RAM usage of 1,7GB.

The test took about 3h on a virtual Intel Xeon Platinum 8168 @ 2.7GHz vcores 16

WER 8,87% with LM enabled.

You can see the tool that I wrote here

I noticed considerable amount of errors related to ’ for example with “i’m” and “i am”, this should happen? Yes the WER increases but at the end is the same meaning.

At the moment I can’t build for CUDA, hopefully soon I got access to a CUDA device :confused:

(kdavis) #5

Cool! Nice having a second pair of eyes on the WER.

We had a slightly lower value 8.26%, but basically it seems about the same.

As to the problems with apostrophes, yes we’ve noted the same. I’d guess it’s a hard problem to solve as when spoken quickly it can sometimes be unclear if a person said “i am” or “i’m”.

If you have any ideas on how one could solve it, we’re “all ears.”

(Carlos Fonseca) #6

Well no at the moment :confused:

What about this one ”perform’d”? There are a few with 'd

I’m still collecting Spanish from Librivox so, I’m not experienced with the creation of the LM, if I think I got something that can improve the apostrophe issue of course I’ll share.

(kdavis) #7

“perform’d”, interesting. I wasn’t aware that this was a word until just looking it up perform’d. We build our language model from SLR11; I wonder if there is some prevalence of “perform’d” there?

(kdavis) #8

I just grep’ed the SLR11 text and there are 175 lines that contain “perform’d”. So that’s the source of the “perform’d” problem.

(Carlos Fonseca) #9

Here’s a list with the 'd issue

million’d
emerg’d
poison’d
impress’d
pierc’d
remov’d
rebuk’d
steel’d

I better share the result, I’m not native so I may be missing couple more.

I run the WER test again and noticed that few of them also are appearing in the LibriSpeech clean test corpus

(Carlos Fonseca) #10

Here’s the wer result https://pastebin.com/1Wrp3pVH

I don’t know if boy's, infant's and thee's are correct.

#11

When I validate I pay special attention to small things like whether the person said “I’m” or “I am” but I have no idea if other validators do. I think if you’re clicking through quickly you may miss stuff like that. So there may perhaps be incorrectly transcribed clips in the dataset contributing to this.

(kdavis) #12

I haven looked for all the strings you mention. But I’ve found examples of all the ones I’ve searched for in SLR11. For example…

beyond the green within its western close a little vine hung leafy arbor rose where the pale lustre of the moony flood dimm’d the vermillion’d woodbine’s scarlet bud and glancing through the foliage fluttering round in tiny circles gemm’d the freckled ground
…
amidst them next a light of so clear amplitude emerg’d that winter’s month were but a single day were such a crystal in the cancer’s sign
…
a cleric lately poison’d his own mother and being brought before the courts of the church they but degraded him
…

I seems like this is a common construct in older forms of English and SLR11 contains many texts that are in public domain, as they are old enough to pass in to public domain, and thus reflect this old construct.

It seems like we could get a pretty good boost by simply using newer texts in place of SLR11 . However, then we have the legal question of how to obtain modern texts that are still open.

(Carlos Fonseca) #13

If we correct the existing text? I think is not too hard since they are easy to spot. The question is, is there any legal issue editing the existing text?

(kdavis) #14

Editing shouldn’t be problematic

(Carlos Fonseca) #15

Well I said it will be easy to spot, but not easy to correct them hahaha, is worse than I thought.

I can take it, but will need the help of native speakers, for example “worse’n” I changed it to “worse than”.

Changing to “better than” here makes no sense.

A GIRL LIKE THAT OUGHT TO DO SOMETHIN BETTER’N THAN STAY HERE IN SOUTH HARNISS AND KEEP STORE

(Carlos Fonseca) #16

I found Spanish and French sentences, should I remove them or is there any reason to mix languages? Sometimes is mixed, for example “he said hola amigos”.

(kdavis) #17

I have an idea, which I’ve not had time to test yet.

I’ve made a number of language models with different parameters. In particular in some of them I’ve limited the vocabulary to the N most frequent words, with various N’s, of SLR11.

As the various “million’d”, “emerg’d”… are not common they’ll likely not be in the N most frequent words and the language model will not exhibit this strange behavior.

But I need to test this idea with some WER runs.

(Carlos Fonseca) #18

If you share them I can run the WER test :slight_smile:

(Carlos Fonseca) #19

Here’s the librispeech-lm-norm.txt.gz cleaned, I’ve removed a lot of Spanish and French. One more to test :slight_smile:

What about using Google’s BERT to brute force sentences ? Even can try Spacy to generate sentences that makes sense using the existing ones.

(kdavis) #20

Unfortunately it’s 16 TB of language models; sharing is a bit hard. :confused: