I’ve been running some math with the 15 locales who have more validated hours today, I’ve calculated how many sentences they have and how many validated hours.
Based on this we can estimate how many sentences we need to cover the current hours without making any repetition (one recording per sentence).
Sentences difference: Numbers in negative indicate that we will need at least that number of sentences more to cover the current hours (so potentially we would need more to get more clips without repetitions)
Additional hrs we could accommodate: Numbers in negative indicate we already have that numbers of hours of repeated sentences, that won’t be used by deep speech training.
|Locale||Current hours||Current sentences||Sentences to cover current hrs||Sentences difference||Additional hrs we could accommodate|
If you are in a locale with a negative difference, I would encourage to prioritize mobilizing your communities and networks on getting more sentences, through the sentence collector or getting some technical people to review our recently published script to mass extract sentences from wikipedia.
We need to avoid people keep recording the same sentences over and over again.
Thanks everyone for your support!
Update: Edited to prevent any notion that duplicate recordings are “not useful”, that’s not at all the case.
Deep Speech is in large parts still a research project. The team is in the constant process of learning and optimising how the “golden standard” training dataset looks like, for our own engine, but also to cater for the needs of the broader research community.
The more clarity we gain, the better we can design and further develop data collection via Common Voice.
To be very clear: All recorded and validated hours are valuable and will be included in the Common Voice dataset. We just want to incorporate feedback we’ve gotten and thus have been putting even more emphasis on sentence diversity and volume with tool adjustments, new approaches and CTAs like the above.
We keep learning