# Do the Common Voice datasets contain multiple audio samples for the same text in the same language?

**URL:** <https://discourse.mozilla.org/t/do-the-common-voice-datasets-contain-multiple-audio-samples-for-the-same-text-in-the-same-language/55215>\
**Category:** Common Voice\
**Tags:** dataset\
**Created:** [March 4, 2020, 7:33pm UTC](https://discourse.mozilla.org/t/do-the-common-voice-datasets-contain-multiple-audio-samples-for-the-same-text-in-the-same-language/55215 "2020-03-04T19:33:03Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hyno](https://avatars.discourse-cdn.com/v4/letter/h/e19adc/32.png) [@Hyno](https://discourse.mozilla.org/u/Hyno)\
**Post date:** [March 4, 2020, 7:33pm UTC](https://discourse.mozilla.org/t/do-the-common-voice-datasets-contain-multiple-audio-samples-for-the-same-text-in-the-same-language/55215/1 "2020-03-04T19:33:03Z")

</div>

Do the Common Voice datasets contain multiple audio samples for the same text in the same language?

Would spare me quite some GBs to download. Thanks!

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [March 5, 2020, 12:05pm UTC](https://discourse.mozilla.org/t/do-the-common-voice-datasets-contain-multiple-audio-samples-for-the-same-text-in-the-same-language/55215/2 "2020-03-05T12:05:46Z")

</div>

Yes, there are some languages that have more than one recording per sentence.

---

<div class="post-metadata">

**Author:** ![Hyno](https://avatars.discourse-cdn.com/v4/letter/h/e19adc/32.png) [@Hyno](https://discourse.mozilla.org/u/Hyno)\
**Post date:** [March 5, 2020, 12:11pm UTC](https://discourse.mozilla.org/t/do-the-common-voice-datasets-contain-multiple-audio-samples-for-the-same-text-in-the-same-language/55215/3 "2020-03-05T12:11:33Z")

</div>

Thanks! Would that be the case for the English dataset? And is it the way the recording is designed to have multiple recordings for every sentence or is it just arbitrary?

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [March 5, 2020, 12:16pm UTC](https://discourse.mozilla.org/t/do-the-common-voice-datasets-contain-multiple-audio-samples-for-the-same-text-in-the-same-language/55215/4 "2020-03-05T12:16:05Z")

</div>

English has some repetitions, but we have been working with languages to make sure they have enough sentences so we keep recordings of the same sentences to the minimum, ideally just one recording per sentence.

---

<div class="post-metadata">

**Author:** ![Hyno](https://avatars.discourse-cdn.com/v4/letter/h/e19adc/32.png) [@Hyno](https://discourse.mozilla.org/u/Hyno)\
**Post date:** [March 5, 2020, 12:38pm UTC](https://discourse.mozilla.org/t/do-the-common-voice-datasets-contain-multiple-audio-samples-for-the-same-text-in-the-same-language/55215/5 "2020-03-05T12:38:47Z")

</div>

Thanks again. I’m looking for a dataset that has english speech audio samples of the same sentence spoken by different speakers. Is it possible to obtain these via Common Voice?

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [March 5, 2020, 12:44pm UTC](https://discourse.mozilla.org/t/do-the-common-voice-datasets-contain-multiple-audio-samples-for-the-same-text-in-the-same-language/55215/6 "2020-03-05T12:44:51Z")

</div>

You can download [the English dataset](https://voice.mozilla.org/datasets) and check the index file, order by sentence and pick the ones with more than one recording.

Please, let us know if this is useful, we would like to know how people are using our dataset 🙂

---

<div class="post-metadata">

**Author:** ![txopi](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/txopi/32/24593_2.png) [@txopi](https://discourse.mozilla.org/u/txopi)\
**Post date:** [March 6, 2020, 4:43pm UTC](https://discourse.mozilla.org/t/do-the-common-voice-datasets-contain-multiple-audio-samples-for-the-same-text-in-the-same-language/55215/7 "2020-03-06T16:43:00Z")

</div>

If ideally it should be one recording per sentence and we need 10.000 hours recorded, how many sentences should we prepare in each language? If we suppose each recording lasts 10 seconds, ideally we need 3,6 million sentences on each language?

If such a number isn’t affordable for an small language (for example Basque), how many recordings and sentences could give us an usable speech-to-text system? I don’t ask for an exact number, of course, just a very approximated one, to clarify which are our objectives.

At this moment, there are 6.212 Basque sentences, so if I’m right, ideally we shouldn’t record more than 17 hours for now. If more recordings are done, the 638 speakers could be wasting their time, because that work is not really going to help to create a better speech-to-text system. Is this right?

We already recorded 99 hours, **many** people has recorded the same sentences and the volunteers keep recording and validating the same material time after time. Should we recommend Basque contributors to stop making new recordings until more sentences are available?

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [March 6, 2020, 4:47pm UTC](https://discourse.mozilla.org/t/do-the-common-voice-datasets-contain-multiple-audio-samples-for-the-same-text-in-the-same-language/55215/8 "2020-03-06T16:47:36Z")

</div>

> [@txopi](#):
>
> If such a number isn’t affordable for an small language (for example Basque), how many recordings and sentences could give us an usable speech-to-text system? I don’t ask for an exact number, of course, just a very approximated one, to clarify which are our objectives.

The current tests that we have done suggest that English will need around 2000 validated hours for a basic general model to be trained. Different languages will most likely have different numbers, we will only be able to know as we collect data and train models.

For major languages we have been collecting 1-3M sentences to have buffer to avoid repetitions. On the tests that Deep Speech have been doing for English, French, German and Mandarin, the quality of the model was higher when only one recording was done.

There were some math over this topic:

> [@My language is now collecting voice, what do I need to know?](https://discourse.mozilla.org/t/my-language-is-now-collecting-voice-what-do-i-need-to-know/35582):
>
> open_book [Mozilla Voice Community Playbook](https://common-voice.github.io/community-playbook/): The source of truth for setting up and maintain self-sustainable communities. Hi, I want to open this topic because more and more languages are moving to the voice collection phase (thanks to be able to [get the minimum number of sentences](https://discourse.mozilla.org/t/readme-how-to-see-my-language-on-common-voice/31530)!) Now that your language is available for voice collection you might wonder, now what? Keep in mind that in order to properly train Deep Speech algorithm for a general (near-human) recognition model there are …

---

<div class="post-metadata">

**Author:** ![Hyno](https://avatars.discourse-cdn.com/v4/letter/h/e19adc/32.png) [@Hyno](https://discourse.mozilla.org/u/Hyno)\
**Post date:** [April 14, 2020, 7:39pm UTC](https://discourse.mozilla.org/t/do-the-common-voice-datasets-contain-multiple-audio-samples-for-the-same-text-in-the-same-language/55215/9 "2020-04-14T19:39:25Z")

</div>

Hi!

It turned out that there are many audio samples with the same sentence in the English dataset. I believe there are 16k sentences that have at least 5 or more recordings. However, I have not yet checked if all of them are unique.

---

<div class="post-metadata">

**Author:** ![isomorph70](https://avatars.discourse-cdn.com/v4/letter/i/7ab992/32.png) [@isomorph70](https://discourse.mozilla.org/u/isomorph70)\
**Post date:** [April 20, 2020, 11:45pm UTC](https://discourse.mozilla.org/t/do-the-common-voice-datasets-contain-multiple-audio-samples-for-the-same-text-in-the-same-language/55215/10 "2020-04-20T23:45:12Z")

</div>

Here is an example of a sentence recorded multiple times. [https://ipfs.io/ipfs/QmcA1QMu7UXcBJTAPE1b44fhYv9CWtr75KAhoachq4ajhC/en/D/e/n/ma/rk\_/is\_a\_pioneer\_of\_renewable\_wind\_energy/¤](https://ipfs.io/ipfs/QmcA1QMu7UXcBJTAPE1b44fhYv9CWtr75KAhoachq4ajhC/en/D/e/n/ma/rk_/is_a_pioneer_of_renewable_wind_energy/%C2%A4)