Multi-language Dataset Beta Release

Finally.
Thanks for release.

What languages can we expect to be released?
And about those which won’t be - what’s missing for them?

Happy new year!

We are going to release all of the languages that have data in them as of October 2018 which includes 16 languages. You can see the full list of languages here https://voice.mozilla.org/en/languages

Hello Everyone, the new dataset is back up and ready for use!

5 Likes

Hi,
thanks for your hard work on making this release happen!
I imagine you’re getting tons of requests for access to the voice data right now. Is giving access a manual process (i.e., giving access only after review of each form submission by a human)?

Hi there,

If you would like voice access you can fill out the form above and we will be sending a link via email at the beginning of each day. Each of the sentences you hear has been reviewed by 2 humans to ensure its correctness. Does that answer your question?

Thanks, that fully answers my question. I filled out the form above earlier today (shortly after your post), but haven’t receive any email, so I was just wondering if it had to be approved first. Again, thanks for your work on this and I’m looking forward to working with the data!

You should be getting your email shortly!

1 Like

Hi @lsaunders,
I’ve found there is an audio file broken in zh-TW dataset.
53777c75a47473ca6101ac395e74d3a8e9b66f2ad58ce3d7defc1a22761f5f0b7072ddf8d62fd06be02a4843587ea1322c29f90b61edf99cc608981306dc35e4.mp3
This audio is in other.tsv.

Finally,
thanks for release.

1 Like

Thanks for the heads up @areyliu6!
@gweber Do you need any further information about the break? Lets review in the next sprint meeting.

I downloaded the kab dataset but I can’t find the transcript (sentences). I got only the audio files.
I’m going to train the first dataset using deepspeech to show it on an event we are going to organize to show the importance and recruit more recorders from Kabylia.

Thanks again for the release.

The sentences are in the clips.tsv file. If you want to get them split up by language, validity & bucket, you need to run the CorporaCreator on the file.

1 Like

Thanks again for help :heart_eyes::heart_eyes::heart_eyes: wonderfull

But when I downloaded the audio files, there is no file clips.tsv. Is this file downloadable lownly?

Question from some local community member about the data:

Are we delivering the voices which only had been verified for multiple times on the site?
If yes, then what’re the differences of voices listed in valid.tsv and in other .tsv (besides invalid)?

You can find that in the first post in this thread:

The clips.tsv files contains the number of votes each clip got and the audio files are everything we have for this language up to this point (which for that release is 2018-12-19 I think). The valid.tsv only contains clips which have at least 2 up-votes and more up- than down-votes.

1 Like

Thanks @lsaunders gweber

Hi there. I just want to tell thank you for the really great job. We have been waiting for this release so long and finally you did it :grinning:

1 Like

Today we have released a new version of the dataset and keep improving the automation of the process.