The multi-language dataset is now available to the Common Voice community as a beta release! This release includes all new, multi-language data that has been collected in 2018.
There are two reasons for choosing a community-focused beta release. First, the data in this release is raw. The Common Voice team will continue improving the way the data is bundled across languages, but we also want to get the dataset in the hands of those who want to start using it immediately.
Second, before a wider release, we need the help and expertise of this community to make the data better for everyone. With your help, we are targeting a full release on the Common Voice site by the end of January.
Mozilla’s DeepSpeech team has created a CorporaCreator repository on GitHub with tools for processing the Common Voice dataset. To help clean the data you can either write or improve a preprocessor for a language (here is the one shared across languages) or you can post a comment about irregularities you may have noticed in the dataset. In particular, we are looking for irregularities like:
- Numbers. There should be no digits in the source text because they can cause problems when read aloud. The way a number is read depends on context and might introduce confusion in the dataset. For example, the number “2409” could be accurately read as both “twenty-four zero nine”; and " two thousand four hundred nine".
- Abbreviations and Acronyms. Abbreviations and acronyms like “USA” or “ICE” should be avoided in the source text because they may be read in a way that does not coincide with their spelling. Additionally, there may be multiple accurate readings for a single abbreviation. For example, the acronym “ICE”; could be pronounced “I-C-E” or as a single word.
- Punctuation. Special symbols and punctuation should only be included when absolutely necessary. For example, an apostrophe is included in English words like “don’t” and “we’re” and should be included in the source text, but it is unlikely you’ll ever need a special symbol like “@” or “#.”
- Foreign letters. Letters must be valid in the language being spoken. For example, “ж” is a letter in the Russian alphabet but is never used in English and so should never appear in any English source text.
To get started, you will need to download the dataset’s clips.tsv file and follow the instructions in the included README. This will only give you access to the text data.
For access to the full dataset, including voice clip audio, you will need to fill out this form.
Reviewing and cleaning the Common Voice data will help everyone who uses it – from academics to small companies and all the makers who need CC0 data – to move forward with a voice-enabled project. The Common Voice team is committed to building a dataset of clean and stable data so we can practice appropriate version control and provide everyone with a way to recreate any testing they need to do in the future.
Thank you for being a part of this project!
