EDIT: Since this thread got bigger than expected here is a summary of the project, that I will update from time to time. You can still find the old first post below. This thread is about using a speech corpus from the EU Parliament. The dataset uses Speeches from 1996 - 2011 and is available in 20 …

Great, I won’t start with any of this before next week, so feel free to be quicker :slight_smile:

Given that things seem settled on this, I went ahead and added a subset of the Dutch sentences, following the guidelines used for German, and adding a couple of other restrictions myself. The pull request is there: https://github.com/mozilla/voice-web/pull/2643 Merging this is especially relevant …

Have been playing with the Danish part today. Lets see what I can do tomorrow.

:tada: Congratulations to the Dutch community for getting the EuroParl corpus cleaned up and incorporated (250K sentences) Adding Dutch sentences extracted from EuroParl by FremyCompany · Pull Request #2643 · common-voice/common-voice · GitHub

As I commented on the PR, it would be good if everyone who has been cleaning-up their Europarl corpus work together to publish a guide for other communities to do the same. What are the steps you took to get it done? What tools did you use and how? What other tips and tricks you learned through th…

Updated fund 2 errors in filter.sh script I have extracted a bunch of Danish sentences from the Europarl dataset. cat europarl-v7.da-en.da | source filter.sh >europarl-filter-da.txt shuf europarl-filter-da.txt | ruby uniq_stem.rb >europarl-v7-da.txt where the filter.sh script is grep -v '[^a-zæø…

@stergro Can we update the first message on this topic to note all the information we need if other languages are doing the extraction? Total number of sentences Source of the sentences What is the clean-up applied? (including our general criteria for sentences ) How many sentences were reviewed as…

Hi, There is no Catalan data on Europarl Dataset, but months ago we translated Spanish data to Catalan using Apertium (an OSS machine translation engine). From 2,123,835 Spanish source strings, we get 1,965,734 Catalan translated strings. I wonder if, after filtering them and doing QA, is It possi…

I just updated the first post with a table of all languages that have already imported the corpus and some information about the QA-process.

Using the Europarl Dataset with sentences from speeches from the European Parliament

Common Voice

dabinat March 6, 2020, 6:50pm 43

Planning one more pass on the English wiki text and then I’ll start on Europarl for English.

Topic		Replies	Views
Europarl Datensatz mit hunderttausenden Sätzen aus EU-Debatten Deutsch (de)	8	1016	December 23, 2019
Polish dataset from Europarl - help needed Common Voice	14	1226	July 17, 2021
Mithilfe benötigt für Massenimport aus dem Datensatz des Europäischen Parlamentes Deutsch (de)	7	1182	March 6, 2020
Polish dataset download Common Voice dataset	49	4853	April 13, 2020
Question about CV Sentence Extractor quality and your experience Common Voice	18	1609	August 30, 2023

Using the Europarl Dataset with sentences from speeches from the European Parliament

Related topics