Using the Europarl Dataset with sentences from speeches from the European Parliament

Common Voice
#9

No I was talking about the things they say in their speeches. I am not sure how much this really a problem in this dataset, but I know that a few MEPs used words like “scum” for certain groups in their speeches.

Good point, I will do that. I will also filter out sentences with letters that are not part of the German alphabet, this filters out many words that are hard to pronounce.

IMO the sentences are equal in quality to the sentences from wikipedia. But they can’t be used without preprocessing them. For example in the German dataset many sentences start with some letters indicating the original language (like EN: blabla). One should delete things like this first but after that it looks fine.

#10

Right. I know in the past, the consensus was to remove that kind of potentially offending language. Now, we have tooling in place to report, and I think we have also ways for people to express “I don’t want to see offending language”.

Maybe @nukeador can complete my answer?

#11

What’s the estimated percentage of problematic sentences?

Again, if we have a lot of sentences, we can run a QA process on them to understand this percentages, as we did for the wikipedia extraction.

1 Like
#12

I’d be interested to take a look at the English side of things. But there doesn’t appear to be a standalone English version. Is the English content the same in all of the packages, so I could just choose one at random and extract the English translations? Or would I need to extract from all languages?

#13

I am not sure. I compared the english file from the dutch and the german collection and the beginning of these files looks identical, but they don’t have the same size, the dutch one is much bigger. (297 mb vs 307 mb)

Edit: looks like the biggest file is the fr-en collection, but the english file there is just as big as in the en-nl collection.

After searching through the file for some typical topics I think the percentage of problematic sentences is not very high. There are a lot of sentences with strong opinions about all kind of political topics, but almost all of them use a acceptable language. I am for the QA process instead of the sentence collector.

#14

Which might still not be what we might want to show on Common Voice though. Even if it’s acceptable language, the context within a sentence might be heavily opinionated and I personally think Mozilla should refrain from displaying potentially weird political issues. Of course some will be submitted through the Sentence Collector. Do we know of any way to filter out some potentially more far left/right politicians from those datasets? (This is my opinion and I’m totally fine if y’all decide differently)

An example (and that could also be about a far left topic, just what came to mind here):

“All foreigners are …” is bad language, “All foreigners should be deported” is not per se bad language, but still might create a weird dissonance for people on Common Voice. I’m sure some assume that the sentences are vetted “by Mozilla” and therefore would associate Mozilla with these sentences.

Just my 2 cents :slight_smile:

#15

There are sentences live on the site right now along the lines of “He said [controversial statement]” or “He believed [controversial opinion]”.

Are these ok because they are referencing what a person said and not saying it directly as if it was a fact?

#16

It’s a thin line, I fully agree there :slight_smile:

#17

The new swiping-mode of the sentence-collector makes the review process much quicker and it would filter out the worst sentences. I would be willing to review maybe 10 000 sentences in German. (I already reviewed that much for the Esperanto sentence collection) We would need at least another 19 people doing the same to import the complete dataset for one language. Likely more since sentences need more than two votes when people disagree.

That being said I recommend everyone to download the dataset and search for some words, topics and phrases that come to your mind that could be problematic. As far as I can see it there are very few really problematic sentences.

In the Europarl dataset most controversial opinions are part of a longer sentence like: “Mister President I have to say that …” and this puts the opinion in a context that makes it easier to be read by someone who doesn’t like it. But there will be some people who will complain about some sentences since they are all highly political. But I could live with that.

#18

Happy to hear that :slight_smile:

I didn’t review it, so if most of them are in this format or alike, I’m totally fine with a full import and relying on the reporting function.

#19

Are there any notable reactions to controversial sentences that exists in the dataset right now? Did you guys get any angry mails yet?

Most sentences are only recorded by one person, so the impact of a bad sentence is likely not very high. One could also delete some topics with a blacklist as we go, based on the things we find over time.

#20

Here is a sample file with 300 random English sentences from fr-en, the only thing I changed before creating this was deleting sentences longer than 14 words.:

#21

Interesting, will take a look at the Dutch sentences.
Concerning validating, swiping is nice, but you need a touch screen. On a desktop/laptop I’d like to be able to have more sentences like 10 or 20 on 1 page, have a “Check all” option and validate, instead of clicking all one by one which is tedious.

3 Likes
#22

I started a Thread in the german section of this forum about this issue to discuss what the german community wants:

Concerning the sentence collector:

True, I would love to have this too.
You can use the Selenium IDE Browser plugin to automate smaller work steps for now. You can record and play clicks on a website with it. For example when all good sentences are successfully reviewed and only the bad sentences with one downvote are left, then you can use it do automate the second downvote. But be careful with it!

#23

I know, I have used iMacro to automate tasks in Firefox and it works very well. In this case I first need to read (only) 5 sentences before I can hit a macrobutton to validate, if the sentences are correct. I read faster than hitting 5, 10 or 20 buttons, so it would be nice to have more sentences on 1 page and click 1 button to validate.

2 Likes
#24

I would ask to avoid any kind of automation tool for the sentence collector. The whole point of the tool is to enforce human review of each sentence to ensure quality or we will end up with a bad corpus for voice collection, delaying the whole process.

If you have a big public domain corpus (> 500K) coming from a trusted source, please reach out independently to me and we can figure out a different QA process than the sentence collector. But note we currently don’t have the team bandwidth to have a process for smaller corpus that ensure the high quality we are looking for.

Thanks!

#25

Also, as I commented over Slack, we probably want to remove the 60K Dutch sentences from the collector and see if we can follow a QA process for all languages that doesn’t involve individual review from large and trusted sources of text.

#26

I created a pull request for the German corpus:

Anyone who wants to help with the review process is welcome to help :slight_smile:

This corpus has around 379 k sentences after cleanup. Am I too quick here, would you prefere another process?

#27

Yes, let’s have a separate process here, this is too complex to just have it on a PR.

I’ll reach out directly to explore which options we have for this corpus.

1 Like
#28

Sounds good. I personally reviewed a thousand sentences out of the 60K I added for Dutch (so more than 1.5% of then), and didn’t find a single sentence that would be bad to have in the corpus. The two worst sentences I found were a sentence where a space was missing (“overPakistan” instead of “over Pakistan”) and another one that said without context “nuclear plants are bombs waiting to explode” but honestly I don’t think anybody would be truly upset if either sentence ended up in the dataset. So, I validated all the sentences for myself, but was planning on letting a second review happen, but if there’s a different process I’m fine with this as well. I can also provide longer sentences from that dataset, similarly to what has been done for German, if the German sentences are deemed ok (they should all be translations of each other in the end).