Feedback about validated sentences

[nukeador (Rubén Martín [❌ taking a break from Mozilla])]

We are coming from here

Export Sentence Collector - 2019-02-07

It’s disappointing that sentences with errors are still making it through despite the review process.

e.g.

The only way too fly is first class
A mimim is a musical term corresponding to a half note
According to the poet, April is the cruellest month
Cathy took two hundred and ten pounds out of her bank account,
Doctor’s write out many prescriptions for patients.
He looked at the handkerchief more, closely.
His central heating had stopped working,
Once the jews harp began emitting its barbaric rhythms, Michael was helpless.
The correct pronunciation is.
The drawer drew a lifelike image of the man.

I suspect that no matter what the process is there is always a % of errors. What we need to ensure is that this % is the lowest possible and enable mechanisms to fix them when encountered.

@dabinat I don’t know how deeply you checked on errors for this export but can you make an estimation on the % of sentences with minor (misspelled) and major (invalid characters) errors?

Imo, this pull requests shows that a couple of things are missing:

• We need a language specific place where we can discuss contribution rules within the community (including a language specific set of rules that can be complemented by the community). This place should ideally be integrated in the sentence-collector tool.
• We need a way to directly correct a wrong sentence in such a way that the original author notices it.
• Since it is too late for the sentences that are already in use, we need an official way to correct the wrong ones.
• We need a whitelist for allowed acronyms. Only for acronyms of which there really is just one correct pronunciation (such as BMW, BKA or DDR but not IT, DECT).

Referring to the German part of this PR, these are the main issues:

• Someone circumvented the prohibitions of acronyms by inserting dashes (“I-T-Sicherheit”, “B-K-A”).
• When I reviewed sentences, I tried to correct all sentences that I considered incorrect. That is, I added the corrected versions to the queue. As a result, some of the sentences in the corpus are present in both their correct and their incorrect form. Examples:
  Der frühe Vogel kann mich mal.
  Der frühe Vogel kann mich Mal.
  Die Tanzlehrerin ist vierfache deutsche und zweifache Europameisterin im Standard- und Latein-Tanz.
  Die Tanzlehrerin ist vierfache deutsche und zweifache Europameisterin im Standart Latein Tanz.
  Greifswald ist eine sogenannte Universitätsstadt.
  Greifswald ist eine so genannte Universitätsstadt.
• There are some sentences with regional slang. E.g.
  Da kiekste wa?
  Shouldn’t an STT system recognize such a sentence as
  Da guckst du, was?
  This is a point which needs discussion at such a language specific place, which I requested above.

@nukeador That list was the entirety of the errors I found.

But the frustrating part is that many of these I saw in my review queue and voted down, yet they still ended up getting approved. I have seen other errors since and voted them down, but are they all going to end up being approved in the next batch?

[nukeador (Rubén Martín [❌ taking a break from Mozilla])]

We need a language specific place where we can discuss contribution rules within the community (including a language specific set of rules that can be complemented by the community). This place should ideally be integrated in the sentence-collector tool.

Feel free to open a topic here and we can figure out how to expose it from the tool.

We need a way to directly correct a wrong sentence in such a way that the original author notices it.

Yes, that’s planned in our roadmap.

Since it is too late for the sentences that are already in use, we need an official way to correct the wrong ones.

There is a cleanup planned for all the sentences in the repo, @gregor will be running it.

We need a whitelist for allowed acronyms. Only for acronyms of which there really is just one correct pronunciation (such as BMW, BKA or DDR but not IT, DECT).

No acronyms are allowed, because we don’t really know how people are going to read them. They can be clear for you but we can’t ensure how other people will read them. I see you opened a topic about this.

But the frustrating part is that many of these I saw in my review queue and voted down, yet they still ended up getting approved. I have seen other errors since and voted them down, but are they all going to end up being approved in the next batch?

Well, if a sentences got a negative vote but then other 2 people voted positive, it will get validated. There is little we can do if people are voting positive here, we can however identify who voted on a specific sentence (checking the sentences json) and try to provide these people with feedback about it.

We can also think how to implement a system that allow us catch and fix this small percentage of wrong sentences that end up validated. Maybe from the voice web site? Other?

[dabinat]

Yes, I think both the recording and validation pages on the CV site should have a way to flag up a sentence. Maybe once it gets x flags it ends up being automatically added to the review queue again?

I also think it would be useful to be able to comment on sentences, ideally in a way that can be seen by everyone, not just the original author. Because sometimes there are edge cases outside of the rules and it would be helpful to engage with others and get their opinions. For instance, I had a sentence today that was something like “One way to annoy Gill is to spell her name Jill.” The G should be pronounced like a J but I don’t know how obvious that is to people who aren’t British. The sentence doesn’t break the rules but it might end up that the majority of readers get it wrong.

[dabinat]

There’s also another factor that the rules don’t cover but is bound to come up sooner or later: what is classed as offensive? What is offensive to some may not be to others, so that’s why discussion between validators is needed IMO.

[Michael_Maggs]

I think you have mis-scored many of those clips. For example:

According to the poet, April is the cruellest month

• TS Eliot being called out for an alleged spelling error. Whatever next?

Cathy took two hundred and ten pounds out of her bank account,

• Common British idiom, and the comma at the end doesn’t matter.

His central heating had stopped working,

• Common British idiom, and the comma at the end doesn’t matter.

The correct pronunciation is.

• Seems a valid clip. Not a full sentence admittedly, but quite readable.

The drawer drew a lifelike image of the man.

• “Drawer: a person or thing that draws” (Collins Dict). Nothing wrong with this at all.

By my count, you’ve mis-validated half the examples you gave. I also find it rather worrying that you give “One way to annoy Gill is to spell her name Jill” as being “an edge case outside the rules.” No it’s not: it a perfectly straightforward English sentence that 100% of British English speakers would read correctly without hesitation. It brings in two very common spelling of the same name, and of course we hope will be read properly, not guessed at by readers who may never have seen one of the spellings. If I see a US word I don’t know how to pronounce, I skip it: isn’t that what the skip button is for? May I suggest you bear in mind when reviewing sentences, too, that not all versions of English use the same orthography and grammar as you do? We are building a corpus of many different varieties of English (of which US and UK are only two examples).

Michael

[Michael_Maggs]

Having said that, I quite agree that clearer validation rules are needed, and ideally some way of discussion and feedback for individual clips. It seems there’s quite a bit of disagreement between reviewers, as is evident by the consistency lowish ‘accuracy’ scores across the validation leaderboard.

[dabinat]

Trailing commas are a mistake. Why accept sloppiness? If we’re going to intentionally allow mistakes through, why bother validating at all?

Also, I think you’ll agree that cutting a sentence short makes it harder to

[dabinat]

Just wanted to add, regarding this:

I also find it rather worrying that you give “ One way to annoy Gill is to spell her name Jill ” as being “an edge case outside the rules.” No it’s not: it a perfectly straightforward English sentence that 100% of British English speakers would read correctly without hesitation. It brings in two very common spelling of the same name, and of course we hope will be read properly, not guessed at by readers who may never have seen one of the spellings.

I have validated almost 70,000 recordings and I can assure that’s not what happens. Case in point: the real sentence in the corpora “The Qt framework is pronounced ‘cute’.” Almost everyone gets this wrong. I get the impression a lot of users don’t read the sentence first before hitting the record button. And I don’t think many users bother to listen to their clips or re-record.

Also, I never said that sentence was bad. I said I wasn’t sure if it was too ambiguous and wanted to check with others.

[Michael_Maggs]

I agree that trailing commas are not helpful, and I’ve suggested on Github that they should be rejected by the upload filter: https://github.com/mozilla/voice-web/issues/1815

[dabinat]

Great, thank you for submitting that. Although Sentence Collector actually has its own separate repo: https://github.com/Common-Voice/sentence-collector/issues

[Michael_Maggs]

Ah, thanks. Didn’t know that. Will repost there tomorrow if need be.

[jf99]

@gregor Feel free to contact me (also on Github) when you start your cleanup. I have identified at least a dozen mistakes in the German sentences added through the collector tool.