I feel that listeners are too lenient with samples

Common Voice
#1

I think I’m being fairly lenient with what I will accept when doing listening tests, but I take a different approach than others I think:
I will only accept a sample if I hear all the words in the displayed text, and if I could recognize all of them on the first listen with my eyes closed.

With this approach, I end up with about an 85% rate of acceptance on the English samples, while the top hundred or so listeners seem to have an acceptance rate in the ballpark of 95%. I’m not sure what level of quality the validation could have if around 95% of these samples are accepted. Many of the samples are inaudible or full of buffer underruns, and many of the speakers clearly do not recognize the words they are attempting to pronounce, or are generally not fluent in any English dialect.

Maybe the listeners should be coached or prompted to have at least some standards for accepting samples.

1 Like
#2

That’s not an acceptance rate. It’s the ratio of how often you agreed with other validators.

One problem with your approach: everyone is biased towards their own accent. So what is unclear to you on the first listen could be perfectly clear to someone else. I validate on whether or not the person said all of the words correctly. With some accents it takes me a couple of listens to determine this.

The whole purpose of Common Voice is to get a wide range of accents, particularly strong accents that are under-served by current speech recognition systems.

2 Likes
#3

But I do agree with you that Mozilla should provide detailed recording and validation guidelines.

1 Like
#4

Maybe it would be more effective to have verifiers transcribe the sentence they hear, instead of just a check?

#5

I’m pretty sure that would lead to many orthographic issues. If practical at all, these transcriptions would need an extra validation pass where they are proof-read by others.

#6

I agree that some validators seem quite lenient. I’m hoping some clearer guidelines might help: Discussion of new guidelines for recording validation

1 Like