Until now I kept myself away from Wiki* and thus Sentence Extractor, but I’m getting out of resources (sigh)… I scanned some random samples in Turkish “Vikipedi” and found many of them are out of topic for Common Voice, has many foreign names, chemical substance names, short entries giving a list (…

Thanks for bringing this up and creating a Discourse post instead of an issue. [image] bozden: Which parts of the rules are most important? I would suggest to have a look at the DE rules file, we spent quite some time improving it recently and I think it’s at a quite good state these days. I…

Thanks again… And sorry for posting in github without seeing a question template. And overall we want < 5% error rate … I find this value too high. For me, even 1% is high, which would introduce 15k bad sentences to the current 50k… Many of our contributors are from higher education and would co…

[image] bozden: I find this value too high. For me, even 1% is high, which would introduce 15k bad sentences to the current 50k… Many of our contributors are from higher education and would complain about it. I think with German we’re below 2% at this point. In the end you can do quite a lot …

Thank you for all these info @mkohler , I’ll give a shot in the following weeks… Actually there are many agglutinative languages which would have the same problem, a single word here becomes a long sentence in English for example - like German’s compound words . That is also a main hindrance in n-gra…

@mkohler , the following issue and related PR gave me an idea on the workflow. The problem they are facing is from a Wikipedia import as far as I can see… [dataset] French, proper nouns. garbage sentence removal opened 02:03PM - 07 Sep 22…

[image] bozden: The problem they are facing is from a Wikipedia import as far as I can see… To be fair, the Wiki extract for French was done a long time ago, before there were further possibilities to create more granular rules. [image] bozden: Wikipedia already gives 3 sentences/article…

I’m aware that the import is from long ago, it has nothing to do with the tool. But it is a good example of how things can go bad. I would still recommend doing the rules/blocklist and see where this brings you. That’s for sure, you need to get a good set to review in any case. But generally I…

[image] mkohler: If it’s only removals, this should be easy to review, and might be something we could do. So better workflow suggestion: This can easily be an extension to SC I think… “Wiki QC” menu item… +Two people say OK, than it is OK… Exports can go automatically into it… (sorry that t…

[image] bozden: But generally I’m really not a fan of that approach because that means that future exports on new articles can’t be done automatically. I think they can go through the same workflow. wiki_delta_yyyymmdd.txt like. But that would not be automatic :wink: [image] bozden: …

Question about CV Sentence Extractor quality and your experience

Common Voice

mkohler (Michael Kohler) August 18, 2022, 8:49pm 7

haha, it’s been years since I’ve last heard that. In all seriousness, of course the block list will filter out valid words, but it will also filter out hard to pronounce foreign words. In the end it’s a tradeoff.

Topic		Replies	Views
Bulk sentences submission from Wikipedia Common Voice sentence-collection	4	685	August 12, 2024
Future of the Sentence Extractor - Your input is required Common Voice sentence-collection	11	1892	May 28, 2021
[Technical feedback needed] Wikipedia extractor script beta Common Voice sentence-collection , feedback	75	9245	July 1, 2020
About the new English Sentences Common Voice feedback , issue	37	3793	May 31, 2019
Extending our sentence collection capabilities Common Voice sentence-collection , announcements	19	3896	September 11, 2019

Question about CV Sentence Extractor quality and your experience

Related topics