Hey @mkohler , I’m trying cv-sentence-extractor for Turkish. There are some more-or-less technical questions. Q1. As far as I can see, when set, the broken_whitespace rule leaves sentences with sub-strings like " !" filtered out. Shouldn’t it fix them instead? Q2. I could not see any construct (e.…

Thanks for creating this thread. [image] bozden: Q1 . As far as I can see, when set, the broken_whitespace rule leaves sentences with sub-strings like " !" filtered out. Shouldn’t it fix them instead? Then you would have to provide how it needs to be fixed and then there is not much differen…

Thank you for the answers @mkohler … [image] mkohler: Then you would have to provide how it needs to be fixed and then there is not much difference to using replacements which allows for way more flexibility. Yes, it is one possibility. I saw many language rules use them, and I thought it is …

Btw, I handled #1 & #2 with with replacements and with empty broken_whitespace setting. Trying to find a solution for #3 now…

[image] bozden: (Although there is a “clean” function which replaces possible Unicode alternatives, some pass through…) Oh, it was in the wrong order in the code: word_usage.py 9e684bdf9 # Remove any double spaces introduced by last regex …

[image] bozden: Suppose I want to black-list “Bülent”, but there is no option to separate by apostrophes in cv-sentence-connector. So, Bülent’in etc will pass though Ah, yes. That currently does not exist. If this would be implemented, I’d suggest to keep it very generic so it would also work…

I think an optional flag, default off would do it. Similar to cvtools. It would only be temporary for token checking. But the same problem will exist, we need to replace alternative apostrophes temporarily to be able to split. Only those who want this will enable this, as per the readme… Here com…

Hey @mkohler , what is the strategy on parenthesis? They are not wanted by Common Voice, but I see many rule sets which allow them. (I do not.) My question about “How much can we replace?” was related to this. There are many cases where the original/latin name/pronunciation etc is given in parenth…

[image] bozden: Hey @mkohler , what is the strategy on parenthesis? There is nothing special. And as long as there are several different places where validation is done, it will not be possible to do a common approach for this. That being said, if it helps to have such a rule specifically for…

Some questions on cv-sentence-extractor

Common Voice

mkohler (Michael Kohler) June 17, 2023, 11:21am 6

Ah, yes. That currently does not exist. If this would be implemented, I’d suggest to keep it very generic so it would also work for other languages.

Topic		Replies	Views
Bulk sentences submission from Wikipedia Common Voice sentence-collection	4	685	August 12, 2024
Question about CV Sentence Extractor quality and your experience Common Voice	18	1636	August 30, 2023
Cleaning up sentence corpus Common Voice sentence-collection	24	2446	July 17, 2019
Sätze überprüfen - Abkürzungen filtern Deutsch (de)	4	1136	October 20, 2022
[Technical feedback needed] Wikipedia extractor script beta Common Voice sentence-collection , feedback	75	9245	July 1, 2020

Some questions on cv-sentence-extractor

Related topics