Hello Users in uzbek while reading sentences corpus have noticed many sentences with a lot of mistakes. The source of those texts are Telegram chats. I suspect that the same user has two or more accounts with which he\she approves the sentences automatically. Is there anyway to remove all senten…

Thanks for reporting this. I took a random example from your list, and checked its source. Here’s the full list of sentences that got submitted with the source "Telegram public chats ": https://commonvoice.mozilla.org/sentence-collector/sentences/uz?source=Telegram%20public%20chats . These are 115k…

Hello Michael! I couldn’t open this link, I’m getting 502 Bad Gateway server eror: https://commonvoice.mozilla.org/sentence-collector/sentences/uz?source=Telegram%20public%20chats . I requested edit access for this sheet of words to be reviewed. Can you please approve it?

[image] Abbosjon_Kudratov: I couldn’t open this link, I’m getting 502 Bad Gateway server eror: https://commonvoice.mozilla.org/sentence-collector/sentences/uz?source=Telegram%20public%20chats . Yeah, that query might time out. I shared edit access with you on the doc.

[image] mkohler: avoid Hello Michael! I have finished reviewing this file and we have 87% result of correct sentences. Is this a bit low quality? What shall we do now to remove bad sentences or we remove all texts from this source?

[image] Abbosjon_Kudratov: I have finished reviewing this file and we have 87% result of correct sentences. Is this a bit low quality? What shall we do now to remove bad sentences or we remove all texts from this source? Thanks for doing this! 87% correct sentences is in my opinion way too hi…

Alright. I guess we will need help from our Uzbek community to report occuring cases. I noticed only 2 patterns in all wrong sentences: contains cyrillic(russian) characters (this regex should match Russian letters [а-яА-Я]) starts with lowercase and contains only 1 word Can we find and remove t…

Yes, I think we can do that. Could you give me one valid and one invalid example for each of those so I can better test it?

Examples of right senteces: U borishga biroz tortindi. Hozir partiya tashkilotchimiz Bo‘ron to‘xtagan bo‘lsa edi Examples of wrong senteces: hammasi [only one word, starts with lowercase and has no meaning as a sentence] U borishga biroz tortindi :grin::sob::+1: [contains emojis] U borishga …

Thank you! This has now been taken care of.

One more thing I’ve just stumbled upon - it’s Uzbek sentences that contain one of these characters: ў, ş, қ, ғ [none of these exist in Uzbek official alphabet today. I suppose these appeared as a result of converting old Uzbek cyrillic texts with wrong software] For example, this sentence: Lekin …

Sentence Collector need help to remove

Common Voice

mkohler (Michael Kohler) December 9, 2021, 4:43pm 12

Taken care of. I also added it to the validation so that new sentences in the Sentence Collector can’t be uploaded if they contain those characters.

Topic		Replies	Views
Remove all sentences in sentence collector for Ukrainian Common Voice sentence-collection	19	1204	December 27, 2019
Remove all sentences in sentence collector for Abkhazian Common Voice sentence-collection	6	578	October 18, 2019
Need help with batch deleting (300k+) in sentence collector inappropriate samples Common Voice sentence-collection	3	655	April 11, 2021
Question about CV Sentence Extractor quality and your experience Common Voice	18	1638	August 30, 2023
Sentence collection for Belarusian Common Voice sentence-collection	8	1940	July 7, 2020

Sentence Collector need help to remove

Related topics