Hello Users in uzbek while reading sentences corpus have noticed many sentences with a lot of mistakes. The source of those texts are Telegram chats. I suspect that the same user has two or more accounts with which he\she approves the sentences automatically. Is there anyway to remove all senten…

Thanks for reporting this. I took a random example from your list, and checked its source. Here’s the full list of sentences that got submitted with the source "Telegram public chats ": https://commonvoice.mozilla.org/sentence-collector/sentences/uz?source=Telegram%20public%20chats . These are 115k…

Hello Michael! I couldn’t open this link, I’m getting 502 Bad Gateway server eror: https://commonvoice.mozilla.org/sentence-collector/sentences/uz?source=Telegram%20public%20chats . I requested edit access for this sheet of words to be reviewed. Can you please approve it?

[image] Abbosjon_Kudratov: I couldn’t open this link, I’m getting 502 Bad Gateway server eror: https://commonvoice.mozilla.org/sentence-collector/sentences/uz?source=Telegram%20public%20chats . Yeah, that query might time out. I shared edit access with you on the doc.

[image] mkohler: avoid Hello Michael! I have finished reviewing this file and we have 87% result of correct sentences. Is this a bit low quality? What shall we do now to remove bad sentences or we remove all texts from this source?

[image] Abbosjon_Kudratov: I have finished reviewing this file and we have 87% result of correct sentences. Is this a bit low quality? What shall we do now to remove bad sentences or we remove all texts from this source? Thanks for doing this! 87% correct sentences is in my opinion way too hi…

Alright. I guess we will need help from our Uzbek community to report occuring cases. I noticed only 2 patterns in all wrong sentences: contains cyrillic(russian) characters (this regex should match Russian letters [а-яА-Я]) starts with lowercase and contains only 1 word Can we find and remove t…

Yes, I think we can do that. Could you give me one valid and one invalid example for each of those so I can better test it?

Examples of right senteces: U borishga biroz tortindi. Hozir partiya tashkilotchimiz Bo‘ron to‘xtagan bo‘lsa edi Examples of wrong senteces: hammasi [only one word, starts with lowercase and has no meaning as a sentence] U borishga biroz tortindi :grin::sob::+1: [contains emojis] U borishga …

Thank you! This has now been taken care of.

One more thing I’ve just stumbled upon - it’s Uzbek sentences that contain one of these characters: ў, ş, қ, ғ [none of these exist in Uzbek official alphabet today. I suppose these appeared as a result of converting old Uzbek cyrillic texts with wrong software] For example, this sentence: Lekin …

Sentence Collector need help to remove

Common Voice

mkohler (Michael Kohler) November 24, 2021, 10:08am 4

Yeah, that query might time out. I shared edit access with you on the doc.

Topic		Replies	Views
Remove all sentences in sentence collector for Ukrainian Common Voice sentence-collection	19	1155	December 27, 2019
Remove all sentences in sentence collector for Abkhazian Common Voice sentence-collection	6	563	October 18, 2019
Need help with batch deleting (300k+) in sentence collector inappropriate samples Common Voice sentence-collection	3	641	April 11, 2021
Question about CV Sentence Extractor quality and your experience Common Voice	18	1612	August 30, 2023
Sentence collection for Belarusian Common Voice sentence-collection	8	1905	July 7, 2020

Sentence Collector need help to remove

Related topics