Hi all, One of the things we are realizing when getting sentences from big sources (like wikipedia) is that, in general, they contain complex words (or foreign ones) that are not optimal for a user experience when reading on the app. To tackle this issue we have been evaluating filtering them from…

One simple idea would be to take a corpus of “everyday” language (or whichever is your target domain) and make a frequency list and basically exclude anything in Wikipedia which includes words beneath a certain rank. I’m not sure how well it would work in practice, but it would certainly be a reason…

Foreign doesn’t always mean unpronounceable and some non-English terms are in common use among English speakers (e.g. a la carte, coup d’etat), so removing all of these may not be the best approach. Plus I think it’s good to have coverage of major country / city names. A simple way of doing this wo…

Another recommendation coming from Deep Speech team: Take Wikipedia text in language X Find all the unique words Sort them from the most frequently used to least frequently used. Take the top N words. Create a corpus from Wikipedia rejecting sentences that have words not in the N most frequent wor…

Derivation and inflection for agglutinative languages can produce unique words within the same language and it’s the case for afroasiatic (hamito-semitic) languages: kabyle and other berber languages, hebrew, arabic, amharic… In such a situation, we need more processings such as lemmatization/racin…

One of the things that I was observing for Spanish is that complex or too technical words usually appear in just one sentence. I suspect it’s safe to remove just sentences with unique words as an initial step in cleaning them up.

I’ll try with the kab corpora on Wikipedia and analyse the result. Kab wiki is still poor.

[image] nukeador: Another recommendation coming from Deep Speech team: Take Wikipedia text in language X Find all the unique words Sort them from the most frequently used to least frequently used. Take the top N words. Create a corpus from Wikipedia rejecting sentences that have words not in t…

[image] Michael_Maggs: Are the Wiki extractions done separately from the validity-checking? Maybe part is done by the Deep Speech team and part by the Common Voice team? No, the process is currently just run by the Common Voice team, and we can include new rules in the extraction, it’s only t…

Balancing most common words vs sentences number

Common Voice

nukeador (Rubén Martín [❌ taking a break from Mozilla]) July 3, 2019, 10:02am 4

Is there a methodology to identify this per language or this is something we’ll have to get from an external source?

Cleaning up sentence corpus

Topic		Replies	Views
Cleaning up sentence corpus Common Voice sentence-collection	24	2154	July 17, 2019
Czech Wikipedia extraction concerns Common Voice sentence-collection	9	856	February 1, 2020
Bulk sentences submission from Wikipedia Common Voice sentence-collection	4	623	August 12, 2024
Question about CV Sentence Extractor quality and your experience Common Voice	18	1592	August 30, 2023
How unique should a sentence be? Common Voice sentence-collection	7	1096	May 15, 2019

Balancing most common words vs sentences number

Related topics