Important: Please go to this topic for the most updated information about the sentence extractor and how to use it . Hello everyone, As we announced a few months ago, as an effort to get sentences from large sources of data, we found a legal way to extract some wikipedia sentences for our corpus …

Michael has just created a chat room over Matrix just for this topic. Feel free to join so we can discuss options more sync and see how we can bring Basque extraction as soon as possible. Thanks!

I just ran a new scraping run where I added numbers under the replacements like: replacements = [ [" 1 ", " ien “], [” 2 ", " twa "], … ] To be sure it doesn’t replace all numbers, only the loose ones, I added a space before and after the numbers. This resulted in a 1.000 sentences more on a t…

[image] Fjoerfoks: I just ran a new scraping run where I added numbers under the replacements like: replacements = [ [" 1 ", " ien “], [” 2 ", " twa "], … ] Out of curiosity, how far did you go? And how much does that incrase the time to run the script?

I went from 1-31, 40, 50-90, 100, 200-900 and 1000. I didn’t time it, but I can’t say it took way more time. My idea now is to create some sort of whitelist with most used numbers, the opposite of the word blacklist, and put like the top 100 in the list of replacements. If someone knows the correct…

I’m not sure if that’s a good idea. If there is a sentence like “In year 2001…” it will become “In year two zero zero one…”?

That’s why I added the spaces before and after the number to separate them from larger numbers like years. It works fine that way.

Ah, OK, I got it! Thank you for your quick answer :slight_smile:

I applied the extractor to Chinese wiki pages and yielded very few sentences. Since Chinese characters and punctuation are quite different from European ones, I would like to know more about the extractor before doing more experiments: Does the extractor respect sentence separators in Chinese, e.…

[image] hyxibg5lez: I applied the extractor to Chinese wiki pages and yielded very few sentences. May I ask which Chinese wiki pages? For zh-CN a wiki extraction has already been done, so we can’t redo that: common-voice/server/data/zh-CN/wiki.zh-cn.txt at master · common-voice/common-voice ·…

[image] mkohler: May I ask which Chinese wiki pages? I chose a dialect of Chinese, zh-yue, which has far less wiki pages for experiment. It corresponding to zh-HK in the CV project. I got zh_yuewiki-20200520-pages-articles-multistream.xml, a 266M file, from Wikipedia. The rules file was basic…

[Technical feedback needed] Wikipedia extractor script beta

Common Voice

nukeador (Rubén Martín [❌ taking a break from Mozilla]) July 1, 2020, 2:07pm 81

We are moving to this topic to list all the things about the sentence extractor

Topic		Replies	Views
Future of the Sentence Extractor - Your input is required Common Voice sentence-collection	11	1838	May 28, 2021
Bulk sentences submission from Wikipedia Common Voice sentence-collection	4	617	August 12, 2024
Question about CV Sentence Extractor quality and your experience Common Voice	18	1573	August 30, 2023
About the new English Sentences Common Voice feedback , issue	37	3344	May 31, 2019
Scraper - Automatic sample sentences extracted in Pull Request Common Voice	1	1571	March 5, 2020

[Technical feedback needed] Wikipedia extractor script beta

Related topics