Sentence Extractor - Current Status and Workflow Summary

[mkohler (Michael Kohler)]

@Blito

I was looking at Spanish as an example, and I see that there are 10836 sentences here. Are those all the sentences that were added to the database using the sentence-extractor ? Is there a way to see how many sentences are there in the database total (including those that were manually added using the sentence-collector)?

I see the export date for Spanish is 2020-06-11.

The file you’re referencing (sentence-collector.txt - 10836 sentences) is the export from the Sentence Collector. Those are the manually uploaded sentences. That’s also what the date 2020-06-11 refers to.

The Wikipedia extract (done through the Sentence Extractor - see this PR) you can find in the wiki.es.txt file. That contains 1172326 sentences. That import was done on July 19th 2019, so we could only use articles created in roughly the last year if we re-did the export.

@Blito

I’m trying to understand where it fits in the broader project, and whether it’s something worth investing more time in.

I’d say it definitely is, as it allows way more sentences to be extracted, and it also would in theory support other data sources. However not all languages have a lot of articles on Wikipedia, that’s why the Sentence Collector is helpful too. The most important issue for the Sentence Extractor to look into is Improve sentence separation · Issue #11 · common-voice/cv-sentence-extractor · GitHub if you want to have a look and have any feedback for that.