# Scraper - Automatic sample sentences extracted in Pull Request

**URL:** <https://discourse.mozilla.org/t/scraper-automatic-sample-sentences-extracted-in-pull-request/55217>\
**Category:** Common Voice\
**Created:** [March 4, 2020, 8:58pm UTC](https://discourse.mozilla.org/t/scraper-automatic-sample-sentences-extracted-in-pull-request/55217 "2020-03-04T20:58:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mkohler](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/mkohler/32/21244_2.png) [@mkohler](https://discourse.mozilla.org/u/mkohler)\
**Post date:** [March 4, 2020, 8:58pm UTC](https://discourse.mozilla.org/t/scraper-automatic-sample-sentences-extracted-in-pull-request/55217/1 "2020-03-04T20:58:30Z")

</div>

Hi everyone

I’ve just merged a change to the infrastructure of the Common Voice Scraper. For every Pull Request there will be an automatic extraction of about 5000 sentences for you to validate your code changes. Additionally some automated tests are run to validate that your changes won’t break anything.

**Edit - 2021-04-24:** Note that [GitHub does not automatically run](https://github.blog/2021-04-22-github-actions-update-helping-maintainers-combat-bad-actors/) the pipeline if you are a first time contributor. If your sample extraction doesn’t get approved within a day, please reach out to us on [Matrix](https://matrix.to/#/#common-voice-sentence-extractor:mozilla.org?web-instance%5Belement.io%5D=chat.mozilla.org).

This will look like this:

 ![Screenshot 2020-03-04 at 20.21.07](https://us1.discourse-cdn.com/flex001/uploads/mozilla/original/3X/2/f/2fabcbcefcdf4c9654087000bd8b108fd47d981a.png)

The extraction takes between 5 and 10 minutes. Once it’s done, GitHub will tell you with a green mark:

 ![Screenshot 2020-03-04 at 20.36.18](https://us1.discourse-cdn.com/flex001/uploads/mozilla/original/3X/9/2/929f90ef7af0c613c39ade62b456b31b7779c1d7.png)

To find the extracted sentences, click on “Show all checks” on the right side and click on the “Detail” link for the “Sample Extraction / extract” row. This will lead you to

 ![Screenshot 2020-03-04 at 20.37.13](https://us1.discourse-cdn.com/flex001/uploads/mozilla/original/3X/5/f/5fab91de94d14e2bceedaed4a17b39542ef5a81c.png)

where you can inspect the logs if wanted, or more importantly, download the extracted sentences through the top right corner “Artifacts” dropdown. It will download a zip file which contains one text file with all the extracted sample sentences. Note that these come from Wikipedia, so please do _not_ upload them to the database, this will be done by the team to extract all sentences we can, and not just the sample. However with this sample you can validate your rules and further adjust them.

For this to work, make sure that

- the rules file name is the ISO code, such as `en.toml` or `hi.toml`
- the blacklist file has the same format, such as `en.txt` or `hi.toml`

Also note that the extraction command locally will need to be changed for the language parameter, passing `en` instead of `english` and so forth:

```auto
cargo run -- extract -l en -d path/to/files

```

If you’d like to contribute another language, please head over to [[Technical feedback needed] Wikipedia extractor script beta](https://discourse.mozilla.org/t/technical-feedback-needed-wikipedia-extractor-script-beta/42983) to learn more. I’m also happy to answer any questions over at [chat.mozilla.org](http://chat.mozilla.org) in the #common-voice-scraper:mozilla.org room.

Also feel free to ask any questions regarding this sample extraction here.

Thanks!  
Michael

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [March 5, 2020, 12:03pm UTC](https://discourse.mozilla.org/t/scraper-automatic-sample-sentences-extracted-in-pull-request/55217/2 "2020-03-05T12:03:53Z")

</div>

Thanks for this @mkohler, this is fundamental step into making content extraction accessible to non-technical contributors!