# About the new English Sentences

**URL:** <https://discourse.mozilla.org/t/about-the-new-english-sentences/38591>\
**Category:** Common Voice\
**Tags:** feedback, issue\
**Created:** [April 24, 2019, 12:16pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591 "2019-04-24T12:16:19Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://avatars.discourse-cdn.com/v4/letter/c/dfb087/32.png) [@Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://discourse.mozilla.org/u/Codigo_Logo_Programacao_e_Inteligencia_Artificial)\
**Post date:** [April 24, 2019, 12:16pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/1 "2019-04-24T12:16:19Z")

</div>

I really like that we have new sentences now, I’m seing a lot of new proper names, which for me as a non-native struggles a lit bit to pronounce, but it’s ok because it’ll improve my English, I have doubts about abbreviations such as “Inc” should I say just Inc or ‘in case’, I said it ‘inc’ just in case, pun intended.

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [April 24, 2019, 12:36pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/2 "2019-04-24T12:36:36Z")

</div>

Do you have an example where “Inc” is included?

---

<div class="post-metadata">

**Author:** ![Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://avatars.discourse-cdn.com/v4/letter/c/dfb087/32.png) [@Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://discourse.mozilla.org/u/Codigo_Logo_Programacao_e_Inteligencia_Artificial)\
**Post date:** [April 24, 2019, 12:44pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/3 "2019-04-24T12:44:44Z")

</div>

It was about the name of a company I don’t recall much of it now.

---

<div class="post-metadata">

**Author:** ![dabinat](https://avatars.discourse-cdn.com/v4/letter/d/8dc957/32.png) [@dabinat](https://discourse.mozilla.org/u/dabinat)\
**Post date:** [April 24, 2019, 3:12pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/4 "2019-04-24T15:12:12Z")

</div>

I’ve seen it a couple of times. It’s from the wiki sentences.

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [April 24, 2019, 3:19pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/5 "2019-04-24T15:19:33Z")

</div>

OK, I understand that the code didn’t recognized that as an acronym (no upper cases) or an abbreviation (missing dot)

---

<div class="post-metadata">

**Author:** ![joshua.landau.ws](https://avatars.discourse-cdn.com/v4/letter/j/c37758/32.png) [@joshua.landau.ws](https://discourse.mozilla.org/u/joshua.landau.ws)\
**Post date:** [April 24, 2019, 1:42am UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/6 "2019-04-24T01:42:19Z")

</div>

Some sentences are ridiculous. Here are some.

> The characters for Kyoto are 京都 and Osaka’s are 大阪.

> The township is in Schuylkill Valley School District.

> Abolhasan Saba, Esmaeil Ghahremani and Ali-Naqi Vaziri were among his students.

> Bundesliga club Kaiserslautern on a one-year contract.

> It is also close to the Naskapi reserved land of Kawawachikamach.

Niche words are important, but “_niche_” is a niche word, not “_Kawawachikamach_”. Surely there’s a better way to get dictionary coverage that doesn’t involve foreign place names and twenty-letter scientific Latin terms.

---

<div class="post-metadata">

**Author:** ![dabinat](https://avatars.discourse-cdn.com/v4/letter/d/8dc957/32.png) [@dabinat](https://discourse.mozilla.org/u/dabinat)\
**Post date:** [April 24, 2019, 2:50am UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/7 "2019-04-24T02:50:20Z")

</div>

I posted an issue about this here so feel free to chime in with any comments: [https://github.com/mozilla/voice-web/issues/1958](https://github.com/mozilla/voice-web/issues/1958)

1.5 million sentences were imported so even if only 3% are bad, that’s quite a lot. I’m working on a script to filter out most of them but some manual validation will still be needed.

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [April 24, 2019, 1:06pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/8 "2019-04-24T13:06:16Z")

</div>

We agree, thanks for flagging, we are looking into improving and fixing this.

---

<div class="post-metadata">

**Author:** ![Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://avatars.discourse-cdn.com/v4/letter/c/dfb087/32.png) [@Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://discourse.mozilla.org/u/Codigo_Logo_Programacao_e_Inteligencia_Artificial)\
**Post date:** [April 24, 2019, 3:47pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/9 "2019-04-24T15:47:31Z")

</div>

Hi it’s me again, well I’ve come across some Japanese words in this sentence “The characters for Kyoto are 京都 and Osaka’s are 大阪.” When I don’t know how to pronounce a word I look it up with Google Translate, but this could lead to some mismatch I think. Was the inclusion of non-english words planned, is it a plan to collect voice data to other languages?

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [April 24, 2019, 3:49pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/10 "2019-04-24T15:49:38Z")

</div>

Merging all messages about this in this topic.

Yes, @gregor is looking into it, for now please skip these sentences.

Thanks!

---

<div class="post-metadata">

**Author:** ![Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://avatars.discourse-cdn.com/v4/letter/c/dfb087/32.png) [@Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://discourse.mozilla.org/u/Codigo_Logo_Programacao_e_Inteligencia_Artificial)\
**Post date:** [April 24, 2019, 3:51pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/11 "2019-04-24T15:51:16Z")

</div>

I’ll do that, Roger that!

---

<div class="post-metadata">

**Author:** ![Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://avatars.discourse-cdn.com/v4/letter/c/dfb087/32.png) [@Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://discourse.mozilla.org/u/Codigo_Logo_Programacao_e_Inteligencia_Artificial)\
**Post date:** [April 24, 2019, 4:09pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/12 "2019-04-24T16:09:09Z")

</div>

I’m recording some clips and sometimes there are sentences which are confusing, I think an option to disable these sentences for recording would be welcome,  
example: [‘The original center ran perpendicular to W. Club Blvd.’]  
I can’t record news clips without having to record this sentence, that’s why I think this option would be needed, I just recorded a silent clip.

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [April 26, 2019, 1:30pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/13 "2019-04-26T13:30:59Z")

</div>

I want to let you know that thanks to @dabinat great work we have filter-out a lot of sentences with issues (at the end there were just around 8%).

This changes should be reflected in the next deployment.

Thanks everyone for your valuable feedback to improve the project 🙂

Cheers.

---

<div class="post-metadata">

**Author:** ![Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://avatars.discourse-cdn.com/v4/letter/c/dfb087/32.png) [@Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://discourse.mozilla.org/u/Codigo_Logo_Programacao_e_Inteligencia_Artificial)\
**Post date:** [April 26, 2019, 7:03pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/14 "2019-04-26T19:03:57Z")

</div>

We also thank the Common Voice team for this awesome project!

---

<div class="post-metadata">

**Author:** ![JAGulin](https://avatars.discourse-cdn.com/v4/letter/j/a88e4f/32.png) [@JAGulin](https://discourse.mozilla.org/u/JAGulin)\
**Post date:** [May 10, 2019, 7:26am UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/15 "2019-05-10T07:26:33Z")

</div>

> [@Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](#):
>
> I can’t record news clips without having to record this sentence, that’s why I think this option would be needed, I just recorded a silent clip.

@Codigo_Logo_Programacao_e_Inteligencia_Artificial  
Just in-case it wasn’t mentioned before, there is a “skip” button which will give you a new phrase to record (so you still have 5). Isn’t it available to you?

> <https://github.com/common-voice/common-voice/issues/1727>
>
> There are lots of clips that are just background noise where the user clicked re…cord then stop very quickly. My theory for why these clips occur is that users want to skip the sentence but don’t know about the Skip button. I have even occasionally heard users say “I don’t want to say that” instead of the recording.
> 
> So I think the Skip button should be made more prominent. I think it should be turned into an actual button with a border and maybe a colored background and put in the top right corner of the sentence box so it’s easier to notice when reading the sentence.

As for “Inc” it would mean “incorporated” when about a company.  
[Inc. - Wikipedia](https://en.wikipedia.org/wiki/Inc).  
The intention may have been to remove these strings, but when it’s a standard part of a company name like “Acme Inc” I’d say it could be read as “ink”. The safe bet is to skip, though.

---

<div class="post-metadata">

**Author:** ![Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://avatars.discourse-cdn.com/v4/letter/c/dfb087/32.png) [@Codigo\_Logo\_Programacao\_e\_Inteligencia\_Artificial](https://discourse.mozilla.org/u/Codigo_Logo_Programacao_e_Inteligencia_Artificial)\
**Post date:** [May 10, 2019, 7:42am UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/16 "2019-05-10T07:42:09Z")

</div>

Yeah I’m seeing this button now. Ok got it.

---

<div class="post-metadata">

**Author:** ![Michael\_Maggs](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/michael_maggs/32/26145_2.png) [@Michael\_Maggs](https://discourse.mozilla.org/u/Michael_Maggs)\
**Post date:** [May 22, 2019, 12:50pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/17 "2019-05-22T12:50:24Z")

</div>

While it’s fantastic to have so many English sentences from Wikipedia, we shouldn’t assume that _everything_ should come from there. WP sentences are typically straight facts which are often boring to record and review. They frequently include really obscure non-English proper names (such as villages in Russia) that aren’t at all useful for the dataset and are exceptionally hard for volunteers to read. And they mostly lack the proper names that we _do_ need such as common English language given names.

If nobody objects I’ll re-start uploading sentences from interesting public domain books, with personal names replaced by script to increase name diversity.

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [May 22, 2019, 12:52pm UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/18 "2019-05-22T12:52:42Z")

</div>

> [@Michael\_Maggs](#):
>
> They frequently include really obscure non-English proper names (such as villages in Russia) that aren’t at all useful for the dataset and are exceptionally hard for volunteers to read. And they mostly lack the proper names that we _do_ need such as common English language given names.

Do you have ideas on how we can optimize the wikipedia extraction to avoid this issue?

---

<div class="post-metadata">

**Author:** ![dabinat](https://avatars.discourse-cdn.com/v4/letter/d/8dc957/32.png) [@dabinat](https://discourse.mozilla.org/u/dabinat)\
**Post date:** [May 23, 2019, 4:00am UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/19 "2019-05-23T04:00:43Z")

</div>

I have been trying to filter out letter sequences that don’t tend to occur often in English. For example, there are no English words (that I know of) that contain the letter sequences “uuk” or “ijp”, so it filters sentences with words containing these letter sequences.

You can find my script here: [https://github.com/dabinat/cvtools/blob/master/sentence\_validator.py](https://github.com/dabinat/cvtools/blob/master/sentence_validator.py)

(I also have a PR awaiting approval with these changes: [https://github.com/mozilla/voice-web/pull/2040](https://github.com/mozilla/voice-web/pull/2040) )

---

<div class="post-metadata">

**Author:** ![dabinat](https://avatars.discourse-cdn.com/v4/letter/d/8dc957/32.png) [@dabinat](https://discourse.mozilla.org/u/dabinat)\
**Post date:** [May 23, 2019, 4:03am UTC](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591/20 "2019-05-23T04:03:51Z")

</div>

> [@Michael\_Maggs](#):
>
> WP sentences are typically straight facts which are often boring to record and review.

Yes, the Wikipedia stuff has improved word coverage a lot but it’s all past-tense, third-person fact description. We still need other sources for diversity.

I’m happy to review any sentences you upload.

[Next page](https://discourse.mozilla.org/t/about-the-new-english-sentences/38591.md?page=2)