# Basque dataset ready

**URL:** <https://discourse.mozilla.org/t/basque-dataset-ready/34355>\
**Category:** Common Voice\
**Tags:** sentence-collection\
**Created:** [December 18, 2018, 7:24am UTC](https://discourse.mozilla.org/t/basque-dataset-ready/34355 "2018-12-18T07:24:49Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![txopi](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/txopi/32/24593_2.png) [@txopi](https://discourse.mozilla.org/u/txopi)\
**Post date:** [December 18, 2018, 7:24am UTC](https://discourse.mozilla.org/t/basque-dataset-ready/34355/1 "2018-12-18T07:24:49Z")

</div>

I know the [sentence collection tool](https://discourse.mozilla.org/t/sentence-collection-tool-development-topic/33390/) is coming, and it will help sentence upload and revision. But during last two months, we have collected more than 6.000 CC0 sentences in Basque language and we already have checked, fixed, cleaned and reviewed them. The dataset is small but contains a diverse grammar and lexicon.  
Here it is a [pull request](https://github.com/mozilla/voice-web/pull/1704) with those first Basque sentences.  
NOTE: the [website is already translated to Basque](https://pontoon.mozilla.org/eu/common-voice/).

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [December 18, 2018, 11:21am UTC](https://discourse.mozilla.org/t/basque-dataset-ready/34355/2 "2018-12-18T11:21:14Z")

</div>

Thanks for this work @txopi

Note that we are currently improving our processes to include sentences in the corpus and we are working on the final guidelines to accept sentences that will be applied to the sentence collection tool.

This means we will probably need to wait a bit to add sentences following this improved processes to make sure they are 100% useful for the speech engine learning, but definitely Basque will have already a lot covered thanks to this work 🙂

Cheers.

---

<div class="post-metadata">

**Author:** ![txopi](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/txopi/32/24593_2.png) [@txopi](https://discourse.mozilla.org/u/txopi)\
**Post date:** [January 19, 2019, 3:58pm UTC](https://discourse.mozilla.org/t/basque-dataset-ready/34355/3 "2019-01-19T15:58:51Z")

</div>

Did you changed your decision? I realised some days ago Basque appeared as Launched language in [voice.mozilla.org](http://voice.mozilla.org)!

Nobody seems to read Slack chat so I decided to ask for [some fixes](https://github.com/mozilla/voice-web/pull/1755) in the sentences (found in part thanks to the Sentence Collector). I also asked to load [Basque accents](https://github.com/mozilla/voice-web/issues/1761).

I think this changes should be done before Basque people starts participating on Common Voice. @nukeador, shouldn’t we make a step forward or backward as soon as possible and resolve Basque languages situation? Please, help!

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [January 21, 2019, 11:01am UTC](https://discourse.mozilla.org/t/basque-dataset-ready/34355/4 "2019-01-21T11:01:16Z")

</div>

> [@txopi](#):
>
> I think this changes should be done before Basque people starts participating on Common Voice. @nukeador, shouldn’t we make a step forward or backward as soon as possible and resolve Basque languages situation? Please, help!

Let me look into this, thanks for flagging.

---

<div class="post-metadata">

**Author:** ![txopi](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/txopi/32/24593_2.png) [@txopi](https://discourse.mozilla.org/u/txopi)\
**Post date:** [January 25, 2019, 9:16am UTC](https://discourse.mozilla.org/t/basque-dataset-ready/34355/5 "2019-01-25T09:16:28Z")

</div>

Did you reach any conclusion?

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [January 25, 2019, 11:47am UTC](https://discourse.mozilla.org/t/basque-dataset-ready/34355/6 "2019-01-25T11:47:54Z")

</div>

Hey @txopi

Did you run the sentences through the sentence collection tool to ensure all of them are valid? We understand the initial work was done before we announced the change in the process to submit and review sentences, so as long as these sentences are all valid we should be fine.

Cheers.

---

<div class="post-metadata">

**Author:** ![txopi](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/txopi/32/24593_2.png) [@txopi](https://discourse.mozilla.org/u/txopi)\
**Post date:** [January 25, 2019, 10:25pm UTC](https://discourse.mozilla.org/t/basque-dataset-ready/34355/7 "2019-01-25T22:25:44Z")

</div>

Yes, I did, but keep in mind that when I uploaded 60 of them [there was a bug](https://github.com/Common-Voice/sentence-collector/issues/51) so I uploaded even some sentences with acronyms. To be absolutely sure all the sentences accomplish the rules, I would need to remove the Basque sentences from the Sentence Collector. That option exists? If so, I don’t know where.

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [January 25, 2019, 10:27pm UTC](https://discourse.mozilla.org/t/basque-dataset-ready/34355/8 "2019-01-25T22:27:57Z")

</div>

> [@txopi](#):
>
> Yes, I did, but keep in mind that when I uploaded 60 of them [there was a bug](https://github.com/Common-Voice/sentence-collector/issues/51) so I uploaded even some sentences with acronyms. To be absolutely sure all the sentences accomplish the rules, I would need to remove the Basque sentences from the Sentence Collector. That option exists? If so, I don’t know where.

Can you coordinate with @mkohler to test all your sentences through the sentence collector? We should make sure the PR doesn’t have these invalid sentences.

/cc @gregor

---

<div class="post-metadata">

**Author:** ![mkohler](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/mkohler/32/21244_2.png) [@mkohler](https://discourse.mozilla.org/u/mkohler)\
**Post date:** [January 29, 2019, 5:44pm UTC](https://discourse.mozilla.org/t/basque-dataset-ready/34355/9 "2019-01-29T17:44:12Z")

</div>

> [@txopi](#):
>
> Yes, I did, but keep in mind that when I uploaded 60 of them [there was a bug](https://github.com/Common-Voice/sentence-collector/issues/51) so I uploaded even some sentences with acronyms. To be absolutely sure all the sentences accomplish the rules, I would need to remove the Basque sentences from the Sentence Collector. That option exists? If so, I don’t know where.

That bug has been fixed in the meantime. And on Monday we have deleted all the sentences that were submitting during the testing period, so none of the sentences previously submitted (before Monday) should be in anymore.

I see there are currently 6212 sentences for Basque. I suppose those are all the new ones that are complying?

---

<div class="post-metadata">

**Author:** ![txopi](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/txopi/32/24593_2.png) [@txopi](https://discourse.mozilla.org/u/txopi)\
**Post date:** [January 30, 2019, 1:24am UTC](https://discourse.mozilla.org/t/basque-dataset-ready/34355/10 "2019-01-30T01:24:46Z")

</div>

As you can see [in other topic](https://discourse.mozilla.org/t/common-voice-sentence-collection-tool-launch/35427/8) of this forum, @gregor is about to accept a PR that fixes this.  
Thank you for your answer @mkohler!

---

<div class="post-metadata">

**Author:** ![txopi](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/txopi/32/24593_2.png) [@txopi](https://discourse.mozilla.org/u/txopi)\
**Post date:** [January 31, 2019, 9:06pm UTC](https://discourse.mozilla.org/t/basque-dataset-ready/34355/11 "2019-01-31T21:06:08Z")

</div>

All is ready for Basque!  
We already started making recordings 😍