[Legal] [Sentence extraction] Belarusian texts from euroradio.fm

Common Voice
#1

Hello!

I am working on creating a Belarusian text corpus for Common Voice.
We have received permission from Euroradio (https://euroradio.fm) - a large belarusian internet media - to use their texts under CC-0 licence for the Common Voice.

  1. Do we need to put this permission into a formal document for you? If so, how should it look like? Can you provide an example of such document?

  2. Can you please also guide, how the process of uploading their texts into Common Voice should be performed?
    We can put all their texts into single file with 1 sentence per line and upload this file to Sentence Collector.

  3. Can we skip the process of validating sentences? All the texts that are published on Euroradioā€™s web site are checked by professional belarusian linguists - so there should be almost no mistakes and the text quality is good

Thanks!

2 Likes
#2

Hello! Iā€™m EM, the Product Lead for CV. So excited to hear about this! I just needed to check in with legal about the corpus guidelines. I promise to get back to you tomorrow with that :slight_smile: - @heyhillary can help with 2 + 3.

#3

Hello :wave:t6:

Thanks for your questions.

For question 2:
Check out this guide on how to add bulk submissions https://github.com/common-voice/common-voice/blob/main/docs/SENTENCES.md#bulk-submission.

For question 3:
We ask that you validate a sample of the sentences. This post explains how the Europal Dataset with Speeches from European Parliament were validated: Using the Europarl Dataset with sentences from speeches from the European Parliament

If you have any questions, we are happy to help.

#4

Hello! We are excited to get you started with these sentences! We can get you a template to share the details of the euroradio agreement the week of July 6th :slight_smile: our legal expert is going on leave right now. Thank you :slight_smile:

1 Like
Sentence collection for Belarusian ā€“ request for advice
Sentence collection for Belarusian ā€“ request for advice
#5

Hi @Em.Lewis-Jong, any updates on this? If the legal team is available now, could you please share the agreement template with @Aliaksandr? While weā€™re in progress crawling sentences from the Euroradio website, weā€™d like to resolve the legal issues as well.

1 Like
#6

Hey, apologies for the delay. We are finalising the agreement template and should get back to you soon.

#7

Just in case, here is the pull request. Would be great if this bulk submission could be part of the next release, as the Belarusian community is running out of sentences again: the previous 70K dataset has mostly been exhausted by now.

1 Like
#8

Hi, I saw that unfortunately the Euroradio sentences didnā€™t make their way into yesterdayā€™s release. Just wanted to emphasize that it is still very important for us to make this batch of sentences available for recording as early as possible. The Belarusian community is currently one of the most active on Common Voice, we went from zero to 270 recorded hours in under two months, and weā€™ve got strong support from major media in the country. To keep up the momentum, more sentences are needed, as ~155K existing sentences (Wikipedia + fiction) have already been recorded by the contributors. We have an informal agreement with Euroradio; please correct if my understanding is wrong ā€“ itā€™s the lack of template and formal approval that blocks the Euroradio sentences submission from going into production.

@heyhillary @phire ā€“ please advise: is there any chance that the pull request could be merged ahead of the schedule and rolled out to the website asap? For the Belarusian community, it would be a great motivator, otherwise new sentences and new voices will miss the dataset release at the end of July, they will shift to the next release in half a year, and therefore the activity will drop significantly.

3 Likes
#9

Could you share more of your experience? How did your team manage to pull this off?

I used a template in the past that was crafted by our lawyer, here is a post discussing this. Copyright waive form for sentence collection

I attached the template for you.
Of course this still needs the green light from Mozilla @heyhillary @phire ŠžŃ‚ŠŗŠ°Š·-Š¾Ń‚-Š°Š²Ń‚Š¾Ń€ŃŠŗŠ¾Š³Š¾-ŠæрŠ°Š²Š°.zip (9.8 KB)

1 Like
#10

Hey, Iā€™m so sorry for the delay. The document has to go through some through checks before we can share it. Apologies on the impact on the community organsing.

#11

Thank you @heyhillary, and no worries ā€“ we will wait until all the legal nuances are sorted out.

Thank you @daniel.abzakh for sharing the template.

Disclaimer: my only responsibility is preparing the sentences, so Iā€™m not fully aware of all aspects of the process. @Aliaksandr_Sh, who leads the team, may be in a better position to advise on this.

Basically, there exists a non-profit entity say.by whose mission is to support and encourage everyday use of Belarusian, both in personal and professional communication. Within their partnership with a few local companies, say.by offer automated assessment of the employeesā€™ level of proficiency in Belarusian. The test platform includes ASR functionality, which is currently in PoC stage. Common Voice is viewed as a convenient vehicle to collect voice data that would improve this component of the platform. After the Wikipedia export was merged, which unblocked Belarusian in Common Voice, say.by reached out to local media with a press release. Publishing the press release to a total audience of several hundred thousand people provided a strong initial inflow of contributors. A dedicated website was rolled out to provide guidance to new participants and track recording / validation progress for Belarusian. There also exist promotional accounts in Instagram and Telegram, and a contributorsā€™ chat.

One more point: since the 2020 events in Belarus, there has been a surge of participation in various non-governmental initiatives. Common Voice is no exception: while Belarusian language is relatively neglected by the state, supporting oneā€™s native tongue by contributing to Common Voice may be regarded as both a moral high ground and a safe way to express oneā€™s views. This is another factor that drives contribution activity for Belarusian.

3 Likes
#12

@mytmpaccount2015 @Aliaksandr_Sh
I appreciate your response!
I actually enjoyed reading your message, itā€™s very informative.

I shared this with our group responsible for the Abkhazian part of Common Voice.

Could we arrange a meeting between our teams in a Skype/Zoom meeting? We think it would be very beneficial for us to hear more about your experience and how you organize all these efforts?

Sincerely,
Nart.

1 Like
#13

Hey @mytmpaccount2015 and @Aliaksandr

The good news, the template is now ready for sharing. My deepest apologies for the wait, it had to go through an extensive review process.

Is it possible if you could direct message me your email. For me to share your form for Euradio ?

Once you have the from please let us know if Euradio are happy. Please also share with us who and the contact details of the person signing the form from Euradio.

2 Likes
#14

Hi @heyhillary, great news! Just DMā€™ed the email.

#15

Thank you ! just emailed @Aliaksandr with the contract. Thanks for your patience.

1 Like
#16

@heyhillary

I am also waiting for the template, can I have access to it?

#17

Hey Daniel,

Apologies for not initially following up with you.

Could you send me your email via dm please for me to share the template ?

Also, we will be hosting a how-to guide for the template on the refreshed Community Playbook so everyone can access this process.

#18

Hello Hillary,

I just sent you my email via dm.

Thank you,
Daniel.

1 Like
#19

Hey Daniel,

Cool, just emailed you.

Kind regards,

Hillary

1 Like
#20

Hi Daniel,

Apologies for the late reply. You can reach me out on telegram @LordOfTheBoards so we could arrange the call.

Basically everything what @mytmpaccount2015 has mentioned is true. And few other details:

  • the messaging to the audience was just the right one ā€œthe future of Belarusan language is in your voiceā€.
  • Common Voice website is incredibly user friendly (not taking into account few bugs with voice recordings).

And this was a great teamwork by Belarusan community. And for Belarus it has shown a great social effect.

3 Likes