Sentence collector copyright issues

Common Voice
#7

Then just remove those marked as invalid by Razmik, he found many mistakes.

Thanks!

#8

Thanks

This is taken care of.

1 Like
#9

The Polish review tab is currently filled with segments from Lord of The Rings, which is very much not public domain. Didn’t even bother slicing it into sentences… Username is narid, source is from the book. (again!).

Sentence collection for Belarusian
#10

Thanks for reporting this. These have been removed.

1 Like
#11

Japanese language collector have the following problems:

Perhaps this is a problem with the corpus.

I went to the source page and checked the "Public Domain version" and it contains the above text. These sources are famous cartoons and games, and they are obviously not in the public domain. The "Public Domain version" file has a [Manga] flag, but some of the sentences are not. Honestly, I can't determine how much of the offending text is in the mix.

Text Corpus Link Collection
#12

@sinumade thanks for reporting. I’m not a lawyer, so I can’t really answer that. @mbranson @jscowcroft any advise here?

#13

ru got some scripts parsed from opensubtitles.org. It’s clearly marked in source, so should be easy to parse.

#14

Thanks for flagging, we’ll take a look on this end and get back to you here. cc @

#15

@sinumade thanks again for flagging this, after review from our Mozilla Legal counterparts it’s been determined that this corpus is not fit for CC0 contribution and all usage should be removed from Common Voice.

@mkohler will work to remove this from the sentence collector, and any sentences that were merged to the primary platform for voice contribution will also be removed from the dataset. cc @phirework who’ll need to take that action.

[ACTION REQUIRED] New Sentence Collector Infrastructure and Improvements
#16

Japanese language collection, again.

Are the reviewers, users Rrock9312 and Rrock2139 the same person?

I checked common-voice/sentence-collector.json. Probably the current Japanese source text isn't all in the public domain. That's a shame.

  • If the current 5,528 sentence is deleted, will the voice no longer be able to be recorded? (i.e., less than 5,000 sentences)
  • If the full text is not available, will it also be removed from the dataset, meaning the dataset cannot be provided? (so, we can't use already recorded voices?)
  • Perhaps the current Sentence Collector has the above source and the text I've added, but are there any other texts? I'd like to check if there is one ...... I don't trust the old collection. (Of course, I want a third party to verify the source I used too.)
  • We may need to verify the source provided by navta. This person participates in other languages as well (at least in the English collection).
  • Seriously, we should ask for volunteers to verify all language sources (I really want the Mozilla staff to verify it, but they'll need time to do so). There are too many resources to lose, including the voice, the dataset itself, etc.
    • Go to the source and check the license. Randomly extract text from the source and search the web for it (to check that the source is not an unauthorized reproduction).
    • However, it's useless unless it's verified by a trusted person. Just pressing the confirm button is not proof.
  • commonvoice.mozilla.org should mention this danger. They are now reading text that is not in the public domain. Users of the dataset are using data that they can't use.

This is insane, and a betrayal to the people.

#17

Ping @mkohler for the mentioned copyright violations
Both Rrock users look suspicious, and there is also some asdf user? Could you check if any of them have any sentences marked invalid? If they have none, I’d say those are most definitely invalid reviews and should be deleted (or japanese submissions are without any mistakes, but I have a hard time believing in that)
If there remain less than 5k sentences, I’m afraid japanese collection will have to stop once again (but either way, 5k is awfully little to expect any great amount of useful collecting to be taking place)
If something has already been recorded, what to do then is a question for more qualified people than me, but it could unfortunately end in lot of bad ways
Second party checking the sources would be great, but now reviews at least show the sentence sources, which is still better than nothing
Mozilla staff can’t viably verify all of the sources for all of the languages, even if only because they don’t speak those. The best you can do is to defer to the local communities

1 Like
#18

@mkohler Hindi dataset, https://tatoeba.org/eng as source, five random sentences with this source were checked, all were incompatible licence (CC-BY 2.0 FR)

#19

Both the Japanese and Hindi issues have been resolved

#20

Sorry, I can’t access to ‘kinto’ url, so can’t provide in requested format.
Ukrainian sentences contain a lot of text from a single book. And they have very bad quality.

Source: cetation of Володимир Білінський
The book is
“Білінський Володимир Броніславович. КРАЇНА МОКСЕЛЬ, або МОСКОВІЯ”
Very “holy-war” and politized one.

Sentence:
“Ростовсько-суздальська земля до приходу Рюриковичів зі своїми ватагами давно була заселена фінськими племенами”
Link

Sentence:
“Початковий київський літопис доволі точно позначає місця проживання цих племен: він знає”
Link

Sentence:
“Здавалося б, за церковними канонами, «царство Боже й царство кесареве мусять залишатися навіки розділеними»”
Link

Sentence:
“Карамзін був палким великоросом на службі в імперії й Государя”
Link

Link to google book (not sure if it’s legal copy)
Google Book

1 Like
Питання щодо якості тексті та авторських прав
#21

And yet another one. It contains a lot of old/rare words. The meaning of some of them I even don’t know.

Book “Меч Арея” by Іван Білик. Link to online copy here

Sentence:
“І не питаєш, чого ради смо прийшли?”
Link

Sentence:
“Пощо тобі князь? — жваво всміхнувся отрок у сорочці.”
Link
Link to source

Sentence:
“Богдан пропустив його слова повз вуха, й усі троє посідали на коней.”
Link

Also, there are a lot of pending sentences from this book. All marked as
“Source: cetation of Іван Білик”. It would be nice to clear them automatically.

These two books kill all pleasure from the project. Unfotunately, 80% of sentences now come from them.
Could you resolve it somehow, please?

#22

@mkohler For the Swedish dataset, there are now a lot of sentences (more than 1000) in the queue with https://tatoeba.org/ given as source and CC-BY 2.0 FR as the stated license.

#23

я б теж підняв питання про прибирання цієї ЄРЕСІ з базового набору

1 Like
#24

Hi I added some sentences from Tatoeba to the Swedish sentences collector but I have since learned that the lichens is incompatible with Mozilla common voice. So they need to be removed.

#25

Both Ukranian and Swedish has been taken care of and the migration should be deployed in a few minutes.

1 Like
#26

Just realized that there are quite a lot of subtitles from the movie The Big Lebowski in the Swedish dataset that likely come from Opensubtitles and has been there at least since August 2019.

Examples:
Jag är The Dude.
Jag kan inte skicka en räkning till kinesen som pissade på mattan.

(Some of them are easily identified by the use of “The Dude” which apart from the copyright issue obviously isn’t ideal for capturing pronunciation in Swedish since most Swedish-speaking people would still use English pronunciation for that. Also there are some offensive sentences in there.)