Thanks for reporting this. These have been removed.
1 Like
Japanese language collector have the following problems:
- “username”: “navta”, “source”: “http://www.edrdg.org/wiki/index.php/Tanaka_Corpus”
- “sentence”: “あきらめたら、そこで試合終了ですよ。”
- From SLAM DUNK.
- Ref: あきらめたら、そこで試合終了ですよ。 - Google 検索
- “sentence”: “我が生涯に一片の悔いなし。”
- From 北斗の拳.
- Ref: 我が生涯に一片の悔いなし。 - Google 検索
- “sentence”: “僕は新世界の神となる。”
- From DEATH NOTE.
- Ref: 僕は新世界の神となる。 - Google 検索
- “sentence”: “あんたらの名前なんか興味ないね。どうせこの仕事が終わるとお別れだ。”
- From ファイナルファンタジーVII.
- Ref: あんたたちの名前なんか興味ないね。 - Google 検索
- There are a few changes.
- “sentence”: “あきらめたら、そこで試合終了ですよ。”
Perhaps this is a problem with the corpus.
I went to the source page and checked the "Public Domain version" and it contains the above text. These sources are famous cartoons and games, and they are obviously not in the public domain. The "Public Domain version" file has a [Manga] flag, but some of the sentences are not. Honestly, I can't determine how much of the offending text is in the mix.
@sinumade thanks for reporting. I’m not a lawyer, so I can’t really answer that. @mbranson @jscowcroft any advise here?
ru got some scripts parsed from opensubtitles.org. It’s clearly marked in source, so should be easy to parse.
Thanks for flagging, we’ll take a look on this end and get back to you here. cc @
@sinumade thanks again for flagging this, after review from our Mozilla Legal counterparts it’s been determined that this corpus is not fit for CC0 contribution and all usage should be removed from Common Voice.
@mkohler will work to remove this from the sentence collector, and any sentences that were merged to the primary platform for voice contribution will also be removed from the dataset. cc @phirework who’ll need to take that action.
Japanese language collection, again.
- “username”: “navta”
- “source”: “http://d.hatena.ne.jp/satoru_net/20151030/1446184756”
- This source is an unauthorized reproduction of ATR 503 sentences.
- Corpus Name: ATR 503 sentences (Japanese name: ATR音素バランス503文)
- Original is paid for: ATRデジタル音声データベース｜ATR音声言語データベース｜ATR-Promotions
- “source”: “https://github.com/voice-statistics/voice-statistics.github.com/blob/master/assets/doc/balance_sentences.txt”
- Creator: 日本声優統計学会
- Corpus Name: 音素バランス文
- License: CC BY-SA 4.0; Creator said CC-BY-SA ライセンスで配布しています．
- “source”: “https://github.com/matbahasa/TALPCo/blob/master/data_jpn.txt”
- Current: https://github.com/matbahasa/TALPCo/blob/master/jpn/data_jpn.txt
- Corpus Name: TUFS Asian Language Parallel Corpus
- License: CC BY 4.0; Corpus said TALPCo is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.
- “source”: “http://d.hatena.ne.jp/satoru_net/20151030/1446184756”
Are the reviewers, users Rrock9312 and Rrock2139 the same person?
I checked common-voice/sentence-collector.json. Probably the current Japanese source text isn't all in the public domain. That's a shame.
- If the current 5,528 sentence is deleted, will the voice no longer be able to be recorded? (i.e., less than 5,000 sentences)
- If the full text is not available, will it also be removed from the dataset, meaning the dataset cannot be provided? (so, we can't use already recorded voices?)
- Perhaps the current Sentence Collector has the above source and the text I've added, but are there any other texts? I'd like to check if there is one ...... I don't trust the old collection. (Of course, I want a third party to verify the source I used too.)
- We may need to verify the source provided by navta. This person participates in other languages as well (at least in the English collection).
- Seriously, we should ask for volunteers to verify all language sources (I really want the Mozilla staff to verify it, but they'll need time to do so). There are too many resources to lose, including the voice, the dataset itself, etc.
- Go to the source and check the license. Randomly extract text from the source and search the web for it (to check that the source is not an unauthorized reproduction).
- However, it's useless unless it's verified by a trusted person. Just pressing the confirm button is not proof.
- commonvoice.mozilla.org should mention this danger. They are now reading text that is not in the public domain. Users of the dataset are using data that they can't use.
This is insane, and a betrayal to the people.
Ping @mkohler for the mentioned copyright violations
Both Rrock users look suspicious, and there is also some asdf user? Could you check if any of them have any sentences marked invalid? If they have none, I’d say those are most definitely invalid reviews and should be deleted (or japanese submissions are without any mistakes, but I have a hard time believing in that)
If there remain less than 5k sentences, I’m afraid japanese collection will have to stop once again (but either way, 5k is awfully little to expect any great amount of useful collecting to be taking place)
If something has already been recorded, what to do then is a question for more qualified people than me, but it could unfortunately end in lot of bad ways
Second party checking the sources would be great, but now reviews at least show the sentence sources, which is still better than nothing
Mozilla staff can’t viably verify all of the sources for all of the languages, even if only because they don’t speak those. The best you can do is to defer to the local communities
1 Like
@mkohler Hindi dataset, https://tatoeba.org/eng as source, five random sentences with this source were checked, all were incompatible licence (CC-BY 2.0 FR)
Both the Japanese and Hindi issues have been resolved
Sorry, I can’t access to ‘kinto’ url, so can’t provide in requested format.
Ukrainian sentences contain a lot of text from a single book. And they have very bad quality.
Source: cetation of Володимир Білінський
The book is
“Білінський Володимир Броніславович. КРАЇНА МОКСЕЛЬ, або МОСКОВІЯ”
Very “holy-war” and politized one.
Sentence:
“Ростовсько-суздальська земля до приходу Рюриковичів зі своїми ватагами давно була заселена фінськими племенами”
Link
Sentence:
“Початковий київський літопис доволі точно позначає місця проживання цих племен: він знає”
Link
Sentence:
“Здавалося б, за церковними канонами, «царство Боже й царство кесареве мусять залишатися навіки розділеними»”
Link
Sentence:
“Карамзін був палким великоросом на службі в імперії й Государя”
Link
Link to google book (not sure if it’s legal copy)
Google Book
1 Like
And yet another one. It contains a lot of old/rare words. The meaning of some of them I even don’t know.
Book “Меч Арея” by Іван Білик. Link to online copy here
Sentence:
“І не питаєш, чого ради смо прийшли?”
Link
Sentence:
“Пощо тобі князь? — жваво всміхнувся отрок у сорочці.”
Link
Link to source
Sentence:
“Богдан пропустив його слова повз вуха, й усі троє посідали на коней.”
Link
Also, there are a lot of pending sentences from this book. All marked as
“Source: cetation of Іван Білик”. It would be nice to clear them automatically.
These two books kill all pleasure from the project. Unfotunately, 80% of sentences now come from them.
Could you resolve it somehow, please?
@mkohler For the Swedish dataset, there are now a lot of sentences (more than 1000) in the queue with https://tatoeba.org/ given as source and CC-BY 2.0 FR as the stated license.
я б теж підняв питання про прибирання цієї ЄРЕСІ з базового набору
1 Like
Hi I added some sentences from Tatoeba to the Swedish sentences collector but I have since learned that the lichens is incompatible with Mozilla common voice. So they need to be removed.
Both Ukranian and Swedish has been taken care of and the migration should be deployed in a few minutes.
1 Like
Just realized that there are quite a lot of subtitles from the movie The Big Lebowski in the Swedish dataset that likely come from Opensubtitles and has been there at least since August 2019.
Examples:
Jag är The Dude.
Jag kan inte skicka en räkning till kinesen som pissade på mattan.
(Some of them are easily identified by the use of “The Dude” which apart from the copyright issue obviously isn’t ideal for capturing pronunciation in Swedish since most Swedish-speaking people would still use English pronunciation for that. Also there are some offensive sentences in there.)
The sentences have been removed from the Sentence Collector. Thanks for reporting!
Cantonese (yue):
There are at least 200 sentences from https://hkbus.fandom.com/wiki/九巴286C線, but that site is licensed under CC BY-SA 3.0.
Also the sentences submitted are not Cantonese, but Standard Written Chinese, which is a standardized form of Mandarin.
Thanks for reporting, these sentences have been removed.