# Using OSCAR corpus sentences

**URL:** <https://discourse.mozilla.org/t/using-oscar-corpus-sentences/60766>\
**Category:** Common Voice\
**Created:** [May 26, 2020, 8:40am UTC](https://discourse.mozilla.org/t/using-oscar-corpus-sentences/60766 "2020-05-26T08:40:04Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![jmontane](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/jmontane/32/23655_2.png) [@jmontane](https://discourse.mozilla.org/u/jmontane)\
**Post date:** [May 26, 2020, 8:40am UTC](https://discourse.mozilla.org/t/using-oscar-corpus-sentences/60766/1 "2020-05-26T08:40:04Z")

</div>

I wonder if we can import sentences from [OSCAR project](https://oscar-corpus.com/) to Common Voice. Content is crawled from Internet, but it’s packaged and released under CC0.

---

<div class="post-metadata">

**Author:** ![stergro](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/stergro/32/40229_2.png) [@stergro](https://discourse.mozilla.org/u/stergro)\
**Post date:** [May 26, 2020, 8:51am UTC](https://discourse.mozilla.org/t/using-oscar-corpus-sentences/60766/3 "2020-05-26T08:51:56Z")

</div>

I experimented with this corpus a while ago. It is hard to filter sentences or add line breaks after full stops because the files are all huge. But it is possible to use it and you can get many good sentences out of it.

I am not so sure if using this could lead to copyright problems. One thought I had, was just using sentences shorter than four words or so. It is very hard to claim copyright at such short sentences and the dataset often lacks short and easy sentences right now.

---

<div class="post-metadata">

**Author:** ![repodiac](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/repodiac/32/37585_2.png) [@repodiac](https://discourse.mozilla.org/u/repodiac)\
**Post date:** [May 26, 2020, 9:03am UTC](https://discourse.mozilla.org/t/using-oscar-corpus-sentences/60766/4 "2020-05-26T09:03:23Z")

</div>

Thanks for the pointer - didn’t know of OSCAR until now. Haven’t checked out the HUMONGOUS (!) files neither… yet.

But regarding the CC0 related initial question: I would be rather careful! The dataset itself might be public domain, but it’s (single) pieces of content might not! If there is not a clear and transparent documentation on the crawling process, I would rather assume that it is illegal in most of the jurisdictions or countries (at least in Europe).

Crawling a website without permission is not allowed, afaik, if you use the contents for your own works and not for display/browsing only. But I’m not a lawyer or legal expert and copyright and IP law is one big minefield - in most countries…

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [May 26, 2020, 12:14pm UTC](https://discourse.mozilla.org/t/using-oscar-corpus-sentences/60766/5 "2020-05-26T12:14:46Z")

</div>

Unfortunately we can’t use OSCAR, as you pointed out:

> [@repodiac](#):
>
> The dataset itself might be public domain, but it’s (single) pieces of content might not!

When we reviewed it with our legal team we were advised to not to use it because of this legal risk.

---

<div class="post-metadata">

**Author:** ![jmontane](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/jmontane/32/23655_2.png) [@jmontane](https://discourse.mozilla.org/u/jmontane)\
**Post date:** [May 28, 2020, 7:36am UTC](https://discourse.mozilla.org/t/using-oscar-corpus-sentences/60766/6 "2020-05-28T07:36:43Z")

</div>

Ok, I understand it. What a pitty ☹

---

<div class="post-metadata">

**Author:** ![stergro](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/stergro/32/40229_2.png) [@stergro](https://discourse.mozilla.org/u/stergro)\
**Post date:** [May 28, 2020, 7:48am UTC](https://discourse.mozilla.org/t/using-oscar-corpus-sentences/60766/7 "2020-05-28T07:48:52Z")

</div>

Could you ask the legal if short sentences, like four words or fewer could be a possibility? There are exceptions for small passages in almost every copyright of the world.

Right now we mainly have long and complicated sentences, so mixing short sentences could benefit the dataset. Short sentences can be reviewed quickly in the sentence collector if you don’t want to risk automated imports.

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [May 28, 2020, 11:47am UTC](https://discourse.mozilla.org/t/using-oscar-corpus-sentences/60766/8 "2020-05-28T11:47:07Z")

</div>

Let me check that again with our legal team, thanks!

---

<div class="post-metadata">

**Author:** ![nukeador](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/nukeador/32/20533_2.png) [@nukeador](https://discourse.mozilla.org/u/nukeador)\
**Post date:** [June 4, 2020, 4:32pm UTC](https://discourse.mozilla.org/t/using-oscar-corpus-sentences/60766/9 "2020-06-04T16:32:20Z")

</div>

OK, so I got back from our legal team and they advise not to do it even for short sentences, it still introduces some risks we don’t feel comfortable with and it’s better to avoid.

Cheers.