# Where is the documentation regarding Mozilla Common Voice database?

**URL:** <https://discourse.mozilla.org/t/where-is-the-documentation-regarding-mozilla-common-voice-database/81476>\
**Category:** Common Voice\
**Tags:** issue\
**Created:** [June 10, 2021, 7:21am UTC](https://discourse.mozilla.org/t/where-is-the-documentation-regarding-mozilla-common-voice-database/81476 "2021-06-10T07:21:08Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![makoto\_wada\_jp](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@makoto\_wada\_jp](https://discourse.mozilla.org/u/makoto_wada_jp)\
**Post date:** [June 10, 2021, 7:21am UTC](https://discourse.mozilla.org/t/where-is-the-documentation-regarding-mozilla-common-voice-database/81476/1 "2021-06-10T07:21:08Z")

</div>

I can not find any documentation regarding Mozilla Common Voice except for: [Common Voice \> How does it work?](https://commonvoice.mozilla.org/en/about).  
So here are the things I would like to know about the database.

1. Where can I find a documentation and revision history of the database (v1, v2, … v5.1, v6.1) ?
2. What does each \*.tsv file represent especially the following:
  - invalidated.tsv
  - other.tsv
  - reported.tsv
  - validated.tsv  
and which of the tsv file represent **Clip Graveyard** mentioned in [Common Voice \> How does it work?](https://commonvoice.mozilla.org/en/about)?

3. What does each header represent in the \*.tsv file, especially “segment” & “reason”.  
I presume, “up\_votes” and “down\_votes” are related to “\>= 2 Yes votes” and “\>= No votes” mentioned in [Common Voice \> How does it work?](https://commonvoice.mozilla.org/en/about).
4. The header for reported.tsv is different from the rest, but I guess I will know why if what each file represents.

**HEADER (Y = Exist, N = Does Not Exist)**

| file | client\_id | path | sentence | sentence\_id | up\_votes | down\_votes | age | gender | accent | locale | segment | reason |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| reported.tsv | **Y** | **Y** | **Y** | N | **Y** | **Y** | **Y** | **Y** | **Y** | **Y** | **Y** | N |
| the rest of \*.tsv | N | N | **Y** | **Y** | N | N | N | N | N | **Y** | N | **Y** |

1. Is the database download incremental or full version?  
In other words,
  - a) Can I just download v6.1 and it will contain everything from v1 … v5.1?  
&nbsp;&nbsp;&nbsp;&nbsp;or
  - b) Do I need to download v1, v2, … v5.1, v6.1 to construct the full database

---

<div class="post-metadata">

**Author:** ![Joshua\_G](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/joshua_g/32/49155_2.png) [@Joshua\_G](https://discourse.mozilla.org/u/Joshua_G)\
**Post date:** [June 12, 2021, 6:30am UTC](https://discourse.mozilla.org/t/where-is-the-documentation-regarding-mozilla-common-voice-database/81476/2 "2021-06-12T06:30:59Z")

</div>

For a privacy browser, a little transparency would be appreciated. You lose your privacy “title” and you’ll be using chrome on this message board.

---

<div class="post-metadata">

**Author:** ![ftyers](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/ftyers/32/59280_2.png) [@ftyers](https://discourse.mozilla.org/u/ftyers)\
**Post date:** [June 12, 2021, 6:03pm UTC](https://discourse.mozilla.org/t/where-is-the-documentation-regarding-mozilla-common-voice-database/81476/3 "2021-06-12T18:03:02Z")

</div>

1. I don’t think there is any.
2. invalidated = has downvoted, other = excluded, reported = has been reported by a user, validated = has been upvoted enough
3. segment = if the clip is part of a particular subcorpus (e.g. the target segments subcorpus), reason = why the clip/transcript was rejected
4. it’s different because you have the reason the clip was reported.
5. each release is a full dump, not incremental. you only need to download the latest release.

Hope this helps!

Fran

---

<div class="post-metadata">

**Author:** ![makoto\_wada\_jp](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@makoto\_wada\_jp](https://discourse.mozilla.org/u/makoto_wada_jp)\
**Post date:** [June 14, 2021, 7:57am UTC](https://discourse.mozilla.org/t/where-is-the-documentation-regarding-mozilla-common-voice-database/81476/4 "2021-06-14T07:57:14Z")

</div>

@ftyers Thank you for your answer. Yes it is very helpful. I do have additional question:

> [@ftyers](#):
>
> 1. invalidated = has downvoted, other = excluded, reported = has been reported by a user, validated = has been upvoted enough

Regarding **other.tsv = excluded** , was it excluded because it doesn’t have enough upvotes or downvotes to be either in **validated.tsv** or **invalidated.tsv** , repsectively?

> [@ftyers](#):
>
> 1. segment = if the clip is part of a particular subcorpus (e.g. the target segments subcorpus) …

For **segment** , I see that all **\*.tsv** that is not **reported.tsv** is either empty “” (2,131,602 cases) or has the value “Benchmark” (49,384 cases [**validated.tsv** [32,726 cases] + **invalidated.tsv** [4,071 cases] + **other.tsv** [12,573 cases] + **test.tsv** [14 cases]) for CCommon Voice Corpus 6.1 English.

So if you happen to know what the “Benchmark” is, then please let me know. I am a little confused with the word “Benchmark” when the database has **train.tsv** , **dev.tsv** , and **test.tsv**. In other words, **test.tsv** sounds like the “Benchmark”.

---

<div class="post-metadata">

**Author:** ![makoto\_wada\_jp](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@makoto\_wada\_jp](https://discourse.mozilla.org/u/makoto_wada_jp)\
**Post date:** [June 14, 2021, 8:09am UTC](https://discourse.mozilla.org/t/where-is-the-documentation-regarding-mozilla-common-voice-database/81476/5 "2021-06-14T08:09:59Z")

</div>

@Joshua_G Thank you for your comments, I am relatively new to Mozilla Discourse so please excuse me if I sound unsavvy with the following questions:

> [@Joshua\_G](#):
>
> For a privacy browser, a little transparency would be appreciated.

1. What do you mean by privacy browser?
2. What is the transparency that is required?

> [@Joshua\_G](#):
>
> You lose your privacy “title” and you’ll be using chrome on this message board.

1. For the above, I am using a chrome browser. However I do not understand what you mean by “title” in the above passage?

---

<div class="post-metadata">

**Author:** ![ftyers](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/ftyers/32/59280_2.png) [@ftyers](https://discourse.mozilla.org/u/ftyers)\
**Post date:** [June 14, 2021, 1:37pm UTC](https://discourse.mozilla.org/t/where-is-the-documentation-regarding-mozilla-common-voice-database/81476/6 "2021-06-14T13:37:51Z")

</div>

Yes _Benchmark_ is the fixed-vocab subset (0…9 and yes/no), e.g.

```auto
other.tsv:e0230fbbdc872252b94443dc3fa5d50c0c8845ba44f6485ff57a2b7b0cf5db94cd1b69c7ca10634e630917815aaa4e6787a034eb82f6c1dfe406d5a2930078ae	common_voice_en_22430937.mp3	seven	0	0	fourties	male en	Benchmark

```

---

<div class="post-metadata">

**Author:** ![makoto\_wada\_jp](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@makoto\_wada\_jp](https://discourse.mozilla.org/u/makoto_wada_jp)\
**Post date:** [June 15, 2021, 10:33am UTC](https://discourse.mozilla.org/t/where-is-the-documentation-regarding-mozilla-common-voice-database/81476/7 "2021-06-15T10:33:34Z")

</div>

@ftyers Thank you for all your comments thus far. For your information, when I was searching for other information regarding Common Voice Dataset … I found:

- [README.md](https://github.com/common-voice/cv-dataset)

- [changelog (Revision History)](https://github.com/common-voice/cv-dataset/blob/main/CHANGELOG.md)

I just wanted to share the information with you.

---

<div class="post-metadata">

**Author:** ![heyhillary](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/heyhillary/32/49240_2.png) [@heyhillary](https://discourse.mozilla.org/u/heyhillary)\
**Post date:** [November 16, 2021, 3:23pm UTC](https://discourse.mozilla.org/t/where-is-the-documentation-regarding-mozilla-common-voice-database/81476/8 "2021-11-16T15:23:38Z")

</div>

Hey @makoto_wada_jp

There is also the Common Voice metadata available here: [https://github.com/common-voice/cv-dataset](https://github.com/common-voice/cv-dataset)

---

<div class="post-metadata">

**Author:** ![makoto\_wada\_jp](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@makoto\_wada\_jp](https://discourse.mozilla.org/u/makoto_wada_jp)\
**Post date:** [January 24, 2022, 8:04am UTC](https://discourse.mozilla.org/t/where-is-the-documentation-regarding-mozilla-common-voice-database/81476/9 "2022-01-24T08:04:35Z")

</div>

@heyhillary Yes. Thank you for pointing it out. The link you’ve provided ([https://github.com/common-voice/cv-dataset](https://github.com/common-voice/cv-dataset)) is the same as the link for [README.md](https://github.com/common-voice/cv-dataset) right above your reply.

- [https://github.com/common-voice/cv-dataset](https://github.com/common-voice/cv-dataset)  
||
- [README.md](https://github.com/common-voice/cv-dataset)

Sorry for the confusing way of linking but I much appreciate your response.

---

<div class="post-metadata">

**Author:** ![heyhillary](https://sea1.discourse-cdn.com/flex001/user_avatar/discourse.mozilla.org/heyhillary/32/49240_2.png) [@heyhillary](https://discourse.mozilla.org/u/heyhillary)\
**Post date:** [January 24, 2022, 11:12am UTC](https://discourse.mozilla.org/t/where-is-the-documentation-regarding-mozilla-common-voice-database/81476/10 "2022-01-24T11:12:40Z")

</div>

Hey @makoto_wada_jp,

No worreis, I didn’t realise that it was the same link.

just to note, we have recently updated the Common Voice about page to visbilise the documentation regarding the dataset.

> **[Mozilla Common Voice](https://commonvoice.mozilla.org/en/about?tab=how-validate#playbook)**
