CommonVoice Hebrew

Musicode · January 7, 2024, 3:30pm

Hello.
Who manages CommonVoice in Hebrew?

There is a lot of material from the Bible in the data set. This material is with cantillation notes (“Taamey hamikra”), which are non-standard characters, and should not be in the data set.

Also there are a lot of punctuation characters (dots and lines). Do those match the rules?

שלום.
מי מנהל את CommonVoice בעברית?

יש במערך הנתונים הרבה חומר מהתנ"ך. החומר הזה הוא עם טעמי המקרא, שאלו תווים לא סטנדרטיים, ולא אמורים להיות במערך הנתונים.

כמו כן יש הרבה חומר מנוקד. האם ניקוד תואם את הכללים?

bozden · January 7, 2024, 10:50pm

Hey @Musicode, FYI: Hebrew rules are here:

github.com

common-voice/common-voice/blob/main/server/src/core/sentences/validation/languages/he.ts

import {
  ERR_NO_ABBREVIATIONS,
  ERR_NO_NUMBERS,
  ERR_NO_FOREIGN_SCRIPT,
  ERR_TOO_LONG,
  ValidatorRule,
  ERR_NO_SYMBOLS,
} from '../../types/validators'

const tokenizeWords = require('talisman/tokenizers/words')

// Minimum of words that qualify as a sentence.
const MIN_WORDS = 1

// Maximum of words allowed per sentence to keep recordings short enough.
const MAX_WORDS = 14

// We want to create a pattern that rejects everything except what is allowed.
// Here we specify only valid character classes, i.e. these will be placed in
// `[^...]`.

This file has been truncated. show original

Here is the related issue which resulted in the above rules:

Hope these help…

Gina_Moape · January 10, 2024, 11:49am

Hi @Musicode

Thanks @Bulent_Daldalan.

Kindly note that community members are responsible for quality assurance and validation. To exclude sentences of this nature from the dataset, community members can downvote them as they encounter them during validation.

Thanks