Musicode
January 7, 2024, 3:30pm
1
Hello.
There is a lot of material from the Bible in the data set. This material is with cantillation notes (“Taamey hamikra”), which are non-standard characters, and should not be in the data set.
Also there are a lot of punctuation characters (dots and lines). Do those match the rules?
שלום.
יש במערך הנתונים הרבה חומר מהתנ"ך. החומר הזה הוא עם טעמי המקרא, שאלו תווים לא סטנדרטיים, ולא אמורים להיות במערך הנתונים.
כמו כן יש הרבה חומר מנוקד. האם ניקוד תואם את הכללים?
bozden
January 7, 2024, 10:50pm
2
Hey @Musicode , FYI: Hebrew rules are here:
import {
ERR_NO_ABBREVIATIONS,
ERR_NO_NUMBERS,
ERR_NO_FOREIGN_SCRIPT,
ERR_TOO_LONG,
ValidatorRule,
ERR_NO_SYMBOLS,
} from '../../types/validators'
const tokenizeWords = require('talisman/tokenizers/words')
// Minimum of words that qualify as a sentence.
const MIN_WORDS = 1
// Maximum of words allowed per sentence to keep recordings short enough.
const MAX_WORDS = 14
// We want to create a pattern that rejects everything except what is allowed.
// Here we specify only valid character classes, i.e. these will be placed in
// `[^...]`.
show original
Here is the related issue which resulted in the above rules:
opened 11:39PM - 17 Jul 23 UTC
closed 08:48AM - 18 Jul 23 UTC
This is not a feature request nor a bug. I'm in touch with Francis and they sugg… ested I post the issue here.
**Description**
Someone submitted - with the best of intentions I believe - a large corpus of sentences from the Hebrew Bible including very complex diacritics (called Hebrew cantillation). These sentences are archaic, not in any means whatsoever spoken language, and terribly hard to read. This becomes too much of a burden and quite redundant for volunteers to reject each of these sentences, and I would dare to say this is a blocker for contributions for the Hebrew language altogether.
**Describe the solution you'd like**
I would like your help in removing sentences from this source (all from The Westminster Leningrad Codex) from the Hebrew corpus altogether.
**Describe alternatives you've considered**
Have volunteers I am trying to recruit to add more sentence go over all these sentences - this is unfortunately a no-go and seems redundant.
I can also suggest adding a button to report a given source. With enough votes disqualifying data from a specific source, this can aid in future, similar issues. However, I suspect this is seldom the case. In addition, the source can easily vary - if for each batch of sentences a different citation was used, for instance, if the chapter number was mentioned for each bunch of sentences from that chapter in a public domain text.
**Additional context**
This is how it looks like for a user coming in to volunteer in sentence review:
What you see here is a sentence from the Hebrew Bible, with a mix of regular diacritics ([niqqud](https://en.wikipedia.org/wiki/Niqqud)) and special diacritics that appear only in the bible ([cantillation](https://en.wikipedia.org/wiki/Hebrew_cantillation)). Though the sentence is intelligible to modern Hebrew speakers, it is light-years from being spoken language and thus often very difficult to read properly.
Hope these help…
gina
January 10, 2024, 11:49am
3
Hi @Musicode
Thanks @Bulent_Daldalan .
Kindly note that community members are responsible for quality assurance and validation. To exclude sentences of this nature from the dataset, community members can downvote them as they encounter them during validation.
Thanks