Issues in the Romanian dataset

[bozden (Bülent Özden)]

Now, Q3 - in parts…

@elearningbakery

Would it be possible to have a look at the sentences which have been validated, the ones which need to be validated, and the ones which still need to be recorded, get a frequency list of content words (nouns, verbs, adjectives, and adverbs) from them, compare it with a list from a bigger Romanian corpus (maybe even roTenTen21), …

You can look at the data ONLY in the releases, or see them during validation. When you look the metadata in the release:

• You will see *_sentences.tsv files for text-corpus only
• Each other metadata file also has some sentence column valid for that data.

First part is already there (or not). Please see the explanations and a problem in this issue.

nouns, verbs, adjectives, and adverbs
compare it with a list from a bigger Romanian corpus

That would require language knowledge of course, maybe some NLP tooling for that particular language. I don’t have them, so I do generic tokenization and count of words for the frequencies in Analyzer text-corpus tab.

You should do these knowing the language, but anyway, I uploaded the intermediate files for you (v20.0 ro)… If our results do not merge, please break the glass (issue).

@elearningbakery

and then use LLMs to generate example sentences with the content words which are not yet present enough in the CV data? Would LLM-generated sentences of up to 14 words each be acceptable for the CV?

Nope, AI generated sentences are not allowed. Here is some info:

ChatGPT-Generated Sentences in MCV

Hi community Recently, we’ve had people attempt to submit ChatGPT-generated sentences to MCV. We are excited by the opportunities that large language models (LLMs) provide for innovation, and we understand that many of our communities face a significant challenge generating open source sentences. However, presently, we are not accepting AI generated content. LLMs have, for the most part, been trained on data scraped without consent from the internet, and there are ongoing conversations about w…

I think its time to talk about AI generated sentences again (read answers/resolution)

It is degenerative, and we will see in LLMs in a few years (actually dementia is already there).