There are some sentences, that I reported many times on audio clip checking tab but I continue to meet them again and again. What is the reason for that?
Are reports on "Scripted Speech. Validate readings " tab always marked as “potencial audio clip problems” or it depends on the reason I choose, so if I choose “Grammatical / spelling error” the sentence actually will be reported and if I choose “Offensive speech. The clip has disrespectful or offensive language.”, the clip will be reported? So what about “other reason”. Will some report with that reason be sent to “Reported clips” or to “Reported sentences”?
Also: how and by who these reports are reviewed, especially for languages the team doesn’t have native speakers?
SCS and SPS act differently in that respect due to their natures.
In SPS the first 3 take them out of circulation, “other” will only remove them from you.
In SCS - nothing is taken out of circulation, reports get just logged and in releases “reported.tsv” file include them. So it will need a post process to remove them - by dataset consumer.
See two related issues (mentioned multiple times, search please)
The second one is sentence validation related but also valid for Listen page…
If something is totally wrong, I vote NO to remove them from validated…
A remark from my experience: Half of the reports can be wrong or biased. E.g. for English someone can report sentences because of -ise/-ize difference, or I saw someone reported this sentence: “We need freedom!”
Hi @Libra. Thanks for checking in about this and thanks @bozden for answering. Just to phrase another way, if you are logged in when validating, you shouldn’t receive the same clip after you’ve reported it in Scripted Speech. In Scripted Speech, a reported clip isn’t removed from another user’s review queue, rather the information about clip reports is included in the release data. Since the clips aren’t removed, the platform can only tell if you have seen a clip before if it is able to recognize you as the same user who reported the clip last time. For Scripted Speech clip reports, whoever downloads the dataset decides how to handle the reported clips.
It is a little bit strange behavior and should be clarified on the report tab, while reporting the sentence/recording.
Even more strange honestly. It means that every my report was to this ‘reported.csv’ and not to the team, that will check my reports? It is frustrating…
For instance, I met many Russian sentences like
“???? ????? ???????? ????.”
“? ??? ?? ????.”
etc
and reported them with ‘other’ reason adding the comment ‘perhaps encoding problems’, and thought it will be reported and stored and will be checked by someone after sometime.
I thought I help to other contributors not get sentences with mistakes, but all hundrends and thousands of reported sentences are still in circullation and other contributors all the time meet bad Russian sentences.
What is “Report” mechanism used for, If in fact it is just an analog of ‘Skip’ button. Why does ‘report’ feature even exist? And what is ‘comment’ field in this other reason is used? Is it even stored?
It is hard to take part in this project, when every 4th or 5th sentence is incorrect and has some mistake and there is no any instrument for taking them out of the circultation…
Of course things like that happen and will continue to happen
Hi @Libra. We’re working on improving some of our report features. The goal of the current report design was to protect valid clips against being taken out of circulation in the event that the reports are problematic, such as the ones described here
It is also problematic for the team to review some reports, such as those about the content of the prompt, as we don’t have speakers of every language on the platforms. The idea of passing that information along with the released data is to enable researchers to address those for the languages they are working with.
That being said, I think having an option to report something as having a technical error might be useful for cases like the encoding issue you mentioned. Errors like those would be good for the team to review. We are currently working on improvements to the platforms and I’ll bring this idea up in the weekly meeting.
