No --validate_label_locale specified, your might end with inconsistent dataset

[Alan_Belem]

Hello

I am training my own model with portuguese dataset from Mozilla Comon Voice. I am using Colab for that.

Running this line:
!/content/DeepSpeech/bin/import_cv2.py --filter_alphabet /content/DeepSpeech/data/alphabet.txt ‘/content/drive/My Drive/pt-language’

I got this issue:
Loading TSV file: /content/drive/My Drive/pt-language/test.tsv
Importing mp3 files…
WARNING: No --validate_label_locale specified, your might end with inconsistent dataset.
WARNING: No --validate_label_locale specified, your might end with inconsistent dataset.

Can someone help me, please?
Thanks.

[othiele (Olaf Thiele)]

Please search before you post, you are not the first to get that message …

What and how to report if you need support

We are happy for every new user of DeepSpeech. As it is under active development, stuff changes quickly and you might have problems running it. We are here to help and might point to this thread if needed Please do the following before posting: Search this forum or Google for your problem Read the documentation for your version here Check Github issues for more technical problems If you still struggle, feel free to open a new thread in this forum (don’t answer to this post). …

[lissyx ((slow to reply) [NOT PROVIDING SUPPORT])]

@Alan_Belem As @othiele said, please make some efforts and search: https://deepspeech.readthedocs.io/en/v0.8.2/search.html?q=validate_label_locale&check_keywords=yes&area=default

We have docs, but if you dont read them or people don’t give feedback like “i searched the docs and have not found” , we can’t improve the project.

[Alan_Belem]

You are right.

I read the docs, I found this " Some importers might require additional code to properly handled your locale-specific requirements. Such handling is dealt with --validate_label_locale flag that allows you to source out-of-tree Python script that defines a validate_label function. Please refer to util/importers.py for implementation example of that function. If you don’t provide this argument, the default validate_label function will be used. This one is only intended for English language, so you might have consistency issues in your data for other languages."

For sure it has to do with my issue, because I am not training for English language.
But I don’t understand what I have to do.
I saw the importers.py code, it really had mentions about validate_label, but I don’t know where I should change.

[lissyx ((slow to reply) [NOT PROVIDING SUPPORT])]

@Alan_Belem

I saw the importers.py code, it really had mentions about validate_label, but I don’t know where I should change.

As documented --validate_label_locale allows you to pass a locale-specific python file containing a function like validate_label to perform cleanup locale-specific.

[Alan_Belem]

It worked. I looked for the validate function, find it, then wrote --validate_label_locale /validatefunction.py, as you said and as the docs said.
Now it is training.
Thank you very much!

[Alan_Belem]

Obs: For who have the same issue, “validationfunction.py” was just an eg. For me the name was “validate_locale_fra.py”