Duplicate sentences in Portuguese

Common Voice
1

Hi! I found that some sentences are basically the same in Portuguese, with only differences in capitalization or punctuation that don’t change how they’re pronounced.

I’ve created a simple Python script, available in this gist, that reads the TSV file, normalizes the sentences and groups them by their normalized version. I found 143 duplicated sentences, appearing 295 times in total. The gist also contains the results for the Portuguese validated.tsv from CV27.

Some of them differ only in capitalization, such as these city names, which are not full sentences at all:

  • São José dos Campos (2ac9674cbdfe135882312fd715a5b3f5fadf3f414c005598a9389a91dac5e8b8) and São José Dos Campos (78c59c69545b52fa0627591cd4fe7007c2176e6f29e04df5e855689f91c690dd)
  • São José dos Pinhais (af8f8c3e65a289c45ac11e7be0c709d92e93c3a817b6141937e758605277cc6e) and São José Dos Pinhais (e1305a973a050cd2e9a1da7584f0bc00df822d30045b060afa77f7cc3f1d8e32)

There are also some that differ only by a period at the end, which doesn’t change the pronunciation, such as “Me manda por WhatsApp” (676105b1f37866161e3b69cbac8e46200e47c90325be92c342de6a304f923d2b) and “Me manda por WhatsApp.” (a1d253681c564cb1a448353c94b9a785746cb6bc0078d0bd22d93d41fbb9aedb).

Are there any plans to clean up duplicates like these? Or perhaps to add a validator to prevent duplicate entries from being added to the database in the first place?

2

Hi @turicas, yes this is true.

As I explained in the meeting, each sentence (and question in SPS) is hashed (language code + sentence is hashed to make it cross-language), and that hash is used as the primary key. A single symbol change will result in a different hash, thus uniqueness control is lost.

Are there any plans to clean up duplicates like these?

Although we plan some data sanitation/cleaning, this is not part of it. Most of these have attached audio, and they can only be deleted as a whole.

Or perhaps to add a validator …

Actually it would need a normalizer - (e.g. removing all punctuation and converting to lowercase) - and unfortunately normalization is mostly language specific, and no single method/library covers the whole universe of 7000+ possible languages, or cover/implement all - it would be very heavy and “costly”.

One other option would be using similarity distance metrics (e.g. Levenshtein, Cosine, …) but these are too costly algorithms and they should be re-evaluated for each new sentence - against every existing sentence (think of adding 1000 new sentences to 1M existing).

I use a light/modified version of Levenshtein for offline pre-processing scraped books (books from authors who died 70+ years ago). Get text corpus from last release, get new ones, set a threshold and filter out those similar - it takes a while. But again, it is not again a totally dependable measure. E.g. think of these:

  • Right.
  • Right!
  • Right?
  • All right
  • …

Where do you put your threshold? They are valid and all produce another intonation, have different meaning, these are all language dependent, need to be tested and fine-tuned, etc…

So, we think that it is the dataset consumer’s job to postprocess the data for a specific application/model. In this case, if you think that they interfere with your model, you can use postprocessing algorithms (normalization + distance measurements) to auto-blacklist them before training and then measure the results.

I’ll be very happy to talk about alternatives, as always - please keep these coming :saluting_face: !

3

Here is Portuguese text-corpus analysis from v22.0
Ref: Text-corpus tab of: Common Voice Dataset Analyzer | cv-toolbox

image
image838×683 26.3 KB

As you can see the difference between normalized and originals are very small, it is just noise for the model (which is mostly good).

The main problem here is the size of the text corpus. I’d focus on the text corpus, add many more pre-processed, quality and longer sentences, and even that noise will diminish…

4

Yes, the intonation changes, so “.” is different from “?” but it’s the same as “” and “”. I think the normalization steps (at least for Portuguese) could be:

  • Replace \s+ (any kind of space, one or more times) with " " (one single space, ASCII code 32) – this will avoid typos like double spaces and will also normalize TABs and other space chars (such as 0xA0) to one, well-defined, space
  • Strip spaces from the start and end of the sentence
  • Put everything to lowercase
  • If the last character is “.” and the penultimate is not “.” (skip “…”), remove it
  • Calculate the hash based on the last step.

With these rules, most of the duplicate sentences I found would not be allowed again and the cost of checking will be almost the same.

What do you think? I could implement a function like this in Python and run through the Portuguese TSVs to check which sentences would be avoided as duplicate - even if we can’t do nothing for the ones already on the database, we avoid further duplicates.

5

Yes, but we cannot generalize most of them, e.g. think of Korean Hangul. As I said, normalization is language specific and must be handled as post-processing, i.e. pre-processing prior to training.

These are now handled on the client side, before sending to the server, double handled on BE, checked against the DB and duplicates will be reported to front-end.

We do that after the above steps…

This we should not do, for the sake of language models. You can do it on your workflow, but everyone has another workflow targeting another type of model.

Unfortunately, we cannot generalize such rules for thousands of languages.

Please have a look at these repositories for related info, where multiple languages are supported:

6

Thanks for the reply! I disagree regarding the lowercase (at least for Portuguese), but I understand there must be a post-processing (pre-training) step where the dataset user will cleanup the way she wants.