Cleaning up sentence corpus

[dabinat]

@nukeador

One of the things that I was observing for Spanish is that complex or too technical words usually appear in just one sentence. I suspect it’s safe to remove just sentences with unique words as an initial step in cleaning them up.

I have a script to calculate word usage here:

word_usage.py

If you let me know what the total number of words should be (or the minimum number of occurrences), I could build up a list for English pretty quickly. The script should work fine with other languages too.