Some questions on cv-sentence-extractor

[mkohler (Michael Kohler)]

Thanks for creating this thread.

@bozden

Q1 . As far as I can see, when set, the broken_whitespace rule leaves sentences with sub-strings like " !" filtered out. Shouldn’t it fix them instead?

Then you would have to provide how it needs to be fixed and then there is not much difference to using replacements which allows for way more flexibility.

@bozden

Q2 . I could not see any construct (e.g. regex) to replace multiple spaces, do I miss it (Rust is new to me)?

Yes, that currently does not exist. Especially for Wikipedia I doubt that that is necessary. Do you encounter a lot of that? I can see how a double space can slip through, but more than that should be caught by editorial processes? For the double space using a replacement might be a good option instead.

@bozden

Q3 . I’m having trouble with the apostrophe’s. I used cvtools 's word_usage.py as instructed to get the word frequencies. I used it with --strip-apostrophes setting. In Turkish, proper names with suffixes are divided by them, like [Bülent’in], [Bülent’e], [Bülent’lerden] etc. (Although there is a “clean” function which replaces possible Unicode alternatives, some pass through…)

I’m not sure if @dabinat is still active?

@bozden

How can I handle similar stuff in cv-sentence-extractor (wrt. apostophes)?

Not sure I understand this question, can you elaborate?