bug about remove_markup about gensim

[gojomo]

Thank you for your report & code-to-reproduce!

In looking over the code, the more basic problem might be that RE_P1 (for REF tags) assumes that any /> ends the tag, as if there'd never be any kind of nested tag – but in your example fragments, the <NOWIKI/> or <BR/> tags do appear. Your suggested fix seems it would only remedy the problem in the few cases you've seen – while the unexpected nesting of any other tags ending /> risk triggering the same problem.

It looks like the RE_P9 (described as "external links"?) and RE_P10 (math) share a similar assumption that any /> must end of the tag-of-interest, rather than some nested tag, and thus might be susceptible to the same issue.

I think it'd be better to tune those regexes to not assume the absence of all nested tags, but that might risk other side-effects, or require other re-ordering of steps – I'm not sure why the existing regexes work the way they do, and processing HTML or Wikipedia's weird wikitext format with regexes is an inherently clunky & hard-to-maintain approach.

It might be most robust to move some form of RE_P11 ("All other tags") up in the process, but narrowed to leave any specific tags of interest.

[seadog-www]

Thank you for your report & code-to-reproduce!

In looking over the code, the more basic problem might be that RE_P1 (for REF tags) assumes that any /> ends the tag, as if there'd never be any kind of nested tag – but in your example fragments, the <NOWIKI/> or <BR/> tags do appear. Your suggested fix seems it would only remedy the problem in the few cases you've seen – while the unexpected nesting of any other tags ending /> risk triggering the same problem.

It looks like the RE_P9 (described as "external links"?) and RE_P10 (math) share a similar assumption that any /> must end of the tag-of-interest, rather than some nested tag, and thus might be susceptible to the same issue.

I think it'd be better to tune those regexes to not assume the absence of all nested tags, but that might risk other side-effects, or require other re-ordering of steps – I'm not sure why the existing regexes work the way they do, and processing HTML or Wikipedia's weird wikitext format with regexes is an inherently clunky & hard-to-maintain approach.

It might be most robust to move some form of RE_P11 ("All other tags") up in the process, but narrowed to leave any specific tags of interest.

Yes, my suggestion is not perfect. Do you have better method for processing HTML or Wikipedia's weird wikitext format without regexes?