Doesnt extract li tags content with an id about trafilatura

[adbar]

Hi @alroythalus, a screenshot is not enough to diagnose the problem here. Could you please provide an URL for the first case or copy the critical part of the HTML code?

[alroythalus]

yes sure @adbar. This is the site in the 1st case https://www.kia.com/us/en/privacy and this in the case image https://docs.github.com/en/site-policy/privacy-policies/github-privacy-statement.

It misses out on a lot of core information, the screenshot I sent above is just one example. You can just run it against this site https://www.kia.com/us/en/privacy and have a look. Most of the bullet points are omitted.

[alroythalus]

I am using this config of trafilature.

extract( original_html, include_formatting=True, include_tables=True, include_comments=False, include_links=False, output_format="XML" )

[adbar]

Thanks, these are quite complex web pages but let's see if there is something we can do.

[adbar]

A quick summary of ongoing developments:

• the Gihub case is now solved as far as I can see
• the Kia case occurs because the main text elements are contained by a <template> tag which is wrongly deleted. I can fix this but then another problem arises: due to the complexity of the page the extraction heuristics are not optimal and non-p elements get discarded.