kaqqao / nutch-element-selector Goto Github PK

This is probably a problem with my setup rather than your plugin.

I have nutch-2.3.1 and have installed your plugin to get rid of a bunch of navigation elements, breadcrumbs, footers components from my standard web pages.

I've set my nutch-site.xml property as follows (also using tika for pdf and word documents)

plugin.includes protocol-httpclient|urlfilter-regex|element-selector|index-(basic|more)|query-(basic|site|url)|indexer-solr|nutch-extensionpoints|parse-(text|html|tika|js)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)

And here are the elements I am trying to grab with the blacklist, also in nutch-site.xml

parser.html.selector.blacklist header,footer,div.breadcrumbs,div.searchForm,table.jobTable A comma-delimited.... I've tried with parse-html included and excluded. I read that if you enable tika don't use parse-html. But I thought that for your plugin to work, parse-html must be enabled.

Any guidance would be helpful.

Thanks very much in advance.

Hi there.

This is great work. I have it working on Nutch-2.4 (Feb 2021).

Question: why would such an important plugin such as this not have been integrated into the 1.x stream?

Is there any way to make this work with Nutch-1.18

Thanks,
Colin