Scarce Documentation for OpenNLP Integration about lucenenet

[rclabo]

The timing of your question couldn't be better.

Today @paulirwin mentioned in a separate issue #460 the following:

Unfortunately, while IKVM has been a reasonable go-to way to quickly support Java-based apps in the past, it has been abandoned by its main contributor in 2017 and has no .NET Core/NET Standard support.

FYI for everyone that ikvm-revived is now IKVM proper, and has .NET 6+ support as of v8.7. https://github.com/ikvmnet/ikvm/releases/tag/8.7.0

(You are probably already aware that the LuceneNET OpenNLP integration requires IKVM because OpenNLP is a seperate Apache project that is only available as Java. )

[paulirwin]

Also in a POC console app, I was able to use IKVM 8.7 and the <IkvmReference> csproj support to load the opennlp 1.9.4 jar directly without OpenNLP.NET (on an arm64 macOS machine, no less). In case that helps anyone.

[NightOwl888]

BTW - I was just looking at the code example you posted and noticed a big issue. You won't be able to use StandardAnalyzer because it uses the StandardTokenizer to tokenize the text, which pulls out all of the punctuation. For OpenNLP tokenization you must use the OpenNLPTokenizer to set up your token stream.

[rclabo]

I haven't used that specific feature of Lucene.NET, but when I bump into similar issues with other features one thing I find helpful is to look at the unit tests for the feature. Often, that helps me understand the coding pattern for using the feature. It may be worth a look.

[NightOwl888]

The OpenNLP project is not ported from Java, instead it uses IKVM to convert the bytecode directly into IL. So, there is nothing we add over and above the Java implementation of OpenNLP 1.9.1.

The documentation for OpenNLP 1.9.1 is on the OpenNLP website.

As far as the Lucene TokenStream APIs are concerned, the documentation (which is a bit scant) is in the Getting Started section. I concur with @rclabo that the best place to see examples are in the tests.

For such a task, I would also recommend using the test framework package to verify your analyzer implementation is compatible with Lucene.NET. And of course to look at other tests to work out how to use it because the docs are not great.

[rclabo]

Also, if you get it figured out, we'd greatly appreciate you posting some example code here or submitting a few additional unit tests that demonstrate the use more clearly. That will help the next person. Lucene and LuceneNET are really quite amazing but it sometimes takes some research to figure out how to use various aspects. Contributing back some of that knowhow would be greatly appreciated.

[GAInTheHouse]

I used @rclabo 's recommendation and tried to utilize the unit tests. However I ran into an exception I assume is because I'm not using .Net4.8:
System.TypeLoadException: Could not load type 'System.Reflection.Emit.MethodToken' from assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'

Does the OpenNLP integration not support .Net6.0?

@rclabo Lucene.Net is indeed an amazing library and I'd love to contribute to it once I figure it out!

[NightOwl888]

Also in a POC console app, I was able to use IKVM 8.7 and the <IkvmReference> csproj support to load the opennlp 1.9.4 jar directly without OpenNLP.NET (on an arm64 macOS machine, no less). In case that helps anyone.

Yeah, unfortunately, all of the latest versions that have ARM 64 support (and macOS x64) stopped working on .NET Framework. So, there isn't yet a version of IKVM that works everywhere. Support for mac was added in 8.7.0. It also appears that 8.7.0 and higher will get a type initialization exception if you don't explicitly create an object before using any of the converted libraries.

<MavenReference> is really how we should be handling this so people who depend on Lucene.Net.Analysis.OpenNLP can combine it with other Maven packages and it will resolve the IKVM compiled dependency versions so there are no conflicts.

[NightOwl888]

I have updated the OpenNLP docs in master, but the API is now so advanced from where we were in beta 16 that the docs wouldn't be correct if we updated them.

Your complaint was warranted, though. There were docs in Lucene 8.2.0 that we were missing and I have added them here:
https://github.com/apache/lucenenet/blob/43e0e894b9b40f5b28064a13ef98874a82c15330/src/Lucene.Net.Analysis.OpenNLP/overview.md

I also created a demo project showing how to build a filter for NER using OpenNLP and Sentiment using Stanford CoreNLP: https://github.com/NightOwl888/lucenenet-opennlp-mavenreference-demo

For integration with OpenNLP on .NET Core until the official release, you can use the feed here: https://dev.azure.com/lucene-net-temp2/Lucene.NET/_artifacts/feed/lucene-net-temp-2.

[GAInTheHouse]

This is great! Thank you @NightOwl888