consulthys / elastixray Goto Github PK

As done in elyzer, add another option to view the state of the analysis at each step, i.e. after the tokenizer and each token filter.

For example, if we analyze the tokens A big brown fox with the standard analyzer, the output should show something similart o this:

1. Initial text: A big brown fox
2. Tokenizer: standard
   Tokens: [A, big, brown, fox]
3. Token filter: lowercase
   Final tokens: [a, big, brown, fox]