[Discussion] AnnotationTable.TokenizedAnnotationTable about arctokenization

[kMutagene]

i am thinking of a complete rework of the parsing. I think we should use ARCtrl's composite column model.

[kMutagene]

iirc ARCtrl parses annotation tables like this:

• pattern match and assign grouping
• everything not assignes is Freetext

if that is true, then it should be easy to use for tokenization as well, by filling these composite columns with CvParams in an additional step.

Sounds good? @HLWeil

[HLWeil]

Yup that's pretty much it.

It sounds fine with me, provided that it doesn't fail in some specific cases which should be checked. But as a starting point for getting your tokens for further use it should be good!

[kMutagene]

Closing this as we use ARCtr's ARCTable parser now, which we then tokenize. See #48