Lengths of unicode text and generated ascii text are different about medkit

[khuynh11]

With the warning, we are able to see the characters that may need to be preprocessed but it is not always mandatory / necessary. The idea is to log this information to alert the user (we display the ascii conversion made during regexpmatcher operation) but it may be not blocking if these characters are not used in the rules.

If you want to make the conversion yourself to have more control or because your rules are based on some specific characters, you can preprocess the text before using regexpmatcher. In your example, quotes and degree may be managed in a preprocessing step..

Here is a code example with the preprocessing in medkit:

from medkit.core.text import TextDocument
from spacy import displacy
from medkit.text.spacy.displacy_utils import medkit_doc_to_displacy
import json
f = open('trainset.json')

data = json.load(f)

from medkit.text.ner import RegexpMatcher, RegexpMatcherRule

regexp_rules = [
    RegexpMatcherRule(regexp=r"\binsuffisance r.nale\b", label="phenotype"),
]

regexp_matcher = RegexpMatcher(rules=regexp_rules)

from medkit.text.preprocessing import Normalizer, NormalizerRule, QUOTATION_RULES

preprocessing_rules = QUOTATION_RULES
preprocessing_rules.append(NormalizerRule("°", "deg"))

preprocessing_op = Normalizer(output_label="clean_segment", rules=preprocessing_rules)

colors = {'phenotype': "#85C1E9"}
options = {"ents": ['phenotype'], "colors": colors}

for i in range(0, len(data)):
    s = ' '.join(data[i]['token'])
    #s = str(s).encode(encoding = 'ascii', errors = 'replace')
    #s = str(s)
    doc = TextDocument(text=s)
    normalized_segments = preprocessing_op.run([doc.raw_segment])
    entities = regexp_matcher.run(normalized_segments)
    if(len(entities)>0):
        for entity in entities:
            doc.add_annotation(entity)
        displacy_data = medkit_doc_to_displacy(doc)
        displacy.render(displacy_data, manual=True, style="ent", options=options)
        break

The quotes are already proposed / managed in medkit with QUOTATION_RULES but degree has not been integrated because it is the same character as number (n°).

If you have suggestion on how it can be more managed in medkit, we are open to improvements.

[BastienRance]

Thank you @khuynh11 , this is a great solution !

[khuynh11]

Thanks for your feedback, I close the issue.