Solution of complex Named Entity Recognition tasks (and subtask Nested NER) based on modern Large Language Models (LLMs).
You should form python dictionaries for every text and labels. Let's look at an simplified example from Russian Drug Reaction Corpus (RuDReC).
- Input text:
Это старый-добрый Римантадин, только в сиропе.
- Labels:
Римантадин - Drugname, сиропе - Drugform
Russian:
Ты решаешь задачу NER. Извлеки из текста слова, относящиеся к каждой из следующих сущностей: Drugname, Drugclass, DI, ADR, Finding.
English:
You are solving the NER problem. Extract from the text words related to each of the following entities: Drugname, Drugclass, DI, ADR, Finding.
You can use one of two supported version.
raw_entities = {
'Drugname': ['Римантадин'],
'Drugclass': [],
'Drugform': ['сиропе'],
'DI': [],
'ADR': [],
'Finding': []
}
short_form_output=True (available with Nerel-BIO and MultiCoNER)
raw_entities = {
'Drugname': ['Римантадин'],
'Drugform': ['сиропе']
}
MODEL_INPUT_TEMPLATE = {
'prompts_input': "### Задание: {instruction}\n### Вход: {inp}\n### Ответ: ",
'output_separator': "Ответ: "
}
Or english version
MODEL_INPUT_TEMPLATE = {
'prompts_input': "### Task: {instruction}\n### Input: {inp}\n### Answer: ",
'output_separator': "Answer: "
}
instruction_ner/utils/instruct_dataset.py
class Instruction(TypedDict):
instruction: str
input: str
output: str
source: str
raw_entities: dict[str, list[str]]
id: str
{'instruction': 'Ты решаешь задачу NER. Извлеки из текста слова, относящиеся к каждой из следующих сущностей: Drugname, Drugclass, DI, ADR, Finding.',
'input': 'Это старый-добрый Римантадин, только в сиропе.\n',
'output': 'Drugname: Римантадин\nDrugclass: \nDrugform: сиропе\nDI: \nADR: \nFinding: \n',
'source': '### Задание: Ты решаешь задачу NER. Извлеки из текста слова, относящиеся к каждой из следующих сущностей: Drugname, Drugclass, DI, ADR, Finding.\n### Вход: Это старый-добрый Римантадин, только в сиропе.\n### Ответ: ',
'raw_entities': {'Drugname': ['Римантадин'],
'Drugclass': [],
'Drugform': ['сиропе'],
'DI': [],
'ADR': [],
'Finding': []},
'id': '1_2555494.tsv'}
instruction_ner/utils/
- Russian Drug Reaction Corpus (RuDReC)
- NEREL-BIO (Nested Named Entities)
- CoNLL-2003
- MultiCoNER II (2023) (HF, fine and coarse level mapping of the tags)
instruction_ner/metric.py
You can use the implemented functions with the output of inference_instruct calculate metrics.
import pandas as pd
from utils.rudrec.rudrec_utis import ENTITY_TYPES
from metric import calculate_metrics_from_dataframe
prediction = pd.read_json('prediction.json')
prediction.head(3)
id | extracted | target | |
---|---|---|---|
0 | 8_1443820.tsv | {'Drugname': [], 'Drugclass': [], 'Drugform': ['таблетки'], 'DI': [], 'ADR': [], 'Finding': []} | {'Drugname': [], 'Drugclass': [], 'Drugform': ['таблетки'], 'DI': [], 'ADR': [], 'Finding': []} |
1 | 1_2555494.tsv | {'Drugname': ['Римантадин'], 'Drugclass': [], 'Drugform': ['сиропе'], 'DI': [], 'ADR': [], 'Finding': []} | {'Drugname': ['Римантадин'], 'Drugclass': [], 'Drugform': ['сиропе'], 'DI': [], 'ADR': [], 'Finding': []} |
2 | 1_618967.tsv | {'Drugname': [], 'Drugclass': [], 'Drugform': [], 'DI': [], 'ADR': [], 'Finding': []} | {'Drugname': [], 'Drugclass': [], 'Drugform': [], 'DI': [], 'ADR': [], 'Finding': []} |
from metric import calculate_metrics_from_dataframe
metrics = calculate_metrics_from_dataframe(prediction, ENTITY_TYPES)
{'Drugname': {'precision': 0.9670250896057347,
'recall': 0.9195637355146558,
'f1': 0.9426974143955277}, ...}
Tables with metrics for implemented datasets (link)
Instruction LLM for NER performs well on flat entities, but performs poorly on datasets with large tagset and nested entites. It's hard to output all entities in a single response due to performance limitations.
Thus, LLM and encoder model produce comparable results on flat-ner datasets with incredibly different training and inference times.
- poteminr/llama2-rudrec adapter model (LoRA)
- poteminr/llama2-rudrec-merged merged with base model
- poteminr/mistral-rudrec adapter model (LoRA)
and other models on HF such as T5 Llama: poteminr