stanleylsx / entity_extractor_by_ner

基于Tensorflow2.3开发的NER模型，都是CRF范式，包含Bilstm(IDCNN)-CRF、Bert-Bilstm(IDCNN)-CRF、Bert-CRF，可微调预训练模型，可对抗学习，用于命名实体识别，配置后可直接运行。

Nice to see you.👋

entity_extractor_by_ner's Issues

label和token是基于train数据学出来的，test出现新的则无法使用

TypeError: 'NoneType' object is not callable

Traceback (most recent call last):
File "main.py", line 74, in
train = Train(configs, dataManager, logger)
File "/root/ner/engines/train.py", line 55, in init
self.ner_model = NerModel(configs, vocab_size, num_classes)
File "/root/ner/engines/model.py", line 23, in init
self.pretrained_model = TFBertModel.from_pretrained('./bert-base-chinese')
File "/root/miniconda3/lib/python3.8/site-packages/transformers/modeling_tf_utils.py", line 2919, in from_pretrained
model.build() # build the network with dummy inputs
File "/root/miniconda3/lib/python3.8/site-packages/transformers/modeling_tf_utils.py", line 1134, in build
if self.built or call_context().in_call:
TypeError: 'NoneType' object is not callable

更换预训练模型，比如albert

up主，请问代码中，如果要更换预训练模型albert，需要除了需要修改system.config之外，还需要修改那些地方呢？

equests.exceptions.SSLError:

raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/bert-base-chinese (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:852)'),))
这种问题怎么解决哇，求救
what can i do to solve this problem

为什么每次predict的结果会不同呢

请教下关系抽取？

您的程序目前实现的是实体抽取，没有关系抽取？

Finetune-Bert-CRF

大佬，为什么使用Finetune-Bert-CRF训练时，precision跟f1一直都是-1？

自己重新训练报错

您好 再更换自己数据集从头训练·时，固定在2%的位置跳这个错 但是不是很理解原因 劳请提点一下
2%|▏ | 20/1022 [00:18<15:17, 1.09it/s]
Traceback (most recent call last):
File "C:/Users/lzh/Desktop/entity_extractor_by_ner-master/entity_extractor_by_ner-master/main.py", line 72, in
train(configs, dataManager, logger)
File "C:\Users\lzh\Desktop\entity_extractor_by_ner-master\entity_extractor_by_ner-master\engines\train.py", line 151, in train
X_train_batch, y_train_batch, batch_pred_sequence, configs, data_manager)
File "C:\Users\lzh\Desktop\entity_extractor_by_ner-master\entity_extractor_by_ner-master\engines\utils\metrics.py", line 34, in metrics
y = [str(data_manager.id2label[val]) for val in y_true[i] if val != data_manager.label2id[data_manager.PADDING]]
File "C:\Users\lzh\Desktop\entity_extractor_by_ner-master\entity_extractor_by_ner-master\engines\utils\metrics.py", line 34, in
y = [str(data_manager.id2label[val]) for val in y_true[i] if val != data_manager.label2id[data_manager.PADDING]]
KeyError: -1

转换类型有问题

tensor_targets = tf.convert_to_tensor(targets, dtype=tf.int32)这里会报错
改成tensor_targets = tf.convert_to_tensor(targets, dtype=tf.int64)就可以使用

内存消耗较大，出现内存交换

Allocation of 12826795200 exceeds 10% of free system memory.

我这里用200M左右的数据做的训练，60M做dev，batch_size已经搞得很小了。

请问代码里有什么办法或者参数可以解决吗？

感谢！

ValueError: Tensor conversion requested dtype int32 for Tensor with dtype int64: <tf.Tensor 'targets:0' shape=(32, 300) dtype=int64>

用下载的代码源码改为训练模式训练，报错
ValueError: in user code:

/home/ml/entity_extractor_by_ner/engines/model.py:34 call  *

ValueError: Tensor conversion requested dtype int32 for Tensor with dtype int64: <tf.Tensor 'targets:0' shape=(32, 300) dtype=int64>

使用微调的时候，训练老是0，为什么

训练为啥前几个 准确率啥的都是-1呢？是程序的设定吗？

3%|▎ | 20/725 [00:33<17:52, 1.52s/it]training batch: 20, loss: 29.96524, precision: -1.000 recall: 0.000 f1: -1.000 accuracy: 0.852
6%|▌ | 40/725 [01:04<17:31, 1.53s/it]training batch: 40, loss: 24.01720, precision: -1.000 recall: 0.000 f1: -1.000 accuracy: 0.889
8%|▊ | 60/725 [01:36<17:31, 1.58s/it]training batch: 60, loss: 21.02634, precision: -1.000 recall: 0.000 f1: -1.000 accuracy: 0.821
11%|█ | 80/725 [02:09<17:02, 1.59s/it]training batch: 80, loss: 17.43504, precision: -1.000 recall: 0.000 f1: -1.000 accuracy: 0.859
14%|█▍ | 100/725 [02:42<17:20, 1.66s/it]training batch: 100, loss: 12.46207, precision: -1.000 recall: 0.000 f1: -1.000 accuracy: 0.908
17%|█▋ | 120/725 [03:16<17:41, 1.75s/it]training batch: 120, loss: 9.91527, precision: 1.000 recall: 0.029 f1: 0.057 accuracy: 0.923
19%|█▉ | 140/725 [03:49<15:39, 1.61s/it]training batch: 140, loss: 11.09897, precision: 0.636 recall: 0.146 f1: 0.237 accuracy: 0.915

Could not generate token2id file

python版本

是不是必须用Python3.7才能无错安装依赖

ValueError: Tensor's shape (300, 800) is not compatible with supplied shape (768, 800)

root@0213952a61fb:/home/entity_extractor_by_ner# python main.py
2021-12-29 03:24:28.233606: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-12-29 03:24:29
++++++++++++++++++++++++++++++++++++++++CONFIGURATION SUMMARY++++++++++++++++++++++++++++++++++++++++
Status:
mode : train
++++++++++++++++++++++++++++++++++++++++
Datasets:
datasets fold: data/example_datasets
train file: train.csv
validation file: dev.csv
vocab dir: data/example_datasets/vocabs
delimiter : b
use bert: True
use bilstm: True
finetune : True
checkpoints dir: checkpoints/finetune-bert-bilstm-crf
log dir: data/example_datasets/logs
++++++++++++++++++++++++++++++++++++++++
Labeling Scheme:
label scheme: BIO
label level: 2
suffixes : ['ORG', 'PER', 'LOC']
measuring metrics: ['precision', 'recall', 'f1', 'accuracy']
++++++++++++++++++++++++++++++++++++++++
Model Configuration:
embedding dim: 768
max sequence length: 300
hidden dim: 200
CUDA VISIBLE DEVICE: 0
seed : 42
++++++++++++++++++++++++++++++++++++++++
Training Settings:
epoch : 300
batch size: 32
dropout : 0.5
learning rate: 0.001
optimizer : Adam
checkpoint name: model
max checkpoints: 3
print per_batch: 20
is early stop: True
patient : 5
++++++++++++++++++++++++++++++++++++++++CONFIGURATION SUMMARY END++++++++++++++++++++++++++++++++++++++++
loading vocab...
dataManager initialed...
mode: train
loading data...
1112231it [00:48, 22721.77it/s]
loading data...
223833it [00:09, 22922.39it/s]
training set size: 23181, validating set size: 4636
2021-12-29 03:25:29.113422: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-12-29 03:25:29.114056: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-12-29 03:25:29.147051: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-12-29 03:25:29.147614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 23.70GiB deviceMemoryBandwidth: 871.81GiB/s
2021-12-29 03:25:29.147632: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-12-29 03:25:29.149236: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-12-29 03:25:29.149262: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-12-29 03:25:29.149942: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-12-29 03:25:29.150171: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-12-29 03:25:29.150271: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2021-12-29 03:25:29.150685: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-12-29 03:25:29.150785: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-12-29 03:25:29.150797: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-12-29 03:25:29.151450: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-12-29 03:25:29.151470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-12-29 03:25:29.151477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
Some layers from the model checkpoint at bert-base-chinese were not used when initializing TFBertModel: ['mlm___cls', 'nsp___cls']

• This IS expected if you are initializing TFBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).

• This IS NOT expected if you are initializing TFBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  All the layers of TFBertModel were initialized from the model checkpoint at bert-base-chinese.
  If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertModel for predictions without further training.
  Restored from checkpoints/bilstm-crf/model-14
  ++++++++++++++++++++training starting++++++++++++++++++++
  epoch:1/300
  0%| | 0/725 [00:02<?, ?it/s]
  Traceback (most recent call last):
  File "main.py", line 72, in
  train(configs, dataManager, logger)
  File "/home/entity_extractor_by_ner/engines/train.py", line 82, in train
  logits, log_likelihood, transition_params = ner_model(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 1012, in call
  outputs = call_fn(inputs, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 828, in call
  result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 871, in _call
  self._initialize(args, kwds, add_initializers_to=initializers)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 725, in _initialize
  self._stateful_fn._get_concrete_function_internal_garbage_collected( # pylint: disable=protected-access
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 2969, in _get_concrete_function_internal_garbage_collected
  graph_function, _ = self._maybe_define_function(args, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 3361, in _maybe_define_function
  graph_function = self._create_graph_function(args, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 3196, in _create_graph_function
  func_graph_module.func_graph_from_py_func(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/func_graph.py", line 990, in func_graph_from_py_func
  func_outputs = python_func(*func_args, **func_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 634, in wrapped_fn
  out = weak_wrapped_fn().wrapped(*args, **kwds)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 3887, in bound_method_wrapper
  return wrapped_fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/func_graph.py", line 977, in wrapper
  raise e.ag_error_metadata.to_exception(e)
  ValueError: in user code:

  /home/entity_extractor_by_ner/engines/model.py:43 call *
  outputs = self.bilstm(outputs)
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/wrappers.py:539 call **
  return super(Bidirectional, self).call(inputs, **kwargs)
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer.py:1008 call
  self._maybe_build(inputs)
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer.py:2710 _maybe_build
  self.build(input_shapes) # pylint:disable=not-callable
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/wrappers.py:694 build
  self.forward_layer.build(input_shape)
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/recurrent.py:578 build
  self.cell.build(step_input_shape)
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/utils/tf_utils.py:272 wrapper
  output_shape = fn(instance, input_shape)
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/recurrent.py:2344 build
  self.kernel = self.add_weight(
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer.py:623 add_weight
  variable = self._add_variable_with_custom_getter(
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/training/tracking/base.py:805 _add_variable_with_custom_getter
  new_variable = getter(
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py:130 make_variable
  return tf_variables.VariableV1(
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py:260 call
  return cls._variable_v1_call(*args, **kwargs)
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py:206 _variable_v1_call
  return previous_getter(
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py:67 getter
  return captured_getter(captured_previous, **kwargs)
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py:712 variable_capturing_scope
  v = UnliftedInitializerVariable(
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py:264 call
  return super(VariableMetaclass, cls).call(*args, **kwargs)
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py:227 init
  initial_value = initial_value()
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/training/tracking/base.py:81 call
  return CheckpointInitialValue(
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/training/tracking/base.py:117 init
  self.wrapped_value.set_shape(shape)
  /usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py:1215 set_shape
  raise ValueError(

  ValueError: Tensor's shape (300, 800) is not compatible with supplied shape (768, 800)

WARNING:tensorflow:Unresolved object in checkpoint: (root).ner_model.embedding.embeddings
WARNING:tensorflow:Unresolved object in checkpoint: (root).ner_model.dense.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).ner_model.dense.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).ner_model.bilstm.forward_layer.cell.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).ner_model.bilstm.forward_layer.cell.recurrent_kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).ner_model.bilstm.forward_layer.cell.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).ner_model.bilstm.backward_layer.cell.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).ner_model.bilstm.backward_layer.cell.recurrent_kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).ner_model.bilstm.backward_layer.cell.bias
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.

GPU 使用率

感谢分享。训练过程中GPU 使用率一直在30%到60%之间，有没有什么办法可以提高？

将bert微调后的模型进行迁移，迁移后预测就结果就会错误，而在原来的机器上是正常的

train_dataset, val_dataset = data_manager.get_training_set()两个返回TensorSliceDataset无法用len函数

train.py:
train_dataset, val_dataset = data_manager.get_training_set()两个返回TensorSliceDataset无法用len函数
报错TypeError: object of type 'TensorSliceDataset' has no len()
tensorflow版本v2.2.0
我试了几个版本，都不支持len

ValueError: Shapes (27, 27) and (8, 8) are incompatible

请问我自己训练模型的时候报这个错要怎么改呢

要是有测试集测试就完美了

因为现在都是训练集的各项指标看起来不错，但是在测试集上效果怎么样也就是泛化性还不是很清楚，不过已经非常好了，万分感谢！

bert

你好，能不能先下载好bert使用啊？如果能代码里面应该怎么改？怎么下载会比较快？请大佬赐教

ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.

Traceback (most recent call last):
File "main.py", line 70, in
dataManager = DataManager(configs, logger)
File "/root/ner/engines/data.py", line 56, in init
self.tokenizer = BertTokenizer.from_pretrained(huggingface_tag)
File "/root/miniconda3/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1672, in from_pretrained
resolved_vocab_files[file_id] = cached_path(
File "/root/miniconda3/lib/python3.8/site-packages/transformers/file_utils.py", line 1271, in cached_path
output_path = get_from_cache(
File "/root/miniconda3/lib/python3.8/site-packages/transformers/file_utils.py", line 1494, in get_from_cache
raise ValueError(
ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.

how to solve it

没有GPU也可以跑吗

请问没有GPU可以跑吗

如果不用bert，用的是什么做embd呢？

如果不用bert，用的是什么做embd呢？

请问我想用自己数据文件来命名实体，再写入文件，但是总是显失多个loop,并且出不来结果，这该怎么办

bert模型位置

你好请问bert模型自动下载，是下载到哪里了呢，我看项目文件里没有呀

如何训练自己的数据集

你好，请问训练自己的数据集时，checkpoints_dir和checkpoints_name要怎么改呢

acc 全为0

为什么每个实体标签的 acc都为0呢

请问是否可以用tf-serving部署？

RT

替换为自己的数据后，出现错误

用您的数据跑是可以的，但是换成自己的数据出现了这个错误
ValueError: Input [] is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.

tensorflow版本与tensorflow-addons版本冲突

换分词库

我可以不用结巴分词，用自己的分词库吗？

Idcnn+crf 无法训练

3%|▎ | 20/725 [00:03<02:07, 5.53it/s]training batch: 20, loss: 21.37336, precision: 0.000 recall: 0.000 f1: 0.000 accuracy: 0.890
6%|▌ | 40/725 [00:07<01:55, 5.94it/s]training batch: 40, loss: 1131.92053, precision: 0.000 recall: 0.000 f1: 0.000 accuracy: 0.856
8%|▊ | 60/725 [00:11<02:00, 5.52it/s]training batch: 60, loss: 208.67366, precision: 0.000 recall: 0.000 f1: 0.000 accuracy: 0.889
11%|█ | 80/725 [00:15<01:57, 5.47it/s]training batch: 80, loss: 19.80162, precision: 0.000 recall: 0.000 f1: 0.000 accuracy: 0.880
14%|█▍ | 100/725 [00:18<01:43, 6.02it/s]training batch: 100, loss: 15.38830, precision: 0.000 recall: 0.000 f1: 0.000 accuracy: 0.899
17%|█▋ | 120/725 [00:22<01:45, 5.73it/s]training batch: 120, loss: 17.61979, precision: 0.000 recall: 0.000 f1: 0.000 accuracy: 0.875
各项数据都为0 ，数据是本项目提供的，仅仅是改成训练模式。

predict接口 cpu下推理速度比GPU下快

请教一个问题，predict.py代码在cpu下的性能压测竟然比GPU下快，不知道楼主有没有遇到过，谢谢。

将自己的数据集分为四类替换进去并且修改了suffix里面的标签为自己的标签 ，出现如下错误

你好，请问你改成tf.data之后，GPU的利用率以及训练效率有提升么

导入自己标注的文本，显示数据集为0的问题

采用自建的数据集训练时，标签的数量能否大于三个呢

用了自己的标注数据集 结果报错

Traceback (most recent call last):
File "D:/code/entity_extractor_by_ner-master/main.py", line 72, in
train(configs, dataManager, logger)
File "D:\code\entity_extractor_by_ner-master\engines\train.py", line 50, in train
train_dataset, val_dataset = data_manager.get_training_set()
File "D:\code\entity_extractor_by_ner-master\engines\data.py", line 249, in get_training_set
df_train['label_id'] = df_train.label.map(lambda x: -1 if str(x) == str(np.nan) else self.label2id[x])
File "C:\ProgramData\Anaconda3\envs\entity_extractor_by_ner-master\lib\site-packages\pandas\core\series.py", line 3828, in map
new_values = super()._map_values(arg, na_action=na_action)
File "C:\ProgramData\Anaconda3\envs\entity_extractor_by_ner-master\lib\site-packages\pandas\core\base.py", line 1300, in _map_values
new_values = map_f(values, mapper)
File "pandas/_libs/lib.pyx", line 2228, in pandas._libs.lib.map_infer
File "D:\code\entity_extractor_by_ner-master\engines\data.py", line 249, in
df_train['label_id'] = df_train.label.map(lambda x: -1 if str(x) == str(np.nan) else self.label2id[x])
KeyError: 'B-EQ'

EQ是我自己的实体类别 system.config里已经修改suffix为我的实体类别了 但还是报错 是不是还有其他地方需要修改 谢谢

导入自建数据集，出现training set size: 0, validating set size: 0

你好！自己标注的数据集有1528个数据。自动生成的label2id有20个，token2id有450多个。
但是运行会出现如下情况：
1528it [00:00, 1527018.47it/s]
validating set is not exist, built...
training set size: 0, validating set size: 0
导致错误：
tensorflow.python.framework.errors_impl.InvalidArgumentError: buffer_size must be greater than zero. [Op:ShuffleDatasetV3]

Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.

Traceback (most recent call last):
File "main.py", line 70, in
dataManager = DataManager(configs, logger)
File "/root/ner/engines/data.py", line 56, in init
self.tokenizer = BertTokenizer.from_pretrained(huggingface_tag)
File "/root/miniconda3/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1672, in from_pretrained
resolved_vocab_files[file_id] = cached_path(
File "/root/miniconda3/lib/python3.8/site-packages/transformers/file_utils.py", line 1271, in cached_path
output_path = get_from_cache(
File "/root/miniconda3/lib/python3.8/site-packages/transformers/file_utils.py", line 1494, in get_from_cache
raise ValueError(
ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.

how to solve it

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[24,95] = -7 is not in [0, 100)出现这个问题应该怎么办

当我训练完bilstm-crf和idcnn-crf模型的时候，开始训练bert-bilstm-crf模型，老是报错

自己准备好训练集之后出现这种情况，请问怎么解决

Traceback (most recent call last):
File "G:/BERT+BiLSTM/entity_extractor_by_ner-master/main.py", line 72, in
train(configs, dataManager, logger)
File "G:\BERT+BiLSTM\entity_extractor_by_ner-master\engines\train.py", line 81, in train
inputs=model_inputs, inputs_length=inputs_length, targets=y_train_batch, training=1)
File "H:\解释器\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 985, in call
outputs = call_fn(inputs, *args, **kwargs)
File "H:\解释器\lib\site-packages\tensorflow\python\eager\def_function.py", line 780, in call
result = self._call(*args, **kwds)
File "H:\解释器\lib\site-packages\tensorflow\python\eager\def_function.py", line 807, in _call
return self._stateless_fn(*args, **kwds) # pylint: disable=not-callable
File "H:\解释器\lib\site-packages\tensorflow\python\eager\function.py", line 2829, in call
return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
File "H:\解释器\lib\site-packages\tensorflow\python\eager\function.py", line 1848, in _filtered_call
cancellation_manager=cancellation_manager)
File "H:\解释器\lib\site-packages\tensorflow\python\eager\function.py", line 1933, in _call_flat
cancellation_manager=cancellation_manager)
File "H:\解释器\lib\site-packages\tensorflow\python\eager\function.py", line 550, in call
ctx=ctx)
File "H:\解释器\lib\site-packages\tensorflow\python\eager\execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0,22] = -16 is not in [0, 256)
[[{{node cond_5/else/_1/cond/GatherV2_1}}]] [Op:__forward_call_7380]

Function call stack:
call

Process finished with exit code 1

代码问题

您好！我改成mode=train训练模式之后，在训练过程中就报错

无法下载模型，怎么操作呢？

由于网络无法直接下载模型。
手动下载后报错：OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5'] found in directory chinese or from_pt set to False

代码有点问题

data.py的prepare方法，我输出的X矩阵和y向量都为空

关于数据集标注

你好，我想尝试用这个模型来训练自己的训练集，但是我的领域里暂时没有大量的语料库或者词典，所以要自己打标签和BIO标注。请问哪种工具可以把数据集的标签打得和模型中的tokenid、vocabid、dev.csv和train.csv的格式一致？还是需要手动打tag？我的语料都是英文的。

idcnn训练时出现准确率和召回率全0

如图，尝试训练IDCNN-CRF模型时，出现准确和召回率全0且训练无改善，数个Epoch后loss变为NaN的情况，应该怎么解决？

文章出现\t 和空格时会报错

文章出现\t 和空格时会报错

token, token_id = row.split('\t')[0], int(row.split('\t')[1])
IndexError: list index out of range