rokid / elmo-chinese Goto Github PK

我用自己的数据训练了一下这个模型，但是我不太知道如何可以反应它的效果

TODO

76, in <module> main(args) File "train_elmo.py", line 66, in main train(options, data, n_gpus, tf_save_dir, tf_log_dir) File "/data/sde/jiaxin_hu/git_project/ELMo-chinese/bilm/training.py", line 766, in train allow_soft_placement=True)) as sess: File "/data/sde/jiaxin_hu/git_project/ELMo-chinese/bin/testenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1494, in __init__ super(Session, self).__init__(target, graph, config=config) File "/data/sde/jiaxin_hu/git_project/ELMo-chinese/bin/testenv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 626, in __init__ self._session = tf_session.TF_NewSession(self._graph._c_graph, opts) tensorflow.python.framework.errors_impl.InternalError: Failed to create session.

请问用您传的中文数据集跑elmo时出现这个错误

Traceback (most recent call last):
  File "train_elmo.py", line 72, in <module>
    main(args)
  File "train_elmo.py", line 62, in main
    train(options, data, n_gpus, tf_save_dir, tf_log_dir)
  File "/mnt/disk2/data/wp/Word2vec/model/ELMo-chinese/bin/bilm/training.py", line 838, in train
    for batch_no, batch in enumerate(data_gen, start=1):
  File "/mnt/disk2/data/wp/Word2vec/model/ELMo-chinese/bin/bilm/data.py", line 469, in iter_batches
    num_steps, max_word_length)
  File "/mnt/disk2/data/wp/Word2vec/model/ELMo-chinese/bin/bilm/data.py", line 311, in _get_batch
    :how_many]
ValueError: could not broadcast input array from shape (18,50) into shape (19,50)

最后发现是 bilm/data.py 下的 get_bacth 函数的:

  inputs[i, cur_pos:next_pos] = cur_stream[i][0][:how_many]
                if max_word_length is not None:
                    char_inputs[i, cur_pos:next_pos] = cur_stream[i][1][:
                                                                        how_many]

这一段报错。
由于没看懂 get_batch 的逻辑，自己不会改，请问能指点一下吗，谢谢

您好！
请问 options.json 里面的 261 改成 262 是什么原因？我看第五步 输出word embedding到hdf5文件 usage_token.py 好像没有用到这个 options.json 文件
多谢了！

UnicodeCharsVocabulary类中_convert_word_to_char_ids函数您注释是chinese也可以成为255个ids，但汉字字符应该明显多于255个呀，所以用utf解码变成ids是不是不合适？

ELMO不是上下文相关吗