GithubHelp home page GithubHelp logo

doc_event_role's Issues

有关k-sentence reader的问题

原文还没有看完，刚看到数据构造这里。
k-sentence reader是对于一个有N条句子的篇章，构造{seq[1], ..., seq[N], seq[1] + seq[2], ..., seq[N -1] + seq[N, ...}这样的训练样本吗？这样的话样本数量会非常大，是n^2级别的，解码的时候不会特别慢吗？

Reproducing score

Hi @xinyadu , thanks a lot for sharing the code. I am trying to reproduce the scores from Table 1 of the paper. I am grateful for your documentation and followed the steps outlined in this repo as closely as possible and am able to run the experiment all the way through. But the score I get at the end of the experiment does not match the scores reported in Table 1.

The steps I took were:

1. Environment setup/download spacy model/download

$ git clone https://github.com/xinyadu/doc_event_role.git
$ cd doc_event_role
$ touch environment.yml  # add the requirements in this file
$ cat environment.yml 
name: muc_seq_acl
channels:
  - defaults
dependencies:
  - python==3.5.6
  - pip==20.1.1
  - spacy==2.0.12
  - cudatoolkit==9.2
  - pip:
    - torch==0.4.1
    - pytorch-pretrained-bert==0.6.2
    - typing==3.7.4.3
$ conda env create -f environment.yml
$ conda activate muc_seq_acl

2. Download spacy

$ python3 -m spacy download en_core_web_sm

3. Download Glove

Download glove.6B.100d.txt in doc_event_role/model/code/utils/glove.6B.100d.txt

4. Train

$ cd model/code
$ mkdir data_seq_tag_pairs
$ python gen_seq_tag_pairs.py --div train
$ mkdir model_save
$ mkdir model_out
$ python main.py --config config/example.config
$ python seq_to_extracts.py --seqfile model_out/multi_bert.out  # Generate preds.json

The Dev scores I see during the training are:

Dev: time: 193.15s, speed: 15.20st/s; p: 28.1324, r: 70.0204, f: 40.1383
Dev: time: 124.92s, speed: 15.35st/s; p: 62.9229, r: 39.4083, f: 48.4639
Dev: time: 124.47s, speed: 15.36st/s; p: 66.1268, r: 43.9025, f: 52.7702
Dev: time: 133.18s, speed: 15.39st/s; p: 52.8802, r: 66.1970, f: 58.7939
Dev: time: 126.86s, speed: 15.37st/s; p: 63.4102, r: 48.3327, f: 54.8542
Dev: time: 124.93s, speed: 15.33st/s; p: 65.0108, r: 44.1431, f: 52.5823
Dev: time: 123.39s, speed: 15.36st/s; p: 74.8237, r: 37.4043, f: 49.8758
Dev: time: 125.21s, speed: 15.34st/s; p: 72.0788, r: 46.6892, f: 56.6702
Dev: time: 131.71s, speed: 15.31st/s; p: 57.4742, r: 69.6806, f: 62.9915
Dev: time: 129.72s, speed: 15.36st/s; p: 60.7873, r: 62.0204, f: 61.3977
Dev: time: 136.96s, speed: 15.33st/s; p: 47.4962, r: 73.2321, f: 57.6211
Dev: time: 128.23s, speed: 15.33st/s; p: 63.9869, r: 58.7949, f: 61.2811
Dev: time: 131.07s, speed: 15.36st/s; p: 58.6679, r: 62.1833, f: 60.3745
Dev: time: 126.07s, speed: 15.35st/s; p: 65.7156, r: 50.6334, f: 57.1970
Dev: time: 123.92s, speed: 15.33st/s; p: 72.3295, r: 41.9134, f: 53.0724

5. Test scores

$ python eval.py --goldfile ./data/processed/test.json --predfile ./model/code/pred.json

================Exact Match=================
Prec, Recall, F-1
PerpInd
55.1020 39.8649 46.2611
PerpOrg
40.9091 57.1429 47.6821
Target
45.3744 63.4483 52.9105
Victim
46.5753 71.5789 56.4315
Weapon
44.8980 72.1311 55.3459
MACRO average:
46.5718 60.8332 52.7557
===============Head Noun Match===============
Prec, Recall, F-1
PerpInd
59.5745 42.5676 49.6552
PerpOrg
47.7612 65.4762 55.2330
Target
60.2804 78.6207 68.2397
Victim
49.6503 72.6316 58.9815
Weapon
48.9362 75.4098 59.3548
MACRO average:
53.2405 66.9412 59.309

What I am not sure about is, does this score correspond to the Multi-Granularity Reader score from Table 1 of the paper or does this correspond to some other row? Was the config file used to run the Multi-Granularity Reader experiment different from example.config?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.