uclnlp / jack Goto Github PK
View Code? Open in Web Editor NEWJack the Reader
License: MIT License
Jack the Reader
License: MIT License
Read in data in format #1 as gold data and some format of predictions (question: how is system output stored?), and calculate accuracy/ranking metrics.
Features:
You all know you want them https://github.com/uclmr/tfrnn/blob/master/tfrnn/hooks.py
On the same note, would be great to connect to TensorFlow summary writes to monitor progress on TensorBoard.
As part of #2. Maybe something like
"question": "dataset_for(parsing, ?)",
"support": [ "list of paragraphs created by a heuristic" ]
"answer": ["list of answers from wikipedia"]
We should also generate cloze questions from paragraphs that mention the same entity
"question": "we used the ? corpus for our parsing model",
"support": [ "list of paragraphs created by a heuristic involving parsing, but removing the sentence that is the basis for the cloze question" ]
"answer": ["answer as taken from the cloze question sentence"]
We can also create cloze questions from the highlights.
Somehow pip doesn't seem to find a tensorflow version for the wercker VM.
Models from DeepMind "Teaching Machines to Read and Comprehend" paper
http://arxiv.org/abs/1506.03340
https://github.com/thomasmesnard/DeepMind-Teaching-Machines-to-Read-and-Comprehend
Create a multihop QA dataset with NL supporting facts, exploiting the link structure from wikipedia.
So far I have a few todos
I believe there is a relatively task independent model that relies on the following assumptions:
The model may only have a few parameters that capture spatial consistence, weigh the word embedding dimensions etc. Further, one could build in a simple latent "coreference" mechanism.
Implement loaders that save into the format of #1. To do:
(Add as you see fit).
Whoever is assigned (or assigns themselves) to one of these should create a new issue specific to that format.
Loader for SNLI data into quebab format, with different options for support
If we do code on this more collaboratively we should follow a consistent style guide such as PEP8. Easy to install https://pypi.python.org/pypi/pep8 and run it on the code. PyCharm also complains and automatically fixes certain style issues. Maybe get a formatter for your own editor if it isn't PyCharm.
As Sebastian suggested, would be great to have some tiny toy versions of the datasets for debugging and validation.
Load format in #1 into a python data-structure. Could just be a dictionary and simple json parsing for now.
Load FB-15k and FB15K-237 into our format in #1. This requires the definition of a support neighbourhood for a given entity (and question). For example, we can use all entities that are direct neighbours:
"question": "born_in(BarackObama, ? )",
"support": [
"BarackObama was born in Hawaii",
"president_of(BarackObama, USA)"
]
"candidates": { "filename": "filename for file with list of candidates" }
Part of #2.
In future we could also support a more "global" (Johannes) way of viewing support by providing as support the list of all facts, for example by a filename pointing to the facts.
"support" : {"filename":"filename of file with all training/test facts"}
Options for conversion:
For answers:
Where should this be? cannon?
For short-answer data sets like squad, the quebap way to approach this problem would be to generate candidate answer spans. This can be done offline and allows multiple-choice style models to function on these datasets without modification.
There's a few ways to do this. One is checked into preprocess/suggest_span_candidates.py, using pos-tag sequences of the training answers. However, coverage is not great -- using the top 300 pos-sequences, we get about 70% coverage, and 14% or so answers have pos-tag squences unique to that instance.
There are other ways to do this which might be complimentary to this approach. One would be to use the parse trees to make candidate suggestions.
And one disadvantage with the postag approach is that the sequence ['DT'] somehow sneaks in, and then now all determiners are possible answers. In some cases it would make sense to take a look at the word, and if it's say, 'the', not suggest it.
Would be great to have a simple web user interface against our models.
will require a API for the models (and an integrated IR step).
It would be good to have names for the individual components of a quebap file (and not call instances 'quebaps') . Here is a proposal (sort of obvious but it's good to have it official somewhere):
Error analysis to use for the analysis and discussion part of papers
Leave this for some time in future, e.g.
Add training loop from tfrnn
I created a lightweight reading architecture and script here:
https://github.com/uclmr/quebap/blob/master/quebap/model/reader.py#L430
You can get help for the script like so
python3 quebap/model/reader.py -h
and running it would look like
python3 quebap/model/reader.py --train quebap/data/NYT/naacl2013_train.quebap.json --batch_size 100 --epochs 10 --train_end 1 --model boe
The main "architecture" choice is to wrap all model dependent state, conversion between quebap instances and tensorflow, and batching, into the Batcher
class. Everything on top of that is stateless construction of TF nodes based on input TF nodes. So composing a reader will look like this:
https://github.com/uclmr/quebap/blob/master/quebap/model/reader.py#L334
Currently the reader supports two models:
Very lightly tested. Training loop, evaluation etc work, but are still very much just a placeholder. Biggest missing parts:
convert_backward(convert_forward(data))
.To push for model reuse and sharing I hope that we could try to get all our models into this framework where possible (it makes very few assumptions beyond multiple choice so this should be doable), and extend/improve as we go.
Instead of / in addition to having it as part of the batcher
@narad @JohannesMaxWel Is there a schema or example for the output format? Would the output format be an object at the top level with instances
, as for the input format, or just the instance list?
paper from Felix Hill et al.
cloze-style QA
As a safety check for these format readers, probably makes sense to release a proper JSON schema, and also to have a script to check that everything we're producing with these readers adheres to it. I have a small script that can do the verification, that I wrote in eqa-tools, so simple enough to bring that over.
But I actually don't know how to specify the schema file when there are lots of nesting. I actually wound up using an online tool to reverse-engineer the schema file from the JSON for eqa-tools!
Prepare the MovieQA dataset for our shared dataset format.
It's from this paper here.
Note: One of the main takeaways of our reading group on this paper was that the video segments don't actually contribute a lot to model performance. It's probably best to skip the videos and only focus on the text.
It would be great to have some code for dumping out data statistics to help us brainstorm strategies and compare datasets at a glance.
Some natural things if your answer is a span:
what is the length distribution for the span,
what are the postags like,
how frequently do they align to constituents,
what constituents do they align to
Please add other useful statistics here.
See #2.
S
We are using pytest for now. CI finds test__.py files in the source tree and executes test__ functions in those (standard configuration). We have a simple test in quebap/projects/modelF/test_evaluation.py
.
Decision to be made:
test
source tree.I like the simplicity of 1 and don't like to create too many directories, and synchronise directory structures. So I am voting for 1...
Readers should use a global root directory variable to search for data. On local machines this could be set to ./quebap/data
(instead of ./quebap/quebap/data
). On cannon/emerald it should be set to /cluster/project2/mr/data/
.
Tried just adding "tensorflow" as done with numpy, didn't work on wercker. Probably needs a version.
Ideally should work a little on several datasets, reading in format #1. Several versions:
Quite similar (identical) to the MCTest baseline.
Check if json contains "candidates": "#/globals/candidates"
If so always use global candidates
Else use local candidates
Goal: write simple version of model F for NYT data
(simplest possible proof-of-concept for MF models)
At least for tokenization and pos-tagging.
Something like
{
"question": "Where was Barack Obama born",
"support": [
"Barack Obama was born in Hawaii",
"Barack Obama is the president of the USA"
],
"candidates": [
"Hawaii",
"USA",
"whatever"
],
"answer": "Hawaii"
}
Comments:
support
is a list of support documents. For now these can be strings, but we should also support a structured version with additional offset annotation. In many settings we have several question on the same support documents, in which case this will create a lot of duplication. We should therefore also allow support documents to be filenames.candidates
is a list of candidate answers. Again, to avoid duplication I can imagine this also to be a filename of a list of entities (for FB15k where usually the candidate entities in each question are identical).For some datasets we may have provenance information. That is, for the correct answer we know where the answer is mentioned (in the form of an answer span). We can include this information in the answer representation:
"answer": { "value": "Hawaii", "doc_id": 0, "span": [26, 32] }
Personally I would also like to produce an alignment between the question and the context around the answer (in order to score it), but I don't think this needs to be in this format now.
See https://tticnlp.github.io/who_did_what/
They report results for lots of baselines already
Loading word vectors, etc
implement the memNN model paper
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.