Build tables of information by extracting facts from indexed text corpora via a simple and effective query language.
Home Page: http://allenai.org/software/interactive-knowledge-extraction/
License: Apache License 2.0
Mon Mar 16 23:58:17 2015
Originally opened as https://github.com/allenai/okcorpus/issues/34
Per Oren -- "This has proved critical in previous systems like ReVerb."
Wed Mar 18 23:58:30 2015
Originally opened as https://github.com/allenai/okcorpus/issues/44
OKC should save your query history, just like bash saves your command history. Query logs should be persisted across browser sessions. You should be able to view the query log and scroll through it, then click on one to re-execute it. When named patterns are available, there should be a link to create a new named pattern from an entry in the query log.
Thu Mar 19 00:33:07 2015
Originally opened as https://github.com/allenai/okcorpus/pull/46
@afader, you already have all the context for this.
dirkgr included the following code: https://github.com/allenai/okcorpus/pull/46/commits
Mon Feb 16 20:36:29 2015
Originally opened as https://github.com/allenai/okcorpus/pull/2
Example code!
afader included the following code: https://github.com/allenai/okcorpus/pull/2/commits
Wed Feb 18 17:07:41 2015
Originally opened as https://github.com/allenai/okcorpus/issues/5
I realize this isn't what you want to target immediately, but it's worth considering now lest it become more difficult in the future.
Presently you always show the first capture group--and from that capture group you build a dictionary. What if the "dictionary" could have multiple columns? For example, someone might search for (JJ* NN+) is a part of (JJ* NN+) and get rows such as (mining, processing) and (liquidity, growth). This might be sufficient for the tool to be applied to create tables.
Thu Mar 5 23:01:28 2015
Originally opened as https://github.com/allenai/okcorpus/issues/13
Error: 8
a.k.a. Error: 4
Also: capture group NullPointerException
Wed Feb 18 19:09:50 2015
Originally opened as https://github.com/allenai/okcorpus/issues/8
OpenRegex supports pattern hierarchies which allows people to write complex but understandable patterns. I understand that BlackLab doesn't support adding new types to its index without reindexing, but hierarchical patterns could be compiled down into base patterns. It's an open question how fast such complex queries would be.
For example, it would be nice to run the following ReVerb extractor in BlackLab.
https://github.com/allenai/taggers/blob/master/examples/chunkedextractor/reverb-parts.taggers
https://github.com/allenai/taggers/blob/master/examples/chunkedextractor/reverb-extr.taggers
Wed Feb 18 16:30:20 2015
Originally opened as https://github.com/allenai/okcorpus/issues/3
@rodneykinney used the pipeline library on https://github.com/allenai/auto-openie so that it optionally processed the raw input and built and index. If this was already complete, it skipped to using the processed index.
It might be a good fit for the dictionary-builder. Or it might be overly complicated.
Wed Mar 18 23:03:14 2015
Originally opened as https://github.com/allenai/okcorpus/issues/41
JJ parsing or some other query that returns multiple pages of hitsinformation extraction or some other query that has one page of hits
Wed Mar 18 16:42:32 2015
Originally opened as https://github.com/allenai/okcorpus/issues/40
Right now, we are using Brown clusters for a notion of word similarity in OKC. The benefit of this is that the clusters are discrete and prefix-based, so we can easily add them as token annotations in BlackLab. The disadvantages are:
I was playing around with word2vec, and the results on the ACL corpus are great. For example, here are the phrases most similar to information_extraction:
Word Cosine distance
------------------------------------------------------------------------
text_mining 0.826766
question_answering 0.796270
knowledge_extraction 0.772408
ontology_learning 0.764749
automatic_summarization 0.752856
automatic_text_summarization 0.748179
ie 0.742152
entity_extraction 0.737166
knowledge_discovery 0.737004
fact_extraction 0.735164
image_retrieval 0.731172
information_retrieval 0.725849
biomedical_information_extraction 0.723953
information_fusion 0.721746
applications_such_as_information_extraction 0.721209
literature_mining 0.716144
relation_extraction 0.713560
text_summarization 0.710251
named_entity_recognition 0.708391
record_linkage 0.705008
text_understanding 0.704244
such_as_information_retrieval 0.698317
such_as_information_extraction 0.698231
ontology_population 0.697375
information_extraction_systems 0.694957
many_natural_language_processing_tasks 0.686736
opinion_mining 0.684321
open_domain 0.683641
entity_recognition 0.682989
cross_-_language_retrieval 0.681546
question_answering_systems 0.681245
document_classification 0.680587
computer_vision 0.679659
named_entity_extraction 0.679576
textual_entailment_recognition 0.677568
named_-_entity_recognition 0.675705
natural_language_processing 0.675098
data_mining 0.674606
applications_such_as_question_answering 0.674527
opinion_extraction 0.673862
Here are the most similar phrases to outperforms:
Word Cosine distance
------------------------------------------------------------------------
performs_better_than 0.914765
significantly_outperforms 0.894385
outperformed 0.879147
improves_over 0.859142
performs_much_better_than 0.842823
performs_worse_than 0.832298
consistently_outperforms 0.831501
outperforms_both 0.828320
clearly_outperforms 0.821622
performs_significantly_better_than 0.817346
is_superior_to 0.816786
beats 0.806530
performed_better_than 0.805698
performs_slightly_better_than 0.804796
outperforming 0.800169
performs_as_well_as 0.792212
still_outperforms 0.788039
outperforms_all 0.785506
does_better_than 0.784221
also_outperforms 0.782234
achieves_better_performance_than 0.774746
surpasses 0.770587
significantly_outperformed 0.770434
works_better_than 0.765581
can_outperform 0.762990
model_outperforms 0.762042
slightly_outperforms 0.759185
does_not_outperform 0.758555
our_model_outperforms 0.757645
substantially_outperforms 0.752425
underperforms 0.751935
method_outperforms 0.749556
always_outperforms 0.740629
performs_best 0.736030
performs_comparably_to 0.735505
performed_worse_than 0.734069
approach_outperforms 0.733126
is_consistently_better_than 0.729947
models_outperform 0.723421
even_outperforms 0.722097
So, word2vec is:
I think that using this information could have a big impact on usability, since you can stop having to think in terms of single words, but still get a good notion of similarity.
The downside is that multiword phrases cannot be indexed like Brown clusters.
To add word2vec similarity into OKC, we can do the following:
similarPhrases(phrase: Seq[String], threshold: Double): Seq[Seq[String]] that returns the phrases that have similarity within the given thresholdsimilarPhrases are combined into a disjunction, which can then be queried against the index
Thu Mar 5 19:20:10 2015
Originally opened as https://github.com/allenai/okcorpus/pull/11
It's my first foray into react!
The image is half-size, which looks a little lumpy on normal displays, but super sweet on retina displays.
This is kind of a joke, so I don't really expect to merge this, but I'd still appreciate @codeviking's comments if I messed anything up.
@afader, FYI
dirkgr included the following code: https://github.com/allenai/okcorpus/pull/11/commits
Tue Mar 3 22:35:48 2015
Originally opened as https://github.com/allenai/okcorpus/pull/10
I'm getting a build error when I try to run this. This fixes it.
@afader, do you want to review pull requests of this magnitude?
dirkgr included the following code: https://github.com/allenai/okcorpus/pull/10/commits
Mon Mar 9 20:20:08 2015
Originally opened as https://github.com/allenai/okcorpus/pull/20
Started moving the code from dictionaries to tables. Instead of doing this in one giant leap, I'm going to break it into small parts.
This first part does two things:
Dictionary class with a Table class.Table objects, but keeps the same functionality as before (i.e. only works with 1-column tables).Will comment more below...
afader included the following code: https://github.com/allenai/okcorpus/pull/20/commits
Wed Mar 18 00:13:57 2015
Originally opened as https://github.com/allenai/okcorpus/issues/39
Doing a search for we compare .* NNP times out, and then uses lots of CPU, even after the request has been canceled.
OKC should die gracefully
* and +
Fri Mar 13 22:17:22 2015
Originally opened as https://github.com/allenai/okcorpus/issues/23
Wed Mar 18 23:52:17 2015
Originally opened as https://github.com/allenai/okcorpus/issues/43
When you click the "add" button, okc should record:
In the Tables pane, there should be a little info button that you can click on to view a particular row's provenance.
Also, when exporting the table, you should also get the provenance info with it.
Wed Feb 18 16:58:36 2015
Originally opened as https://github.com/allenai/okcorpus/issues/4
It'd be neat if there were a feature that would generalize POS tags. For example, if you wrote a pattern person VBG you could select a dropdown that would offer to generalize VBG to /VB./ (a regular expression matching any verb).
Tue Mar 17 18:42:19 2015
Originally opened as https://github.com/allenai/okcorpus/issues/35
We want to have an instance of okcorpus running on some EC2 machine that answers to http://okcorpus.dev.ai2. This is useful for doing quick demos.
Right now the deployment is a little clunky:
AWS_* env variablescurl 'http://10.12.0.4/ddns/?host=okcorpus' in order to set the domain namesbt deploy need to pass the deploy.user.ssh_keyfile argument
Fri Mar 13 23:22:47 2015
Originally opened as https://github.com/allenai/okcorpus/issues/28
Thu Mar 5 22:57:49 2015
Originally opened as https://github.com/allenai/okcorpus/issues/12
look in lib/ for the ugliness
Fri Mar 6 23:02:54 2015
Originally opened as https://github.com/allenai/okcorpus/issues/16
Suppose you have a pattern like languages such as (NNP) and (NNP) that matches the text languages such as English and Spanish. You often want to return English and Spanish as two separate rows, instead of as a single row with two columns.
Mon Mar 16 16:59:58 2015
Originally opened as https://github.com/allenai/okcorpus/pull/29
Here's my work from last week, wanna review in person @chrisc36?
afader included the following code: https://github.com/allenai/okcorpus/pull/29/commits
Fri Mar 6 23:00:37 2015
Originally opened as https://github.com/allenai/okcorpus/issues/15
Port our familiar friends from other regular expression languages:
E? - match E zero or one timeE{m,n} - match E at least m times and no more than n timesE{m} - match E exactly m times
Mon Mar 16 22:30:19 2015
Originally opened as https://github.com/allenai/okcorpus/pull/32
Adds the functionality from #26
localStorage takes place in TableManagerTableManager serializes and saves all of the tables on every change. This is inefficient, but I think it should be fine for now.DictApp component makes a call to TableManager when loading its initial state.afader included the following code: https://github.com/allenai/okcorpus/pull/32/commits
Fri Mar 6 23:30:12 2015
Originally opened as https://github.com/allenai/okcorpus/issues/19
or maybe "review later"
Fri Feb 13 23:16:45 2015
Originally opened as https://github.com/allenai/okcorpus/pull/1
merging code for ingesting new text
afader included the following code: https://github.com/allenai/okcorpus/pull/1/commits
Fri Mar 13 00:16:19 2015
Originally opened as https://github.com/allenai/okcorpus/pull/21
@afader, this adds a simply download button to the dictionaries.
@codeviking, do you have any idea how to make this thing work in Safari? I tried a bunch of stuff, most promisingly FileSaver.js, but it fails on both Chrome and Safari for two different reasons. I'd love to go over it with you. I'm sure there is a way to make it work, because the FileSaver.js demo works just fine on Safari.
dirkgr included the following code: https://github.com/allenai/okcorpus/pull/21/commits
Wed Feb 18 17:22:13 2015
Originally opened as https://github.com/allenai/okcorpus/issues/6
Presently there's no way to inspect the POS tags beneath tokens. A popover with details about the sentence or tokens would help.
Fri Mar 13 22:18:53 2015
Originally opened as https://github.com/allenai/okcorpus/issues/24
Dirk: balsalmic
Thu Mar 19 21:30:19 2015
Originally opened as https://github.com/allenai/okcorpus/issues/49
Thu Mar 19 00:36:15 2015
Originally opened as https://github.com/allenai/okcorpus/issues/47
In the readme, in filenames, etc.
Fri Mar 6 23:29:28 2015
Originally opened as https://github.com/allenai/okcorpus/issues/18
Wed Feb 18 18:43:57 2015
Originally opened as https://github.com/allenai/okcorpus/pull/7
schmmd included the following code: https://github.com/allenai/okcorpus/pull/7/commits
Mon Mar 16 17:21:29 2015
Originally opened as https://github.com/allenai/okcorpus/issues/30
When this is resolved, the corresponding workaround in the code should be removed.
Fri Mar 13 22:14:55 2015
Originally opened as https://github.com/allenai/okcorpus/issues/22
We need to merge in Chris' ML changes and make sure they work with tables.
Fri Mar 13 22:21:42 2015
Originally opened as https://github.com/allenai/okcorpus/issues/25
We need a button to export and import tables to CSV files.
Fri Mar 20 00:38:54 2015
Originally opened as https://github.com/allenai/okcorpus/pull/50
Fixes #28.
@afader? Lots of code reviews when you get back :-)
dirkgr included the following code: https://github.com/allenai/okcorpus/pull/50/commits
Tue Mar 17 21:29:50 2015
Originally opened as https://github.com/allenai/okcorpus/issues/38
Here's a weird UI bug
Starting from no tables, execute a search:
Then add a table:
Then go back to the search:
You get a goofed up results table. Once the target changes, the query needs to be re-executed, I think.
Thu Mar 19 00:07:11 2015
Originally opened as https://github.com/allenai/okcorpus/issues/45
We should be able to write queries that combine the outputs of multiple patterns. For example intersect($pattern1, $pattern2) should return the intersection of the captured groups of both patterns.
This will require:
intersect($x, $y), potentially making multiple calls to BlackLabShort list of operators:
Wed Feb 18 23:19:31 2015
Originally opened as https://github.com/allenai/okcorpus/issues/9
Presently the tool is focused around using a single corpus. We will ultimately want to support multiple corpora at once. We'll need a tag for the corpus name, which can become an additional column in the search results.
Fri Mar 6 22:59:35 2015
Originally opened as https://github.com/allenai/okcorpus/issues/14
Wed Mar 18 23:49:05 2015
Originally opened as https://github.com/allenai/okcorpus/issues/42
Mon Mar 16 17:23:12 2015
Originally opened as https://github.com/allenai/okcorpus/issues/31
Workaround code should be removed when this is fixed.
Tue Mar 17 18:45:09 2015
Originally opened as https://github.com/allenai/okcorpus/pull/36
This is just a little plumbing to make deploying this code a little easier. It needs to be cleaned up (see #35). But now we have an OKC demo running at http://okcorpus.dev.ai2/
afader included the following code: https://github.com/allenai/okcorpus/pull/36/commits
Thu Mar 19 14:08:47 2015
Originally opened as https://github.com/allenai/okcorpus/issues/48
Right now there is no way to search for periods, commas, brackets, etc. We should have ways to quote or escape these.
Mon Mar 16 22:48:53 2015
Originally opened as https://github.com/allenai/okcorpus/pull/33
Adds new QExpr type QAnd, new QExpr supertypes QAtom and QCapture to make some pattern matching cases easier, and adds getCaptureGroups, getQueryLength, getQueryString to QueryLanguage.
chrisc36 included the following code: https://github.com/allenai/okcorpus/pull/33/commits
Fri Mar 13 22:22:26 2015
Originally opened as https://github.com/allenai/okcorpus/issues/26
When you close the browser, you don't want to lose all your history.
@jkinkead what was your suggestion here? I thought you mentioned something on our walk to lunch.
Fri Mar 13 22:23:39 2015
Originally opened as https://github.com/allenai/okcorpus/issues/27
Fri Mar 6 23:29:00 2015
Originally opened as https://github.com/allenai/okcorpus/issues/17
Tue Mar 17 20:56:12 2015
Originally opened as https://github.com/allenai/okcorpus/pull/37
afader included the following code: https://github.com/allenai/okcorpus/pull/37/commits
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.