samyadaleh / cl-toolbox Goto Github PK

A Parsing-as-Deduction system that parses with different parsing algorithms and some related tools for educational purposes.

Java 99.94% Batchfile 0.05% Shell 0.01%

parsing deductions earley cfg java cyk-parser earley-parser earley-algorithm topdown computational-linguistics

cl-toolbox's Introduction

CL-Toolbox

Overview

A Parsing-as-Deduction system implemented in Java that parses everything and some related tools for educational purposes.

Check the documentation in /doc/doc.md for the list of currently supported algorithms and formalisms and all implementation details.

It prints out full parsing traces along with applied rules and antecedence items. Use it to generate examples, play around with grammars etc.

Without Installation

If you just want to get parsing traces of the most common algorithms, please visit http://cl-taskbox.de/. You can insert grammar and input for all algorithms listed there, afterwards click Help > Solve Task to get the full trace.

Installation

If you want to tinker with the code, for me it worked like this:

use your favorite IDE to clone/checkout the project
add the dependencies to your classpath/build path, preferable with the build.gradle file
run the JUnit tests in src.test to verify that everything is working. You should see some success messages and parsing outputs.

Logged-in GitHub users can download the latest jar from the Action tab.

Dependencies

See build.gradle.

Usage

Run it in your IDE.

If you downloaded or produced a jar, call with java -Dfile.encoding="UTF-8" -jar ... to correctly display special characters. Or call CL-Toolbox.bat (Windows) or CL-Toolbox.sh (Linux). They work with the same parameters, but you don't have to type in the encoding.

Video series about background and demonstration : English German

Examples:

java -jar CL-Toolbox.jar anbn.cfg "a a b b" cfg-topdown

true

1	[S,0]	axiom	{}
2	[a b,0]	predict S -> a b	{1}
3	[a S b,0]	predict S -> a S b	{1}
4	[b,1]	scan	{2}
5	[S b,1]	scan	{3}
6	[a b b,1]	predict S -> a b	{5}
7	[a S b b,1]	predict S -> a S b	{5}
8	[b b,2]	scan	{6}
9	[S b b,2]	scan	{7}
10	[b,3]	scan	{8}
11	[a b b b,2]	predict S -> a b	{9}
12	[a S b b b,2]	predict S -> a S b	{9}
13	[ε,4]	scan	{10}

Where anbn.cfg is:

N = {S}
T = {a, b}
S = S
P = {S -> a S b, S -> a b}

CL-Toolbox.bat ancb.tag "a c b" tag-cyk --success

true

1	[β,.1⊤,0,-,-,1]	lex-scan a	{}
3	[α2,.1⊤,1,-,-,2]	lex-scan c	{}
7	[β,.2⊤,1,1,2,2]	foot-predict	{}
12	[α1,.2⊤,2,-,-,3]	lex-scan b	{}
15	[β,ε⊥,0,1,2,2]	move-binary	{1, 7}
16	[α2,ε⊥,1,-,-,2]	move-unary	{3}
19	[β,ε⊤,0,1,2,2]	null-adjoin	{15}
21	[α2,ε⊤,0,-,-,2]	adjoin α2[ε,β]	{16, 19}
23	[α1,.1⊤,0,-,-,2]	substitute α1[.1,α2]	{21}
25	[α1,ε⊥,0,-,-,3]	move-binary	{12, 23}
27	[α1,ε⊤,0,-,-,3]	null-adjoin	{25}

Where ancb.tag is:

N = {S, T}
T = {a, b, c}
S = S
I = {α1 : (S T b), α2 : (T c)}
A = {β : (T a T*)}

Features

prints full traces or only successful traces
automatically transforms grammars into more expressive formalisms
can convert grammars to fit the algorithm
displays derivated trees for all algorithms
for TAG displays item trees on mouseover

cl-toolbox's People

Contributors

Stargazers

Watchers

Forkers

vtqveant yoginya drjoeavg kouklinaa

cl-toolbox's Issues

Add Astar Parsing for CFG

Make sure that string represenations of items are not cut off if entries are empty

Update Tree class to represent crossing edges

Add terminals/leaves separately, in bracket format allow pointers to the leaves.
Tree parser should be able to handle both formats.
Tag/ grammar parser needs check. No crossing edges allowed for Tag. (Or let's try once and see what happens. :))

Add CYK parsing for C2F

It should be a simple extention of CYK for CNF.
Maybe just convert a grammar to C2F in this stage if general CYK parsing is not available yet.

Display Trees

from bracket format or from Tree-class (I can convert them into each other) or from deduction items - from Tree class is best
with dot: top, bottom, la, ra, lb, rb
with crossing edges (maybe needs other format for that or pointers to words in sentence, see how TiGer does that)
with indices at the or between the terminals to display items

For the returned chart allow to pass it to each item kind that interprets them and create the tree or create a new chart interpreter

CFG remove epsilon productions

Make LeftCorner for CFG faster

It is so slow it is not even useable with a toy grammar.

Extend recognizers to real parsers

Store the pointers of antecedence items.
One item may can be derived in different ways. Instead of collecting sets of pointers, collect sets or lists of sets of pointers (or lists of lists, whatever).

Parse MCFG

Make that Deduction beside true or false also/optionally can return the chart

Add CFG Unger Parsing

The challenge is how to get all combinations of picking up to n items.

My idea is that: Get all numbers from 0 to 2^chartlength. From their binary representation take all with cross sum = n. Map them to the chart, 1 means the item is picked, 0 it's not. Pass the picked items to the deduction rule as always.

The next challenge is how to correctly generate consequences from them. My idea is to pass a deduction rule to the dynamic rule. And the number of antecedences has to be the number of rhs symbols. Hence n. Order the items by indices and then check if their order matches the rhs and also check if the indices match.

Create optional parameter -success that prints only successful items from a parsing trace

Marry JUnit

I don't want to live without it anymore. I miss it so much. <3

Constructor CFG from PCFG

CFG remove chain rules

In GUI when hovering over item display it as tree

It would be so cool if the text hovered over would be highlighted. It would be even cooler if parts of the items that belong together are highlighted with the same color when hovering over it.

I imagine the tree displayed in a new window that is only shown on hovering.

Maybe also display the underlying definition of the rule that was applied.

Maybe create a cursor that moves down through the whole chart, so the user gets the impression of an animated parsing process. And it takes my job of explaining. :)