vida-nyu / domain-discovery-d4 Goto Github PK

The default input file for the ter-index step D4 is listed as text-columns.txt. This should be columns/.

Allow user to switch on/off output printed to the console by different D4 components.

Add step for replacing local domains as equivalence classes to the the main D4 workflow.

The strong domain discovery step can be modified in the following way:

• Cluster local domains that support each other. Each cluster forms a strong domain
• For each strong domain rank terms based on the number of columns (in the strong domain) that they occur in
• Use steepest drop to group terms based on their weights

Add option to compute frequency of an equivalence class for each column C as either

• min. column frequency of all EQ terms in C
• max. column frequency of all EQ terms in C
• sum of frequencies for all EQ terms in C (default)
• average (rounded) of frequencies for all EQ terms in C

We should keep track of term frequencies in the column files (and the terms index and compressed term index). This would allow us to use similarity measures for terms/equivalence classes that are based on some notion of tf-idf.

The current implementation for SimilarTermIndexGenerator is rather naive. It merges all equivalence classes in a connected component based on similarity between pairs of equivalence classes. This approach has the strong disadvantage of potentially merging dis-similar equivalence classes because similarity is not transitive.

One improvement could be to pick equivalence classes as strong seeds and then merge them with all other equivalence classes that are similar to the seed. While this could still merge dis-similar equivalence classes there is the guarantee that they all at least satisfy the similarity threshold with the seed equivalence class.

Heiko,

Can you please add a reference to our VLDB paper in the readme? This may be useful for people that try to use the tool.

Thanks,
Juliana