d2klab / harvestingtablesinthewild Goto Github PK

We should find a common data format in which we can store extracted, parsed and augmented tables.
For now - to keep things simple - we are just using JSON data with the following format:

    obj = {
        'url': url,
        'timestamp': datetime.now().isoformat(), # ISO 8601
        'page_title': page_title,
        'table': table,
    }

    # Example table:

    # COUNTRY, SIZE
    # Finland  5.5
    # Germany  80
    # Norway
    # USA      506

    # table = {
    #   "Country" : ["Finland","Germany","Norway", "USA"],
    #   "Size": ["5.5", "80", "", "506"],
    # }

In this thread we can collect some ideas for other data storage formats.

The data format should be extended with the following fields:

• tableOrientation
• language
• columnNumber
• rowNumber

As Eelis mentioned:

I've seen some utf-8 characters being used in place of similar looking ascii characters. We might want to do some character normalization before this. Example package: https://github.com/woodgern/confusables

Originally posted by @EelisK in #22 (comment)

So we should look into character normalization at least for get_term_set, but also think about if the regular table cells and headers need to be normalized.

Maybe we can already create a small graph out of the ingested data, e.g. with edges such as URL and Domain.

get_term_set function should also remove stub words ("a", "the", "of").
For a start, just for English.

Currently, we have no detection for the type of tables ("tableType") we are crawling.
Also the "headerPosition" field is hardcoded ATM.

For inspiration, see:

• Header classification: https://github.com/JulianEberius/dwtc-extractor/blob/master/src/main/java/webreduce/extraction/basic/BasicExtractionAlgorithm.java#L255
• Table classification: https://github.com/JulianEberius/dwtc-extractor/blob/2a7e1c0b8e8328b52278c2dfba020b9ccfc12839/src/main/java/webreduce/extraction/mh/TableClassification.java#L82

As discussed in our last meeting, we agree that it does not make sense to just randomly crawl all possible website on the web - and most likely accrue a lot of junk.

Instead, we should come up with a strategy that biases the selection of websites towards high-quality data sources.

We need to take the items out from the Kafka message queue and store them in ArangoDB database.
Initial work for this has been done in:
#11

Since we have Kafka message queue now, we also need to figure out how to put the items from the scrapy pipeline into Kafka.

Ref:
https://github.com/TeamHG-Memex/scrapy-kafka-export

Add a message queue for the ingestion of the harvested tables into the database. We have chosen Apache Kafka for this purpose.

Take some of the tables from data/test-urls.txt and make sure the output matches the expected output.

With the crawling strategy introduced in it can happen (or is even likely) that the same URL is visited multiple times (e.g. when lots of sites point to https://en.wikipedia.org/index.php), which should obviously be avoided.

Ref.:
https://github.com/scrapy-plugins/scrapy-deltafetch
https://github.com/TeamHG-Memex/scrapy-crawl-once

• Select interesting websites from Alexa 500 ranking: https://www.alexa.com/topsites
• https://ourworldindata.org/ -> not accessible since its rendered client side => FUTURE WORK
• Extract common websites from CommonCrawl
• Use Wikipedia List / Category pages as seeds, e.g. https://en.wikipedia.org/wiki/Category:Lists_by_economic_indicators

DWTC-Extractor has a good algorithm with filtering heuristics:

https://github.com/JulianEberius/dwtc-extractor/blob/master/src/main/java/webreduce/extraction/basic/BasicExtractionAlgorithm.java