sebastian-nagel/crawler-commons
A set of reusable Java components that implement functionality common to any web crawler
Repositories
sebastian-nagel/nutch
Mirror of Apache Nutch
sebastian-nagel/warc-crawler
Process web archives (WARC format) with StormCrawler and index content into OpenSearch
sebastian-nagel/jwarc
Java library for reading and writing WARC files with a typed API
sebastian-nagel/sitemap-performance-test
sebastian-nagel/nutch-test-single-node-cluster
sebastian-nagel/PyAthena
PyAthena is a Python DB API 2.0 (PEP 249) client for Amazon Athena.
sebastian-nagel/webarchive-commons
Common web archive utility code.
sebastian-nagel/cld2
Compact Language Detector 2
sebastian-nagel/compact_enc_det
compact_enc_det - Compact Encoding Detection
sebastian-nagel/konstanz-in-zahlen
Konstanz in Zahlen: Jährliche Zahlen und Fakten zur Stadt Konstanz
sebastian-nagel/docker-hadoop
Apache Hadoop docker image
sebastian-nagel/ossym2022-robotstxt-experiments
Experiments and metrics about robots.txt captures, presentation at #ossym2022
sebastian-nagel/surt
Sort-friendly URI Reordering Transform (SURT) python module
sebastian-nagel/zip2gz
Create a file tree with the raw data from a zip file in usable format
sebastian-nagel/nutch-1
sebastian-nagel/pga-declarations
Declarations of terms of major social media platforms. Maintained by the Platform Governance Archive team, University of Bremen.
sebastian-nagel/sfm-facebook-harvester
sebastian-nagel/selenium_test_demo_tu2txt
sebastian-nagel/storm-crawler
Web crawler SDK based on Apache Storm
sebastian-nagel/cc2dataset
Easily convert common crawl to a dataset of caption and document. Image/text Audio/text Video/text, ...
sebastian-nagel/news-please
news-please - an integrated web crawler and information extractor for news that just works.
sebastian-nagel/duckdb-web
DuckDB-Web - Source code of duckdb.org
sebastian-nagel/simhash-py
Simhash and near-duplicate detection
sebastian-nagel/twarc-csv
A plugin for twarc2 for converting tweet JSON into DataFrames and exporting to CSV.
sebastian-nagel/introduction-to-python
sebastian-nagel/sfm-instagram-harvester
sebastian-nagel/sfm-web-harvester-browsertrix
sebastian-nagel/browsertrix-crawler
Run a high-fidelity browser-based crawler in a single Docker container
sebastian-nagel/wdc-page
This repository contains the source files of the Web Data Commons website and is used to maintain the site. The Web Data Commons project extracts structured data from the Common Crawl