kulturarvscluster / getting-started-with-netarkivet-and-sparklyr

Some short code snippets and tutorials for getting started with Sparklyr and an ETL for the Danish Netarchive

Short introductions to R/Sparklyr and the Netarkiv ETL

These code snippets illustrates how to begin with R and metadata from the Danish Web Archive.

List of snippets

first-rmarkdown.Rmd

An example on how to use RMarkdown for developing reproducible code and documentation.

test-rmarkdown.Rmd

A test file for the above snippet.

first-sparklyr.Rmd

An example on how to use Sparklyr

first-metadata-extract.Rmd

An example on how to extract metadata from metadata WARC (And ARC) files generated by an internet harvester.

first-load-of-ETL-data.Rmd

An example on how to load the above metadata into Spark.

first-sparklyr-on-cluster.Rmd

An example on how to do the same on the DeIC National Cultural Heritage Cluster, Royal Danish Library.

Author

• Per Møldrup-Dalum - The Royal Danish Library - Kulturarvscluster