mitlibraries / tdm-pit

Process for Indexing Thesis

Process for Indexing Thesis

This is the main worker process used for indexing items from Fedora in Elasticsearch.

Installing

pit uses Pipenv to manage dependencies. Once you have pipenv installed:

$ git clone https://github.com/mitlib-tdm/pit.git
$ cd pit
$ pipenv install --dev

Using pit

The pit run subcommand will start the worker process. This will watch Fedora's ActiveMQ instance for newly added items and add them to the Elasticsearch index. Type $ pit run --help for a full description.

The pit reindex subcommand will reindex the full theses collection. Type $ pit reindex --help for a full description.

Developing

There are several Makefile targets that can be used for developing. make test and make coverage will run the tests and output the test coverage. make update will update all the dependencies. make release will increase the version number, create a new tag and build a new docker image with a corresponding tag.

tdm-pit's Issues

Copyright date is frequently empty

Currently, a missing copyright date will result in an error while indexing a document.

Use Fedora ID not tied to host

Since the URI for a thesis could change based on how services are deployed in kubernetes, it's been proposed to just use the handle ID as the identifier in elasticsearch. We can guarantee it will be unique locally.