Description

Enable rule-based tokenization that regroups neighboring tokenized terms that logically belong together under the same entities. Think compound words or full names.

Objectives

1. Edit tokenization functions to allow a variable number of parameters to allow for specific rules and exceptions during tokenization.

Description

Connect to MongoDB with a Python API and test CRUD operations for a single document with a text field.

Objectives

1. Study the fundamentals of a Python API for MongoDB. PyMongo is recommended.
2. Successfully connect to a live session.
3. Insert, read, update, and delete a single document with a text field to test functionality of database.

Description

Create basic frontend UI with widgets for file upload.

Objectives

• pip install streamlit
• Add title for project with a logo.
• Add file upload widget with label. Allowed extensions are currently .txt, doc, docx, pdf.
• Add button for wiki creation.

Description

Compute scores for how objective or subjective a piece of text is. The sign indicates which direction the text leans. For example, a positive score could mean the text is more likely to be verifiable fact than opinion. The magnitude just indicates the degree of being objective or subjective.

Objectives

1. Use a library with such functionality e.g. TextBlob. This feature is usually part of sentiment analysis libraries.
2. Write functions that allow objective-subjective scoring on the sentence, paragraph, and whole document level.

Description

Write helper functions that would read in text from easily readable formats.

Objectives

1. Read in single and multiple .txt files.
2. Read all .txt files from a path to a directory.
3. Read text from a given URL.

Description

Retrieve answers from user questions by mining for relevant information from given set of documents. The implementation for this one is quite open-ended.

Objectives

No specific steps. Recommended to checkout functionality from the QA pipeline on HuggingFace.

Description

Using some kind of clustering algorithm to predict a class per document. Classes may be genre, topic, usefulness, etc. Finding the closest cluster per document relies on a distance metric.

Objectives

1. Implement different clustering algorithms to classify documents into an arbitrary set of classes. Text similarity would be a good starting point as the distance metric utilized.
2. Use zero-shot learning (ZSL) to classify documents from a group of pre-determined classes. HuggingFace has a pipeline for that. Checkout the comments in here.

Description

Get familiar with knowledge graphs and their embeddings through code.

Objectives

1. Compare and contrast different Python libraries for knowledge graph embeddings (KGE). The ones that are highly usable together with custom Python code are AmpliGraph (Tensorflow), TorchKGE (PyTorch), and PyKEEN.
2. Finalize a library to stick to based on prior Python experience.

Description

Use an NLP library to tokenize textual data.

Objectives

1. Use an NLP library's built-in function for tokenization to handle individual strings.
2. Extrapolate tokenization to sentence-level and paragraph level.

Description

Perform OCR on images of text to recognize and transform the text into digital format.

Objectives

1. Familiarize with the functions of a library e.g. pytesseract.
2. Write a wrapper function that grayscales the image and then utilizes the appropriate OCR function.
3. Not necessary but may help: add more steps for image preprocessing such as denoising, if that improves OCR accuracy.

Description

Flag non-stop words with part of speech.

Objectives

1. Use an NLP library's built-in POS tagging function wrapped inside a helper function.
2. Add function to filter words with certain POS tags.

Description

Build ranking functionality for a set of documents. The ranking criteria can be as simple as the number of keywords mined. Other criteria include relevance to a queried topic and readability level.

Objectives

1. Find some metrics that can be used to score documents for ranking. For example, textstat has functions that judge reading level by grade.
2. Write a function that ranks documents based on different scoring metrics. The function can simply return a reordered list of indexes or names of documents.
3. Add parameters in function for order direction (highest or lowest score first), skip the top n documents, limit the return to the top n ranked documents, and other editions if you want.

Description

Extract keywords from a document.

Objectives

Use a stable library such as KeyBERT or an established (and sometimes more simple) algorithm like TF-IDF.

Description

Notify users via text or email of changes to their uploaded stash of documents. This is meant for a group contribution where more than one person can upload extra or delete documents from an initial stash of documents afterwards.

Objectives

1. Find a suitable Python API (highly recommend Twilio).
2. Test a single SMS message.
3. Test a single email.