VoucherRedaction (VoucherVision + Redaction)

Table of Contents
About
Roadmap and New Features List
Try our public demo!
Installing VoucherVision
Create a Desktop Shortcut to Launch VoucherVision GUI (MacOS)
Create a Desktop Shortcut to Launch VoucherVision GUI (Windows)
Run VoucherVision
Custom Prompt Builder
Expense Reporting
- Expense Report Dashboard
User Interface Images

About

VoucherVision - In Beta Testing Phase 🚀

For inquiries, feedback (or if you want to get involved!) please complete our form.

Overview:

Initiated by the University of Michigan Herbarium, VoucherVision harnesses the power of large language models (LLMs) to transform the transcription process of natural history specimen labels. Our workflow is as follows:

Text extraction from specimen labels with LeafMachine2.
Text interpretation using Google Vision OCR.
Text redaction using RoBERTa for Sequence Classification
LLMs, including GPT-3.5, GPT-4, PaLM 2, and Azure instances of OpenAI models, standardize the OCR output into a consistent spreadsheet format. This data can then be integrated into various databases like Specify, Symbiota, and BRAHMS.

For ensuring accuracy and consistency, the VoucherVisionEditor serves as a quality control tool.

Thanks to all of our collaborating institutions!

Roadmap and New Features List

VoucherVision Features

Feature	Sub-features	Done
Location information redaction		✓
Update all GPT models		✓
Zip output files option		✓
Expense tracking	Dashboard More granular GPT support Project-based and cumulative tracking	✓ ✓ ✓
Hugging Face Spaces	Working and refactored Unified with local version Visualize locations on a map Tested with 300 images Optimize for +300 images	✓ ✓ ✓ ✓
Modular Prompt Builder	Build, save, load, submit Test column order Standard prompts: long, medium, short Test shorter prompt effectiveness	✓ ✓
API Availability Test/Visualization	Lock-out unsupported methods	✗
Load existing OCR		✗
Save/load VV settings		✓
LLM Cost Calculator		✓
FAQs page		✗
Detect and display hardware		✓
Filename Prefix Handling	Needs more options and a GUI test	✗
Validation Tools	JSON formatting World Flora Online Wikipedia links USDA GRIN links Plants of the World Online links	✓ ✓ ✓ ✓ ✓

Supported LLM APIs

API	Sub-versions	Available
OpenAI	GPT 4 GPT 4 Turbo 1106-preview GPT 4 Turbo 0125-preview GPT 4 32k GPT 3.5 GPT 3.5 Instruct	✓ ✓ ✓ ✓ ✓ ✓
OpenAI (Microsoft Azure)	GPT 4 GPT 4 Turbo 1106-preview GPT 4 Turbo 0125-preview GPT 4 32k GPT 3.5 GPT 3.5 Instruct	✓ ✓ ✓ ✓ ✓ ✓
MistralAI	Mistral Tiny Mistral Small Mistral Medium	✓ ✓ ✓
Google PaLM2	text-bison@001 text-bison@002 text-unicorn@001	✓ ✓ ✓
Google Gemini	Gemini-Pro	✓
Local LLMs	GPU Mixtral 8x7B Instruct v0.1 GPU Mistral 7B Instruct v0.2 CPU Mistral 7B Instruct v0.2 GGUF	✓ ✓ ✓

Supported Locally Hosted LLMs

LLM	Requirements	Versions	Available
MistralAI	24GB+ VRAM GPU	Mixtral 8x7B Instruct v0.1 Mixtral 7B Instruct v0.2	✓ ✓
MistralAI	CPU Inference	Mixtral 7B Instruct v0.2 GGUF via llama.cpp	✓
Meta	24GB+ VRAM GPU	Base Llama2 7B chat hf	✓ ✗

VoucherVisionEditor

Feature	Done
Streamline the startup procedure	✗
Add configurable dropdown menus for certain fields	✗
Make sure that VVE can accommodate arbitrary column names	✗
Remove legacy support (version 1 prompts)	✗
Taxonomy validation helper	✓
Visualize locations on a map (verbatim and decimal)	✓
More support for datum and verbatim coordinates	✗
Compare raw OCR to values in form to flag hallucinations/generated content	✗
Accept zipped folders as input	✓
Flag user when multiple people/names/determinations are present	✗

Package Information:

The main VoucherVision tool and the VoucherVisionEditor are packaged separately. This separation ensures that lower-performance computers can still install and utilize the editor. While VoucherVision is optimized to function smoothly on virtually any modern system, maximizing its capabilities (like using LeafMachine2 label collages or running Retrieval Augmented Generation (RAG) prompts) mandates a GPU.

NOTE: VoucherVision runs on computers that do not have a GPU, but the LeafMachine2 collage will run slower.

Try our public demo!

Our public demo, while lacking several quality control and reliability features found in the full VoucherVision module, provides an exciting glimpse into its capabilities. Feel free to upload your herbarium specimen and see what happens! VoucherVision Demo

Installing VoucherVision (using PIP)

Prerequisites

Python 3.10.4 or later
Optional: an Nvidia GPU + CUDA for running LeafMachine2

Installation - Cloning the VoucherVision Repository

First, install Python 3.10, or greater, on your machine of choice. We have validated up to Python 3.11.
- Make sure that you can use pip to install packages on your machine, or at least inside of a virtual environment.
- Simply type pip into your terminal or PowerShell. If you see a list of options, you are all set. Otherwise, see either this PIP Documentation or this help page
Open a terminal window and cd into the directory where you want to install VoucherVision.
In the Git BASH terminal, clone the VoucherVision repository from GitHub by running the command:
```
git clone https://github.com/Gene-Weaver/VoucherVision.git
```
Move into the VoucherVision directory by running cd VoucherVision in the terminal.
To run VoucherVision we need to install its dependencies inside of a python virtual environmnet. Follow the instructions below for your operating system.

About Python Virtual Environments

A virtual environment is a tool to keep the dependencies required by different projects in separate places, by creating isolated python virtual environments for them. This avoids any conflicts between the packages that you have installed for different projects. It makes it easier to maintain different versions of packages for different projects.

For more information about virtual environments, please see Creation of virtual environments

Installation - Windows 10+

Installation should basically be the same for Linux.

Virtual Environment

Still inside the VoucherVision directory, show that a venv is currently not active
```
python --version
```
Then create the virtual environment (venv_VV is the name of our new virtual environment)
```
python3 -m venv venv_VV
```
Or depending on your Python version...
```
python -m venv venv_VV
```
Activate the virtual environment
```
.\venv_VV\Scripts\activate
```
Confirm that the venv is active (should be different from step 1)
```
python --version
```
If you want to exit the venv later for some reason, deactivate the venv using
```
deactivate
```

Installing Packages

Install the required dependencies to use VoucherVision
cd into VoucherVision

pip install -r requirements.txt

If you do NOT have a GPU, then you are all set. Otherwise...

Make sure that your GPU can be recognized. While in the terminal/powershell, type
```
python
```
This opens a Python script. Import torch
```
import torch
```
Make sure the GPU is found
```
torch.cuda.is_available()
```
Exit the Python instance
```
exit()
```
If torch.cuda.is_available() returned True, then you should be set. Otherwise, you need to make sure that your CUDA version is compatible with the PyTorch version. It's usually a good idea to leave the CUDA drivers alone and find the right PyTorch version since installing/updating CUDA can be non-trivial.
- Example: If torch.cuda.is_available() returned False, I would first check my CUDA version. In a terminal, type
```
nvidia-smi
```
- If this throws an error, then you do not have CUDA installed. Please see the troubleshooting steps below.
- Otherwise, look for CUDA Version: XX.X. In this example, we saw CUDA Version: 12.1
- Go to https://pytorch.org/get-started/previous-versions/, search for 12.1 (or your CUDA version) and find the conda installation version. There are MacOS options too.
- We need a PyTorch version greater than 2.X.X. If none exists, then your CUDA version may be too old.
- When I searched for 12.1, I found this: pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu121
- Install your matching version
- Cheat sheet:
  - CUDA 11.8
```
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118
```
  - CUDA 12.1
```
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu121
```
- Verify the installation
- Now we should have the right PyTorch version. Check to see if torch.cuda.is_available() returns True by following the same procedure as above

If your CUDA version < 11.8, then you should probably upgrade to 12.1 If you need help, please submit an inquiry in the form at LeafMachine.org

To run LLMs locally also install pip install flash-attn==1.0.9. Windows is not yet compatible with newer versions. For Linux, you can try pip install flash-attn --no-build-isolation and use a 2.X version.

Installing VoucherVision (using conda)

First, install Anaconda using default settings
- Anaconda
Open the Anaconda Powershell Prompt (Windows) or the terminal (macOS/Linux)
Install Mamba in the base environment. We will use Mamba because it it much faster!
```
conda install mamba -n base -c conda-forge
```

Make sure Conda and Mamba are up to date

conda update conda

conda update mamba -c conda-forge

mamba update --all

mamba clean --all

Create a new Conda environment using Mamba

mamba create --name vouchervision python=3.10 git -c conda-forge

Activate the Conda environment
```
conda activate vouchervision
```
Use cd to move to the directory where you want VoucherVision to live
Clone the VoucherVision repository
```
git clone https://github.com/Gene-Weaver/VoucherVision.git
```
Move into the VoucherVision home directory
```
cd VoucherVision
```

Then we need to install some packages
```
pip install -r requirements.txt
```

If you do NOT have a GPU, then you are all set. Otherwise...

Make sure that your GPU can be recognized. While in the terminal/powershell, type
```
python
```
This opens a Python script. Import torch
```
import torch
```
Make sure the GPU is found
```
torch.cuda.is_available()
```
Exit the Python instance
```
exit()
```
If torch.cuda.is_available() returned True, then you should be set. Otherwise, you need to make sure that your CUDA version is compatible with the PyTorch version. It's usually a good idea to leave the CUDA drivers alone and find the right PyTorch version since installing/updating CUDA can be non-trivial.
- Example: If torch.cuda.is_available() returned False, I would first check my CUDA version. In a terminal, type
```
nvidia-smi
```
- If this throws an error, then you do not have CUDA installed. Please see the troubleshooting steps below.
- Otherwise, look for CUDA Version: XX.X. In this example, we saw CUDA Version: 11.7
- Go to https://pytorch.org/get-started/previous-versions/, search for 11.7 (or your CUDA version) and find the conda installation version. There are MacOS options too.
- We need a PyTorch version greater than 2.X.X. If none exists, then your CUDA version may be too old.
- When I searched for 11.7, I found this: mamba install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia
- Install your matching version (conda install can be very slow)
- Cheat sheet:
  - CUDA 11.7
```
mamba install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia
```
  - CUDA 11.8
```
mamba install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=11.8 -c pytorch -c nvidia
```
  - CUDA 12.1
```
mamba install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia
```
- Verify the installation
```
mamba list cudnn
```
- Now we should have the right PyTorch version. Check to see if torch.cuda.is_available() returns True by following the same procedure as above
- Check torch.cuda.is_available() one more time just to be sure.

If your CUDA version < 11.8, then you should probably upgrade to 12.1 If you need help, please submit an inquiry in the form at LeafMachine.org

Create a Desktop Shortcut to Launch VoucherVision GUI (Windows)

We can create a desktop shortcut to launch VoucherVision. In the ../VoucherVision/ directory is a file called create_desktop_shortcut.py. In the terminal, move into the ../VoucherVision/ directory and type:

python create_desktop_shortcut.py

Or...

python3 create_desktop_shortcut.py

Follow the instructions, select where you want the shortcut to be created, then where the virtual environment is located.

Note If you ever see an error that says that a "port is not available", open run.py in a plain text editor and change the --port value to something different but close, like 8502. Sometimes the connection may not close properly. Also make sure that the previous terminal is closed before re-launching.

Create a Desktop Shortcut to Launch VoucherVision GUI (MacOS)

We can create a desktop shortcut to launch VoucherVision. In the ../VoucherVision/ directory is a file called create_desktop_shortcut_mac.py. In the terminal, cd into the ../VoucherVision/ directory and type:

python create_desktop_shortcut_mac.py

Or...

python3 create_desktop_shortcut_mac.py

Now go look in the ../VoucherVision/ directory. You will see a new file called VoucherVision.app. Drag this file into the Applications folder so that you can open VoucherVisionEditor just like any other app.

NOTE: The instructions below have not been updated to reflect the new code as of Feb. 14, 2024. Stay tuned for updated instructions

Run VoucherVision

In the terminal, make sure that you cd into the VoucherVision directory and that your virtual environment is active (you should see venv_VV on the command line).

Type:

python run_VoucherVision.py

or depending on your Python installation:

python3 run_VoucherVision.py

If you ever see an error that says that a "port is not available", open run_VoucherVision.py in a plain text editor and change the --port value to something different but close, like 8502.

Setting up API key

VoucherVision requires access to Google Vision OCR and at least one of the following LLMs: OpenAI API, Google PaLM 2, a private instance of OpenAI through Microsoft Azure. On first startup, you will see a page with instructions on how to get these API keys. Nothing will work until you get at least the Google Vision OCR API key and one LLM API key.

Check GPU

Press the "Check GPU" button to see if you have a GPU available. If you know that your computer has an Nvidia GPU, but the check fails, then you need to install an different version of PyTorch in the virtual environment.

Run Tests

Once you have provided API keys, you can test all available prompts and LLMs by pressing the test buttons. Every combination of LLM, prompt, and LeafMachine2 collage will run on the image in the ../VoucherVision/demo/demo_images folder. A grid will appear letting you know which combinations are working on your system.

Starting VoucherVision

"Run name" - Set a run name for your project. This will be the name of the new folder that contains the output files.
"Output directory" - Paste the full file path of where you would like to save the folder that will be created in step 1.
"Input images directory" - Paste the full file path of where the input images are located. This folder can only have JPG or JPEG images inside of it.
"Select an LLM" - Pick the LLM you want to use to parse the unstructured OCR text.
- As of Nov. 1, 2023 PaLM 2 is free to use.
"Prompt Version" - Pick your prompt version. We recommend "Version 2" for production use, but you can experiment with our other prompts.
"Cropped Components" - Check the box to use LeafMachine2 collage images as the input file. LeafMachine2 can often find small handwritten text that may be missed by Google Vision OCR's text detection algorithm. But, the difference in performance is not that big. You will still get good performance without using the LeafMachine2 collage images.
"Domain Knowledge" is only used for "Version 1" prompts.
"Component Detector" sets basic LeafMachine2 parameters, but the default is likely good enough.
"Processing Options"
- The image file name defines the row name in the final output spreadsheet.
- We provide some basic options to clean/parse the image file name to produce the desired output.
- For example, if the input image name is MICH-V-3819482.jpg but the desired name is just 3819482 you can add MICH-V- to the "Remove prefix from catalog number" input box. Alternatively, you can check the "Require Catalog..." box and achieve the same result.
Finally you can press the start processing button.

Azure Instances of OpenAI

If your institution has an enterprise instance of OpenAI's services, like at the University of Michigan, you can use Azure instead of the OpenAI servers. Your institution should be able to provide you with the required keys (there are 5 required keys for this service).

Custom Prompt Builder

VoucherVision empowers individual institutions to customize the format of the LLM output. Using our pre-defined prompts you can transcribe the label text into 20 columns, but using our Prompt Builder you can load one of our default prompts and adjust the output to meet your needs. More instructions will come soon, but for now here are a few more details.

Load, Build, Edit

The Prompt Builder creates a prompt in the structure that VoucherVision expects. This information is stored as a configuration yaml file in ../VoucherVision/custom_prompts/. We provide a few versions to get started. You can load one of our examples and then use the Prompt Builder to edit or add new columns.

Instructions

Right now, the prompting instructions are not configurable, but that may change in the future.

Defining Column Names Field-Specific Instructions

The central JSON object shows the structure of the columns that you are requesting the LLM to create and populate with information from the specimen's labels. These will become the rows in the final xlsx file the VoucherVision generates. You can pick formatting instructions, set default values, and give detailed instructions.

Note: formatting instructions are not always followed precisely by the LLM. For example, GPT-4 is capable of granular instructions like converting ALL CAPS TEXT to sentence-case, but GPT-3.5 and PaLM 2 might not be capable of following that instruction every time (which is why we have the VoucherVisionEditor and are working to link these instructions so that humans editing the output can quickly/easily fix these errors).

Prompting Structure

The rightmost JSON object is the entire prompt structure. If you load the required_structure.yaml prompt, you will wee the bare-bones version of what VoucherVision expects to see. All of the parts are there for a reason. The Prompt Builder UI may be a little unruly right now thanks to quirks with Streamlit, but we still recommend using the UI to build your own prompts to make sure that all of the required components are present.

Mapping Columns for VoucherVisionEditor

Finally, we need to map columns to a VoucherVisionEditor category.

Expense Reporting

VoucherVision logs the number of input and output tokens (using tiktoken) from every call. We store the publicly listed prices of the LLM APIs in ../VoucherVision/api_cost/api_cost.yaml. Then we do some simple math to estimage the cost of run, which is stored inside of your project's output directory ../run_name/Cost/run_name.csv and all runs are accumulated in a csv file stored in ../VoucherVision/expense_report/expense_report.csv. VoucherVision only manages expense_report.csv, so if you want to split costs by month/quarter then copy and rename expense_report.csv. Deleting expense_report.csv will let you accumulate more stats.

This should be treated as an estimate. The true cost may be slightly different.

This is an example of the stats that we track:

run	date	api_version	total_cost	n_images	tokens_in	tokens_out	rate_in	rate_out	cost_in	cost_out
GPT4_test_run1	2023_11_05__17-44-31	GPT_4	0.23931	2	6749	614	0.03	0.06	0.20247	0.03684
GPT_3_5_test_run	2023_11_05__17-48-48	GPT_3_5	0.0189755	4	12033	463	0.0015	0.002	0.0180495	0.000926
PALM2_test_run	2023_11_05__17-50-35	PALM2	0	4	13514	771	0	0	0	0
GPT4_test_run2	2023_11_05__18-49-24	GPT_4	0.40962	4	12032	811	0.03	0.06	0.36096	0.04866

Expense Report Dashboard

The sidebar in VoucherVision displays summary stats taken from expense_report.csv.

User Interface Images

Validation test when the OpenAI key is not provided, but keys for PaLM 2 and Azure OpenAI are present:

Validation test when all versions of the OpenAI keys are provided:

A successful GPU test:

Successful PaLM 2 test:

kanelindsay / voucherredaction Goto Github PK

voucherredaction's Introduction