pharringtonp19 / llmft

LLMs for Causal Inference

llmft

Purpose

Producing causal effects by fine-tuning Large Language Models on text data.

Example

In the Right to Counsel at Scale, we estimate the impact of legal representation (for those facing eviction) on housing court outcomes and housing stability more importantly. We do so via an instrumental variable approach where the instrument is whether free legal representation is available in the tenant's zip code.

$$y_i = \beta(\big(\mathbb{E}[D_i \vert X_i, Z_i] - \mathbb{E}[D_i \vert X_i]\big)$$

We estimate the conditional expectation functions via fine-tuned LLMs where $X_i$ is the landlord's complaint against the tenant.

llmft's Issues

Decoder Model

• Implement compute_recall
  • Currently this involves a separate forward pass of the model. We should be able to combine this with evaluation_loss
  • model.generate isn't working atm
• flash-attention implementation
• Improve logging format: log_predictions
  • Can we make this interactive via gradio?
• Plot learning rate
• Early Stopping
• two 'yes' tokens

Tokenizers Parallelism

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using tokenizers before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Enable Sharing Compiled Models Across Executions of the notebook

Inference

• Conformal Prediction
• Closed Form

Focus Loss Function

• Raises an error with the decoder model

Experiment # 2

• Create New Dataset (positive health didn't work)
• Cross Tabs to evaluate what the model learns

To Do

• Encoder Model

  • Class Imbalance

• Decoder Model

  • Add a callback that can record the models output to various prompts during training
  • Enable Early Stopping (Callback)

Report Standard Validation Loss

Encoder Model

• Add a k-fold split option

Experiment # 1 (Descriptive)

• Seed