neskuchny / dft

Democratic Fine-tuning with a Moral Graph

Democratic Fine-Tuning: README

Developed by the Institute for Meaning Alignment, funded by OpenAI. Live deployment available at dft.meaningalignment.org.

Table of Contents

• Overview
• Background
• Tech Stack
• Output
• Local Setup
• Database Evolution
• Deployment
• Testing
• Additional Documentation

Overview

Democratic Fine-Tuning (DFT) is an initiative aimed at achieving a fine-tuned model that bridges political, cultural, and ideological boundaries.

This repository hosts code for an application with a new democratic process that takes ~15 minutes to go through. The app is live at dft.meaningalignment.org.

Participants go through the following steps:

1. Dialogue: Participants interact with a chatbot, discussing values they believe ChatGPT should have when responding to contentious questions.
2. Vote on Values: Participants vote on values proposed by their peers.
3. Vote on Wisdom Transition: Participants vote on wether the transition from one value to another represents an increase in wisdom.

This process generates a moral graph, which can be used to find convergence in which values ChatGPT should have in contentious scenarios, while remaining legible and democratically legitimated.

The intricacies of the graph can be explored here. Further insights on the construction of the graph and the data handling can be found in the data section. For a deep dive into the motivations and processes behind DFT, read our introduction blog post.

Background

Our aspiration with DFT is to craft a model universally regarded as "wise." Such a model would resonate with Republicans, Democrat, irrespective of their ideological or cultural bearings. The ultimate goal is to mitigate the prospects of ideological conflicts amplified by models individually fine-tuned based on group or individual preferences. Two novel techniques are employed:

• Value Alignment: Rather than aligning with preferences, the model is aligned with values. These values are sourced from an expansive and diverse demographic. For more on how we define values, please read the following.
• Moral Graph Creation: This graph helps find convergent values.

Subsequent endeavors will focus on fine-tuning the LLM based on these values.

Tech Stack

• Development Language: TypeScript
• Framework: Remix
• Database: PostgreSQL
• Event Queue: Inngest
• Deployment Platform: Vercel

Output

The moral graph, survey data and demographics data we collected can be found here.

• Database Schema: The data collated during the process adheres to our database schema.
• Moral Graph Generation: The code responsible for generating the moral graph is available here.
• Data Export: A moral graph can be exported in JSON format via this endpoint. The export schema is detailed here.

Local Setup

1. Install Dependencies: npm i
2. Generate Prisma Schema: npx prisma generate
3. Environment Configuration: Duplicate .env.example to create .env and populate it with relevant values.
4. Run Development Server: npm run dev

Database Evolution

To update the database schema, execute: npx prisma migrate dev

The database schema can be found here.

Deployment

A push to the main branch of the GitHub repository will trigger an automatic deployment at dft.meaningalignment.org.

Testing

Some tests leverage GPT-4 for dialogue evaluations. These can be initiated with npm run test. Due to their compute-intensive nature, it's advisable to run them only when absolutely necessary.

Additional Documentation

• Remix Documentation
• Vercel Documentation

---

Thank you for your engagement with Democratic Fine-Tuning. We value your contributions and insights.