Code implementation of GPT as a finite-state markov chain
This repository contains a minimalist implementation of a GPT (Generative Pre-trained Transformer) model in PyTorch, named Minmalist-GPT. The implementation is kept minimal for educational purposes and includes functionalities for training on a simple binary sequence.
Minmalist-GPT is designed to demonstrate the core concepts of the GPT architecture in a simplified manner. The implementation includes a basic GPT model, training on a binary sequence, and a visualization of the model's state transitions.
The code is organized into the following main components:
-
minmalist_gpt.py: Defines the GPT model (
GPT
class) and related configurations (GPTConfig
class). -
main.py: Contains functions for training Minmalist-GPT on a binary sequence and visualizing the model's state transitions.
-
states-1.png: Visualization of Minmalist-GPT's initial state transitions.
-
states-2.png: Visualization of Minmalist-GPT's state transitions after training.
torch
graphviz
To use Minmalist-GPT, you can follow the example in main.py
:
# Example usage
from minmalist_gpt import GPT, GPTConfig, plot_model, token_seq_to_tensor, do_training
# Set the vocabulary size, context length, and other configuration parameters
vocab_size = 2
context_length = 3
config = GPTConfig(
block_size=context_length,
vocab_size=vocab_size,
n_layer=4,
n_head=4,
n_embd=16,
bias=False,
)
# Create a new Minmalist-GPT model
gpt = GPT(config)
# Plot the initial state transitions
dot = plot_model()
dot.render('states-1', format='png')
# Prepare training data sequence
seq = list(map(int, "111101111011110"))
print("\nTraining data sequence: ", seq)
X, Y = token_seq_to_tensor(seq)
# Train Minmalist-GPT
do_training(X, Y)
# Plot the state transitions after training
dot = plot_model()
dot.render('states-2', format='png')
This implementation is inspired by the GPT architecture, and special thanks to the authors of the original GPT paper.
This project is licensed under the Apache 2.0 License - see the LICENSE file for details.