godrm / gpt3-tokenizer

GPT3 encoder & decoder tool written in Swift

GPT3 Tokenizer

GPT3 encoder & decoder tool written in Swift

About

GPT-2 and GPT-3 use byte pair encoding to turn text into a series of integers to feed into the model. This is a Swift implementation of OpenAI's original python encoder/decoder which can be found here and based on this Javascript implementation here

Install with Swift Package Manager

Add this repository url to Swift Package Manager

https://github.com/aespinilla/GPT3-Tokenizer.git

Usage

import GPT3_Tokenizer

let gpt3Tokenizer = GPT3Tokenizer()

let text = "hello 👋 world 🌍"

// Encode
let encoded = gpt3Tokenizer.encoder.enconde(text: text)
print(encoded)

// Decode
let decoded = gpt3Tokenizer.decoder.decode(encode: encoded)
print(decoded)