chengchingwen / bytepairencoding.jl Goto Github PK

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Hi, thanks for writing this package! It's been super useful for the GenGPT3.jl package I've been developing for research.

OpenAI recently released newer completion models (gpt-3.5-instruct-turbo, babbage-002 and davinci-002) that no longer use the old GPT-2 tokenizer, but instead use the new cl100k_base encoding. I was wondering if you plan to support this encoding at some point -- or if not, if you have any pointers on how I could implement it? Thanks!

Hi,

do you think it would be possible to break the costructor of BPELearner such that you can directly pass the dictionary with tokens and their frequencies?
Something along lines

function BPELearner(vfiles::Vector{String}, num_sym::Int;
                    min_freq::Int = 2, endsym::String = "</w>",
                    normalizer=UnNormalizer())
    vocab = mapreduce((f)->get_vocab(f; normalizer=normalizer), merge!, vfiles)
  BPELearner(vocab::Dict{String,Int}, num_sym = num_sym; min_freq = min_freq,endsym = ))  
end

function BPELearner(vocab::Dict{String,Int}, num_sym::Int; min_freq::Int = 2,endsym::String = "</w>")    
    stats = Statistic(vocab)
    endsym != "</w>" && set_endsym(endsym)
    new(num_sym, min_freq, endsym, vfiles,
        stats,
        Vector{Pair{String, String}}(undef, num_sym),
        normalizer)
end