Comments (2)
Hi, thanks for trying our model!
There are a couple of things to keep in mind:
- The models are not finetuned on instructions, they're just trained to predict the next words. So the models probably won't be able to follow instructions like "please write an essay about birds". A good comparison with GPT-Neo 1.3B.
- At 1.3B or 2.7B parameters we don't expect models to have amazing abilities yet (e.g., GPT3 davinci, ChatGPT). Again, a good comparison is with a Transformer of similar size (e.g. GPT-Neo 1.3B). Given the promising early results, we're excited about scaling these up and finetuning them with instructions.
- The default options during benchmarking is to use greedy decoding, which is known to be not great for text quality (it tends to produce repetitive text). You can try top_k or top_p sampling.
As an example, with the prompt "Please write an essay about birds", and these options:
generated_text = model.generate(input_ids=input_ids, max_length=max_length,
eos_token_id=tokenizer.eos_token_id,
return_dict_in_generate=True, output_scores=True, top_p=0.9, top_k=50)
I'm getting
Please write an essay about birds.
To begin with, learn to identify birds by making a sketch of a bird in your mind. Draw a bird on paper and copy it with a pen on your sketch pad or at a computer.
After studying birds on paper, select 10 new birds from the drawing or copy your drawing with a pen; then look through the book for an interesting bird illustration.
[It then continues with more text]
With GPT-Neo-1.3B and the same prompt, I'm getting
Please write an essay about birds in this course
The course has a focus on the importance of birds in the food cycle, and, as part of that, asks you to write an essay about birds in one of these units! Students work in
from h3.
Thank you.I am too used to ChatGPT producing excellent results.Going to try.You are doing excellent job.
from h3.
Related Issues (20)
- Correct method to load 2.7B? HOT 4
- Unpickling errors when loading models HOT 1
- Releasing Training and Synthetic benchmarks HOT 3
- What is fftconv_bwd doing? HOT 2
- FFT Conv on Seq > 8192? HOT 1
- inconsistent output from fftconv_func and native pytorch fft HOT 3
- Error Running `generate_text_h3.py` (`CUDA error: CUBLAS_STATUS_NOT_INITIALIZED`) HOT 3
- Having trouble for compiling fftconv HOT 9
- The motivation for not fusioning fff(k) into the kernel HOT 2
- Release of pretraining and fine tuning code HOT 5
- Error when using use_fast_fftconv option in generate_text_h3.py HOT 3
- why dividing kv_f by fft_size? HOT 2
- Question about methodology used for evaluating FlashConv against cuFFT HOT 2
- /bin/bash: line 0: cd: csrc/fused_softmax: No such file or directory
- TypeError: forward() got an unexpected keyword argument 'last_token_only' HOT 2
- Setup for repo unclear HOT 1
- ERROR: CUDA RT call cudaFuncSetAttribute. Failed with invalid device function (98). HOT 2
- use_fast_fftconv generates error HOT 2
- Full version of H3 model? HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from h3.