Comments (6)
Hi @mostafamdy, it seems like the guide is outdated. Thanks for bringing this up! You can refer to the Sampler API docs or the "Example Use" section on the Kaggle model card. For your usecase, there's now a simpler API for plugging-in different samplers:
import keras_nlp
model = keras_nlp.models.GemmaCausalLM('gemma_2b_en')
# Tell KerasNLP to use a "greddy" sampler. Other options are "top_k", "top_p", etc.
# See https://keras.io/api/keras_nlp/samplers/ for more info
model.compile(sampler="greedy")
output = model.generate("What is Keras?", max_length=50)
# You can also initialize a sampler to configure it for your usecase
sampler = keras_nlp.samplers.TopKSampler(k=5, temperature=0.7)
causal_lm.compile(sampler=sampler)
causal_lm.generate(["What is Keras?"])
Does this answer your question?
from keras-nlp.
Ah OK. Your code looks good to me. You seem to be printing out the next token predictions for each input sequence which is why I guess the outputs are different. Can you check if this code generates the right output:
import keras
from keras import ops
import keras_nlp
model = keras_nlp.models.GemmaCausalLM.from_preset('gemma_2b_en')
preprocessor = model.preprocessor
tokenizer = preprocessor.tokenizer
backbone = model.backbone
def loss_fn(y_true, y_pred, prompt=None, index=None):
logits = y_pred
temperature = 1.0
logits = ops.cast(logits, "float32")
# Compute probs and next token value
probabilities = ops.softmax(logits[:, index, :], axis=-1)
next_token = ops.argmax(probabilities, axis=-1)
# Update the prompt
prompt_tokens = tokenizer.tokenize(prompt)
updated_prompt_tokens = ops.concatenate([prompt_tokens, next_token[..., None]], axis=-1)
updated_prompt = tokenizer.detokenize(updated_prompt_tokens)
# Print the updated prompt
print(f"The updated prompt is: {updated_prompt}")
# Get the inputs
prompt = ["The quick brown"]
train_data = preprocessor(prompt, sequence_length=10)
index = ops.min(ops.sum(train_data[0]['padding_mask'], axis=-1)) - 2
# Evaluate the loss function
loss_fn(train_data[1], model(train_data[0]), prompt=prompt, index=index)
# The updated prompt is: [b'The quick brown fox']
# Check outputs match
model.generate(prompt, max_length=5)
# ['The quick brown fox']
from keras-nlp.
Hi @mostafamdy ,
It seems the sampler expects the prompt in the form a List. The {{call_args}}
in base Sampler class not defined properly IMO.
keras-nlp/keras_nlp/samplers/sampler.py
Lines 32 to 33 in 16d3ebb
from keras-nlp.
Thanks @tirthasheshpatel
I want to use sampler in custom loss function is it possible? 😅
Is this code correct?
def custom_loss(y_true,y_pred):
logits=y_pred
temperature=1.0
logits = ops.cast(logits, "float32")
probabilities = keras.activations.softmax(logits / temperature)
next_token=ops.argmax(probabilities, axis=-1)
print(next_token)
txt=tokenizer.detokenize(next_token)
print(f"Greedy search generated text: \n{txt}\n")
from keras-nlp.
I tried to call gemma model like this
preprocessor=keras_nlp.models.GemmaPreprocessor(
tokenizer, sequence_length=SEQ_LEN,
)
model_out = gemma_lm(preprocessor([data[0]]))
and then passed model_out to custom_loss function to generate text
def custom_loss(y_true,y_pred):
logits=y_pred
temperature=1.0
logits = ops.cast(logits, "float32")
probabilities = keras.activations.softmax(logits / temperature)
next_token=ops.argmax(probabilities, axis=-1)
txt=tokenizer.detokenize(next_token)
print(f"Greedy search generated text: \n{txt}\n")
custom_loss("y_true",model_out)
But the output is different from gemma_lm.generate()
from keras-nlp.
Thank you so much ❤️
I tried this code and it's working well, but I have a misunderstanding.
I tried to change the index of logits
for i in range(10):
probabilities = ops.softmax(logits[:, i, :], axis=-1)
next_token = ops.argmax(probabilities, axis=-1)
updated_prompt = tokenizer.detokenize(next_token)
print(updated_prompt)
the output was :
and fox fox,' the, the
When using generate with a sequence length of 10, I received:
The quick brown fox jumps over the sleeping dog
from keras-nlp.
Related Issues (20)
- Distributed training not working (batch size calculation) HOT 7
- Documented `id_to_token` doesn't exist for UnicodeCodepointTokenizer HOT 4
- Add support for returning the attention scores in encoder call. HOT 1
- Add remainig Phi-3 models HOT 5
- Error while running .fit on the PaliGemma example from Kaggle HOT 1
- Merge keras-nlp to keras avoid verison problem HOT 3
- Error on model.summary() with PaliGemma HOT 5
- Does `.enable_lora` work on PaliGemma? HOT 2
- Enable XLA Compatibility for Pretraining BERT with Keras NLP on TensorFlow GPU HOT 3
- Will keras_nlp support qwen2 model in future? HOT 1
- Minor refactoring: cached MHA reimplements code already in base MHA HOT 2
- Documentation rendering issue for PaliGemma
- Add `get_layout_map()` for all backbones HOT 1
- Conversion script modification for Gemma 2 models HOT 1
- TokenAndPositionEmbedding cannot be serialized. HOT 2
- cannot create URI for a fine-tuned Gemma model for Kaggle upload HOT 5
- Polishing T5 inference HOT 1
- When using QLoRA, I get an InvalidArgumentError. HOT 8
- Add `add_bos`, `add_eos` parameters to `SentencePieceTokenizer`.
- 1TB of memory required for model training. HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from keras-nlp.