intel-analytics / text-generation-webui Goto Github PK

This project forked from oobabooga/text-generation-webui

A Gradio Web UI for running local LLM on Intel GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) using IPEX-LLM.

License: GNU Affero General Public License v3.0

Shell 1.36% JavaScript 2.35% Python 91.11% CSS 3.50% Batchfile 0.66% Jupyter Notebook 0.54% Dockerfile 0.47%

text-generation-webui's People

Contributors

Stargazers

Watchers

Forkers

juan-oy chtanch opticblu lishu2006ll zhangky11 mingyu-wei sgwhat

text-generation-webui's Issues

Mistral-7B-Instruct-v0.2 fails for transformers 4.39

Error for 4.39.3

Traceback (most recent call last):
  File "/home/arda/kai/webui/text-generation-webui/modules/callbacks.py", line 61, in gentask
    ret = self.mfunc(callback=_callback, *args, **self.kwargs)
  File "/home/arda/kai/webui/text-generation-webui/modules/text_generation.py", line 375, in generate_with_callback
    shared.model.generate(**kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/generation/utils.py", line 1575, in generate
    result = self._sample(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/generation/utils.py", line 2697, in _sample
    outputs = self(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py", line 1157, in forward
    outputs = self.model(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/ipex_llm/transformers/models/mistral.py", line 148, in mistral_model_forward_4_36
    return MistralModel.forward(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py", line 1042, in forward
    layer_outputs = decoder_layer(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py", line 757, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/ipex_llm/transformers/models/mistral.py", line 528, in mistral_attention_forward_4_36
    return forward_function(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/ipex_llm/transformers/models/mistral.py", line 806, in mistral_attention_forward_4_36_original
    past_key_value.seen_tokens += key_states.shape[-2]
AttributeError: can't set attribute

Can siucceed for 4.38.2

@jason-dai @shane-huang Seems the upstream is continuously upgrading transformers and our implementation may fail for most models...

Add model.half() when model load

@sgwhat To save some memory?

Related issue may: intel-analytics/ipex-llm#10575
If on Arc750

upstream text webui will by default install torch 2.2.2 at this moment

Will have issues here if not downgrading pytorch: https://github.com/analytics-zoo/nano/issues/1227

Llama2-7b doesn't support transformers >=4.38

Traceback (most recent call last):
  File "/home/arda/kai/webui/text-generation-webui/modules/callbacks.py", line 61, in gentask
    ret = self.mfunc(callback=_callback, *args, **self.kwargs)
  File "/home/arda/kai/webui/text-generation-webui/modules/text_generation.py", line 392, in generate_with_callback
    shared.model.generate(**kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/generation/utils.py", line 1592, in generate
    return self.sample(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/generation/utils.py", line 2696, in sample
    outputs = self(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 1176, in forward
    outputs = self.model(
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/anaconda3/envs/text-webui-upstream/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: llama_model_forward_4_36() got an unexpected keyword argument 'cache_position'
Output generated in 0.17 seconds (0.00 tokens/s, 0 tokens, context 72, seed 1344122438)

One week ago they have upgraded to 4.39...
oobabooga@3ce0d92

Conflict dependencies with upstream

accelerate: 0.27.* (upstream) VS 0.21.0 (ipex-llm)
tokenizers: 0.15.2 (by default upstream will install the latest) VS 0.13.3 (ipex-llm)
transformers: 4.38.* (upstream) VS 4.31.0 (ipex-llm)

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble