大佬，为什么我调试出来回复的速度非常慢呢？ about llama3-chinese-chat

[condywl]

并且控制台报错：

`

load model...
load model end.
D:\Development-Projects\Python-Projects\ShareAI\deploy\web_streamlit_for_instruct.py:80: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
logger.warn( # pylint: disable=W4902
--- Logging error ---
Traceback (most recent call last):
File "D:\Soft\Python3\Lib\logging_init_.py", line 1110, in emit
msg = self.format(record)
^^^^^^^^^^^^^^^^^^^
File "D:\Soft\Python3\Lib\logging_init_.py", line 953, in format
return fmt.format(record)
^^^^^^^^^^^^^^^^^^
File "D:\Soft\Python3\Lib\logging_init_.py", line 687, in format
record.message = record.getMessage()
^^^^^^^^^^^^^^^^^^^
File "D:\Soft\Python3\Lib\logging_init_.py", line 377, in getMessage
msg = msg % self.args
~~~~^~~~~~~~~~~
TypeError: not all arguments converted during string formatting
Call stack:
File "D:\Soft\Python3\Lib\threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "D:\Soft\Python3\Lib\threading.py", line 1045, in _bootstrap_inner
self.run()
File "D:\Soft\Python3\Lib\threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "D:\Development-Projects\Python-Projects\ShareAI.venv\Lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 307, in _run_script_thread
self._run_script(request.rerun_data)
File "D:\Development-Projects\Python-Projects\ShareAI.venv\Lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 600, in _run_script
exec(code, module.dict)
File "D:\Development-Projects\Python-Projects\ShareAI\deploy\web_streamlit_for_instruct.py", line 314, in
main(model_name_or_path, adapter_name_or_path)
File "D:\Development-Projects\Python-Projects\ShareAI\deploy\web_streamlit_for_instruct.py", line 288, in main
for cur_response in generate_interactive(
File "D:\Development-Projects\Python-Projects\ShareAI.venv\Lib\site-packages\torch\utils_contextlib.py", line 35, in generator_context
response = gen.send(None)
File "D:\Development-Projects\Python-Projects\ShareAI\deploy\web_streamlit_for_instruct.py", line 80, in generate_interactive
logger.warn( # pylint: disable=W4902
Message: "Both 'max_new_tokens' (=660) and 'max_length'(=862) seem to have been set. 'max_new_tokens' will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)"
Arguments: (<class 'UserWarning'>,)

`

[CrazyBoyM]

可能是显卡性能有点低，可以试试用llama.cpp配合cpu多线程跑的方案

[try1995]

我的也慢，原生的8b很快，控制台报错我把启动脚本里面的deploy\web_streamlit_for_v1.py这个文件81行改了，就好了： logger.warning( # pylint: disable=W4902
f"Both 'max_new_tokens' (={generation_config.max_new_tokens}) "
f"and 'max_length'(={generation_config.max_length}) seem to "
"have been set. 'max_new_tokens' will take precedence. "
'Please refer to the documentation for more information. '
'(https://huggingface.co/docs/transformers/main/'
'en/main_classes/text_generation)'
)