thisserand / llama2_local Goto Github PK

View Code? Open in Web Editor NEW

82.0 82.0 35.0 211 KB

License: MIT License

Python 100.00%

llama2_local's People

Stargazers

Watchers

llama2_local's Issues

Convert this into an API

Hello there,
I am an automation developer and don't have much experience with AI. if you could help with details on how I can convert this into an API it would be useful in my project GPT-Vuln_Analyzer. It would be of great help.

Thank you

Responses become too large

I've been trying the code as a customer service chatbot and works pretty well with few interactions. My issue is that after a few interactions, the model starts to add the following interactions of the conversation into the response.

Something like this:

Customer: Question1
Agent: Answer1
Customer: Question2
Agent: Answer2, Question3, Answer3, Question4, Answer4..

It seems like a stop sequence could help to indicate to the agent that it only has to reply to the customer, instead of write the whole remaining part of the conversation as answer. The problem is that I'm not sure of how to do it? Any suggestion?

Thanks in advance for the help!

safetensors does not contain metadata

windows 11
pytorch 2.0.1
transformers==4.31.0
auto-gptq==0.3.0
langchain==0.0.237
gradio==3.37.0
llama-cpp-python==0.1.73
fire==0.5.0

(llm) PS D:\Codes\llama\llama2_local> python llama.py --model_name="TheBloke/Llama-2-7b-Chat-GPTQ"
The safetensors archive passed at C:\Users\brt/.cache\huggingface\hub\models--TheBloke--Llama-2-7b-Chat-GPTQ\snapshots\b7ee6c20ac0bba85a310dc699d6bb4c845811608\gptq_model-4bit-128g.safetensors does not contain metadata. Make sure to save your model with the save_pretrained method. Defaulting to 'pt' metadata.

can you give me any suggestion?

how can I use for cpu QA model from llama2

llm = CTransformers(model='./llama-2-7b-chat.ggmlv3.q4_K_M.bin', model_type='llama', config={'max_new_tokens': 4096,'temperature': 0.0})

from langchain import PromptTemplate, LLMChain
template = """Answer the question based on the contexts below.If the question cannot be answered using the information provided answer with "I don't know".
Contexts:{text}
Question:What Role does the candidate suits for - 1.Telecaller 2.Medical Coding 3.Software Developer 4.Data Entry?
Answer:
"""
print(template)
prompt = PromptTemplate(template=template, input_variables=["text"])
llm_chain = LLMChain(prompt=prompt, llm=llm)
print("Running...")
text = """

could you give examples around this type of QA.

Problems installing llama2 (failure building wheel)

Hello,

I am having problems installing Llama2. I am trying to install it on Windows and I am stuck on the step where you run:
pip install -r requirements.txt

Note: I made sure to have CMake, MinGW64, VS, and VS Build Tools installed and also have the environment variables including file paths to MingGW/bin and CMake/bin.

I get the following failure:

`Building wheel for llama-cpp-python (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for llama-cpp-python (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [199 lines of output]

  --------------------------------------------------------------------------------
  -- Trying 'Ninja (Visual Studio 17 2022 x64 v143)' generator
  --------------------------------
  ---------------------------
  ----------------------
  -----------------
  ------------
  -------
  --
  Not searching for unused variables given on the command line.
  CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
    Compatibility with CMake < 3.5 will be removed from a future version of
    CMake.

    Update the VERSION argument <min> value or use a ...<max> suffix to tell
    CMake that the project does not need compatibility with older versions.


  -- The C compiler identification is GNU 11.2.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: C:/mingw64/bin/gcc.exe - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- The CXX compiler identification is GNU 11.2.0
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: C:/mingw64/bin/c++.exe - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  CMake Error at CMakeLists.txt:9 (message):
    MSVC is required to pass this check.


  -- Configuring incomplete, errors occurred!
  --
  -------
  ------------
  -----------------
  ----------------------
  ---------------------------
  --------------------------------
  -- Trying 'Ninja (Visual Studio 17 2022 x64 v143)' generator - failure
  --------------------------------------------------------------------------------



  --------------------------------------------------------------------------------
  -- Trying 'Visual Studio 17 2022 x64 v143' generator
  --------------------------------
  ---------------------------
  ----------------------
  -----------------
  ------------
  -------
  --
  Not searching for unused variables given on the command line.
  CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
    Compatibility with CMake < 3.5 will be removed from a future version of
    CMake.

    Update the VERSION argument <min> value or use a ...<max> suffix to tell
    CMake that the project does not need compatibility with older versions.


  -- Selecting Windows SDK version 10.0.22000.0 to target Windows 10.0.22621.
  -- The C compiler identification is MSVC 19.37.32824.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- The CXX compiler identification is MSVC 19.37.32824.0
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Configuring done (18.0s)
  -- Generating done (0.0s)
  -- Build files have been written to: C:/Users/wanyi/AppData/Local/Temp/pip-install-74r1bdtd/llama-cpp-python_5cf10169b309419697c9247200f9fe90/_cmake_test_compile/build
  --
  -------
  ------------
  -----------------
  ----------------------
  ---------------------------
  --------------------------------
  -- Trying 'Visual Studio 17 2022 x64 v143' generator - success
  --------------------------------------------------------------------------------

  Configuring Project
    Working directory:
      C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build
    Command:
      'C:\Users\wanyi\AppData\Local\Temp\pip-build-env-7s_z186d\overlay\Lib\site-packages\cmake\data\bin/cmake.exe' 'C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90' -G 'Visual Studio 17 2022' --no-warn-unused-cli '-DCMAKE_INSTALL_PREFIX:PATH=C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-install' -DPYTHON_VERSION_STRING:STRING=3.10.9 -DSKBUILD:INTERNAL=TRUE '-DCMAKE_MODULE_PATH:PATH=C:\Users\wanyi\AppData\Local\Temp\pip-build-env-7s_z186d\overlay\Lib\site-packages\skbuild\resources\cmake' '-DPYTHON_EXECUTABLE:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\python.exe' '-DPYTHON_INCLUDE_DIR:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\Include' '-DPYTHON_LIBRARY:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\libs\python310.lib' '-DPython_EXECUTABLE:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\python.exe' '-DPython_ROOT_DIR:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310' -DPython_FIND_REGISTRY:STRING=NEVER '-DPython_INCLUDE_DIR:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\Include' '-DPython_LIBRARY:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\libs\python310.lib' '-DPython3_EXECUTABLE:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\python.exe' '-DPython3_ROOT_DIR:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310' -DPython3_FIND_REGISTRY:STRING=NEVER '-DPython3_INCLUDE_DIR:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\Include' '-DPython3_LIBRARY:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\libs\python310.lib' -T v143 -A x64 -DCMAKE_BUILD_TYPE:STRING=Release

  Not searching for unused variables given on the command line.
  -- Selecting Windows SDK version 10.0.22000.0 to target Windows 10.0.22621.
  -- The C compiler identification is MSVC 19.37.32824.0
  -- The CXX compiler identification is MSVC 19.37.32824.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - failed
  -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe
  -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - broken
  CMake Error at C:/Users/wanyi/AppData/Local/Temp/pip-build-env-7s_z186d/overlay/Lib/site-packages/cmake/data/share/cmake-3.27/Modules/CMakeTestCCompiler.cmake:67 (message):
    The C compiler

      "C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe"

    is not able to compile a simple test program.

    It fails with the following output:

      Change Dir: 'C:/Users/wanyi/AppData/Local/Temp/pip-install-74r1bdtd/llama-cpp-python_5cf10169b309419697c9247200f9fe90/_skbuild/win-amd64-3.10/cmake-build/CMakeFiles/CMakeScratch/TryCompile-68uuwd'

      Run Build Command(s): "C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/MSBuild/Current/Bin/amd64/MSBuild.exe" cmTC_09e2c.vcxproj /p:Configuration=Debug /p:Platform=x64 /p:VisualStudioVersion=17.0 /v:n
      MSBuild version 17.7.2+d6990bcfa for .NET Framework
      Build started 2023-09-17 2:37:44 PM.

      Project "C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj" on node 1 (default targets).
      PrepareForBuild:
        Creating directory "cmTC_09e2c.dir\Debug\".
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(541,5): warning MSB8029: The Intermediate directory or Output directory cannot reside under the Temporary directory as it could lead to issues with incremental build. [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
        Creating directory "C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\Debug\".
        Creating directory "cmTC_09e2c.dir\Debug\cmTC_09e2c.tlog\".
      InitializeBuildStatus:
        Creating "cmTC_09e2c.dir\Debug\cmTC_09e2c.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
        Touching "cmTC_09e2c.dir\Debug\cmTC_09e2c.tlog\unsuccessfulbuild".
      ClCompile:
        C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\CL.exe /c /Zi /W1 /WX- /diagnostics:column /Od /Ob0 /D _MBCS /D WIN32 /D _WINDOWS /D "CMAKE_INTDIR=\"Debug\"" /Gm- /RTC1 /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"cmTC_09e2c.dir\Debug\\" /Fd"cmTC_09e2c.dir\Debug\vc143.pdb" /external:W1 /Gd /TC /errorReport:queue "C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\testCCompiler.c"
        Microsoft (R) C/C++ Optimizing Compiler Version 19.37.32824 for x64
        Copyright (C) Microsoft Corporation.  All rights reserved.
        testCCompiler.c
        cl /c /Zi /W1 /WX- /diagnostics:column /Od /Ob0 /D _MBCS /D WIN32 /D _WINDOWS /D "CMAKE_INTDIR=\"Debug\"" /Gm- /RTC1 /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"cmTC_09e2c.dir\Debug\\" /Fd"cmTC_09e2c.dir\Debug\vc143.pdb" /external:W1 /Gd /TC /errorReport:queue "C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\testCCompiler.c"
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003: The specified task executable "CL.exe" could not be run. System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.dir\Debug\cmTC_09e2c.tlog'. [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at System.IO.FileSystemEnumerableIterator`1.CommonInit() [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at System.IO.FileSystemEnumerableIterator`1..ctor(String path, String originalUserPath, String searchPattern, SearchOption searchOption, SearchResultHandler`1 resultHandler, Boolean checkHost) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at System.IO.Directory.GetFiles(String path, String searchPattern) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at Microsoft.Build.Utilities.TrackedDependencies.ExpandWildcards(ITaskItem[] expand) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at Microsoft.Build.Utilities.CanonicalTrackedOutputFiles.InternalConstruct(ITask ownerTask, ITaskItem[] tlogFiles, Boolean constructOutputsFromTLogs) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at Microsoft.Build.CPPTasks.CL.PostExecuteTool(Int32 exitCode) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at Microsoft.Build.CPPTasks.TrackedVCToolTask.ExecuteTool(String pathToTool, String responseFileCommands, String commandLineCommands) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at Microsoft.Build.Utilities.ToolTask.Execute() [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      Done Building Project "C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj" (default targets) -- FAILED.

      Build FAILED.

      "C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj" (default target) (1) ->
      (PrepareForBuild target) ->
        C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(541,5): warning MSB8029: The Intermediate directory or Output directory cannot reside under the Temporary directory as it could lead to issues with incremental build. [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]


      "C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj" (default target) (1) ->
      (ClCompile target) ->
        C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003: The specified task executable "CL.exe" could not be run. System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.dir\Debug\cmTC_09e2c.tlog'. [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at System.IO.FileSystemEnumerableIterator`1.CommonInit() [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at System.IO.FileSystemEnumerableIterator`1..ctor(String path, String originalUserPath, String searchPattern, SearchOption searchOption, SearchResultHandler`1 resultHandler, Boolean checkHost) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at System.IO.Directory.GetFiles(String path, String searchPattern) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at Microsoft.Build.Utilities.TrackedDependencies.ExpandWildcards(ITaskItem[] expand) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at Microsoft.Build.Utilities.CanonicalTrackedOutputFiles.InternalConstruct(ITask ownerTask, ITaskItem[] tlogFiles, Boolean constructOutputsFromTLogs) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at Microsoft.Build.CPPTasks.CL.PostExecuteTool(Int32 exitCode) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at Microsoft.Build.CPPTasks.TrackedVCToolTask.ExecuteTool(String pathToTool, String responseFileCommands, String commandLineCommands) [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]
      C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(741,5): error MSB6003:    at Microsoft.Build.Utilities.ToolTask.Execute() [C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build\CMakeFiles\CMakeScratch\TryCompile-68uuwd\cmTC_09e2c.vcxproj]

          1 Warning(s)
          1 Error(s)

      Time Elapsed 00:00:01.76





    CMake will not be able to correctly generate this project.
  Call Stack (most recent call first):
    CMakeLists.txt:3 (project)


  -- Configuring incomplete, errors occurred!
  Traceback (most recent call last):
    File "C:\Users\wanyi\AppData\Local\Temp\pip-build-env-7s_z186d\overlay\Lib\site-packages\skbuild\setuptools_wrap.py", line 666, in setup
      env = cmkr.configure(
    File "C:\Users\wanyi\AppData\Local\Temp\pip-build-env-7s_z186d\overlay\Lib\site-packages\skbuild\cmaker.py", line 357, in configure
      raise SKBuildError(msg)

  An error occurred while configuring with CMake.
    Command:
      'C:\Users\wanyi\AppData\Local\Temp\pip-build-env-7s_z186d\overlay\Lib\site-packages\cmake\data\bin/cmake.exe' 'C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90' -G 'Visual Studio 17 2022' --no-warn-unused-cli '-DCMAKE_INSTALL_PREFIX:PATH=C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-install' -DPYTHON_VERSION_STRING:STRING=3.10.9 -DSKBUILD:INTERNAL=TRUE '-DCMAKE_MODULE_PATH:PATH=C:\Users\wanyi\AppData\Local\Temp\pip-build-env-7s_z186d\overlay\Lib\site-packages\skbuild\resources\cmake' '-DPYTHON_EXECUTABLE:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\python.exe' '-DPYTHON_INCLUDE_DIR:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\Include' '-DPYTHON_LIBRARY:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\libs\python310.lib' '-DPython_EXECUTABLE:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\python.exe' '-DPython_ROOT_DIR:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310' -DPython_FIND_REGISTRY:STRING=NEVER '-DPython_INCLUDE_DIR:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\Include' '-DPython_LIBRARY:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\libs\python310.lib' '-DPython3_EXECUTABLE:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\python.exe' '-DPython3_ROOT_DIR:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310' -DPython3_FIND_REGISTRY:STRING=NEVER '-DPython3_INCLUDE_DIR:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\Include' '-DPython3_LIBRARY:PATH=C:\Users\wanyi\AppData\Local\Programs\Python\Python310\libs\python310.lib' -T v143 -A x64 -DCMAKE_BUILD_TYPE:STRING=Release
    Source directory:
      C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90
    Working directory:
      C:\Users\wanyi\AppData\Local\Temp\pip-install-74r1bdtd\llama-cpp-python_5cf10169b309419697c9247200f9fe90\_skbuild\win-amd64-3.10\cmake-build
  Please see CMake's output for more information.

  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for llama-cpp-python
Failed to build llama-cpp-python
ERROR: Could not build wheels for llama-cpp-python, which is required to install pyproject.toml-based projects`

Any help or suggestions would be highly appreciated!

Thanks!

How to run other models?

I want to run the model below that has a c++ version.
https://huggingface.co/anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g

The ggml-model-q4_1.bin model is already in the models folder. I execute the program by running
python llama.py --model_name="anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g" --file_name=="ggml-model-q4_1.bin"

It gives me the following error:

Traceback (most recent call last):
  File "D:\llama2_local\llama.py", line 114, in <module>
    fire.Fire(main)
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\fire\core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\fire\core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\fire\core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "D:\llama2_local\llama.py", line 110, in main
    model, tokenizer = init_auto_model_and_tokenizer(model_name, model_type, file_name)
  File "D:\llama2_local\llama.py", line 55, in init_auto_model_and_tokenizer
    model, tokenizer = initialize_gpu_model_and_tokenizer(model_name, model_type=model_type)
  File "D:\llama2_local\llama.py", line 40, in initialize_gpu_model_and_tokenizer
    model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", token=True)
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\models\auto\auto_factory.py", line 493, in from_pretrained
    return model_class.from_pretrained(
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\modeling_utils.py", line 2519, in from_pretrained
    resolved_archive_file = cached_file(pretrained_model_name_or_path, filename, **cached_file_kwargs)
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\utils\hub.py", line 417, in cached_file
    resolved_file = hf_hub_download(
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\huggingface_hub\utils\_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\huggingface_hub\file_download.py", line 1181, in hf_hub_download
    headers = build_hf_headers(
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\huggingface_hub\utils\_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\huggingface_hub\utils\_headers.py", line 121, in build_hf_headers
    token_to_send = get_token_to_send(token)
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\huggingface_hub\utils\_headers.py", line 153, in get_token_to_send
    raise LocalTokenNotFoundError(
huggingface_hub.utils._headers.LocalTokenNotFoundError: Token is required (`token=True`), but no token found. You need to provide a token or be logged in to Hugging Face with `huggingface-cli login` or `huggingface_hub.login`. See https://huggingface.co/settings/tokens.

I don't quite understand how it says that there are no tokens when I was able to run the 7B and 13B GGML models provided in the readme

CUDA extension not installed

I have been trying to run GPTQ models, but i'm getting the error below. Tried the following and nothing worked:

enabled developer mode on windows 10
installed cuda from nvdia webpage
ran conda install -c nvidia cuda

CUDA extension not installed.
Traceback (most recent call last):
  File "D:\llama2_local\llama.py", line 114, in <module>
    fire.Fire(main)
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\fire\core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\fire\core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\fire\core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "D:\llama2_local\llama.py", line 110, in main
    model, tokenizer = init_auto_model_and_tokenizer(model_name, model_type, file_name)
  File "D:\llama2_local\llama.py", line 55, in init_auto_model_and_tokenizer
    model, tokenizer = initialize_gpu_model_and_tokenizer(model_name, model_type=model_type)
  File "D:\llama2_local\llama.py", line 37, in initialize_gpu_model_and_tokenizer
    model = AutoGPTQForCausalLM.from_quantized(model_name, device_map="auto", use_safetensors=True, use_triton=False)
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\auto_gptq\modeling\auto.py", line 94, in from_quantized
    return quant_func(
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\auto_gptq\modeling\_base.py", line 749, in from_quantized
    make_quant(
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\auto_gptq\modeling\_utils.py", line 92, in make_quant
    make_quant(
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\auto_gptq\modeling\_utils.py", line 92, in make_quant
    make_quant(
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\auto_gptq\modeling\_utils.py", line 92, in make_quant
    make_quant(
  [Previous line repeated 1 more time]
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\auto_gptq\modeling\_utils.py", line 84, in make_quant
    new_layer = QuantLinear(
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\site-packages\auto_gptq\nn_modules\qlinear\qlinear_cuda_old.py", line 83, in __init__
    self.autogptq_cuda = autogptq_cuda_256
NameError: name 'autogptq_cuda_256' is not defined

ValueError: Need either a `state_dict` or a `save_folder` containing offloaded weights.

python llama.py --model_name="meta-llama/Llama-2-7b-hf"

Downloading (…)lve/main/config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 609/609 [00:00<00:00, 1.98MB/s]
Downloading (…)fetensors.index.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 26.8k/26.8k [00:00<00:00, 56.4MB/s]
Downloading (…)of-00002.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.98G/9.98G [07:43<00:00, 21.5MB/s]
Downloading (…)of-00002.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.50G/3.50G [02:43<00:00, 21.4MB/s]
Downloading shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [10:29<00:00, 314.61s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:07<00:00, 3.75s/it]
Downloading (…)neration_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 167/167 [00:00<00:00, 756kB/s]
Traceback (most recent call last):
File "/home/prameela/PycharmProjects/llama_local/llama2_local/llama.py", line 114, in
fire.Fire(main)
File "/home/prameela/anaconda3/envs/llama_local/lib/python3.11/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/prameela/anaconda3/envs/llama_local/lib/python3.11/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/home/prameela/anaconda3/envs/llama_local/lib/python3.11/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/prameela/PycharmProjects/llama_local/llama2_local/llama.py", line 110, in main
model, tokenizer = init_auto_model_and_tokenizer(model_name, model_type, file_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/prameela/PycharmProjects/llama_local/llama2_local/llama.py", line 55, in init_auto_model_and_tokenizer
model, tokenizer = initialize_gpu_model_and_tokenizer(model_name, model_type=model_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/prameela/PycharmProjects/llama_local/llama2_local/llama.py", line 40, in initialize_gpu_model_and_tokenizer
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", token=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/prameela/anaconda3/envs/llama_local/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 493, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/prameela/anaconda3/envs/llama_local/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2959, in from_pretrained
dispatch_model(model, **kwargs)
File "/home/prameela/anaconda3/envs/llama_local/lib/python3.11/site-packages/accelerate/big_modeling.py", line 364, in dispatch_model
weights_map = OffloadedWeightsLoader(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/prameela/anaconda3/envs/llama_local/lib/python3.11/site-packages/accelerate/utils/offload.py", line 150, in init
raise ValueError("Need either a state_dict or a save_folder containing offloaded weights.")
ValueError: Need either a state_dict or a save_folder containing offloaded weights.

ModuleNotFoundError: No module named 'gradio'

does anyone get this error? I searched for everything and didn't help. No errors installing requirements (pip install gradio shows it already is). Using python3 instead of python command doesn't help. Python version is 3.11.4.

Necessary prerequisites to compiling llama

here's what i did to make the compilation successful. please add the following to the readme/instructions/guide:

CMake version cmake-3.27.0-windows-x86_64.msi installed to root directory ("C:")
minGW64 version 11.0.0 extracted to root directory ("C:")
set environment path variables for CMake and minGW64
install visual studio build tools. It's way at the bottom under "Tools for Visual Studio" drop down list.
In visual studio, check the "Desktop development with c++", click install.

Then you can compile successfully.

Run after tunnelling

I ran the code according to the instructions on a Ubuntu 22.04.2 LTS-based workstation. Then I created a ssh tunnel from my MacBook to this workstation using the command sudo ssh -N -L 7860:127.0.0.1:7860 user@workstation_ip.

The GUI opens up in my browser on mac, but when I enter the prompt, I get the following error:

Expecting value: line 1 column 1 (char 0)

Can you please help what am I doing wrong?

Always loads the same model

llama.py always loads the model llama-2-7b-chat.ggmlv3.q4_K_M.bin even if I change models.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble