microsoft / jarvis Goto Github PK
View Code? Open in Web Editor NEWJARVIS, a system to connect LLMs with ML community. Paper: https://arxiv.org/pdf/2303.17580.pdf
License: MIT License
JARVIS, a system to connect LLMs with ML community. Paper: https://arxiv.org/pdf/2303.17580.pdf
License: MIT License
While trying to run the models server I run into the following error:
raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
ValueError: Could not load model models/openai/whisper-base with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCTC'>, <class 'transformers.models.auto.modeling_auto.AutoModelForSpeechSeq2Seq'>, <class 'transformers.models.whisper.modeling_whisper.WhisperForConditionalGeneration'>).
I tried redoing all the installation steps, removing and recreating the conda env, and making sure all python dependencies are installed and both api tokens configured. What could be going wrong?
I have quadruple checked my openai api key is correct. I copied and pasted it to avoid typo, I even generated a new key after many attempts/fails. However, no matter what key I generate and write to config.yaml, when I run awesome_chat.py it insists that I have an incorrect openai key.
I also tried running npm run dev and the browser opens and I see the HuggingGPT page and text box, but anytime I submit any prompt, system returns "Something seems seems wrong". Interestingly, localhost is opened to port 9999 when running npm run dev, but in config.yaml it has localhost port set to 8003.
What is it I am doing wrong? I feel like I am so close, but so far. :)
Can Nvidia's 40 series graphics card be used for the current project?
as I can see that the System Requirements is as follows:
Ubuntu 16.04 LTS
NVIDIA GeForce RTX 3090 * 1
RAM > 24GB
Hi,
attempting to install this on windows, had a few issues with installing pytorch at the start but after that i got this:
Traceback (most recent call last):
File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\modeling_utils.py", line 98, in load_state_dict
return safetensors.torch.load_file(checkpoint_file, device="cpu")
File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\safetensors\torch.py", line 100, in load_file
result[k] = f.get_tensor(k)
RuntimeError: shape '[640, 2560]' is invalid for input of size 798399
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\modeling_utils.py", line 102, in load_state_dict
if f.read().startswith("version"):
File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 40678: character maps to <undefined>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\AI\JARVIS\server\models_server.py", line 338, in <module>
pipes = load_pipes(local_deployment)
File "D:\AI\JARVIS\server\models_server.py", line 315, in load_pipes
"control": ControlNetModel.from_pretrained(f"{local_fold}/lllyasviel/sd-controlnet-mlsd", torch_dtype=torch.float16),
File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\modeling_utils.py", line 557, in from_pretrained
state_dict = load_state_dict(model_file, variant=variant)
File "C:\Users\PC\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\modeling_utils.py", line 114, in load_state_dict
raise OSError(
OSError: Unable to load weights from checkpoint file for 'models/lllyasviel/sd-controlnet-mlsd\diffusion_pytorch_model.safetensors' at 'models/lllyasviel/sd-controlnet-mlsd\diffusion_pytorch_model.safetensors'. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.
any ideas?
I downloaded all the models (28 models in 431GB) on my PC.
And I have hybrid/minimal mode running successfully.
But I can't have JARVIS running in local/full mode. (I do have 128GB RAM)
I got a error message saying that it can't load the file that exists actually. And I checked the file read permission with the process running user no problem.
have not a >48g gpu....
weird feeling that is a joke, looking forward for the code
麻烦问一下各位,现在项目完善好了吗?有一块3090,16GB运存,不知道能带的动不?
2023-04-05 20:33:46,053 - awesome_chat - INFO - input:
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/routes.py", line 393, in run_predict
output = await app.get_blocks().process_api(
File "/home/ec2-user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 1108, in process_api
result = await self.call_function(
File "/home/ec2-user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 915, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/ec2-user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/home/ec2-user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/home/ec2-user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "run_gradio_demo.py", line 81, in bot
message = chat_huggingface(all_messages, OPENAI_KEY)
File "/home/ec2-user/SageMaker/JARVIS/server/awesome_chat.py", line 852, in chat_huggingface
task_str = parse_task(context, input, openaikey).strip()
File "/home/ec2-user/SageMaker/JARVIS/server/awesome_chat.py", line 306, in parse_task
return send_request(data)
File "/home/ec2-user/SageMaker/JARVIS/server/awesome_chat.py", line 173, in send_request
return response.json()["choices"][0]["text"].strip()
KeyError: 'choices'
stanford_alpaca是开源模型,可以保证在openai服务不可用的地方依然可以使用到HuggingGPT
Hello Microsoft,
Congratulations on turning the project into working code, looking forward to contributing to this project. This architecture with enough training can possibly be the way to AGI.
There are a few concerns however. The name JARVIS
appears to be a registered trademark under Marvel Entertainment Studios under Class 9 for computer application software for use as personal digital assistants. [reference. 1]. As well as Blackberry has a history of similar trademark under their digital assistant application categories.
python models_server.py --config config.yaml
Traceback (most recent call last):
File "models_server.py", line 23, in
import torchaudio
File "/Users/qinxiaoqiang/miniconda3/envs/jarvis/lib/python3.8/site-packages/torchaudio/init.py", line 1, in
from torchaudio import ( # noqa: F401
File "/Users/qinxiaoqiang/miniconda3/envs/jarvis/lib/python3.8/site-packages/torchaudio/_extension/init.py", line 5, in
from torchaudio._internal.module_utils import fail_with_message, is_module_available, no_op
ImportError: cannot import name 'fail_with_message' from 'torchaudio._internal.module_utils'
config:
inference_mode: huggingface
local_deployment: minimal
If I run command:
python3 awesome_chat.py --config config.yaml --mode cli
I got these results:
[ User ]: show me a joke and a image about cat
[ Jarvis ]: I understand your request. Firstly, I used the GPT-2 model to generate a joke for you. Unfortunately, there was an inference error and I could not generate a joke. Secondly, I used the Google/VIT-Base-Patch16-224 model to classify an image of a cat. However, there was an error in the path or URL, so I could not generate an image. Lastly, I used the PromptHero/OpenJourney model to generate an image of a cat. Unfortunately, there was an error in the path or URL, so I could not generate an image. I apologize for not being able to fulfill your request. Is there anything else I can help you with?
Always got error message like this
Traceback (most recent call last):
File "/JARVIS/server/awesome_chat.py", line 570, in model_inference
inference_result = huggingface_model_inference(model_id, data, task)
File "/JARVIS/server/awesome_chat.py", line 411, in huggingface_model_inference
img_data = image_to_bytes(img_url)
File "/JARVIS/server/awesome_chat.py", line 213, in image_to_bytes
load_image(img_url).save(img_byte, format="png")
File "/anaconda3/envs/jarvis/lib/python3.8/site-packages/diffusers/utils/testing_utils.py", line 268, in load_image
raise ValueError(
ValueError: Incorrect path or url, URLs must start with `http://` or `https://`, and public/<GENERATED>-1 is not a valid path
2023-04-04 18:18:56,609 - __mp_main__ - WARNING - Inference error: {'message': 'Incorrect path or url, URLs must start with `http://` or `https://`, and public/<GENERATED>-1 is not a valid path'}
2023-04-04 18:18:58,304 - __mp_main__ - WARNING - Inference error: unknown error
or error file path like this
2023-04-04 18:18:58,304 - __mp_main__ - WARNING - Inference error: unknown error
[Errno 2] No such file or directory: 'public/images/1788.png'
Traceback (most recent call last):
File "/JARVIS/server/awesome_chat.py", line 570, in model_inference
inference_result = huggingface_model_inference(model_id, data, task)
File "/JARVIS/server/awesome_chat.py", line 346, in huggingface_model_inference
img.save(f"public/images/{name}.png")
File "/anaconda3/envs/jarvis/lib/python3.8/site-packages/PIL/Image.py", line 2429, in save
fp = builtins.open(filename, "w+b")
FileNotFoundError: [Errno 2] No such file or directory: 'public/images/1788.png'
2023-04-04 18:19:02,531 - __mp_main__ - WARNING - Inference error: {'message': "[Errno 2] No such file or directory: 'public/images/1788.png'"}
Hi,
I don't want to use it with OpenAI but I prefer GPT4ALL. Do you have any recommendations about how to do it?
Encounter an error:
Error:OSError: You seem to have cloned a repository without having git-lfs installed. Please install git-lfs and run `git lfs install` followed by `git lfs pull` in the folder you cloned.
How to solve this problem when running
python models_server.py --config config.yaml
Please include dockerfile or other isolated OS environment for improved building. If I get time this or next weekend, I'll try to create one and open a PR, otherwise please tag me if one gets created. Thanks!
When I run the models_server.py in aws , OSError: [Errno 99] Cannot assign requested address.
How can I deploy the service on the cloud server and I download all model in cloud .
And if i set config is hugging face run the server , "However, the inference result was an error, as the service related to ControlNet is not available."
(jarvis) [root@myflink server]# python models_server.py --config config.yaml
Traceback (most recent call last):
File "models_server.py", line 10, in
from diffusers.utils import export_to_video
ImportError: cannot import name 'export_to_video' from 'diffusers.utils' (/root/anaconda3/envs/jarvis/lib/python3.8/site-packages/diffusers/utils/init.py)
I strongly believe that JARVIS deserves its name, and here's why:
Just Analyzing Responses: The core functionality of JARVIS lies in its ability to analyze responses generated by ChatGPT. The library efficiently handles user queries, dives deep into the knowledge base, and makes sense of the generated responses. This simplifies the overall process of interacting with ChatGPT and allows developers to focus on building creative applications.
Verifying Input: JARVIS ensures that the input provided to the language models is processed and verified. It checks for any potential errors or inconsistencies before sending the data to the models. This feature is crucial for maintaining the quality and accuracy of the output generated by ChatGPT.
Synthesizing: The most notable aspect of JARVIS is its ability to synthesize information. It can combine outputs from multiple language models and generate coherent, contextually relevant responses. This makes JARVIS an ideal tool for developers looking to create advanced natural language processing applications.
By harnessing the power of both ChatGPT and HuggingFace, JARVIS provides an unparalleled foundation for developers to build cutting-edge NLP solutions. The library is designed with a focus on simplicity, efficiency, and robustness, making it an ideal choice for those who wish to create versatile AI-driven applications.
Furthermore, JARVIS is open-source and designed to be accessible to developers of all skill levels. We encourage you to contribute to the project, share your ideas, and help improve the library, making it even more powerful and versatile.
Server is up an running. Using gradio python run_gradio_demo.py --config config.gradio.yaml
:
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Running built-in example show me a joke and an image of cat
. Gradio terminal output:
2023-04-06 15:45:43,407 - awesome_chat - INFO - ********************************************************************************
2023-04-06 15:45:43,407 - awesome_chat - INFO - input: show me a joke and an image of cat
2023-04-06 15:45:45,739 - awesome_chat - INFO - [{"task": "conversational", "id": 0, "dep": [-1], "args": {"text": "please show me a joke of cat" }}, {"task": "text-to-image", "id": 1, "dep": [-1], "args": {"text": "a photo of cat" }}]
Traceback (most recent call last):
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/routes.py", line 393, in run_predict
output = await app.get_blocks().process_api(
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 1108, in process_api
result = await self.call_function(
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 915, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "run_gradio_demo.py", line 81, in bot
message = chat_huggingface(all_messages, OPENAI_KEY)
File "/home/user/Developer/JARVIS/server/awesome_chat.py", line 908, in chat_huggingface
response = response_results(input, results, openaikey).strip()
File "/home/user/Developer/JARVIS/server/awesome_chat.py", line 342, in response_results
messages = json.loads(demos_or_presteps)
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid \escape: line 8 column 398 (char 527)
Hey, so I have a game built from HTML/Javascript. I would like to see how well JARVIS could try to understand and then modify the existing game. Make changes... re-run the game... play it for finding bugs, etc.
For this it will need to learn how to press buttons, type text into fields, and
How might this be achieved? What hugging face AI's would I have to pull in?
据以往经验,为chatgpt这种在国内访问已被拒的生态组件做项目,如果做得好,都会受到审查牵连,影响后面的商业化进程,不是能翻就ok的,望重视早日落地成功
(jarvis) m200@DESKTOP-K0LFJU8:~/workdir/JARVIS/web$ npm run dev
> [email protected] dev
> vite
file:///home/m200/workdir/JARVIS/web/node_modules/vite/bin/vite.js:7
await import('source-map-support').then((r) => r.default.install())
^^^^^
SyntaxError: Unexpected reserved word
at Loader.moduleStrategy (internal/modules/esm/translators.js:133:18)
at async link (internal/modules/esm/module_job.js:42:21)
Any idea?
Hi, I only saw OpenAI API key in config.yml. Does it support Azure OpenAI? Or is there any plans to support Azure OpenAI?
Add a project logo: Including a project logo or image can help make the README file more visually appealing and memorable.
1.Add more usage examples: While the existing usage examples are helpful, consider adding more examples to showcase the full capabilities of the project.
OS: macOS 11.5.1 (20G80)
Hardware: MacBook Pro (Retina, 15-inch, Mid 2015)
active environment : jarvis
active env location : /usr/local/anaconda3/envs/jarvis
shell level : 1
user config file : /Users/bruce/.condarc
populated config files :
conda version : 23.1.0
conda-build version : 3.23.3
python version : 3.10.9.final.0
virtual packages : __archspec=1=x86_64
__osx=10.16=0
__unix=0=0
base environment : /usr/local/anaconda3 (writable)
conda av data dir : /usr/local/anaconda3/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/osx-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /usr/local/anaconda3/pkgs
/Users/bruce/.conda/pkgs
envs directories : /usr/local/anaconda3/envs
/Users/bruce/.conda/envs
platform : osx-64
user-agent : conda/23.1.0 requests/2.28.1 CPython/3.10.9 Darwin/20.6.0 OSX/10.16
UID:GID : 501:20
netrc file : None
offline mode : False
empty, no any config
# packages in environment at /usr/local/anaconda3/envs/jarvis:
#
# Name Version Build Channel
ca-certificates 2023.01.10 hecd8cb5_0 defaults
certifi 2022.12.7 py38hecd8cb5_0 defaults
libcxx 14.0.6 h9765a3e_0 defaults
libffi 3.4.2 hecd8cb5_6 defaults
ncurses 6.4 hcec6c5f_0 defaults
openssl 1.1.1t hca72f7f_0 defaults
pip 23.0.1 py38hecd8cb5_0 defaults
python 3.8.16 h218abb5_3 defaults
readline 8.2 hca72f7f_0 defaults
setuptools 65.6.3 py38hecd8cb5_0 defaults
sqlite 3.41.1 h6c40b1e_0 defaults
tk 8.6.12 h5d9f67b_0 defaults
wheel 0.38.4 py38hecd8cb5_0 defaults
xz 5.2.10 h6c40b1e_1 defaults
zlib 1.2.13 h4dc903c_0 defaults
Run the following command
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
Then it shows the following error
> conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: -
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
Examining conflict for torchvision pytorch: 38%|███████████████ | 3/8 [00:04<00:07, 1.48s//failed
UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:
Specifications:
- torchaudio -> python[version='>=2.7,<2.8.0a0|>=3.11,<3.12.0a0']
Your python: python=3.8
If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.
The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versions
Package setuptools conflicts for:
pytorch -> jinja2 -> setuptools
torchvision -> setuptools
python=3.8 -> pip -> setuptools
Package pytorch conflicts for:
torchaudio -> pytorch[version='1.10.0|1.10.1|1.10.2|1.11.0|1.12.0|1.12.1|1.13.0|1.13.1|2.0.0|1.9.1|1.9.0|1.8.1|1.8.0|1.7.1|1.7.0|1.6.0|1.5.1']
torchvision -> pytorch[version='1.10.0|1.10.1|1.10.2|1.11.0|1.12.0|1.12.1|1.13.0|1.13.1|2.0.0|1.9.1|1.9.0|1.8.1|1.8.0|1.7.1|1.7.0|1.6.0|1.5.1|1.7.1.*|1.3.1.*']
Package six conflicts for:
pytorch -> mkl-service[version='>=2.3.0,<3.0a0'] -> six
torchvision -> six
Package libcxxabi conflicts for:
pytorch -> libcxx[version='>=4.0.1'] -> libcxxabi==4.0.1[build='hebd6815_0|hcfea43d_1']
python=3.8 -> libcxx[version='>=4.0.1'] -> libcxxabi==4.0.1[build='hebd6815_0|hcfea43d_1']
Is it possible to integrate the functionality of that project?
https://github.com/facebookresearch/segment-anything
Macs are not using NVIDIA display card, so Mac can not use this right?
On windows 11
16 GB RAM
12th Gen Intel(R) Core(TM) i7-12700F
NVIDIA GeForce RTx3070
Command: python awesome_chat.py --config config.yaml --mode cli
Error:
File ".\JARVIS\server\awesome_chat.py", line 17, in
from diffusers.utils import load_image
ModuleNotFoundError: No module named 'diffusers'
I am getting this error when using the cli:
Traceback (most recent call last):
File "awesome_chat.py", line 970, in
cli()
File "awesome_chat.py", line 941, in cli
answer = chat_huggingface(messages)
File "awesome_chat.py", line 850, in chat_huggingface
task_str = parse_task(context, input).strip()
File "awesome_chat.py", line 297, in parse_task
return send_request(data)
File "awesome_chat.py", line 167, in send_request
return response.json()["choices"][0]["text"].strip()
KeyError: 'choices'
(jarvis)
My prompt was this:
[ User ]: based on this image, assets\example_image.JPG , please tell me what you see.
Server is up an running.
Commit SHA - bc66e5a
Using gradio python run_gradio_demo.py --config config.gradio.yaml
:
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Running built-in example based on the /examples/a.jpg, please generate a video and audio
. Gradio terminal output:
2023-04-06 16:33:43,346 - awesome_chat - INFO - ********************************************************************************
2023-04-06 16:33:43,352 - awesome_chat - INFO - input: based on the /examples/a.jpg, please generate a video and audio
2023-04-06 16:33:47,719 - awesome_chat - INFO - [{"task": "image-to-text", "id": 0, "dep": [-1], "args": {"image": "/examples/a.jpg" }}, {"task": "text-to-video", "id": 1, "dep": [0], "args": {"text": "<GENERATED>-0" }}, {"task": "text-to-speech", "id": 2, "dep": [0], "args": {"text": "<GENERATED>-0" }}]
2023-04-06 16:34:36,874 - awesome_chat - INFO - response: I have carefully considered your request and I have generated a video and audio based on the image you provided. The workflow I used is as follows: First, I used the image-to-text model 'nlpconnect/vit-gpt2-image-captioning' to generate the text 'a cat sitting on a window sill looking out'. Then, I used the text-to-video model 'damo-vilab/text-to-video-ms-1.7b' to generate the video '/videos/293f.mp4'. Finally, I used the text-to-speech model 'facebook/fastspeech2-en-ljspeech' to generate the audio '/audios/19fc.flac'. The complete path of the video and audio are '/videos/293f.mp4' and '/audios/19fc.flac' respectively. I hope this answer your request. Is there anything else I can help you with?
Traceback (most recent call last):
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/routes.py", line 393, in run_predict
output = await app.get_blocks().process_api(
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 1111, in process_api
data = self.postprocess_data(fn_index, result["prediction"], state)
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 1045, in postprocess_data
prediction_value = block.postprocess(prediction_value)
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/components.py", line 4333, in postprocess
self._postprocess_chat_messages(message_pair[1]),
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/components.py", line 4296, in _postprocess_chat_messages
filepath = self.make_temp_copy_if_needed(filepath)
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/components.py", line 245, in make_temp_copy_if_needed
temp_dir = self.hash_file(file_path)
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/components.py", line 217, in hash_file
with open(file_path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'public//videos/293f.mp4'
Hi guys, when I'm running python models_server.py --config config.yaml I get this error:
raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
ValueError: Could not load model models/openai/whisper-base with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCTC'>, <class 'transformers.models.auto.modeling_auto.AutoModelForSpeechSeq2Seq'>, <class 'transformers.models.whisper.modeling_whisper.WhisperForConditionalGeneration'>).
I don't quite understand the requirement being rtx 3090.
Is it the vram that matters?
Server is up an running. Using gradio python run_gradio_demo.py --config config.gradio.yaml
:
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Running built-in example Given a collection of image A: /examples/a.jpg, B: /examples/b.jpg, C: /examples/c.jpg, please tell me how many zebras in these picture?
. Gradio terminal output:
2023-04-06 15:41:17,757 - awesome_chat - INFO - ********************************************************************************
2023-04-06 15:41:17,758 - awesome_chat - INFO - input: Given a collection of image A: /examples/a.jpg, B: /examples/b.jpg, C: /examples/c.jpg, please tell me how many zebras in these picture?
2023-04-06 15:41:30,058 - awesome_chat - INFO - [{"task": "image-to-text", "id": 0, "dep": [-1], "args": {"image": "/examples/a.jpg" }}, {"task": "object-detection", "id": 1, "dep": [-1], "args": {"image": "/examples/a.jpg" }}, {"task": "visual-question-answering", "id": 2, "dep": [1], "args": {"image": "<GENERATED>-1", "text": "How many zebras in the picture?" }}, {"task": "image-to-text", "id": 3, "dep": [-1], "args": {"image": "/examples/b.jpg" }}, {"task": "object-detection", "id": 4, "dep": [-1], "args": {"image": "/examples/b.jpg" }}, {"task": "visual-question-answering", "id": 5, "dep": [4], "args": {"image": "<GENERATED>-4", "text": "How many zebras in the picture?" }}, {"task": "image-to-text", "id": 6, "dep": [-1], "args": {"image": "/examples/c.jpg" }}, {"task": "object-detection", "id": 7, "dep": [-1], "args": {"image": "/examples/c.jpg" }}, {"task": "visual-question-answering", "id": 8, "dep": [7], "args": {"image": "<GENERATED>-7", "text": "How many zebras in the picture?" }}]
2023-04-06 15:41:50,233 - awesome_chat - INFO - response: Based on the inference results, there are two zebras in the picture.
My workflow for your request is as follows:
1. I used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate a text description for each image.
2. Then I used the object-detection model facebook/detr-resnet-50 to detect the objects in the image and generate an image with predicted boxes.
3. Finally, I used the visual-question-answering model dandelin/vilt-b32-finetuned-vqa to answer your question.
For the image A: /examples/a.jpg, the object-detection model detected a cat and a potted plant in the image. The visual-question-answering model predicted that there are 0 zebras in the picture.
For the image B: /examples/b.jpg, the object-detection model detected a zebra in the image. The visual-question-answering model predicted that there is 1 zebra in the picture.
For the image C: /examples/c.jpg, the object-detection model detected three zebras in the image. The visual-question-answering model predicted that there are 2 zebras in the picture.
Therefore, there are two zebras in the picture.
Traceback (most recent call last):
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/routes.py", line 393, in run_predict
output = await app.get_blocks().process_api(
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 1108, in process_api
result = await self.call_function(
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/gradio/blocks.py", line 915, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "run_gradio_demo.py", line 82, in bot
image_urls, audio_urls, video_urls = extract_medias(message)
File "run_gradio_demo.py", line 18, in extract_medias
for match in image_pattern.finditer(message):
TypeError: expected string or bytes-like object
Same error is thrown in the following built-in examples:
Please generate a canny image based on /examples/f.jpg
what is in the examples/a.jpg
generate a video and audio about a dog is running on the grass
based on the /examples/a.jpg, please generate a video and audio
based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image
When running:
python models_server.py --config config.yaml
I get the following error:
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/envs/jarvis/lib/python3.8/site-packages/transformers/modeling_utils.py", line 415, in load_state_dict
return torch.load(checkpoint_file, map_location="cpu")
File "/home/ubuntu/miniconda3/envs/jarvis/lib/python3.8/site-packages/torch/serialization.py", line 795, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/ubuntu/miniconda3/envs/jarvis/lib/python3.8/site-packages/torch/serialization.py", line 1002, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.
Clean install. The server is running.
Ubuntu 22.04
Firefox/Chrome
cd web/
npm run dev
Result of submitting any prompt:
With the following in the console:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8004/hugginggpt. (Reason: CORS request did not succeed). Status code: (null).
In awesome_chat.py
, you implement the inference method huggingface_model_inference
(that uses the Hugging Face Hub under the hood) using requests
and custom code. One thing you can do to deduplicate some code is to use huggingface_hub
library and in particular its InferenceAPI
class. You can find a guide here on how to use it.
For example, you can replace:
if task == "text-to-image":
text = data["text"]
response = requests.post(task_url, headers=HUGGINGFACE_HEADERS, json={"inputs": text})
img_data = response.content
img = Image.open(BytesIO(img_data))
by
if task == "text-to-image":
img = InferenceApi(repo_id=model_id, task=task, token=token)(inputs=data["text"])
Some advantages of using huggingface_hub
are:
huggingface-cli login
, the token is automatically retrievedDisclaimer: I am a maintainer of huggingface_hub
. If you need any help with the integration, I'd be happy to help! 😃
python models_server.py --config config.yaml
Fetching 27 files: 100%|██████████████████████████████████████████████████████████████████████████████| 27/27 [00:00<00:00, 338048.38it/s]
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
WARNING:datasets.builder:Found cached dataset cmu-arctic-xvectors (/home/user/.cache/huggingface/datasets/cmu-arctic-xvectors/default/0.0.1/a62fea1f9415e240301ea0042ffad2a3aadf4d1caa7f9a8d9512d631723e781f)
WARNING:datasets.builder:Found cached dataset cmu-arctic-xvectors (/home/user/.cache/huggingface/datasets/cmu-arctic-xvectors/default/0.0.1/a62fea1f9415e240301ea0042ffad2a3aadf4d1caa7f9a8d9512d631723e781f)
Some weights of DPTForDepthEstimation were not initialized from the model checkpoint at models/Intel/dpt-large and are newly initialized: ['neck.fusion_stage.layers.0.residual_layer1.convolution1.weight', 'neck.fusion_stage.layers.0.residual_layer1.convolution2.bias', 'neck.fusion_stage.layers.0.residual_layer1.convolution2.weight', 'neck.fusion_stage.layers.0.residual_layer1.convolution1.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Traceback (most recent call last):
File "models_server.py", line 342, in <module>
pipes = load_pipes(local_deployment)
File "models_server.py", line 261, in load_pipes
"model": pipeline(task="image-to-text", model=f"{local_fold}/ydshieh/vit-gpt2-coco-en"),
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/transformers/pipelines/__init__.py", line 779, in pipeline
framework, model = infer_framework_load_model(
File "/home/user/anaconda3/envs/jarvis/lib/python3.8/site-packages/transformers/pipelines/base.py", line 271, in infer_framework_load_model
raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
ValueError: Could not load model models/ydshieh/vit-gpt2-coco-en with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForVision2Seq'>, <class 'transformers.models.vision_encoder_decoder.modeling_vision_encoder_decoder.VisionEncoderDecoderModel'>).
pip install controlnet-aux==0.0.1
requirement.txt 前3项如果出现安装问题,可以下载后,copy到javis环境目录
第三个,直接使用上面的pip安装。
Why isn't there a contributing.md file to help give guidance on how or even if we can contribute to this project?
I think adding one would help give guidance on how or if we can contribute to this project.
hi,dear
as the title,if i don't have a openai key ,could i use the rp ?
thx
Hi there,
Love this idea and would love to try and contribute if possible.
Now I'm deploying it on Colab but failed a few times with the error "not enough hardware space".
I'm seeing there're close to 200G space but still running out of space in the end which is not consistant with the description of 80G required for full local model.
Please let me know if you have any questions.
Thanks,
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.