haonanguo / remote-sensing-chatgpt Goto Github PK

I didn't see it in the repository.

I think it would be great if all the required weights get downloaded when running for the very first time. Downloading weights individually feels boring.

Hi HaonanGo
Thank you so much for sharing your work.
I'm bit rookie on this openai running scripts. So forgive me if I may be spamming around here. Not my intent.
I was trying to run your code by executing python rs_chatgpt.py 'my opeanai key' as suggeste to me by chatGPT. But the errors I got led me to change (accordding chatgpt help) in file rs_chatgpt.pt the line 593 from:
bot = RSChatGPT(gpt_name=args.gpt_name,load_dict=load_dict,openai_key=args.openai_key)
to
bot = RSChatGPT(gpt_name=args.gpt_name, load_dict=load_dict, openai_key=getattr(args, 'openai-key'))

and it run something with a subsequent error:

Initializing ImageCaptioning to cuda:0
preprocessor_config.json: 100%|████████████████████████████████████████████| 287/287 [00:00<00:00, 1.00MB/s]
tokenizer_config.json: 100%|███████████████████████████████████████████████| 506/506 [00:00<00:00, 1.97MB/s]
vocab.txt: 100%|█████████████████████████████████████████████████████████| 232k/232k [00:00<00:00, 1.85MB/s]
tokenizer.json: 100%|████████████████████████████████████████████████████| 711k/711k [00:00<00:00, 1.90MB/s]
special_tokens_map.json: 100%|██████████████████████████████████████████████| 125/125 [00:00<00:00, 529kB/s]
config.json: 100%|█████████████████████████████████████████████████████| 4.56k/4.56k [00:00<00:00, 14.7MB/s]
pytorch_model.bin: 100%|█████████████████████████████████████████████████| 990M/990M [01:26<00:00, 11.4MB/s]
Initializing SceneClassification
/home/ibslab/miniconda3/envs/RSChatGPT/lib/python3.11/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.

Traceback (most recent call last):
File "/home/ibslab/PEDRO/Remote-Sensing-ChatGPT/rs_chatgpt.py", line 594, in
bot = RSChatGPT(gpt_name=args.gpt_name, load_dict=load_dict, openai_key=getattr(args, 'openai-key'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ibslab/PEDRO/Remote-Sensing-ChatGPT/rs_chatgpt.py", line 506, in init
self.models[class_name] = globals()class_name
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ibslab/PEDRO/Remote-Sensing-ChatGPT/rs_chatgpt.py", line 209, in init
trained = torch.load('./checkpoints/Res34_AID_best.pth')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ibslab/miniconda3/envs/RSChatGPT/lib/python3.11/site-packages/torch/serialization.py", line 986, in load
with _open_file_like(f, 'rb') as opened_file:
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ibslab/miniconda3/envs/RSChatGPT/lib/python3.11/site-packages/torch/serialization.py", line 435, in _open_file_like
return _open_file(name_or_buffer, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ibslab/miniconda3/envs/RSChatGPT/lib/python3.11/site-packages/torch/serialization.py", line 416, in init
super().init(open(name, mode))
^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './checkpoints/Res34_AID_best.pth'

Then I thought it might be related to not tunning interface.py code, where I also had to change a similar line of code to parse my openai key:
bot = RSChatGPT(load_dict=load_dict, openai_key=getattr(args, 'openai-key'))

and same type of error related to missing file:
FileNotFoundError: [Errno 2] No such file or directory: './checkpoints/Res34_AID_best.pth'

what I'm doing wrong, using the initial code as it is it didn't work, i got:

Traceback (most recent call last):
File "/home/ibslab/PEDRO/Remote-Sensing-ChatGPT/interface.py", line 600, in
bot = RSChatGPT(load_dict=load_dict,openai_key=args.openai_key)
^^^^^^^^^^^^^^^
AttributeError: 'Namespace' object has no attribute 'openai_key'. Did you mean: 'openai-key'?

Requirement already satisfied: mypy-extensions>=0.3.0 in ./.env/lib/python3.12/site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain-community) (1.0.0)

But when running python RSChatGPT-shell.py ==> Alert :
1 - from langchain_community.chat_models import ChatOpenAI. To install langchain-community run pip install -U langchain-community
2- import pkg_resources as pkg ModuleNotFoundError: No module named pkg_resources

root@autodl-container-7e0c4db22a-9f3ec4ab:/RSGPT# python interface.py
usage: interface.py [-h] [--load LOAD] openai-key
interface.py: error: the following arguments are required: openai-key
root@autodl-container-7e0c4db22a-9f3ec4ab:/RSGPT# python rs_chatgpt.py
usage: rs_chatgpt.py [-h] [--image-dir IMAGE_DIR] [--gpt_name {gpt-3.5-turbo,gpt-4}] [--load LOAD] openai-key
rs_chatgpt.py: error: the following arguments are required: openai-key
作者，对您的研究非常感兴趣，想问下您运行过程中出现的openai-key报错要怎么解决呢？谢谢啦

Hi - this is awesome. It looks like the object detection weights on GDrive are from HRNet. Is that intentional, or should these be yolov5?

InstanceSegmentation 里的标签顺序不对，应该是
{'ship': 1, 'storage tank': 2, 'baseball diamond': 3, 'tennis court': 4, 'basketball court': 5, 'ground track field': 6, 'bridge': 7, 'large vehicle': 8, 'small vehicle': 9, 'helicopter': 10, 'swimming pool': 11, 'roundabout': 12, 'soccer ball field': 13,'plane': 14, 'harbor': 15}

Hi, first of all I'd like to thank you for this very interesting project. I've tried to run your code under google colab by adding the whole project and adding the following elements to the checkpoints folder (HRNET_LoveDA_best.pth, last_swint_upernet_finetune.pth, Res34_AID_best.pth, yolov5_best.pt)
when I tried to run the following command :
!python /drive/MyDrive/Remote-Sensing-ChatGPT-main/RSChatGPT-shell.py --openai_key my_open_ai_key --image_dir /drive/MyDrive/Remote-Sensing-ChatGPT-main/test_image.png
I got the following error :

/usr/local/lib/python3.10/dist-packages/langchain/chat_models/init.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

from langchain_community.chat_models import ChatOpenAI.

To install langchain-community run pip install -U langchain-community.
warnings.warn(
Initializing RSChatGPT, load_dict={'ImageCaptioning': 'cuda:0', 'SceneClassification': 'cuda:0', 'ObjectDetection': 'cuda:0', 'LandUseSegmentation': 'cuda:0', 'InstanceSegmentation': 'cuda:0', 'ObjectCounting': 'cuda:0', 'EdgeDetection': 'cpu'}
Initializing ImageCaptioning to cuda:0
2024-02-05 21:51:21.669993: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-05 21:51:21.670040: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-05 21:51:21.671336: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-05 21:51:22.692478: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Initializing SceneClassification
Initializing SceneClassification
/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=None.
warnings.warn(msg)
Initializing LandUseSegmentation
Initializing InstanceSegmentation
Initializing InstanceSegmentation
/usr/local/lib/python3.10/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Initializing Edge Detection Function....
Initializing Image2Canny
All the Available Functions: {'ImageCaptioning': <main.ImageCaptioning object at 0x7af43ee69b10>, 'SceneClassification': <main.SceneClassification object at 0x7af43ee699f0>, 'ObjectDetection': <main.ObjectDetection object at 0x7af430e7b340>, 'LandUseSegmentation': <main.LandUseSegmentation object at 0x7af430e7b250>, 'InstanceSegmentation': <main.InstanceSegmentation object at 0x7af5421f32b0>, 'ObjectCounting': <main.ObjectCounting object at 0x7af3eedcc0d0>, 'EdgeDetection': <main.EdgeDetection object at 0x7af3eed94070>}
/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The class langchain_community.chat_models.openai.ChatOpenAI was deprecated in langchain-community 0.0.10 and will be removed in 0.2.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run pip install -U langchain-openai and import as from langchain_openai import ChatOpenAI.
warn_deprecated(
/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The function initialize_agent was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use Use new agent constructor methods like create_react_agent, create_json_agent, create_structured_chat_agent, etc. instead.
warn_deprecated(
RSChatGPT initialization done, you can now chat with RSChatGPT~
/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1273: UserWarning: Using the model-agnostic default max_length (=20) to control the generation length. We recommend setting max_new_tokens to control the maximum length of the generation.
warnings.warn(

Processed ImageCaptioning, Input Image: image/d3bc52dc.png, Output Text: A satellite image of a large airport with many planes parked on the tar

Processed ImageCaptioning, Input Image: image/d3bc52dc.png, Output Text: A satellite image of a large airport with many planes parked on the tar

Processed run_image, Input image: image/d3bc52dc.png
Current state: [('image/d3bc52dc.png', 'Received.')]
Current Memory: Human: Provide a remote sensing image named image/d3bc52dc.png. The description is: A satellite image of a large airport with many planes parked on the tar. This information helps you to understand this image, but you should use tools to finish following tasks, rather than directly imagine from my description. If you understand, say "Received".
AI: Received.
/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The function __call__ was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.
warn_deprecated(

Entering new AgentExecutor chain...
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Thought: Do I need to use a tool? Yes
Action: Count object
Action Input: image/d3bc52dc.png, plane
Processed Object Counting, Input Image: image/d3bc52dc.png, Output text: plane is not a supported category for the model.

Observation: plane is not a supported category for the model.
Thought:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Thought: Do I need to use a tool? Yes
Action: Get Photo Description
Action Input: image/d3bc52dc.png/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1273: UserWarning: Using the model-agnostic default max_length (=20) to control the generation length. We recommend setting max_new_tokens to control the maximum length of the generation.
warnings.warn(

Processed ImageCaptioning, Input Image: image/d3bc52dc.png, Output Text: A satellite image of a large airport with many planes parked on the tar

Processed ImageCaptioning, Input Image: image/d3bc52dc.png, Output Text: A satellite image of a large airport with many planes parked on the tar

Observation: A satellite image of a large airport with many planes parked on the tar
Thought:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
Thought: Do I need to use a tool? Yes
Action: Detect the given object
Action Input: image/d3bc52dc.png, planeTraceback (most recent call last):
File "/drive/MyDrive/Remote-Sensing-ChatGPT-main/RSChatGPT-shell.py", line 239, in
state=bot.run_image(args.image_dir, [], txt)
File "/drive/MyDrive/Remote-Sensing-ChatGPT-main/RSChatGPT-shell.py", line 218, in run_image
state=self.run_text(f'{txt} {image_filename} ', state)
File "/drive/MyDrive/Remote-Sensing-ChatGPT-main/RSChatGPT-shell.py", line 184, in run_text
res = self.agent({"input": text.strip()})
File "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py", line 145, in warning_emitting_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/langchain/chains/base.py", line 363, in call
return self.invoke(
File "/usr/local/lib/python3.10/dist-packages/langchain/chains/base.py", line 162, in invoke
raise e
File "/usr/local/lib/python3.10/dist-packages/langchain/chains/base.py", line 156, in invoke
self._call(inputs, run_manager=run_manager)
File "/usr/local/lib/python3.10/dist-packages/langchain/agents/agent.py", line 1391, in _call
next_step_output = self._take_next_step(
File "/usr/local/lib/python3.10/dist-packages/langchain/agents/agent.py", line 1097, in _take_next_step
[
File "/usr/local/lib/python3.10/dist-packages/langchain/agents/agent.py", line 1097, in
[
File "/usr/local/lib/python3.10/dist-packages/langchain/agents/agent.py", line 1182, in _iter_next_step
yield self._perform_agent_action(
File "/usr/local/lib/python3.10/dist-packages/langchain/agents/agent.py", line 1204, in _perform_agent_action
observation = tool.run(
File "/usr/local/lib/python3.10/dist-packages/langchain_core/tools.py", line 373, in run
raise e
File "/usr/local/lib/python3.10/dist-packages/langchain_core/tools.py", line 345, in run
self._run(*tool_args, run_manager=run_manager, **tool_kwargs)
File "/usr/local/lib/python3.10/dist-packages/langchain_core/tools.py", line 523, in _run
else self.func(*args, **kwargs)
File "/drive/MyDrive/Remote-Sensing-ChatGPT-main/RSChatGPT-shell.py", line 124, in inference
log_text=self.func.inference(image_path, det_prompt,updated_image_path)
File "/drive/MyDrive/Remote-Sensing-ChatGPT-main/RStask/ObjectDetection/YOLOv5.py", line 25, in inference
out, _ = self.model(image.to(self.device), augment=False,val=True)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/drive/MyDrive/Remote-Sensing-ChatGPT-main/RStask/ObjectDetection/models/common.py", line 307, in forward
y = self.model(im, augment=augment, visualize=visualize)[0]
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/drive/MyDrive/Remote-Sensing-ChatGPT-main/RStask/ObjectDetection/models/yolo.py", line 135, in forward
return self._forward_once(x, profile, visualize) # single-scale inference, train
File "/drive/MyDrive/Remote-Sensing-ChatGPT-main/RStask/ObjectDetection/models/yolo.py", line 158, in _forward_once
x = m(x) # run
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/drive/MyDrive/Remote-Sensing-ChatGPT-main/RStask/ObjectDetection/models/common.py", line 276, in forward
return torch.cat(x, self.d)
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 244 but got size 243 for tensor number 1 in the list.