hangover3832 / comfyui-hangover-nodes Goto Github PK

kosmos2 can output bboxes to corresponding keywords that were recognized in the input image, could be useful in ComfyUI.

Feel free to contribute.

This makes it very difficult to work. Fix please

After installation, the following error occurs upon the first launch.

Traceback (most recent call last):
  File "/mnt/teratera/git/ComfyUI/nodes.py", line 1810, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/mnt/teratera/git/ComfyUI/custom_nodes/ComfyUI-Hangover-Nodes/__init__.py", line 1, in <module>
    from .ms_kosmos2 import NODE_CLASS_MAPPINGS
  File "/mnt/teratera/git/ComfyUI/custom_nodes/ComfyUI-Hangover-Nodes/ms_kosmos2.py", line 17, in <module>
    model = AutoModelForVision2Seq.from_pretrained("microsoft/kosmos-2-patch14-224")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 434, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 873, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/teratera/venv/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 579, in __getitem__
    raise KeyError(key)
KeyError: 'kosmos-2'

Cannot import /mnt/teratera/git/ComfyUI/custom_nodes/ComfyUI-Hangover-Nodes module for custom nodes: 'kosmos-2'

Error occurred when executing MS kosmos-2 Interrogator:

'MsKosmos2' object has no attribute 'processor'

ImportError: cannot import name 'Kosmos2Config' from 'transformers' (A:\uAI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers_init_.py)

Cannot import A:\uAI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Hangover-Nodes module for custom nodes: cannot import name 'Kosmos2Config' from 'transformers' (A:\uAI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers_init_.py)

I really wanted to try this out, but I couldn't get it running. I'm missing the model, and it's not downloading automatically. If I downloaded itmanually, where should I put it?

Error occurred when executing MS kosmos-2 Interrogator:

'MsKosmos2' object has no attribute 'model'

File "***ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "***ComfyUI_windows_portable\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "***ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "***ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Hangover-Nodes\ms_kosmos2.py", line 49, in interrogate
if (self.model == None) or (self.processor == None) or (self.modelname != huggingface_model) or (device != self.device):
^^^^^^^^^^

Is there a way to avoid including prompt in the response?
I am considering using the response directly as a gen image prompt, but it is difficult to use because it contains prompt.

Hello,

Thanks for your time and effort on this node!

I noticed that Microsoft quietly dropped the model for Kosmos 2.5 yesterday here: https://github.com/microsoft/unilm/tree/master/kosmos-2.5
https://huggingface.co/microsoft/kosmos-2.5

The paper was published last September: https://arxiv.org/abs/2309.11419

Curious if you have plans to make it usable with this node or for ComfyUI in any fashion?

Thanks for any info!!

(Edit: Looks like you may be able to just swap in the new model)

Thank you for your selfless dedication; I hope we can resolve this issue.

The model can't be downloaded automatically. You don't specify which folder it should be placed in when manually downloading it. There's only a model name, and no matter where it's placed, it will be hard to find later on. There are too many files with this name.

Can somebody tell me what/who giterhub is, how its related to github and why i am a "member" there?

ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
File "X:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 155, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "X:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 85, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "X:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 78, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "X:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Hangover-Nodes\ms_kosmos2.py", line 51, in interrogate
if (self.model == None) or (self.processor == None) or (self.modelname != huggingface_model) or (device != self.device):
^^^^^^^^^^
AttributeError: 'MsKosmos2' object has no attribute 'model'