Return data from selecting multiple overlapping images about carefree-drawboard

[yuhaoo00]

Embarrassingly, I couldn't reproduce those bugs, which may be caused by something else instead of this new feature. 🤣

And yes, it is very convenient to use PIL or OpenCV to implement various pre-processing, which is exactly what I am doing.

Thank you for all your help! 🍻

[yuhaoo00]

btw, before that, if you want to try the fusing algorithm anyway, you can try these settings:

class Fusing(IFieldsPlugin):
    @property
    def settings(self) -> IPluginSettings:
        return IPluginSettings(
            ...,
            pluginInfo=IFieldsPluginInfo(
                definitions=dict(
                    url0=IImageField(label="Image 0"),
                    url1=IImageField(label="Image 1"),
                    # Your original definitions, leave it blank if there weren't any!
                    ...
                ),
            ),
        )

    async def process(self, data: ISocketRequest) -> List[Image.Image]:
        img_url0 = data.extraData["url0"]
        img_url1 = data.extraData["url1"]
        img0 = await self.load_image(img_url0)
        img1 = await self.load_image(img_url1)
        ...
        return img_fused


class ImagesFollowers(IPluginGroup):
    @property
    def settings(self) -> IPluginSettings:
        return IPluginSettings(
            ...,
            # this modified line means the plugin will always be displayed
            nodeConstraintRules=None,
            ...,
            pluginInfo=IPluginGroupInfo(
                ...
                plugins={
                    "fusing": Fusing,
                },
            ),
        )

By using IImageField, you should be able to see the nice 'image picker's in the plugin, where you can pick up the images on the drawboard, and the selected ones will be placed at first!

Very useful feature! Thanks for these tips! 🤩

[carefree0910]

Ah, this is a good feature that is currently lack of, I'll develop it today and let you know once it is done!

Here's my plan: you can specify an argument in IPluginSettings, with which I'll return the full images to you. Currently, I'll use the largest image as a 'canvas' and 'crop' those images, which is useful for outpainting features.

And in case you don't know: I implemented an internal method for all plugins to download images, so maybe you can try:

class Fusing(IFieldsPlugin):
    ...
    img0 = await self.load_image(img_url0)
    img1 = await self.load_image(img_url1)

To simplify your implementations!

[carefree0910]

btw, before that, if you want to try the fusing algorithm anyway, you can try these settings:

class Fusing(IFieldsPlugin):
    @property
    def settings(self) -> IPluginSettings:
        return IPluginSettings(
            ...,
            pluginInfo=IFieldsPluginInfo(
                definitions=dict(
                    url0=IImageField(label="Image 0"),
                    url1=IImageField(label="Image 1"),
                    # Your original definitions, leave it blank if there weren't any!
                    ...
                ),
            ),
        )

    async def process(self, data: ISocketRequest) -> List[Image.Image]:
        img_url0 = data.extraData["url0"]
        img_url1 = data.extraData["url1"]
        img0 = await self.load_image(img_url0)
        img1 = await self.load_image(img_url1)
        ...
        return img_fused


class ImagesFollowers(IPluginGroup):
    @property
    def settings(self) -> IPluginSettings:
        return IPluginSettings(
            ...,
            # this modified line means the plugin will always be displayed
            nodeConstraintRules=None,
            ...,
            pluginInfo=IPluginGroupInfo(
                ...
                plugins={
                    "fusing": Fusing,
                },
            ),
        )

By using IImageField, you should be able to see the nice 'image picker's in the plugin, where you can pick up the images on the drawboard, and the selected ones will be placed at first!

[carefree0910]

I've pushed some codes and now you can do something like this after installing the latest codes with pip install -e .:

class Fusing(IFieldsPlugin):
    @property
    def settings(self) -> IPluginSettings:
        return IPluginSettings(
            ...,
            pluginInfo=IFieldsPluginInfo(
                # add this line!
                exportFullImages=True,
                ...
            ),
        )

    ...

By specifying exportFullImages=True, the images will be exported as-is!

However some cutting edge features are being pushed as well so it might be buggy 🤣

[yuhaoo00]

For exportFullImages=True, I found that it seems to return only two raw images (without the user's transformation on the UI), and no intersection between these two.

But anyway, I'm surprised that you directly helped me develop this feature, so thank you very much. 😉 👍

[yuhaoo00]

I just found that the returned <SingleNodeType.IMAGE> provides quite a lot of properties, which can be used to customize the various operations I need.

Maybe, we don't need to add this new feature exportFullImages=True, in order to avoid a series of bugs 😨 (e.g. in drawing masks for inpainting)

[carefree0910]

Ah! You discovered something I missed: I've put the original (raw) image url in the meta -> data -> url field!

But this is not something very stable... So I'll keep the exportFullImages feature if only somebody needs it. 😉

You've mentioned 'a series of bugs', theoretically everything should be the same when exportFullImages=False, but yes, if exportFullImages=True & you try to send masks / blank nodes, it might be buggy. What situation are you in?

And, you've mentioned you want the user's transformation on the UI - this has a stable solution: there's a transform property in the nodeData, with which you can use cv2 to transform the image if you need!

And for the intersection - theoretically, you can create a large canvas, transform & paste the images to the canvas with their transforms, then crop the area you need. It is a little complicated though. 😵‍💫