lllyasviel / ic-light

More relighting!

Style2Paints Research Lvmin Zhang (Lyumin Zhang)

Because GitHub only allows pinning 6 repos, here is a longer quick list for what you are looking for:

Paints-UNDO: Understand Human Behavior to Align True Needs

Omost: Your image is almost there!

IC-Light: More relighting!

LayerDiffuse for Diffusers

LayerDiffuse for Forge

WebUI Forge

LayerDiffuse

Fooocus

sd-webui-controlnet ControlNet 1.1 ControlNet style2paints PaintingLight SingleFileDB more ...

ic-light's Issues

Can the color be controlled?

I found that this thing can change the original color of the item during use. If this function is mainly used for lighting, the original color should not be changed. I hope the following improvements can be made.

is there a paper of this method?

Dear Lvmin,

Thank you for your great work. I really like it.

Is there a paper explaining this method? The explanation at the bottom of the webpage is a bit vague.

I appreciate your help.

Best Wishes,

Zongze

Even lighting (or) No lighting Option - For creating Unlit/evenly lit diffuse maps:

Can an addition be made to completely remove any existing lighting in a image and create a unlit diffuse map from an image, this is an important part of PBR texture process from real life pics and there are different workflows (ex:https://github.com/Unity-Technologies/DeLightingTool?tab=readme-ov-file) that are very convoluted to achieve this.

But with this extension, if it can have an option to add a no lighting or an evenly lit lighting option (no directional light) , it would be a valuable addition to any PBR texture process

Example of potential use case, left is texture with lighting, right is unlit albedo/diffuse map, with lighting effects removed:

A even more detailed option would be to remove direction light from an existing texture, as in above example it was evenly lit, but in most real life PBR uses, light is angled (as in this below pic, stone is lit from right side to left)

Thanks

is there a way to produce non low-key image?

Every image i make is very dramatic low-key image. I never seen normal bright or flash photography made. is it possible?! prompts dont help...

The effect of small accessories is not good

The effect of small accessories is not very good, such as a brooch with pearls and diamonds around it. The resulting effect is too different from the original picture.

Which checkpoint/model is used in the demo on HuggingFace?

Can you please answer that? It seems to work particularly well with IC-Light compared to the models I have tried.

Increasing image dimensions beyond 1024

Is this possible? Currently, typing over 1024 (eg. 2048) simply resets it to the max dimensions after clicking "re-light".

Error loading shm.dll, The specified module could not be found.

(cuda) C:\Users\palam\OneDrive\Desktop\IC light\IC-Light>python gradio_demo.py
Traceback (most recent call last):
File "C:\Users\palam\OneDrive\Desktop\IC light\IC-Light\gradio_demo.py", line 5, in
import torch
File "C:\Users\palam\AppData\Roaming\Python\Python310\site-packages\torch_init_.py", line 141, in
raise err
OSError: [WinError 126] The specified module could not be found. Error loading "C:\Users\palam\AppData\Roaming\Python\Python310\site-packages\torch\lib\shm.dll" or one of its dependencies

Could you explained more about "Consistency using MLPs in latent space

(Remove because accidentally press post), please reply on #13 instead

Preserve the high-frequency component (fine details) in the original image

Nice work! I'm already using it in our indie game dev. I see some people complaining that the AI will change details in the original image, so I just want to mention that we can separate the high-frequency component from the original image and replace it into the processed image.

We can use, e.g. Erasing Appearance Preservation to separate the base color and the high-frequency component, for which I have a Python implementation here

[Errno 2] No such file or directory

Cannot import /Users/Myname/Documents/ComfyUI/custom_nodes/ComfyUI-IC-Light module for custom nodes: [Errno 2] No such file or directory: '/Users/Myname/Documents/ComfyUI/custom_nodes/ComfyUI-IC-Light/init.py'

[bug]Qwen requires transformers>4.37, while req is 4.36.2

Tlntin/Qwen-TensorRT-LLM#80
更正transformer版本后正常

Huggingface page not working

The demo on the huggingface page doesn't work, on pc it says error 500, while on my smartphone it doesn't say anything, it's blank

Relight transparent object incorrectly

As you can see, the relight program treats the transparent part like opaque screen. The relight process only takes reflection light into consideration while for transparent object, both reflection light and transmission light have to be taken into account. Just a suggestion, and I encountered the same problem when I used Switchlight before.

不知原因的错误

因为我只是个美术类的用户，也不懂代码什么的……

[data] Datset used

This is an amazing model, really impressed by this!

I was wondering what dataset you're using to train this, I tried to find an opensource OLAT dataset to recreate switchlight, but didn't have any luck.
Also, any chance you are going to release the training code?

Expecting value: line 1 column 1 (char 0)

It's so cool,I need comyfui version!

It's so cool,I need comyfui version!

[feature] Is it now possible to get different map like albedo metallic and AO map?

As we can already relight it, maybe it's possible to get them all to generate propriate model in UE5? Well I'm not sure whether it can distinguish albedo and metallic.
Well by asking this, is because, of course, it's not suitable to have only one light. A common lighting should have a composite light, including at least front light, backlight, side light and etc. Yet in some case we do want them seperately.

All images must be cutout before further processing. How can I skip the cutout step?

All images must be cutout before further processing. How can I skip the cutout step?

how to use in a1111 webui?

awesome! Will integrate into the Fooocus?

Will this integrate into the image2image in Fooocus?

interested in the method of training re-lighting model

Hi, thanks for the great work!
I'm curious about how is a model like this trained to achieve re-lighting ability. Is there any plan of a paper release?

two stage denosing

Thanks for your wonderful work!
I have a question that, in your codes, the process function contains two stage denosing steps, I don't know why it needs two.

inpainting pipeline

add a version for inpaint pipeline please

We will crop your image based on the center, and it is recommended to increase the scale of the original image

1. We will crop your image based on the center, and it is recommended to increase the scale of the original image
2. The format of the image is webp, which has too many restrictions. It is recommended to change it to png or other formats

can not running on local URL

Hi, when I try to access the local site I can't load the page, my browser (it doesn't matter which one because all the browsers I've tried give the same message) reports this error: "
Unable to reach site The web page at http://0.0.0.0:7860/ may be temporarily unavailable or has been permanently moved to a new web address.
ERR_ADDRESS_INVALID "

sdxl support

are you planning to add support for sdxl?

Has this work been published? Want to know the training details

Great work!! I'd like to know more details about this work.

IC-Light in diffusers

Hello! Will this model be added to diffusers as a pipeline?

Does the background need to be changed in order to achieve lighting rendering?

Thank you very much for your contributions to this field. I have a question: Currently, there are two implementation methods for this algorithm, both of which achieve foreground lighting rendering by replacing the background. Is it currently only possible to achieve lighting rendering on the foreground by replacing the background?

Feature request: Allow for randomizing the seed

I want to be able to set the seed to -1 like I can in the A111 webui (and forge) to get a random seed for each generation.

ERROR: Exception in ASGI application

You can upload files after running, like this.
Is there anyone who knows how to solve solutions?

ERROR: Exception in ASGI application
Traceback (most recent call last):
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\uvicorn\protocols\http\h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\uvicorn\middleware\proxy_headers.py", line 69, in call
return await self.app(scope, receive, send)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\fastapi\applications.py", line 1054, in call
await super().call(scope, receive, send)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette\applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette\middleware\errors.py", line 186, in call
raise exc
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette\middleware\errors.py", line 164, in call
await self.app(scope, receive, _send)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette\middleware\cors.py", line 85, in call
await self.app(scope, receive, send)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette\middleware\exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette_exception_handler.py", line 64, in wrapped_app
raise exc
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette\routing.py", line 756, in call
await self.middleware_stack(scope, receive, send)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette\routing.py", line 776, in app
await route.handle(scope, receive, send)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette\routing.py", line 297, in handle
await self.app(scope, receive, send)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette\routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette_exception_handler.py", line 64, in wrapped_app
raise exc
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette\routing.py", line 72, in app
response = await func(request)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\fastapi\routing.py", line 278, in app
raw_response = await run_endpoint_function(
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\fastapi\routing.py", line 193, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\starlette\concurrency.py", line 42, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\anyio_backends_asyncio.py", line 851, in run
result = context.run(func, *args)
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\gradio\routes.py", line 282, in api_info
return gradio.blocks.get_api_info(config, serialize) # type: ignore
File "C:\Users\naver\AppData\Roaming\Python\Python310\site-packages\gradio\blocks.py", line 504, in get_api_info
serializer = serializing.COMPONENT_MAPPINGtype
KeyError: 'dataset'

How does the subject remain unchanged?

I am curious to how IC-light keeps the "asset" unchanged in the diffusion process, and adds new effects to it.

GPU memory problems

GPU memory problems, 3050 TI 4gb card, used Fooocus and other image generation AIs they worked fine on this video card

Frontal and surrounding Light

That tool is really cool! :)
But we have lights from almost all angles, but not frontal or surrounding, like "I want the faces lighted up"...

Why change the unet.conv_in?

the channel of unet.conv_in is 8/12 in gradio_demo.py/gradio_demo_bg.py, curious how it works in t2i_pipe/i2i_pipe, and how does it get 8/12 channel? Especially if foreground information pass to unet via cross_attention_kwargs.

Front Light (like a Google Photos Portrait Light)

Embed metadata to image and allow for uploading images to use metadata.

Like how the SD webui does it. This would help a lot with making similar images and tweaking them.

Segmentation fault (core dumped)

when i start to infer, it throw the error: Segmentation fault (core dumped)

The size of tensor a (12) must match the size of tensor b (8) at non-singleton dimension 1

when i run gradio_demo.py, it report error "The size of tensor a (12) must match the size of tensor b (8) at non-singleton dimension 1" ,which means model "iclight_sd15_fc.safetensors" shape not equal to base model (realistic-vision-v51)

Speed up version of IC-Light

Hi, first want to say this is very awesome work!

Wondering if you have plans to create a speed-up version, similar to Marigold (https://huggingface.co/prs-eth/marigold-v1-0) v.s. Marigold LCM (https://huggingface.co/prs-eth/marigold-lcm-v1-0)?

Thank you!

Will this integrate into sd-webui-forge?

Could you explained more about "Consistency using MLPs in latent space"

Great work on IC-Light.

Could you elaborate more on how to train IC-Light?

For example, what does the dataset look like? How can the MLP apply consistency? what are the input and output of the MLP?

Thank you.

Requires PULID to keep face consistent

I guess if we one incorporates PULID into workflow face won't change so much anymore.

about albedo

where can I find the function or model to generate albedo map ?

some small badcases

Great work，overall performance is very good! But, when the person's face takes up a relatively small proportion in the image, the face in the generated image collapses significantly. As follows：

Relighting with Foreground Condition

original image	left light	right light

Relighting with Foreground and Background Condition

original image	bg 1	bg 2

How to get consistent face

If the input image is not a close-up portrait, the facial features may not resemble each other.

Normal map calculation reference?

Hi community,

Can anyone refer me to a tech report or paper on calculating normal maps from images with different lighting directions (left, right, bottom, top)? Thanks in advance!

IC-Light/gradio_demo_bg.py

Lines 352 to 380 in 9365406

	ambient = (left + right + bottom + top) / 4.0
	h, w, _ = ambient.shape
	matting = resize_and_center_crop((matting[..., 0] * 255.0).clip(0, 255).astype(np.uint8), w, h).astype(np.float32)[..., None] / 255.0
	def safa_divide(a, b):
	e = 1e-5
	return ((a + e) / (b + e)) - 1.0
	left = safa_divide(left, ambient)
	right = safa_divide(right, ambient)
	bottom = safa_divide(bottom, ambient)
	top = safa_divide(top, ambient)
	u = (right - left) * 0.5
	v = (top - bottom) * 0.5
	sigma = 10.0
	u = np.mean(u, axis=2)
	v = np.mean(v, axis=2)
	h = (1.0 - u ** 2.0 - v ** 2.0).clip(0, 1e5) ** (0.5 * sigma)
	z = np.zeros_like(h)
	normal = np.stack([u, v, h], axis=2)
	normal /= np.sum(normal ** 2.0, axis=2, keepdims=True) ** 0.5
	normal = normal * matting + np.stack([z, z, 1 - z], axis=2) * (1 - matting)
	results = [normal, left, right, bottom, top] + inner_results
	results = [(x * 127.5 + 127.5).clip(0, 255).astype(np.uint8) for x in results]
	return results

401 Client Error: Unauthorized

When I run "python gradio_demo.py",the error is throwed:
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://hf-mirror.com/briaai/RMBG-1.4/resolve/main/model.safetensors

How to solve this problem，thanks！