Front Light (like a Google Photos Portrait Light)

IC-Light

IC-Light is a project to manipulate the illumination of images.

The name "IC-Light" stands for "Imposing Consistent Light" (we will briefly describe this at the end of this page).

Currently, we release two types of models: text-conditioned relighting model and background-conditioned model. Both types take foreground images as inputs.

Get Started

Below script will run the text-conditioned relighting model:

git clone https://github.com/lllyasviel/IC-Light.git
cd IC-Light
conda create -n iclight python=3.10
conda activate iclight
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
python gradio_demo.py

Or, to use background-conditioned demo:

python gradio_demo_bg.py

Model downloading is automatic.

Note that the "gradio_demo.py" has an official huggingFace Space here.

Screenshot

Text-Conditioned Model

(Note that the "Lighting Preference" are just initial latents - eg., if the Lighting Preference is "Left" then initial latent is left white right black.)

Prompt: beautiful woman, detailed face, warm atmosphere, at home, bedroom

Lighting Preference: Left

Prompt: beautiful woman, detailed face, sunshine from window

Lighting Preference: Left

beautiful woman, detailed face, neon, Wong Kar-wai, warm

Lighting Preference: Left

Prompt: beautiful woman, detailed face, sunshine, outdoor, warm atmosphere

Lighting Preference: Right

Prompt: beautiful woman, detailed face, sunshine, outdoor, warm atmosphere

Lighting Preference: Left

Prompt: beautiful woman, detailed face, sunshine from window

Lighting Preference: Right

Prompt: beautiful woman, detailed face, shadow from window

Lighting Preference: Left

Prompt: beautiful woman, detailed face, sunset over sea

Lighting Preference: Right

Prompt: handsome boy, detailed face, neon light, city

Lighting Preference: Left

Prompt: beautiful woman, detailed face, light and shadow

Lighting Preference: Left

(beautiful woman, detailed face, soft studio lighting)

Prompt: Buddha, detailed face, sci-fi RGB glowing, cyberpunk

Lighting Preference: Left

Prompt: Buddha, detailed face, natural lighting

Lighting Preference: Left

Prompt: toy, detailed face, shadow from window

Lighting Preference: Bottom

Prompt: toy, detailed face, sunset over sea

Lighting Preference: Right

Prompt: dog, magic lit, sci-fi RGB glowing, studio lighting

Lighting Preference: Bottom

Prompt: mysteriou human, warm atmosphere, warm atmosphere, at home, bedroom

Lighting Preference: Right

Background-Conditioned Model

The background conditioned model does not require careful prompting. One can just use simple prompts like "handsome man, cinematic lighting".

A more structured visualization:

Imposing Consistent Light

In HDR space, illumination has a property that all light transports are independent.

As a result, the blending of appearances of different light sources is equivalent to the appearance with mixed light sources:

Using the above light stage as an example, the two images from the "appearance mixture" and "light source mixture" are consistent (mathematically equivalent in HDR space, ideally).

We imposed such consistency (using MLPs in latent space) when training the relighting models.

As a result, the model is able to produce highly consistent relight - so consistent that different relightings can even be merged as normal maps! Despite the fact that the models are latent diffusion.

From left to right are inputs, model outputs relighting, devided shadow image, and merged normal maps. Note that the model is not trained with any normal map data. This normal estimation comes from the consistency of relighting.

You can reproduce this experiment using this button (it is 4x slower because it relight image 4 times)

Below are bigger images (feel free to try yourself to get more results!)

For reference, geowizard (geowizard is a really great work!):

And, switchlight (switchlight is another great work!):

Model Notes

iclight_sd15_fc.safetensors - The default relighting model, conditioned on text and foreground. You can use initial latent to influence the relighting.
iclight_sd15_fcon.safetensors - Same as "iclight_sd15_fc.safetensors" but trained with offset noise. Note that the default "iclight_sd15_fc.safetensors" outperform this model slightly in a user study. And this is the reason why the default model is the model without offset noise.
iclight_sd15_fbc.safetensors - Relighting model conditioned with text, foreground, and background.

Also, note that the original BRIA RMBG 1.4 is for non-commercial use. If you use IC-Light in commercial projects, replace it with other background replacer like BiRefNet.

Cite

@Misc{iclight,
  author = {Lvmin Zhang and Anyi Rao and Maneesh Agrawala},
  title  = {IC-Light GitHub Page},
  year   = {2024},
}

	ambient = (left + right + bottom + top) / 4.0
	h, w, _ = ambient.shape
	matting = resize_and_center_crop((matting[..., 0] * 255.0).clip(0, 255).astype(np.uint8), w, h).astype(np.float32)[..., None] / 255.0

	def safa_divide(a, b):
	e = 1e-5
	return ((a + e) / (b + e)) - 1.0

	left = safa_divide(left, ambient)
	right = safa_divide(right, ambient)
	bottom = safa_divide(bottom, ambient)
	top = safa_divide(top, ambient)

	u = (right - left) * 0.5
	v = (top - bottom) * 0.5

	sigma = 10.0
	u = np.mean(u, axis=2)
	v = np.mean(v, axis=2)
	h = (1.0 - u 2.0 - v 2.0).clip(0, 1e5) ** (0.5 * sigma)
	z = np.zeros_like(h)

	normal = np.stack([u, v, h], axis=2)
	normal /= np.sum(normal 2.0, axis=2, keepdims=True) 0.5
	normal = normal * matting + np.stack([z, z, 1 - z], axis=2) * (1 - matting)

	results = [normal, left, right, bottom, top] + inner_results
	results = [(x * 127.5 + 127.5).clip(0, 255).astype(np.uint8) for x in results]
	return results

lllyasviel / ic-light Goto Github PK

ic-light's Introduction

IC-Light

Get Started

Screenshot

Text-Conditioned Model

Background-Conditioned Model

Imposing Consistent Light

Model Notes

Cite

Related Work

ic-light's People

Contributors

Stargazers

Watchers

Forkers

ic-light's Issues

Relighting with Foreground Condition

Relighting with Foreground and Background Condition

Recommend Projects

Recommend Topics

Recommend Org

Jobs