hnmr293 / sd-webui-evviz2

Embedding Vector Visualization for Stable Diffusion web UI

EvViz2 - Embedding Vector Visualizer 2

What is this?

This is an extension for stable-diffusion-webui which visualizes the embedding vector generated by CLIP.

Example

Usage

1. Input prompt as you like.
2. Click Run button.
3. Wait a second.

Detail

Embedding vectors (upper scatter)

This figure shows the embedding vector for each token. Each vector has 768 (for SDv1) or 1024 (for SDv2) dimensions.

Correlations of each token (lower heatmaps)

This figure shows correlations between each token. Calculation is carried out as follows:

1. Compute an embedding vector v from the given prompt. v is typically has dimension (77, 768). For xattn, v is converted by to_k linear layer.
2. For each token t, create a new prompt with the t replaced by padding token. Then compute its embedding vector v_{t}.
3. Let d_{t} = v - v_{t}.
4. Let d_{t,n} is nth row vector of d_{t}. d_{t,n} is a 768(or 1024)-dimensional vector representing t's effect on nth token. Then compute |d_{t,n}| where |x| is norm of a vector x.
5. Repeat procedure 2..3 for all t in the given prompt.

By default, padding token is _</w> (ID=318).

sd-webui-evviz2's Issues

RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'

Traceback (most recent call last):
File "C:\AI\stable-diffusion-webui\extensions\sd-webui-evviz2\scripts\evviz2.py", line 361, in f
v = fn(*args, **kwargs)
File "C:\AI\stable-diffusion-webui\extensions\sd-webui-evviz2\scripts\evviz2.py", line 338, in run
k = mod.to_k(context._context)
File "C:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in call_impl
return forward_call(*args, **kwargs)
File "C:\AI\stable-diffusion-webui\extensions-builtin\Lora\networks.py", line 361, in network_Linear_forward
return torch.nn.Linear_forward_before_network(self, input)
File "C:\AI\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: "addmm_impl_cpu" not implemented for 'Half'

I'm getting this when I "run"

GTX 1070

error in line 319 and line 361 on A1111 1.6

i got this on A1111 1.6

Traceback (most recent call last):
File "D:\AI\stable-diffusion-webui\extensions\sd-webui-evviz2\scripts\evviz2.py", line 361, in f
v = fn(*args, **kwargs)
File "D:\AI\stable-diffusion-webui\extensions\sd-webui-evviz2\scripts\evviz2.py", line 319, in run
sd_model = SDModel(shared.sd_model)
File "D:\AI\stable-diffusion-webui\extensions\sd-webui-evviz2\scripts\evviz2lib\sdhook.py", line 305, in init
assert isinstance(clip, FrozenCLIPEmbedderWithCustomWordsBase)
AssertionError

only standard extensions on like..

controlnet
adetailer
deforum
regional-prompter
supermerger
vectorscope-cc
wildcards

please delete)

[Enhancement] Correlation of own token should not replace with NaN

I found that correlation of own token is not always larger than correlation to other tokens and this information may useful.
So, I suggest to remove line 200 in scripts/evviz2.py .