TypeError: LoraInjectedLinear.forward() got an unexpected keyword argument 'scale' about lora

[justin-prnd]

query = attn.to_q(hidden_states, scale=scale)

The latest diffusers adopts LoRACompatibleLinear module rather than nn.Linear.

The forward method of LoRACompatibleLinear, however, has an additional kwargs scale which is not included in LoraInjectedLinear.

Since LoraInjectedLinear already has a scale parameter as an instance variable, simply modifying the definition of the forward method of LoraInjectedLinear will resolve the issue.

class LoraInjectedLinear(nn.Module):
  ...
  def forward(self, input, scale: float = 1.0):  # FIXED: add dummy scale argument

[xwhzz]

You may need to lower the version of the Python package, such as using diffusers==0.11.0 and transformers==4.25.1.

[justin-prnd]

Modification for inference

After training, I faced a similar issue for inference.

The patch_pipe method would fail with the following error.

...
line 784, in monkeypatch_or_replace_lora_extended
    _module._modules[name] = _tmp
UnboundLocalError: local variable '_tmp' referenced before assignment

I figured out that the error occurred because the monkeypatch_or_replace_lora_extended did not handle the LoRACompatibleLinear module properly.

There would be a better solution, but I fixed the issue by simply adding the LoRACompatibleLinear module on the search target.

def monkeypatch_or_replace_lora_extended(
    model,
    loras,
    target_replace_module=DEFAULT_TARGET_REPLACE,
    r: Union[int, List[int]] = 4,
):
    for _module, name, _child_module in _find_modules(
        model,
        target_replace_module,
        search_class=[nn.Linear, LoraInjectedLinear, LoRACompatibleLinear, nn.Conv2d, LoraInjectedConv2d],
    ):
        _tmp = None

        if _child_module.__class__ in {nn.Linear, LoraInjectedLinear, LoRACompatibleLinear}:
           ...

[justin-prnd]

@justin-prnd where is the LoRACompatibleLinear defined? I get module not defined error?

Defined at
https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/lora.py#L181
and can be imported as

from diffusers.models.lora import LoRACompatibleLinear

if diffusers >= v0.16.0

[shohog]

Build the environment from source.

[ethvedbitdesjan]

Build the environment from source.

I did run

!pip install git+https://github.com/cloneofsimo/lora.git

Is this not building from source?

[haodong2000]

thanks @justin-prnd

[ethvedbitdesjan]

@justin-prnd where is the LoRACompatibleLinear defined? I get module not defined error?