Composition capability of SDXL about tokencompose

[zwcolin]

Hi Junwen,

Thanks for your comments! From both a research or practical perspective, we wouldn't say a problem as being "addressed" simply based on some examples. Metrics have its quantitative measurements -- to truly say that a model addresses multi-category object composition, likely you would need a model so powerful that you can literally give it as many categories of objects as possible and it will generate all these categories of objects for all its generated images. For example, given enough compute, try SDXL with say, MultiGen-10 or MultiGen-15, and the results will very likely be imperfect.

For example, SDXL image generations of the prompt, "a photo of a clock, an elephant, a toilet, a cat, a suitcase, and a helicopter, an umbrella, a tea cup, a dining table, and an astronaut on a horse" will almost always miss certain categories of instances:

For stronger models such as SDXL, you'd likely need to try prompts that involve more categories of instances to see that it is still an issue (and also categories of instances where they do not commonly appear together naturally).

As a research project, we are implementing our approach on SD 1.4 and SD 2.1 because the compute need for training SDXL is beyond our capacity, but we hope our approach can improve any existing model on multi-category instance composition (as we've seen significant improvements on both SD 1.4 and SD 2.1).

Hope this addresses your concerns, and let us know if you have any follow-up questions!

Best,
Colin