dynamic requirements: what's wrong with what I am doing? about luigi

[tgy]

thanks @riga, that was exactly it. i figured it out by myself before your message but did not really understand why making the parameter significant solved it. now i got it :) thanks a lot!

[tgy]

I tried using DynamicRequirements but it's not solving my issue

tagging @riga because he might know what the issue is

[riga]

Hi @tgy ,

your split parameter is not significant, resulting in the luigi scheduler not triggering all three tasks but just a subset of tasks (in your case one or two of them; I think this should be only one though). As a result your second task declares dynamic dependencies that are never run and thus it never makes it to the point where its own run method finishes.

The DynamicDependencies container is just a way of saving resources during repeated completness checks, e.g. in cases with large numbers of dynamic deps with remote targets.

So, in short, setting significant back to True should do the trick.

Hope that helps!