zwegner / make.py Goto Github PK

I've found two equivalent ways of doing something. One of them is more natural but slow, the other is fast but unnatural.

Currently my rules.py file contains something along these lines:

def rules_for_subsmaple(ctx):

  if False: # more natural, but slow

    targets = [ f(ss,e) for ss in subsamples
			for e in enph_files ]
    ctx.add_rule(
      targets,
      [ g(e) for e in enph_files ],
      [ python, "python/subsample.py" ] )
    return targets

  if True: # fast, but unnatural

    # This builds all of the small targets
    big_target = "output/phony/subsamples"
    ctx.add_rule(
      big_target,
      [ g(e) for e in enph_files ],
      [ [ python, "python/subsample.py" ],
        [ "touch", big_target ] ] )

    # This records that the small targets depend on the big one.
    for ss in subsamples:
      for e in enph_files:
        small_target = f(ss,e)
        ctx.add_rule(
          small_target,
          [ big_target ],
          [ "touch", small_target ] ) # without this, `target` would
                     # look out of date relative to its dependencies

Both ways produce 64 target files.

If I build them the slow way, the first time takes a long time. The next time is much faster, but still takes a few seconds. A counter appears at 64 and counts down to 0.

If I build them the fast way, the first time again takes a long time. But in that case, when I rebuild, it takes no time at all.

Do you know why?

Judging from the syntax in the example, this looks wonderful. But how do I use it? Suppose I write a valid rules.py file. Is it just a matter of copying make.py to the same folder and then run python3 make.py?

Also, do I need gnu_make_parse.py? Will it be useful for translating any of my giant unreadable GNU makefiles into this syntax?

I've noticed some in-efficiency in make.py implementation. Running make.py on one of my
projects takes 3 seconds even if everything is up to date and nothing to build.
After some investigation, I was able to reduce the running time from 3s to 0.4s.

The current implementation works by this way:
The worker threads are driven by the main thread.
The main thread travels over the dependency graph over and over again, to see whether there
are works ready to do and queue them. And there is a 10ms sleep between iterations which causes
a major slow down because my project requires many iterations to finish.
And the main thread increases running time further by repetitively doing works like
parsing d files, stat on dependencies.

Some obvious mitigations to these problems are replacing sleep with semaphore and
adding memoization to costly functions, which I've implemented in my branch
account-login@f74284f.

I also have an idea to improve further:
Instead of checking for work by traveling the graph top-down, the working threads
could be driven by the finishing of dependency. When a target finishes, the working thread
checks for its dependent and enqueue it when all its dependencies are ready.
This improvement probably requires rewriting most of the code.

That's my entire question.

I should note, however, that I'm not sure I need this, or even that it's a desirable feature to have. In Makefiles I've mainly (maybe only?) used any secondary commands in a recipe just to echo stuff to the screen.

I'm experimenting with different ways of getting something done, and the fact that each time I change the recipe to something I believe is equivalent, it gets rebuilt, is slowing me down a lot.

I will happily leave behind almost everything I had to learn about Makefiles. But one feature I might miss is the ability to define phony targets. In a makefile, if you say that x depends on y and z, and mark x as "phony", then in order to "make x" it will just ensure that y and z are up to date.

Using make.py I can already fake this, by creating a recipe that merely touches a dummy file (with all such dummies probably kept in a folder called something like "phony make.py targets"). For me to do that is very little trouble, so please don't consider this a high-stakes issue -- but if you've already implemented phony targets I'd love to know how to use them.