How to debug the par-compiler for :cache-level :all about shadow-cljs

[thheller]

Why is :cache-level :all in this question? It is the default. Do you set it specifically or mess with other caching options in some way?

par-compile is the parallel compilation that tries to spread tasks accross multiple threads. Since in CLJS compilation of namespaces can only be done once all its dependencies have been compiled it has to wait. So the error message is saying that it tried to compile "devtools/defaults.cljs" but was still waiting for cljs.core to finish. Seems like you gathered as much.

So, strictly speaking it was cljs.core that had the problem and I don't have a guess for why that might be. Progress is tracked per form, so it seems unlikely that something would block for 30 sec (the default timeout) in that namespace without making progress.

Are you messing with your clojurescript dependency on some way? local or git dep maybe? Anything else out of the ordinary?

Can you post the full log when setting :log {:level :debug} in shadow-cljs.edn top level (not build config, so same level as :builds). Then npx shadow-cljs compile browser-test --verbose? Maybe there is a clue in there somewhere.

[sirmspencer]

Why is :cache-level :all in this question? It is the default. Do you set it specifically or mess with other caching options in some way?

Our project set it to :jars. I wasn't there for that decision at the time. I'm looking to go back to it, and the devtools preloads seems to be hanging.

The project is pretty large + npm modules and that makes the verbose output too long to capture. All I see is cache reads and the same error above.

This might all be random tough. I see a random different set of our app cljs files failing when I built today. Then as soon as I added :log {:level :debug} there were no errors. Even after taking it back out and switching between :jars and :all back and forth I can't get the error again. Maybe there is something that gets cached during hmr with :jars

Is there a way to force a clean build?

Also side note, :jars looks like might even be faster in some cases. All the time saved with a cache read is taken by a cache write. IDK if maybe any of that can be done async to help with build time.

[thheller]

If it is failing do you see it getting stuck at Cache read?

The verbose log is basically showing every "progress event" that is happening. So, say it is failing for cljs/core.cljs, is the last log entry what you see -> Cache read: ... or -> Compile CLJS? -> it started, <- means it finished (and logs how long it took in addition). Important to know wheter it gets stuck at the cache read or compile.

Only guess I have that maybe you have a weird/outdated/custom transit version that somehow messes with caches? That would only apply if it gets stuck at Cache read though.

In either cache I do not see how :cache-level is at all relevant for any of this. As it was failing for cljs.core which is in a jar, so would be cached the same as :jars or :all.

A clean build you can force by rm -rf .shadow-cljs/builds.

Is there maybe something wrong with your disk? Cache read/write is usually faster than compiling from scratch, usually by a lot.

For example if I run a compile with a proper cache I get

-> Cache read: cljs/core.cljs
<- Cache read: cljs/core.cljs (25 ms)

Without

<- Compile CLJS: cljs/core.cljs (1644 ms)
-> Cache write: cljs/core.cljs
<- Cache write: cljs/core.cljs (253 ms)

So in total that is ~275ms for caching (and only ~25ms for subsequent reads), where it would be ~1600ms if skipping was skipped entirely. And all of this is already done async/parallel wherever possible, i.e. compilation doesn't wait for cache to be written.

That is of course on my machine. Totally possible that it looks different for you? All depends on your actual hardware? Where all the output is written to? Maybe running in a Container? Maybe writing to some network drive or so?

[thheller]

Oh and I forgot to ask ... which shadow-cljs version do you use? Thats probably the most important bit. If it is some ancient history thing maybe try upgrading first. It has been a while since the cache stuff was changed in any way, but there definitely were issues if you are on something super old.