Unexpected non-deterministic output when recompiling Javy provider about javy

[saulecabrera]

For #318 option 2 seems reasonable. Personally I think that with this approach, the determinism of our builds is not an issue worth investing any considerable amount of time on, and it could be solved with documentation, stating that builds will be slightly different every time due to the random state, but that there's no runtime effect to be concerned about. Documenting this is also a pointer in case we need to debug any issues related to difference between builds.

[jeffcharles]

The root cause of this issue and #318 is that a random_state property on the JS context struct is set according to the current time during JS context initialization and we run that initialization as part of the Wizer preinitialization steps. That random_state is used as a seed by Math.random().

The JS context struct is forward declared in quickjs.h with the implementation provided in quickjs.c so we can't access the random_state property outside of quickjs.c. The function that sets random_seed, js_random_init, is a static function so it can't be called from outside quickjs.c. That function has one call site, JS_AddIntrinsicBaseObjects, but that calling function can only be called once on a JS context and it has already called when a JS context is initialized. Attempting to call it again results in abort() being invoked which causes a trap.

As far as I can see, we have 4 high-level approaches we could use:

1. Move the QuickJS context initialization out of the Wizer preinitialization
2. Override the implementation of Math.random() with our own implementation
3. Use a deterministic implementation of clock_time_get whenever we use Wizer to initialize a JS context
4. Change the code in quickjs.c, either move the random_state initialization or add a function to set the random state

(1) is the easiest change. Unfortunately it will degrade performance because a new JS context will need to be created at runtime instead of at build time.

I've tested (2) and it also works. Our implementation can bypass random_state entirely so the randomness will be controlled by whatever WASI implementation is provided by the host environment at runtime. The downside with this approach is that the Javy build is still non-deterministic because a different random_state is being set every time we recompile the core crate. But there isn't really any runtime consequence because of that.

For (3), Wizer when called as a Rust library allows specifying implementations of functions that are imported by the module being preinitialized. We can provide deterministic implementations of clock_time_get and other WASI APIs and try to write code to fallback to the default implementation for fd_read. This would mean the results of builds should become deterministic. The results of Math.random() would be fully deterministic from build to build. Unfortunately #318 will remain an issue since random_state will still remain fixed across invocations of the same Wasm module.

(4) is technically an option but we'd be forking QuickJS if we do it which would make updating QuickJS more difficult in the future so I don't think we should pursue this approach.

I'm strongly leaning toward approach (2). I think (3) could make sense to do at some point so the build becomes deterministic but it's less of a pressing concern if we do (2). And (2) is more straightforward compared to (3).

@saulecabrera thoughts?

[jeffcharles]

Alright, for (2), would you prefer the overridden implementation be added in the core crate or the apis crate?

I'd slightly lean toward core because this only happens as a result of the preinitialization we perform in core and wouldn't normally happen so most other users of javy could use the default Math.random(). That said in favour of putting it in apis, it's a standard API, it would be easy for folks to not include our overridden implementation by not enabling the feature that would include it, and it would mean other folks who want to preinitialize their JS contexts could also use our implementation. So I could also see putting it in apis being a completely reasonable decision.

[saulecabrera]

I was thinking adding it at the apis level, and making sure it's enabled by default, that way we'd only need to maintain a single code path that performs random for both: (i) any consumers of javy::Runtime and (ii) the core crate. Assuming that both random implementations are equivalent in functionality, it's not clear to me why someone would prefer the C implementation; I also think that delegating the internals of the implementation to WASI gives more flexibility to other use-cases in terms of determinism.

Does that make sense to you?

[jeffcharles]

Does that make sense to you?

Mostly

Assuming that both random implementations are equivalent in functionality

They should be. Something's wrong if that's not the case.

it's not clear to me why someone would prefer the C implementation

Possibly to reduce the size of the module by not including an additional RNG. We could just delegate directly to random_get but that would likely be slower than using a PRNG seeded by random_get if there are multiple invocations of Math.random(). Also our approach would result in invoking at least one additional hostcall over using the default implementation. Normally just clock_time_get would run as part of JS context initialization. But our implementation would result in clock_time_get running as part of JS context initialization and then random_get would run at least once. Probably a relatively minor concern all said.

I also think that delegating the internals of the implementation to WASI gives more flexibility to other use-cases in terms of determinism

I'm not sure I follow exactly. I was thinking we would be using random_get to seed a PRNG to reduce hostcalls for multiple uses of Math.random(). I was planning on using the rand crate and double-checking the hostcalls it invokes. And that's not too far from the built-in logic, it's just that the initial seeding happens when initializing the JS context instead of on first call to Math.random(). We could just delegate directly to random_get, but I suspect it would be slower than using a PRNG.

[saulecabrera]

Jeff and myself discussed this offline; after that discussion I'm aligned with the direction here. FWIW, I was hoping we could offer a single implementation but the potential performance impact and module size considerations are good points. If the performance/module size are not a concern in the long run, this is something that we can always revisit in the future.

One thing that I did forgot to mention -- since we are going to offer both options, can we add some documentation describing the pros and cons of each and when you should use one vs the other along with the module size and performance measurements for each? My hope is that with this documentation, it should be straightforward to know when you should not use the default.