Prototype native parser scaling mechanism about bblfshd

[dennwc]

I think it may be easier to go from a different angle: keep native drivers single-threaded, but allow running multiple native processes, which are controlled/scaled by Go driver server (the "stdio-to-gRPC proxy" that exists in every driver).

Scaling can be very naive: always keep 1 driver instance alive, if more requests arrive - run more processes inside the same container (up to the number of CPUs). Then, kill them immediately if no other requests are waiting to get a driver. More advance scaling will be done by K8S or other high-level system, allowing to keep the code simple.

This change can also be introduced incrementally, while keeping current bblfshd design.

As the next steps we may drop driver scaling in bblfshd completely and allow running only a single driver per language (multi-process, as proposed). This should still cover existing use case (saturating CPUs of a single server), while allowing to run and scale drivers separately in K8S.

The only missing piece after this is to implement driver discovery for K8S in any suitable way. We may or may not proxy traffic through bblfshd at that point.

[kuba--]

This part should not touch k8s in any sense. In my opinion it's up to us how many CPU we allow to use (GOMAXPROCS). The whole thing sounds to me like a queueing system which we can implement. And we can handle number of processes which can pull tasks from the queue.
if one routine finished the task and wants to return to the pool we can close it or keep running and pull another task.

[dennwc]

I'm mentioning K8S only to illustrate how this change can fit in the whole picture. It's not directly related.

Re: queueing, yes, this is more or less what I propose, at least if you mean implementing it in a Go process and not externally :)