how is this coming along? about scalajs-rx-idb

[l15k4]

Hi Henry,

I'm not very proud of it actually. It had been running in production for a month or two. I caught a lot of bugs. But it is not really self explanatory as I wanted it to be. You must have a really good grasp of monifu internal workings to avoid lock starvation, to see "happened before relationships", etc. IndexedDb turned out to be much more complicated than I expected and because I hadn't any previous experience with it this library turned out to be a first prototype and it would deserve a rewrite.

I left scalajs environment in late january and I won't be back for a while. But I could go through all my scalajs libraries and upgrade them to actual scalajs release and make sure all tests are running this weekend.

I don't know whether I would recommend you to use it. As I said it's a library written by a dude at the time of learning IndexedDb hello-world :-) But imho you might be dealing only with issues caused by composing (map,flatMap) transactions "incorrectly" (stream didn't end but you think it did) which might lead to lock starvation. Otherwise it worked like a charm. Also note that it was built rather for batch operations then individual requests because I created it for Replayed project (open in chrome:-).

If you decide to use it I can promise I'll be supportive, fixing bugs & accepting PRs in my free time...

[bblfish]

Thanks for the info. I suppose upgrading to latest scala.js could help, insofar at least as it would allow it to be opened in IntelliJ for example, and I would not be surprised if others want to take it further from where you left it. The code is certainly very helpful - at least for a ScalaJS programmer - to help understand how to work with IndexDB. Documenting the difficulties you found with IndexDB would be very helpful, the gotchas, etc... That would allow others to go further. ( I have also just read the documentation and am a bit perplexed as to how to use it )

Do you think that it may be better to use akka.js instead of monifu? ( I am trying to understand where each pattern is best and reading Reactive Design Patterns to that purpose ). Akka would be better as a way to protect resources from multiple concurrent writes I suppose, which I somehow understand has been a problem you had ( when you speak of locks etc... )

[l15k4]

Do you think that it may be better to use akka.js instead of monifu?

akka.js is not much of an alternative to monifu. Monifu is a classic rx streaming library with backpressure implementation which is the topmost rx feature. If akka-stream.jsexisted it would be an alternative. And imho backpressure is the n1 requirement for working with DB in browser environment... RX specification is also a good paradigm for working with transaction boundaries which is the biggest challenge in IndexedDb imho, IndexedDb is all about wondering "why the hell throughput just decreased all of a sudden". Hard to describe it really.

[bblfish]

I am still working hard to understand when Akka and Rx are best used, and how and when to use them. It seems that they are just at different levels, since as you point out akka-streams is built on akka.

Still there are some differences I can think of:

• It seems that eventing goes in the opposite direction from Rx. In the actor case one sends a message to an actor that can then change implementation, forward the message, etc..., whereas in the event case one sends some code to the observable and it applies the function for every event it receives.
• As a result the actor can be supervised: which means the number of WebWorkers to deal with it could be made to grow, or it could even be moved to a different machine. ( But since 'akka-streams' is based on akka, one imagines that the actors on which the streams rely would be doing the same thing )
• Akka enables funnelling of messages through one actor, which allows one to create a very basic lock on some part of the data, since the actor can act as a lock on access to that data. Is this part something that would have been useful? You mention "lock starvation" above, that is why I ask.
• Rx has nice features such as filter, map, flatMap, which would be difficult to implement directly in akka. But one can do that with commands as freeMonads. We did that with an LDP implementation with LDPCommand (we only implemented the functor part). This can then be very useful if one want a number of different behaviors. For example we have an actor that uses the File System to save, edit, create or fetch data, and another actor that uses HTTP for storage and they do that for the same commands which are routed through a routing actor depending on the URI involved. I suppose in the case of browser commands one could imagine similar commands with implementations for IndexDB, for the previous widely deployed SQL versions, or even for just in memory storage depending on the browser type, or even a pool of actors with many web workers if cpu usage gets tight somehow.

I am just thinking out aloud here.

[l15k4]

Regarding lock starvation, I'm talking about IndexedDb transaction handling, in the sense that you don't want to open new transaction unless you closed previous one when R/W the same data. It might happen when you are not careful when R/W from/to stream, you must respect onComplete/onError event signaling end of previous transaction to initialize new one... Otherwise it usually leads to huge throughput decrease in IndexedDb caused by something like "lock starvation"...

Regarding Actor model VS Rx, I've been using both for a few years. Imho they are not comparable paradigms...

Akka is all about handling concurrency effectively allowing for stateful modeling often used as caching mechanism, having perfect supervision hierarchy and sophisticated (distributed) messaging with multiple handy features like stashing etc., you'll get a resilient, robust system that is able to utilize resources of xx nodes in cluster. it's only disadvantage is the lack of backpressure aka push-pull implementation forming a stream of events from publisher pushing events only when subscriber pulls them + complete,cancel,error events which is a brief description of RX... Classic implimentations like Monifu or RxJava implements the same specification as akka-stream which leverages akka messaging under the hood which feels quite weird at the beginning but it works almost the same from the user point of view. It is really just for working with stream of data almost the same way as with classic collections but it has the essential backpressure built-in which you need in like 50% of cases when working with streams of data...

For instance working with IndexedDb is basically working with streams of data + you need backpressure because the subscriber/consumer of data can read them with different pace then IndexedDb yields it and the same applies for writing data to IndexedDb. Consumer is setting the pace of reading and IndexedDb is setting the pace of writing... If you used akka it would just overflow buffers because it has no backpressure ability.

But these technologies shouldn't be compared, the only things they have in common is their ability to utilize multiple threads in a thread-safe way, both can handle huge amounts of data because of that and also the buzz word reactive :-)

Also I don't think akka belongs to browsers unless it would form an actor system with one part residing on client and the other one on server both being part of the same scala codebase... On the other hand rx is imho essential technology for browser dev when working with streams of data.

[bblfish]

Thanks a lot @l15k4 for this discussion which is helping a lot.

I think we agree that Rx and Actors are very different tools. So different in fact that one can build Rx on top of Actors :-)

• Actors do not come out of the box with backpressure support - in fact out of the box they are more likely to drop messages - but they can be used to support backpressure by sending messages back to the sender notifying him of overload ( It's tricky to do this right but it's done in the implemenation of akka-streams ).
• Actors are low level constructs to make it easy to create resilient applications, because the actor hierarchy makes it possible to stay responsive and to recover in the event of failure.
• An important difference is that Actors are Message-Driven whereas Rx is Event-Driven.
• Both are asynchronous and non-blocking.
• Actors contain type safe code, but externally hide types ( ! is a function Any => Unit ) whereas Rx is type-safe functional code that is easy to compose.
• Actors are built for Location Transparency, I don't think Rx is
• Actors make delegation easy.

So these two tools are very different, but can presumably be used together, either by building Rx on Actors or simply by having actors return Rxs .

For instance working with IndexedDb is basically working with streams of data + you need backpressure because the subscriber/consumer of data can read them with different pace then IndexedDb yields it and the same applies for writing data to IndexedDb. Consumer is setting the pace of reading and IndexedDb is setting the pace of writing... If you used akka it would just overflow buffers because it has no backpressure ability.

That makes a very good case for backpressure. We may also want to have WebWorkers open connections to the web in read and write. ( I am not sure how well backpressure works over XMLHttpRequests though ).

Regarding lock starvation, I'm talking about IndexedDb transaction handling, in the sense that you don't want to open new transaction unless you closed previous one when R/W the same data. It might happen when you are not careful when R/W from/to stream, you must respect onComplete/onError event signaling end of previous transaction to initialize new one... Otherwise it usually leads to huge throughput decrease in IndexedDb caused by something like "lock starvation"...

This is where I am thinking Actors and Rx could work together (let's forget for the moment about what may or may not belong in the browser). What you have is a problem where a number of different parts of the code need to be aware of which transactions are happening: ie. they need to synchronise on these transactions. Since we don't and can't use Java like synchronise tools on a global object, and furthermore we may want to delegate access to the IndexDB to WebWorkers which means we can't rely on a synchronisation object, and we also want to be reactive, we need to find a reactive method of synchronisation. This is provided by an akka actor, since messages to a certain DB can be funnelled through the actor which can only do one thing at a time, wherever it is located. So I can imagine a role here for Actors returning Rx objects.

Also: In order to reduce lock starvation would it make sense to have a lot of little IndexDB databases? In my use case I am fetching Linked Data off the web, following links around the web. Would it make sense to have on DB per resource on the web?

[l15k4]

Hi @bblfish

I upgraded scalajs version to 0.6.5 in all my scalajs projects... Sorry it took so long, I was gonna do that in June when we were talking about it but it turned out to be quite a shit storm because there were a lot of breaking changes in Monifu, uPickle PhantomJS I had to deal with. It took me almost entire weekend to make it all work again...

Anyway I'm not using it currently so I cannot test it properly as I would half a year ago on the Replayed project.