Executing DB queries on background thread about nativescript-couchbase-plugin

[triniwiz]

I have an idea of what to do just need some time

[triniwiz]

Good idea it’s simple to do this first you need to create your method an a callback so you can use that to make it into a promise on the js side

[triniwiz]

@jerbob92 are you doing it ? because I started something 🙂

[jerbob92]

@triniwiz, yeah you can see it in my fork. Curious what approach you are taking though.

[triniwiz]

🤔 you are changing all the methods into a promise

[jerbob92]

Like I said: curious what approach you are taking though. I'm following the NativeScript core approach.

[triniwiz]

It's ok my i started with was passing the built object e.g Select to the java class then let it execute there then try using Gson to serialize to a string pass that back to the listener then parse it in the js world

[triniwiz]

public class FancyCouchbase {
    private static ExecutorService executorService = Executors.newSingleThreadExecutor();
    private static Gson gson = new Gson();
    public interface QueryListener {
        void onSuccess(String results);

        void onError(String error);
    }

    public void queryInBackground(final Select select, final QueryListener listener) {
        executorService.execute(new Runnable() {
            @Override
            public void run() {
                try {
                    String json = gson.toJson(select.execute().allResults());
                    listener.onSuccess(json);
                } catch (CouchbaseLiteException e) {
                    listener.onError(e.getLocalizedMessage());
                }
            }
        });
    }
}

[jerbob92]

That's only for the query though, how would you make the other methods threaded? The code in my branch already works for Android, only the batch method is hard to make threaded when stuff in the runnable are made out of Promises.

[triniwiz]

Does the ios methods need to be called on another thread or is it fine in your tests ?

[jerbob92]

I didn't get to iOS yet, wanted to have Android stable first. But my plan is to make both of them threaded.

[triniwiz]

For the inBatch I might have an idea

[jerbob92]

Cool! I just noticed I didnt commit everything yet. Fork is updated now.

[triniwiz]

Something like the following should be fine

public void inBatch(final Runnable runnable) {
        executorService.execute(new Runnable() {
            @Override
            public void run() {
                 try {
                        database.inBatch(runnable);
                    } catch (CouchbaseLiteException e) {
                        e.printStackTrace();
                    }
            }
        });
    }

[jerbob92]

That will not work. My code for the inBatch does work like this currently. But since the calls inside the runnable now also use promises, the run() method on the runnable immediately returns, while the actuall calls within those promises didn't run yet. I have to find some way to make the runnable.run() return after those promises have completed. Mixing Java and JavaScript can have its problems 😕

[jerbob92]

I have figured out a solution for the batch problem in my fork. I first tried an approach by using a custom class that implemented the Runnable interface that would just use Thread.pause(50) while the Promise was still running. Technically, this worked, but the native Couchbase inBatch/save/delete methods locks the db to it's current thread, and since the update/create/delete calls in the plugin are now also threaded, and run in a different thread, they will wait for a lock forever since the inBatch already has the lock. And the inBatch will only finish when all the Promises are finished, so that's something you will never get out of.

Second approach was creating new methods for the update/create/delete that will return a BatchAction object, it contains the native document and the action that should be done to it. The inBatch now wants an array of BatchAction, it will create native Java BatchAction objects in a Java list and send that to the native plugin. It will then loop through the BatchAction list in a seperate runnable in the native Couchbase inBatch method with all the actions that are in the BatchAction. This will make sure all actions are executed on the same thread. It's also a lot faster because it prevents going back and forth between native and JS.

Sadly, I'm still having a lot of UI hangs. So going to debug some more now. At least the DB stuff is threaded now. I think the serializing is also quite heavy on the CPU.

[triniwiz]

for serializing have you looked at gson ?

[jerbob92]

How would gson fit into this? Serialize it in the thread and then JSON.parse in JS?

[triniwiz]

Yes :) the performance for JSON.parse should not be bad

[jerbob92]

I'm not sure if the slowness is actually coming from the serialize, I measured it and it's not really that slow.

[jerbob92]

I figured out some of my JSON objects were very large due to some issue on the API which caused some slowness in the JSON.Parse in NS.

Currently our Android threaded version in working great! I just pushed an update to my fork that's feature complete and everything is working. I also figured out that there is some issue in your changeListener implementation, only one worked, so I fixed that too and made the implementation lighter on NS. Since the source doesn't include your Android lib, I added it to mine, which is now included in the repo.

The iOS implementation is on hold right now, but if anyone else would like to pick to up, it would be great.

[triniwiz]

I think we could leave the older api an append lets say async to new methods to help maintain some compatibility with the old couchbase plugin

[jerbob92]

It might be better to release a new major version. But if anyone likes to put time in creating an async version, sure.

[triniwiz]

The same code you did can be used since it’s all promises

[jerbob92]

I'm not certain you can just put async somewhere and it runs promises in sync anywhere.

[jerbob92]

@triniwiz We finished the iOS threaded version. Are you interested in merging it all together to create a 2.0 version?

[triniwiz]

Pr away 🙂

[jerbob92]

@triniwiz sure :)
But I have some cleaning up to do and want to discuss some things:

• Where does the native code live? In the same repo?
• Do we push the native libraries somewhere? (Maven/CocoaPods)
• If we do, who will be the owner of those libraries?
• I changed everything to Promises, because it makes more sense when everything is non-blocking. Do you agree?

[triniwiz]

Native code can live in the platforms directory for both platforms i never tested for swift code

[triniwiz]

We can release it as v2

[jerbob92]

I don't think you can directly add the native libs into your platform folders and have it work.
Android need's an aar right? For iOS we can probably add a Podfile that points to a folder in stead of a version. I will try next week!

[triniwiz]

You can do this for android in the platforms directory and maybe even this for ios

[jerbob92]

I think that's for NativeScript projects, not plugins.

[triniwiz]

So this is wrong ?

[alexandruantonica]

@jerbob92 What is the status with your background thread idea? I have seen your changes and looks good.
Do you intend to create a pull request for this plugin? It would be nice and helpful ;)

Thanks ;)

[jerbob92]

@alexandruantonica We are using our Android fork in production, it's working great :) We already created an iOS version but are not using that one yet due to stability problems.
However, we do intend to create a pull request. If someone else wants to pick that up, here's the fork that contains the Android work: https://github.com/jerbob92/nativescript-couchbase-plugin/commits/feature/threading

[alexandruantonica]

Thank you for your reply. A PR will be great but I would need the iOS version to be ready. Have a great day

On Mon, 4 Nov 2019 at 8:13 am, jerbob92 ***@***.***> wrote: @alexandruantonica <https://github.com/alexandruantonica> We are using our Android fork in production, it's working great :) We already created an iOS version but are not using that one yet due to stability problems. However, we do intend to create a pull request. If someone else wants to pick that up, here's the fork that contains the Android work: https://github.com/jerbob92/nativescript-couchbase-plugin/commits/feature/threading — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14?email_source=notifications&email_token=AC36CG4DNN7A7WATVVPDQGLQR7KUHA5CNFSM4GRZBBS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC6PPVQ#issuecomment-549255126>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC36CG36DVYDCRR2ZQTPTBLQR7KUHANCNFSM4GRZBBSQ> .

--

…

-- *Alexandru-Claudiu Antonica* *Lead Frontend Developer* *TAPPTER LTD* *London, UK* *Phone: *[phone] *Skype* *ID:* alexandru.antonica

[jerbob92]

@alexandruantonica I can push our work on the iOS version, but you will need to do some testing/fixing on your own to get it production ready. Let me know and I will push the work :)

[alexandruantonica]

Please push it and I will take some time to check it out.

On Mon, 4 Nov 2019 at 8:26 am, jerbob92 ***@***.***> wrote: @alexandruantonica <https://github.com/alexandruantonica> I can push our work on the iOS version, but you will need to do some testing/fixing on your own to get it production ready. Let me know and I will push the work :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14?email_source=notifications&email_token=AC36CG57B72XLX5RQ5WDCVDQR7MCRA5CNFSM4GRZBBS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC6QGIA#issuecomment-549258016>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC36CG6GOMTRJU62XHNZGY3QR7MCRANCNFSM4GRZBBSQ> .

--

…

-- *Alexandru-Claudiu Antonica* *Lead Frontend Developer* *TAPPTER LTD* *London, UK* *Phone: *[phone] *Skype* *ID:* alexandru.antonica

[jerbob92]

I just pushed it :)
The native library is in https://github.com/jerbob92/nativescript-couchbase-plugin/tree/feature/threading/native-libraries/ios
The bridging code is in https://github.com/jerbob92/nativescript-couchbase-plugin/blob/feature/threading/src/couchbase-plugin.ios.ts

This will probably not work automatically, we used a private Podspec repo to download the library. But as @triniwiz said, this might not be necessary.

[Pkurto]

@alexandruantonica did you manage to get this to work on both platforms (Android and iOS)?

[alexandruantonica]

@Pkurto No, unfortunately, I was busy and I wasn't able to take a look. If you manage to take a look and do something, let me know.

[Pkurto]

Hi @jerob92,
Can you create a pull request for this plugin to include the android background thread?
It will be amazing to use this feature :)

[jerbob92]

@Pkurto no, my fork changes a lot of methods, so it's not backwards compatible. It's also not really useful to create a pull request when the iOS version is not finished/stable yet. If you do want to use it you can use my fork.

[Pkurto]

@jerbob92 thank you for the quick response and effort to develop this feature. I think this is one of the most important enhancements for this plugin.

I was only suggesting a PR only because I was thinking that the Android code could be added without the iOS alterations.
Also, I am using this plugin in a Project and @triniwiz already corrected an issue related with case insensitive search. If I use your fork I will lose that correction :(

There is any documentation that can help If I manage to get some time to see if I can help with iOS version?

[jerbob92]

@Pkurto That's possible but then in my opinion the plugin is not complete. If you want recent fixes in my fork you can create your own fork of my fork and then merge the commits from @triniwiz.

There's not really any documentation for the iOS version, and we are also not planning on using this plugin anymore so we're not going to put more time into it.

[senner007]

A simple solution is to just call the database queries on a worker thread.
https://docs.nativescript.org/core-concepts/multithreading-model