Performance issues with large databases about tablex

[christoff-linde]

Yoo! Sorry for the delay, been a little busy.

I implemented Pagination and Virtualization 🎉. Now it should feel smoother, I have pushed a minor release 🚀

Insanely cool. It works flawlessly! Thanks for the awesome updates

[christoff-linde]

Hmmm, that all sounds like a solid plan.

I would be keen to try and help out with code if you decide to split up the work into different tasks/PRs. So I will keep an eye on this issue and the repo in general and then will let you know if I'm planning to pick anything up :)

Otherwise I'm happy to just help test stuff :)

[kareemmahlees]

Yoo!
Sorry for the delay, been a little busy.

I implemented Pagination and Virtualization 🎉.
Now it should feel smoother, I have pushed a minor release 🚀

[kareemmahlees]

Welcome back! 🥳

Thank you again for your descriptive and informative issues ⭐

I anticipated this issue because of ( as you have figured out ) the naive way of handling getting rows, so I was planning to handle this because I think of TableX handling all kinds of DB load ( i.e. big or small data ).

And you had the same thoughts and solutions that came up to me while I was thinking about it!

Let me break down how I think of it:

• Frontend
  • As you have mentioned, virtualizing the table view will greatly help with the performance, so I was planning to use Tanstack Virtual for this. ( because we already use the Query & Table so why not :) )
  • I am also thinking of implementing Infinite Scrolling instead of pagination for now, we already use Tanstack Query which I think has some docs about Infinite Scrolling.
    This step though will require some changes in the backend, listed below.
• Backend
  • the get_rows function should be get_rows_paginated taking page as an input in order to work with Infinite Scrolling.
    This change will help us both in the short-term and long-term because in the future I am planning, hopefully, that we implement settings where the user can specify whether he wants pagination+ specifying the limit number of the page or Infinite Scrolling

This change will be quite some work so I might split it into multiple PRs or just smash it into a single one idk yet.

I will keep the issue updated, you are obviously more than welcome if you would like to help in any part or if you would like to continue playing around with TableX and reporting the bugs you encounter. 😄

[christoff-linde]

Something that could be cool could be to have similar behaviour to JetBrains DataGrip, where the results are paginated by default, and you can navigate through the different pages