User model will not scale with embeded lists of references to followers about orca

[DimiMikadze]

@face, sorry for the late response. To be honest, after building Orca, I realized using a relational database like Postgres would make more sense since almost all the data is related to each other. Unfortunately, I don't have time to do this refactoring at the moment.

I agree that having dozens of million users will cause scaling issues, but honestly, I don't think anyone is using Orca at that scale.

Anyway, if you are open to sending a PR improving the user model scalability, I'd be glad to review and merge it.

Thanks once again for the detailed issue!

[DimiMikadze]

hi @face , if the app will have dozens of million active users there might be other issues, but I don't understand what do you suggest?

[face]

@DimiMikadze I'm thinking cache only recent data in the User model (a page worth of data), then for subsequent pages do a query in the Follow document with indexed searches. Of course, apply this same logic to all models that cache references, not just the User model.

[face]

Honestly, mongodb is not the best for graph relationships. This is a very interesting approach using a combination of persistence engines: https://neo4j.com/blog/neo4j-doc-manager-polyglot-persistence-mongodb/

[face]

It's worse than I thought. Mongodb documents have a 16MB limit so follows would simply stop functioning before a user had millions of followers.

The neo4j is overkill, mongodb indexes on the join tables should work. I think we don't even need to cache any recent data in the user Model...just always query the join tables, like Follows. That is what mongo recommends for "One-to-Squillions": https://www.mongodb.com/developer/article/mongodb-schema-design-best-practices/#one-to-squillions

[face]

Hi @DimiMikadze sorry my late response now, I'm on vacation again and playing with this problem.

The solution I came up with was tri database (didn't use orca so I can't open a PR :(. To summarize, this is what I did to have a solution that would scale to millions followers:

1. Use Postgres for auth and a source of truth for any data which must be 100% consistent.
2. Use Cassandra for all social data
3. Use Redis for caching

Postgres is one of the best sources of truth, but doesn't scale or shard well. Cassandra is great at scaling, sharding, and, grouping and finding "related" data. I had never used cassandra before but it's awesome. Although it's not a relational database, it's better at finding related data than a relational database. You can specify part of your combination key for a table to dictate how the data is sorted and written to disk. So when you search for related data, it is a pre-sorted column of data to read from the actual db files. You can't do relational things like joins in queries, so you pre-build structures like a user's feed in a table (which is ultimately what you want for performance and scale).