User model will not scale with embeded lists of references to followers

[face]

What version of Node.js are you using?

v14.18.1

What version of Yarn are you using?

1.22.10

What browser are you using?

Firefox

What operating system are you using?

Fedora 34

Describe the Bug

The user model has links to all followers and following. If one famous person uses the network, and has 15 million followers, their user document will have 171 megabytes of follower links (a mongodb _id is 12 bytes). Fetching the user model will be slow.

Reference to scalability issue in source code: https://github.com/ElevenSymbols/orca/blob/536dc81096fcd2799eb3ed599c392769296b3540/packages/orca-api/src/db/follow.ts#L17

Other embedded references in the User document will cause similar issues. Over time a user might contribute lots of posts, messages, and comments which all cause the User document to grow without bounds.

Expected Behavior

A user with millions of followers will have a small User document in mongodb.

Steps to reproduce

1. Create a popular social network

[DimiMikadze]

hi @face , if the app will have dozens of million active users there might be other issues, but I don't understand what do you suggest?

[face]

@DimiMikadze I'm thinking cache only recent data in the User model (a page worth of data), then for subsequent pages do a query in the Follow document with indexed searches. Of course, apply this same logic to all models that cache references, not just the User model.

[face]

Honestly, mongodb is not the best for graph relationships. This is a very interesting approach using a combination of persistence engines: https://neo4j.com/blog/neo4j-doc-manager-polyglot-persistence-mongodb/

[face]

It's worse than I thought. Mongodb documents have a 16MB limit so follows would simply stop functioning before a user had millions of followers.

The neo4j is overkill, mongodb indexes on the join tables should work. I think we don't even need to cache any recent data in the user Model...just always query the join tables, like Follows. That is what mongo recommends for "One-to-Squillions": https://www.mongodb.com/developer/article/mongodb-schema-design-best-practices/#one-to-squillions

[DimiMikadze]

@face, sorry for the late response. To be honest, after building Orca, I realized using a relational database like Postgres would make more sense since almost all the data is related to each other. Unfortunately, I don't have time to do this refactoring at the moment.

I agree that having dozens of million users will cause scaling issues, but honestly, I don't think anyone is using Orca at that scale.

Anyway, if you are open to sending a PR improving the user model scalability, I'd be glad to review and merge it.

Thanks once again for the detailed issue!

[face]

Hi @DimiMikadze sorry my late response now, I'm on vacation again and playing with this problem.

The solution I came up with was tri database (didn't use orca so I can't open a PR :(. To summarize, this is what I did to have a solution that would scale to millions followers:

1. Use Postgres for auth and a source of truth for any data which must be 100% consistent.
2. Use Cassandra for all social data
3. Use Redis for caching

Postgres is one of the best sources of truth, but doesn't scale or shard well. Cassandra is great at scaling, sharding, and, grouping and finding "related" data. I had never used cassandra before but it's awesome. Although it's not a relational database, it's better at finding related data than a relational database. You can specify part of your combination key for a table to dictate how the data is sorted and written to disk. So when you search for related data, it is a pre-sorted column of data to read from the actual db files. You can't do relational things like joins in queries, so you pre-build structures like a user's feed in a table (which is ultimately what you want for performance and scale).