Apologies if this isn't the right place, but I have moved instances a few times, and in doing so I've accumulated a number of subscriptions from my old instances. I always used the "migrate" feature, and I have roughly 200 communities I'm subscribed to. Most are from various other instances that aren't my own. That being said, I have Lemmy open on "subscribed" and "scaled", as I found this gives me just what I want to see when I boot up the site.

But it is very slow, sometimes even timing out on Jerboa. I thought it was my instance, but I've had this issue on other instances as well. I'd imagine federating all my subscriptions would take some time, but then wouldn't "All" cause even more problems? I've tried to look into this to no avail.

Am I the only one? I'm asking this more out of curiosity than an actual solution.

you are viewing a single comment's thread
view the rest of the comments

[–] wjs018@piefed.social 18 points 9 hours ago* (last edited 9 hours ago) (1 children)

This is pretty much always going to be the case because the database query is much more complicated to filter by your subscriptions rather than the completely unfiltered views that are just sorted a certain way.

I don't have direct experience working in the lemmy codebase, but I do in piefed, and this is just an inherent limitation of how databases work. Basically, there is a big table in the db with all the posts that the instance knows about. As an example, let's say that you want to look at your Subscribed feed and sort it by New. First, the database sorts all the posts by newest first (this step is very quick). Then it will filter out all the entries that are not in a community that you are subscribed to (this step is a lot slower). So, views where the database doesn't need to do that filtering step are a lot faster than others. There are other things that slow down queries as well like blocks or keyword filters.

Smarter admins than I can employ strategies to tune their database for better performance. I don't have any knowledge in this area though, so others would be able to provide more insightful answers.

[–] srasmus@slrpnk.net 8 points 9 hours ago (3 children)

Interesting. So when you query subscriptions, it actually queries everything and then filters it.

I decided to look at the github for Lemmy and found this issue that seems to reflect what youre saying.

I'm having flashbacks of avoiding table scans during my internship. 🙂

[–] SlurpingPus@lemmy.world 5 points 9 hours ago* (last edited 9 hours ago)

Strictly speaking, the db might be looking in an index to choose rows by the communities — but using such a condition is pretty much guaranteed to be slower than not using it, anyway.

The actual answer depends on the actual database organization, of course. Ideally the whole database should be organized around frequent queries.

[–] poVoq@slrpnk.net 4 points 4 hours ago* (last edited 4 hours ago)

It also further links to another issue about individually blocking users and communities. Apparently that is quite inefficient in the current version, so maybe that adds to your problem?

[–] bjoern_tantau@swg-empire.de 2 points 1 hour ago

Lemmy's databse schema was updated for 1.0. I hope that will address many of the performance problems. It's due to release soonish. The beta is just around the corner.