Apologies if this isn't the right place, but I have moved instances a few times, and in doing so I've accumulated a number of subscriptions from my old instances. I always used the "migrate" feature, and I have roughly 200 communities I'm subscribed to. Most are from various other instances that aren't my own. That being said, I have Lemmy open on "subscribed" and "scaled", as I found this gives me just what I want to see when I boot up the site.

But it is very slow, sometimes even timing out on Jerboa. I thought it was my instance, but I've had this issue on other instances as well. I'd imagine federating all my subscriptions would take some time, but then wouldn't "All" cause even more problems? I've tried to look into this to no avail.

Am I the only one? I'm asking this more out of curiosity than an actual solution.

top 11 comments

sorted by: hot top controversial new old

[–] wjs018@piefed.social 15 points 4 hours ago* (last edited 4 hours ago) (1 children)

This is pretty much always going to be the case because the database query is much more complicated to filter by your subscriptions rather than the completely unfiltered views that are just sorted a certain way.

I don't have direct experience working in the lemmy codebase, but I do in piefed, and this is just an inherent limitation of how databases work. Basically, there is a big table in the db with all the posts that the instance knows about. As an example, let's say that you want to look at your Subscribed feed and sort it by New. First, the database sorts all the posts by newest first (this step is very quick). Then it will filter out all the entries that are not in a community that you are subscribed to (this step is a lot slower). So, views where the database doesn't need to do that filtering step are a lot faster than others. There are other things that slow down queries as well like blocks or keyword filters.

Smarter admins than I can employ strategies to tune their database for better performance. I don't have any knowledge in this area though, so others would be able to provide more insightful answers.

[–] srasmus@slrpnk.net 5 points 4 hours ago (1 children)

Interesting. So when you query subscriptions, it actually queries everything and then filters it.

I decided to look at the github for Lemmy and found this issue that seems to reflect what youre saying.

I'm having flashbacks of avoiding table scans during my internship. 🙂

[–] SlurpingPus@lemmy.world 4 points 4 hours ago* (last edited 4 hours ago)

Strictly speaking, the db might be looking in an index to choose rows by the communities — but using such a condition is pretty much guaranteed to be slower than not using it, anyway.

The actual answer depends on the actual database organization, of course. Ideally the whole database should be organized around frequent queries.

[–] Pudutr0n@feddit.cl 8 points 5 hours ago

Happens to me too. Jerboa frequently times out as well. I thought it was because my access point is running on potato hardware, but i guess it's a thing other places too.

[–] paequ2@lemmy.today 8 points 4 hours ago

I've noticed the more subscriptions I add, the slower my feed loads. All does seem to load faster for me as well.

[–] SlurpingPus@lemmy.world 6 points 4 hours ago

To add to other answers, the result for the ‘all’ feed is likely to be cached, either explicitly by the server app or implicitly by the database. Personal feeds are less likely to be cached, since they're only used by individual users.

[–] your_paranoid_neighbour@lemmy.dbzer0.com 5 points 4 hours ago* (last edited 4 hours ago)

Every instance is like this it seems, maybe lemmy doesn't make a batched select at the cache and the db, but idk.

Maybe a new ~~webscale~~ backend could solve this, but i'm just a java backend developer, not a rust one. Maybe could be an interesting project, scala + pulsar + batching, turning lemmy into a microservices hellscape.

Edit: but

[–] nightwatch_admin@feddit.nl 3 points 4 hours ago (1 children)

Same, thought it was just me or my instance (feddit.nl), started after some update. Unsubscribed from a lot of communities and switching to limit posts per page helped a bit. Te-subscribing made it somewhat worse again. IIRC it is some instances or communities specifically.

[–] dabaldeagul@feddit.nl 1 points 17 minutes ago

If you remember which instances are the worst, please let me know. I still get time-outs more often than I'd like... Though it has become less after the server upgrade Ted did.

[–] avidamoeba@lemmy.ca 2 points 4 hours ago

I don't see a noticeable slowness on lemmy.ca with Subscribed+Scaled. That said it could be I don't have enough subs.

[–] angelmountain@feddit.nl 2 points 2 hours ago

Just wanted to add "for me too". Sometimes my app takes a few minutes to load/time out the "subscribed" feed, while the "all" feed works fine and even my notification page has no issues.