Apologies if this isn't the right place, but I have moved instances a few times, and in doing so I've accumulated a number of subscriptions from my old instances. I always used the "migrate" feature, and I have roughly 200 communities I'm subscribed to. Most are from various other instances that aren't my own. That being said, I have Lemmy open on "subscribed" and "scaled", as I found this gives me just what I want to see when I boot up the site.

But it is very slow, sometimes even timing out on Jerboa. I thought it was my instance, but I've had this issue on other instances as well. I'd imagine federating all my subscriptions would take some time, but then wouldn't "All" cause even more problems? I've tried to look into this to no avail.

Am I the only one? I'm asking this more out of curiosity than an actual solution.

top 18 comments

sorted by: hot top controversial new old

[–] wjs018@piefed.social 18 points 16 hours ago* (last edited 16 hours ago) (1 children)

This is pretty much always going to be the case because the database query is much more complicated to filter by your subscriptions rather than the completely unfiltered views that are just sorted a certain way.

I don't have direct experience working in the lemmy codebase, but I do in piefed, and this is just an inherent limitation of how databases work. Basically, there is a big table in the db with all the posts that the instance knows about. As an example, let's say that you want to look at your Subscribed feed and sort it by New. First, the database sorts all the posts by newest first (this step is very quick). Then it will filter out all the entries that are not in a community that you are subscribed to (this step is a lot slower). So, views where the database doesn't need to do that filtering step are a lot faster than others. There are other things that slow down queries as well like blocks or keyword filters.

Smarter admins than I can employ strategies to tune their database for better performance. I don't have any knowledge in this area though, so others would be able to provide more insightful answers.

[–] srasmus@slrpnk.net 8 points 16 hours ago (3 children)

Interesting. So when you query subscriptions, it actually queries everything and then filters it.

I decided to look at the github for Lemmy and found this issue that seems to reflect what youre saying.

I'm having flashbacks of avoiding table scans during my internship. 🙂

[–] SlurpingPus@lemmy.world 5 points 16 hours ago* (last edited 15 hours ago)

Strictly speaking, the db might be looking in an index to choose rows by the communities — but using such a condition is pretty much guaranteed to be slower than not using it, anyway.

The actual answer depends on the actual database organization, of course. Ideally the whole database should be organized around frequent queries.

[–] poVoq@slrpnk.net 5 points 10 hours ago* (last edited 10 hours ago)

It also further links to another issue about individually blocking users and communities. Apparently that is quite inefficient in the current version, so maybe that adds to your problem?

[–] bjoern_tantau@swg-empire.de 2 points 8 hours ago

Lemmy's databse schema was updated for 1.0. I hope that will address many of the performance problems. It's due to release soonish. The beta is just around the corner.

[–] paequ2@lemmy.today 10 points 16 hours ago

I've noticed the more subscriptions I add, the slower my feed loads. All does seem to load faster for me as well.

[–] poVoq@slrpnk.net 10 points 11 hours ago

Aside from general issues others have mentioned, our instance (slrpnk.net) is seeing some especially high database load in the last couple of days and I also noticed the subscribed page to be even slower than usual. I tried to figure out what it causing it, but so far there is no clear smoking gun, but I suspect some AI scrapers found a way to target the Lemmy API directly so our current scraper protections for the webinterface are inadequate.

[–] Pudutr0n@feddit.cl 9 points 16 hours ago

Happens to me too. Jerboa frequently times out as well. I thought it was because my access point is running on potato hardware, but i guess it's a thing other places too.

[–] SlurpingPus@lemmy.world 9 points 15 hours ago (1 children)

To add to other answers, the result for the ‘all’ feed is likely to be cached, either explicitly by the server app or implicitly by the database. Personal feeds are less likely to be cached, since they're only used by individual users.

[–] wjs018@piefed.social 1 points 6 hours ago (1 children)

This is probably true for anonymous (not logged in) users, but might not be if you are logged in. Things like user/community/instance blocks, votes, read/unread status, etc. need to be accounted for when generating the main feed for logged in users, and those are dynamic. It might be cached for some time, but the retention time would likely be short (a couple minutes maybe).

I haven't worked too much with the caching that piefed does, but I know it keeps some info in a redis cache to help speed things up (blocks, votes, etc.). That way getting all that info doesn't need to be a new db query every time except maybe the first time. Some pages also make use of etags so that if the content is unchanged, the request doesn't need to hit the db at all. I imagine lemmy has some mechanisms to do caching as well, but I am unfamiliar with the code at that level.

[–] SlurpingPus@lemmy.world 1 points 5 hours ago

It's likely much faster to fetch the common feed from the database cache or prepared cache like Redis, and apply all this additional data in the app, than do uncached joins. So I'd hope that the apps do this. Especially since you say they use Redis, which of course doesn't do joins and such, unless something changed in the past years.

[–] angelmountain@feddit.nl 6 points 14 hours ago

Just wanted to add "for me too". Sometimes my app takes a few minutes to load/time out the "subscribed" feed, while the "all" feed works fine and even my notification page has no issues.

[–] your_paranoid_neighbour@lemmy.dbzer0.com 6 points 16 hours ago* (last edited 16 hours ago)

Every instance is like this it seems, maybe lemmy doesn't make a batched select at the cache and the db, but idk.

Maybe a new ~~webscale~~ backend could solve this, but i'm just a java backend developer, not a rust one. Maybe could be an interesting project, scala + pulsar + batching, turning lemmy into a microservices hellscape.

Edit: but

[–] poVoq@slrpnk.net 6 points 11 hours ago

What you can try is to clear your browser cache for the main domain. In the past there was a bug in Lemmy that caused Firefox based browsers to accumulate many gigabytes of cache data and that slowed down the loading of the page significantly. In the latest version there are some fixes for this and it shouldn't effect app usage, but I suspect this problem still persists to some extend.

[–] nightwatch_admin@feddit.nl 5 points 16 hours ago (1 children)

Same, thought it was just me or my instance (feddit.nl), started after some update. Unsubscribed from a lot of communities and switching to limit posts per page helped a bit. Te-subscribing made it somewhat worse again. IIRC it is some instances or communities specifically.

[–] dabaldeagul@feddit.nl 4 points 12 hours ago (1 children)

If you remember which instances are the worst, please let me know. I still get time-outs more often than I'd like... Though it has become less after the server upgrade Ted did.

[–] nightwatch_admin@feddit.nl 1 points 7 hours ago

If I remember,or figure out which ones, I’ll let you know, thanks!

[–] avidamoeba@lemmy.ca 3 points 15 hours ago

I don't see a noticeable slowness on lemmy.ca with Subscribed+Scaled. That said it could be I don't have enough subs.