Our instance is currently down and I can't get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can't work around remotely.

I am still trying a few things, but I am not very optimistic that I can get access.

The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can't access it physically during the next few weeks and my usual back up that could restart it is not available either.

As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy's law seems inescapable 😓

I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing 😥

Edit: we might use this "opportunity" to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.

top 50 comments

sorted by: hot top controversial new old

[–] tofu@lemmy.nocturnal.garden 94 points 2 days ago (3 children)

Solarpunks can have a little downtime, as a treat

[–] evasive_chimpanzee@lemmy.world 57 points 2 days ago (1 children)

Seriously, I think a big part of solarpunk ethos is combating the notion that everything has to always be available 24/7. Society pays a lot to deliver every convenience like fruit out of season from the other side of the world.

[–] stabby_cicada@lemmy.blahaj.zone 25 points 2 days ago

And on the punk side of things, anybody familiar with volunteer community action, cooperative events, whatever, knows that sometimes a person doing shit for the community fails to do the shit they were supposed to do because some other shit happened in their life. It happens. We just move on to the backup shit-doer-person or find a workaround for the lack of shit.

And this is the cool thing about a distributed network like Lemmy. If a giant centralized database like FB goes down, its users can't do anything. If our favorite Lemmy node goes down, we can log into other nodes until our home base is back up.

load more comments (2 replies)

[–] ZonenRanslite@feddit.org 45 points 2 days ago (2 children)

Work trip and server problems. Where have I seen this before?

[–] Kris@feddit.org 30 points 2 days ago (1 children)

Yeah, but at least I have not disappeared yet 😅

[–] ZonenRanslite@feddit.org 13 points 2 days ago (3 children)

I see you understand the allusion.

load more comments (3 replies)

load more comments (1 replies)

[–] Sergio@lemmy.world 38 points 2 days ago

Sorry to hear that! My main account is on slrpnk.net. But don't let it stress you out, "solarpunk" says "punk" which implies "DIY" which means this sort of thing happens. Much better to have an instance run by an admin who truly cares, rather than some soulless corporation.

[–] Initiateofthevoid@lemmy.dbzer0.com 34 points 2 days ago (1 children)

Try not to stress too much about it while you're away! If nothing can be done, nothing can be done. Solarpunks need to be resilient to things like this anyway. Community over convenience.

[–] Kris@feddit.org 19 points 2 days ago

Thanks, I am trying, but it really is quite annoying to lose access like that.

[–] PirateFrog@lemmy.dbzer0.com 32 points 2 days ago (3 children)

Alt of ProdigalFrog@slrpnk.net chiming in, thanks for giving us the sitrep, poVoq!

This is an unusually unlucky set of circumstances, but for the Solarpunks who find their way here, let's try to make the best of it. If you don't already have an alternative account somewhere, you may want to create one to use until we're back up and running. lemmy.dbzer0.com, sopuli.xyz, or lemmy.cafe are some solid choices to consider. After that, you may want to favorite poVoq's post here so you can check back on any news.

Even without our home base, we can still share the solarpunky stuff we find and create around the threadiverse (though as Tofu mentioned, maybe this could be a time to take a breather to be refreshed when we come back 😄).

Besides, if the past is any indication, time will fly by and we'll be back before you know it!

[–] Kris@feddit.org 13 points 2 days ago (6 children)

Well, on the plus side lots of lessons learned and I think I might move at least the xmpp server to an external vps to have a backup communication channel.

load more comments (6 replies)

load more comments (2 replies)

[–] grysbok@lemmy.sdf.org 32 points 2 days ago (6 children)

This is making me want to create an alt on slrpnk.net , since you're so chill and communicative about issues.

[–] threelonmusketeers@sh.itjust.works 12 points 2 days ago* (last edited 2 days ago) (1 children)

communicative about issues

Aren't most admins this way?

*glances at your instance name*

Ah...

[–] grysbok@lemmy.sdf.org 9 points 2 days ago* (last edited 2 days ago) (4 children)

Lol. Yeah, sdf.org feels pretty hands-off. They have longevity going for them--I've had an sdf account of some sort since 2009 at the latest. The system started life as a BBS in the 80s. The name comes from the anime "Super Dimension Fortress", which I've never seen.

load more comments (4 replies)

[–] Five@lemmy.cafe 8 points 2 days ago (8 children)

You're very welcome! Kris@feddit.org / @poVoq@slrpnk.net is an incredible admin, and up until now has declined donations, funding the entire instance several years now from his own pocket.

This current downtime is the exception that proves the rule. I hope he's not stressing out too much. We'll all be ready to jump right back into it when he gets back.

load more comments (8 replies)

load more comments (4 replies)

[–] sabreW4K3@lazysoci.al 26 points 2 days ago

Thanks for letting us know

[–] Blaze@lemmy.dbzer0.com 21 points 2 days ago

Thank you for letting us know, good luck!

[–] stabby_cicada@lemmy.blahaj.zone 19 points 2 days ago

Shit happens. Thanks for the update 🙂

[–] Sergio@lemmy.world 19 points 2 days ago (4 children)

Hey fellow slrpnk.net refugees... if you're looking to recreate your list of subscribed communities, here's a good place to start:

click here for a list of communities that are NOT politics, tech, or meme -related.

Most are currently active (except for the ones with a * which were less active last I checked) Sometimes politics, tech or memes sneak in but they're not the focus.

GENERAL DISCUSSION / QUESTIONS

ART / PHOTOS

ANIMALS

COMICS / GRAPHIC NOVELS

ENTERTAINMENT

GENRES / STYLES

HISTORY

INFORMATION / KNOWLEDGE

OTHER

FEDIVERSE

FINDING NEW/GOOD COMMUNITIES ON LEMMY

click here for a list of (mostly non-political) meme communities

MEMES, SOCIAL MEDIA REPOSTS, AND HUMOR

Most of these are currently active. (except for the ones with a * which were less active last I checked). Sometimes these include politics but that's not the sole focus.

(luckily I was posting this to newcomers whenever I had the chance...)

load more comments (4 replies)

[–] ocean@lemmy.selfhostcat.com 18 points 1 day ago (1 children)

I this it’s a law that servers run 100% perfect until the literal day one leaves town with zero way to return home. One of the many reasons I got all my services off of unraid.

Very cool to learn you’re running your own machines. Do you go into detail about this anywhere?

[–] Kris@feddit.org 9 points 1 day ago* (last edited 1 day ago)

We have a small write up about the hardware on our wiki, but it is also down right now.

I think we will share a post-mortem write up of the actual improvements we will do to avoid this in the future.

One thing I will definitly do is to add a KVM remote management console to one of our server boards and move the main firewall into a VM with hardware passthrough of the NICs (this was anyways planned for a 10gbit network upgrade for the second half of 2025). This way I should be able to reboot and even reinstall the main ingress point remotely, so that only the fiber gateway remains as a failure point that requires physical access.

[–] qaz@lemmy.world 18 points 2 days ago

Thanks for the update, good luck with your trip.

[–] FrostyTrichs@crazypeople.online 16 points 2 days ago

Ouch. Best of luck getting it back up and running ASAP.

[–] compostgoblin@sh.itjust.works 12 points 2 days ago

Thanks for letting us know! Try not to worry about it too much, it’s all good

[–] MelodiousFunk@startrek.website 11 points 2 days ago

Thanks for the heads up, try not to stress too much.

[–] fxomt@lemmy.dbzer0.com 10 points 2 days ago (1 children)

Good luck and all the best. Time to touch grass, everyone 😔

load more comments (1 replies)

[–] bacon_pdp@lemmy.world 10 points 2 days ago

It happens, it’ll be fine. Just remember not to stress out too hard about it and it’ll be fine. The solar punk community knows how to survive a server outage in style.

[–] AlexisFR@jlai.lu 9 points 2 days ago (1 children)

That's unlucky, but is it hosted at someone's home?

[–] Kris@feddit.org 23 points 2 days ago (1 children)

Sort of yes. It is a homelab a few people have access to, but currently they are all either on holiday or on extended work related travel. It seemed like an acceptable risk as everything worked smoothly for years.

[–] AlexisFR@jlai.lu 14 points 2 days ago

That makes sense, given the ethos of your instance, I hope you can fix it a soon as reason allows!

load more comments