Selfhosted

For me I'd just say oh well, gotta fix it when I'm home again.

Otherwise I'd probably write some script on the server, which reboots my router when the server either doesn't have internet anymore or can't ping itself.

It didn't happen in more than 5 years...

For critical equipment you need to spend the extra dollar to minimize this kind of stuff

If you expect it to be flaky you could get one of those old school mechanical time switches with the clicky pegs (or a more modern digital equivalent) and just have it set to power down for 1 click, normally 15 mins, at 4am or whenever suits you - minimally technically complicated and guaranteed stability through planned instability!

I had a 4G modem with a web interface many years ago. It was flaky and would often hang. I just had a raspberry pi on my network pinging some known address, if it failed for long enough it'd replay the commands to restart the web interface.

If I'd have the same problem today I'd probably have home assistant power cycle the router with a smart plug.

I’ve got one of those KeepConnect smart plugs which monitors a few different external servers and their own cloud, and automatically power cycles its outlet if things don’t work. They’ve damn near doubled in price since I’ve bought mine but it does work very well for me. Annual fee is reasonable too.

I could build something similar but I have too many projects as it is, and I feel I’d be fiddling with it endlessly just because I can. This is literally set and forget and in the last 2y it’s cycled the outlet 48 times, most of them in the middle of the night, presumably with my cable provider maintenance windows.

If my lab goes down, it sucks, but that's it. I have no critical service running there.

I have some recoverability, but it requires for the main router to run. If it isn't running it's either a HW failure, which I will not fix remotely anyway or power is down. In which case, not much I can do about it neither.

I have router with OpenWRT with Wireguard and main server (NUC) on a smart plug. If the router runs and server is mishaving to the point where I cannot reboot it, I can power cycle it via the smart plug connected to the router.

You mentioned your brother lives 30mins away - well put some tiny server in his house. Having everything at your home is not build for redundancy at all. That's just the risk management, if you absolutely need access to your server, then 1 site is not going to cut it.

0 4   *   *   *    /usr/sbin/reboot

Adjust interval as needed.

Or if you want something a bit faster and less disruptive:

#!/bin/sh

NAME="$0"

logger_cmd () {
  echo $@
  logger -p daemon.info -t "$NAME[$$]" $@
}

if ! which ncat 1>/dev/null
then
  logger_cmd "ncat not found, installing..."
  opkg update && opkg install ncat
fi

chk_conn () {
  echo "Checking connectivity to $@"
  if ncat --send-only --recv-only -w 334ms $@ 2>/dev/null; then
    return 0
  fi

  logger_cmd "Cannot reach $@"
  return 1
}

restart_network_iface() {
    # TODO: Don't restart every minute

    COOLDOWN_LOCK=/tmp/internet-connectivity-watchcat.tmp
    COOLDOWN_SECONDS=300

    cooldown_time_end=$(cat $COOLDOWN_LOCK || echo 0)

    time_now="$(cat /proc/uptime)"
    time_now="${time_now%%.*}"

    cooldown_time_left=$((cooldown_time_end - time_now))

    if [ "$cooldown_time_left" -lt "1" ]
    then
        logger_cmd "Restarting network interface: \"$1\"."
        ifdown "$1"
        ifup "$1"

        cooldown_time_end=$((time_now + COOLDOWN_SECONDS))
        echo $cooldown_time_end > $COOLDOWN_LOCK
    else
        logger_cmd "Skipping interface \"$1\" restart due to cooldown. Cooldown left: $cooldown_time_left seconds"
    fi
}


logger_cmd "Checking internet connectivity..."

if   chk_conn google.com 443 \
  || chk_conn amazon.com 443 \
  || chk_conn facebook.com 443 \
  || chk_conn cloudflare.com 443 \
  || chk_conn telekom.de 443
then
  logger_cmd "Connected to internet."
else
  logger_cmd  "Not connected to internet."
  restart_network_iface "$1"
fi

In restart_network_iface use /usr/sbin/reboot instead of interface up/down and run the script every few minutes via cron or systemd timer. This was written for OpenWrt so if you use that you can use it as-is. For other systems you'd also have to adjust the logger_cmd.

You can place that on another machine and send a signal to a smart plug instead if you're worried of a locked up / frozen router. That said if your router freezes like that, you should probably change it and you should be able to run this script on it.

Its my house so ill just walk over and fix it.

If im out, nothing is too important enough that it cant be fixed in a couple of days.

On holiday, I'll turn on my router's auto-reboot option to reboot daily.

If you use a regular PC as the router you can attach a KVM to remote control and reboot it.

if the primary internet router goes offline but the internet isn’t out (ie a router reboot would fix the problem)

Maybe you just need to give it a simple power cycle remotely? There are devices that do that sort of thing, I have a Digital Loggers Web Power Switch Pro that I've used on-and-off over the years for this purpose.

https://www.digital-loggers.com/

At one point I had to relocate for half a year while needing to remote access a slightly unstable desktop that wouldn't always reboot cleanly and get stuck at the BIOS, it sometimes needed a couple of power cycles to come back online. The Power Switch was perfect for that, I'd log into it remotely and power cycle anything that was plugged into it.

It should work for routers too e.g. it can automatically power cycle something plugged into it based on different conditions like maybe it stops responding to pings or whatever. Or I guess if you had multiple IPs / multiple internet connections the switch itself can stay online and accessed remotely without needing to schedule anything automatic.

Pretty sure there are more pro-level (and more expensive) types of devices to do this sort of thing if you look around

Redundancy. I have two independent firewalls, each separately routing traffic out through two totally independent multi-homed network connections (one cable, one DSL, please god somebody give me fiber someday) that both firewalls have access to. For awhile was thinking of replacing the DSL with starlink until Elon turned out to be such a pile of nazi garbage, so for now DSL remains the backup link.

To make things as transparent as possible, the firewalls manage their IPs with CARP. Obviously there's no way to have a single public IP that ports itself magically from one ISP to another, but on the LAN side it works great and on the WAN side it at least smooths out a lot of possible failure scenarios. Some useful discussions of this setup are here.

I made an 8 outlet box with relays connected to each outlet (might post a how to). That's connected to a Pi via GPIO.

The Pi runs PiKVM, but also has a service that:

• Checks if the router can be pinged
• Checks if the internet can be pinged
• Checks if the router webUI is up

If any of those fail, it toggles the plugs for modem and router.

I run OpnSense on a 5V miniPC. I have a second one and will be setting up CARP, too.

Note: Cellular backup is more involved, but a separate Cellular inbound might not be. I've considered putting one on the Pi above.

The term to look for is out of band management. Typically this will provide serial/console access to a device, and can often perform actions like power cycling. A lot of server hardware has this built in (eg idrac for Dell, IPMI generically). Some users will have a separate oobm network for remotely accessing/managing everything else.

I have a Shelly plug that is programmed internally to turn itself on if off for 20 seconds. Home assistant turns the plug off for 10 seconds if curl ip.me fails for 15 minutes.

My modem is plugged into that.

I've never had a router or firewall crash if I wasn't fucking with it and did something ~~stupid~~ ill-advised, so I don't try that kind of stuff unless I'm home.

• If your problem is brief brownouts or similar

my experience is that some consumer broadband routers have cheap power supplies that leaves them in bad states when PCs will pull through

you could put them on a UPS.

• If your problem is that your router is unstable, you could just replace your router. Like, if you need remote access and you have a flaky router, that seems like a prime choice.

• You could have a power control device or something and have another machine on your network set up so that if it loses Internet connectivity for some sustained period of time, it power-cycles the router.

• If this is for when you're a long ways away, do you have a friend who you'd trust with a key and flipping a switch?

• I expect that there are business-oriented routers that will have integrated watchdog features that will auto-reboot if they hang. I have not gone looking, though.

• Possibly, if it's compatible with your use case, and uptime is critical enough here, having a second, backup server elsewhere, possibly not self-hosted. I mean, your connectivity is always going to be bounded by the reliability of your residential Internet connection otherwise.

I use a cheap Mikrotik LTE Router as a second route. It has the smallest data plan my provider offers - but it's enough for maintenance and if I need more due to the main line being faulty it's the same provider's fault and they pay the bill anyway.

It mainly goes into the OPNsense as a second gateway,but it also allows me to VPN in and reboot the OPN if needed.

If the OPN would be fucked totally in theory I could run the network directly over it,but that would be nasty.

A friend of mine actually has a pretty nifty solution,but he is an absolute pro at these things. He has a small device (don't ask me what SBC exactly) ping and check (I think DNS and a http check is included as well) various stages of his network, including his core switch, firewall and DSL modem. If one of them freezes the device sends a data packet via LoraWAN. He can then send a downstream command to reboot the devices.

At the enterprise level an Opengear appliance fixes this.

Tripplite PDUs have an option to perform a ping test against an ip, and if it stops responding it can power cycle the outlet of your choice.

If you want to get fancy you can advertise routes from two routers into your L3 switch and if a route goes down your switch will use the backup.

I've moved my instant messenger onto a VPS and that has a good uptime. And I'm somewhat okay if my Nextcloud and calendars don't sync. Most important data is synced anyway.

Other than that I've called my ISP a bunch of times to give me a new router, they refused, I canceled the subscription and made a new one and got a new router. And that one is better. And in doubt I'll call a family member.

Old laptop with SIM card and some outbound remote access (eg logmein, Tailscale) is what some people I know use. But it’s an expense. I just have a reboot script.

So far it never happened but just in case I always leave one of my keys at some neighbors. I do it anyway because if something happens like broken pipe, it's good when someone can just enter home without destroying our doors.