And in other news, all @ungleich hosted #netbox instances have been updated to version 4.1.8. Enjoy!

Heads up, all @ungleich maintained #mastodon instances have been upgraded to 4.3.2

Axpo verified - they made a mistake on synchronising a machine on the 220kV line and thus causing an emergency switch to turn off power.
This affected the whole back valley (place10, place6, place7)

Axpo switched over to emergency - we got power back.
Storage cluster for VMs is back in write mode, I/O performance currently degraded.

We got an update that the problem comes from Axpo affecting the grid to which we are also connected.

Affected: place10 (VM, kubernetes based services)
Not affected: place5 (colocation)

The whole region of Glarus is having a power outage.
All systems are still up, running on battery power at the moment. We are in touch with the grid and local power plant to get insight on what is going on.

So #openwrt does not (yet) work with #cdist. A short report can be found on https://ungleich.ch/u/blog/2024-08-02-openwrt-no-cdist-yet/ - let's see what the future brings.

All VMs are back online.

Most virtual machines are back online. ETA for the rest is 20 minutes.

All virtual machines are restarting.

Majority of storage cluster is back online.

Storage cluster is rebooting.

@xtaran There was a power outage in place10. We are investigating the cause and now restarting all affected systems.

There was a power outage in place10. We are investigating the cause and now restarting all affected systems.

I'm so, so, so mad. @ungleich data centers get attacked because people are protesting against climate change. This is starting to become a digital war between big oil and non profits. And @ungleich in the middle. So, so mad. #climatechange #digitalwar #angry

We are under #DDoS attack. One of our customers, a climate organisation is being targeted. Taking counter measurements. This has really become a war of fossil fuels.

DDOS attack has been mitigated. All system back to normal.

We are experiencing a DDOS attack. Our team is working on mitigation.

All VMs have been migrated and are back up.