Made a backup account over on dice.camp . Just in case our #mstdndk server is out for a bit and I need to provide situation updates. You can follow that here:

@leeleedee@dice.camp

Monitoring is slowly getting up and running.. now if only I knew how to read all that! ;-)

#mastodon #postgresql #cloudnativepg #grafana #mstdndk

Velkommen til #mstdndk @lmjhe

One of the nodes of the Midgaard #Kubernetes cluster running #mstdndk is currently down, and awaiting a disk replacement. Luckily clustering works, and you shouldn't feel a thing ;-)

Resiliency for the win! :-)

Tonight is #Halloween, which also means it's the #birthday of #mstdndk! Three years ago exactly, the domain was registered and we deployed #Mastodon with my home grown Helm chart. A couple of horrible crashes and a lot of experience later, we're still alive and kicking! With backups and everything! :mastowink:

Have a great day! 🙂

#mstdndk and the underlying Kubernetes cluster died a couple of days ago. Working on recovering.

Considering switching the #MinIO backend of #mstdndk to #Garage by #deuxfleurs. 3 replicas on #Kubernetes. Anyone with real life experience and/or tips? :-)

https://garagehq.deuxfleurs.fr/

#mastodon #s3

I've been doing things I shouldn't with #Kubernetes. We're using a replicated #MinIO cluster as the storage backend on #mstdndk, which requires a boat load of storage, especially if you forget to specify any kind of retention. So far, the quick workaround for a full disk, was just to expand the filesystem. Since we're replicating across nodes, we're using #OpenEBS #LVM for local storage. Poor partitioning means we're running out of storage on the volume group, but even worse - PVCs sizes were increased before checking if we had space for it. Kubernetes is now stuck in a most unfortunate situation - it can't grow the local filesystem, as the volume group is full and you're not allowed to decrease the size request. What then? Cue https://github.com/etcd-io/auger - a tools that allows you to edit #K8s resources directly in #etcd. Obviously you should never do this, but with steady hands and clinical precision, you can get yourself out of a pickle like mine. Size was reverted and PVCs were unstuck.

Jeg oplever, at det ofte ikke er muligt at se billeder og videoer, der er posted på andre instanser end her på #mstdndk
Jeg prøvede også at uploade nogle eksempel screenshots til dette opslag, men fik en "upload failed" error.

Jeg bruger den officielle Mastodon mobile client til Android og har prøvet med og uden VPN. Er der nogen, der oplever det samme eller ved, om dette kan være en lokal server federation relateret bug? Måske @saustrup ? 🙂

@Molvorin Hej og velkommen til Mastodon og #mstdndk ! Det lyder som om du har nogle interessante hobbyer, og jeg glæder mig til at følge med på vores servers livefeed.

Den bedste måde at finde ligesindede her er at sørge for at bruge og følge hashtags, der beskriver dine interesser. Det skyldes, at fediverset ikke har nogen algoritme til at promovere dine indlæg, hvis de ikke har hashtags.

Når du har fundet de hashtags, der interesserer dig, vil du sandsynligvis finde en masse dejlige mennesker at følge og interagere med, som poster med disse hashtags. Følg så mange mennesker og hashtags, der interesserer dig, som muligt, og på den måde vil du finde dit fællesskab her.

Hav det sjovt 👍!

(Her er nogle hashtags, der kunne være interessante for dig baseret på dit indlæg: #linux #linuxmint #linux_gaming #EliteDangerous #NoMansSky #StarCitizen )

A node crashed today, and unfortunately the remaining three were overburdened with a seemingly spontaneous filesystem check of some rather large volumes, including the ones holding the three database replicas and three media storage replicas for #mstdndk. We're running #Longhorn on #Kubernetes for replicated block storage, but we're doing this on a bit of a budget, which means spinning metal and 1Gbit connections between servers. You can get quite far by very carefully prioritizing the I/O and CPU of certain processes, but once in a rare while it tumbles over. If we had the money, we'd get fit each server with 16TB NVMEs and a 10Gbit backbone, but unfortunately that's not currently the case.

Is there any benefit to running this on Kubernetes? I have no doubt about it, but I'm also convinced that our current problem is replicated block storage and requirements associated with it.

Our current setup is four i7-8700's with 64GB RAM, 1x10TB spinning metal and 2x512GB SSDs.

We're experiencing an issue with the media storage backend on #mstdndk today. We'll look into it as soon as possible, which unfortunately isn't for at least a couple of hours, as I have some stuff to take care of in the real world first ;-)

Hej #mstdndk ,
dejligt at være her. Jeg arbejder i filmbranchen som software udvikler og nørder i min fritid videre i alle mulige retninger. Digital sikkerhed, privatliv og #righttorepair er for tiden mine yndlingsemner. Glæder mig til at dele og snakke om spændende ting her inde ✌️

#introduktion #nypåmastodon

Velkommen til, @fruAnna, @mrfr, @ethan, @madmilo, @jaw, @fynn1, @sindum, @peterhelms og @chang! Nye på #mstdndk, og måske #nypåmastodon? :-) Skynd jer at få billede og profiltekst på, og overvej at lave en #introduktion 🙂

Protip: slå to-faktor authentication til for jeres konto - det kan I gøre lige her: https://mstdn.dk/settings/two_factor_authentication_methods

We'll be updating #mstdndk during the night one of the next days. Hopefully it will be dreadfully boring and you won't even notice, but just in case - if the server goes down, know that it's probably related to the upgrade, and should be fixed shortly.

We'll put out a notice a couple of hours ahead of the change.

Issues from the expired DNSSEC incident are unfortunately still affecting #mstdndk. We're still temporarily disabling DNSSEC for the affected domains, but missed a few in the first round. We'll try to improve our procedure for actually keeping the damned things updated, but there's a first time for everything. Thank you for your patience.

Issues with expired DNSSEC keys has caused a bit of an outage on #mstdndk today. DNSSEC has been temporarily disabled and everything should work as the update propagates. Will get it sorted ASAP.

Velkommen til / welcome #mstdndk @madmilo og @Fodd 👋

Velkommen til / welcome to #mstdndk @jaw

Velkommen til / welcome to #mstdndk, @fynn1 !