Next up was the update of the Grafana CNPG cluster from 16.2 to 17.6. Yet again, like for the harbor cluster, the minor update from 16.2 to 16.10 has worked, but the update from 16.10 to 17.6 has failed. The primary came up again without issue, but the secondary is failing with timeline divergence again.
I don't think I will keep CNPG in the long run. It doesn't seem to actually give me anything positive.
Ok I fudged up
Trying to recover the cluster as it stands, I deleted the forgejo db to recover it from the backups...
Thing is, I forgot to configure the backups when migrating from gitea, woops
Result, I managed to salvage some about-to-be-deleted filesystem from a node, and now I'm trying to restart the node momentarily to export a dump... FML
#homelab #selfhosted #cluster #k3s #cnpg #postgres #database #backup #recovery #filesystem #mistake
That's why I debug in prod /s
Found out a new issue with the k3s cluster, some of my pods started getting `Evicted` because of `DiskPressure`
But why ? There is plenty of disk space available ? All of my data is stored on secondary drives, so they shouldn't even clog the main disk !
Turns out, yes, all the big data is stored on secondary disks, but not my databases... Including their backups !
Some of my machines have ~30Gb of root space and an additional ~500Gb for big storage, so those little roots are getting filled up quite quickly (6 months) with those dbs...
Oh well, down to writing a new StorageClass for that effect I guess
#selfhosted #homelab #production #debug #kubernetes #k3s #cnpg #postgres #db #database #storage #linux #container
Adventures in #selfhosting!
I just finished failing to do a "correct" and "proper" upgrade of cloud-native postgres, #cnpg, from using standard #longhorn volume backups to the barman cloud plugin.
I got the plugin loaded accord to the migration docs, but couldn't get it to write to #s3, nor could the pods become ready. I worked at it for hours, but I saw lots of other people online and recently having the same issues and log messages that I was.
The reason I did this in the first place was that I noticed that I had some duplicate backup jobs causing issues with #fluxcd reconciliation.
In the end, I gave up and went back to the original longhorn backups, which have worked and I've already done disaster recovery with (don't ask), and deleted the duplicates.
Currently I'm waiting for the previous primary/write node to fully restart and clear out the barman side car. Then I'll turn flux back on and hopefully things will be good.
Nice, I finally powered back on the NAS, it everything came back up
Longhorn backups, CloudNativePG backups, MinIO, OpenmediaVault, Wireguard
The full stack came back without a hiccup
(Small sike on me tho', I had to SSH into the machine to restart the containers and wireguard because I had forgotten to enable them at boot...)
#selfhosted #homelab #kubernetes #k3s #longhorn #wireguard #cnpg #postgres #minio #openmediavault #backups #sike #nas #ssh
Today's #adventuresInSelfHosting, I was having trouble with #cloudnativePG where I'd always have n-1 pods stable and they'd constantly restart between the replicas in a massive loop, then eventually switch with the primary and continue the process. So, I set the instance count to 3 so there'd at least be one stable read replica at a time.
I finally found out what was wrong. I have my #cnpg cluster in the postgresql-system namespace and I happened to see that I had an operator running in the default cnpg-system namespace. I don't know how long it had been there, so both it and the one in my namespace were both competing for the state. Deleting and cleaning up that old cluster brought immediate stability.
I also realized that I wasn't overriding the default #php configuration for #pixelfed, so when I uploaded an image taking on my phone, the web server would restart. I bumped the php memory up to 1GB for now. For the expected userbase for the upcoming #keyboardvagabond #fediverse space, this should be fine.
Right now the services are running well, but I need to do more testing and get mastodon into an "interesting" state for new visitors. Pixelfed seems the hardest for me in terms of getting content onto the server so that it doesn't look barren.
The todo list for now is:
It's getting close! The services are essentially ready, just not necessarily turned on for signups until I'm ready for a pre-launch or full launch. I want to make sure things are in a good state.
But with the 2 node #kubernetes #cluster, I think things should be good!
By then end, it should look like:
Great productivity tonight !
Upgraded Immich from 1.132.1 to 1.135.3
Migrated its database from pgvector.rs to VectorChord
Upgraded to VectorChord 0.4.3 from 0.3.0
That was actually relatively easy once I started taking my time and reading the migration documentation
Now everything is healthy and up to date, let's see how those performance improvements translate to realworld applications
#immich #selfhosted #homelab #kubernetes #vectorchord #pgvector #psql #cnpg #rtfm
Had to downgrade Immich back to a pre 1.133 (here 1.132.3) because of the problem with the pgvector.rs to VectorChord migration on CloudNativePG...
This time, it seems quite complex, and will need more proper time allocated...
Det funkar! Det funkar! CloudNative PG backups i MinIO S3-compatible object storage.
#Selfhosted #Selfhosting #Kubernetes #K8s #MinIO #ObjectStorage #CloudNativePG #CNPG
Första stateful applikationen live i #Kubernetes-klustret! #FreshRSS med CNPG för att hantera databasen. Får bli lite hackigt med uppsättningen av första användaren bara, för att den inte ska nollställa lösenordet. Eller så lagrar jag adminanvändarens lösenord i en secret kanske.
Tappar lite plugins som jag inte löst än. Enklaste sättet att få in dem är antagligen bara att göra en egen image antar jag.
#PostgreSQL #CloudNativePG #CNPG #selfhosted #selfhosting #homelab
Great new v1.26 release of CloudNative-PG out, providing scalable and highly available Postgres on Kubernetes.
https://cloudnative-pg.io/releases/cloudnative-pg-1-26.0-released/
Glad to see the in-place upgrades and extensions. Intrigued about the backup plugins, as I'm interested in integrated logical backups (pg_dump).
Check it out!
So yesterday about this time, I was #poolside somewhere in #Türkiye or #Turkey if you will, when our fearless moderator @leeleedee notified me that https://mstdn.dk was down. The root cause was the rack mounted Synology RS1221+ NAS in my basement, serving as the S3 backup backend for the #CNPG (Cloud Native Postgres) which was failing to get back online after a reboot. When I arrived home today, all it took was a hard reset, and everything is running fine again. This really shouldn't happen, so I'll put it through some rigorous testing in the coming days.
#CloudNativePG : Julian présente la fonctionnalité qui permet d'y installer des #plugins à partir d'un exemple concret =>
https://dali.bo/202503_cloudnativepg-6
#PostgreSQL #SGBD #DBA
#opensource #logicielsLibres
#CNPG
@CloudNativePG
Déployer #PostgreSQL dans un cluster #Kubernetes : que faire en cas de saturation de l'espace disque ? 💥
Dans sa série dédiée à l'opérateur #CloudNativePG, Pierrick vous propose une solution => https://dali.bo/202503_cloudnativepg-5
#SGBD #DBA
#opensource #logicielsLibres
#cnpg CloudNativePG
#pg_bench
Cluster rebuild project
Ok so the databases for the last two services have been shrunk* by removing their current backup targets and forcing a checkpoint.
The #Sharkey db is 25gb currently, I should look if there are ways to shrink it.
I still need to figure out exactly what is happening with backups that cause the db to grow like that. Something isn't being released properly
Also the Ceph dashboard ingress ate shit, not quite sure why but I'm able to expose the service via a load balancer and that works
#Kubernetes #Homelab #Ceph #CNPG #Postgres
Lancer une bouée à un éléphant est tout un art !
Cette semaine, Pierrick évoque différentes méthodes de #sauvegarde d'instance #PostgreSQL avec #cloudNativePG => https://dali.bo/202502_cloudnativepg-4
#SGBD #DBA
#opensource #logicielsLibres
#cnpg @cloudnativepg
#pg_dump #PITR #backup #barman_cloud
Notre série dédiée à #cloudNativePG continue !
Cette fois, Julian présente des solutions de stockage et de performance =>
https://dali.bo/202502_cloudnativepg-3
#PostgreSQL #SGBD
#opensource #logicielsLibres
#cloud #CNPG
CloudNativePG
#longhorn #pgbench
2e article dédié à #cloudnativepg 🚢 🐘
Cette fois, Pierrick aborde les services, le déploiement de #pgAdmin4 dans #kubernetes, et le rôle des labels => https://dali.bo/202501_cloudnativepg-2
Currently on my regular service update again, and I saw a nice update in CloudNativePG: https://github.com/cloudnative-pg/cloudnative-pg/pull/4404
It's a new feature which will safely stop a cluster when the WAL files cannot be written due to disk space exhaustion. I had that problem when I set up my first cluster. It was throwing some quite hard to understand error messages.
