cool, hab jetzt auch noch meine server VM disk von sata zu virtio umgestellt 😎
fühlt sich snappier an, kann aber auch nur wunschdenken sein 🙃

die KI meinte ich hätte jetzt ein datacenter production grade setup 😂
#server #homelab

Adventures in PKI: ​:blobCat_nom_wire:​

Ok so here is the story so far as a recap....
* The starting point was Crowdsec. Crowdsec has three components: agents which parse logs/events, remediation engines, which act on decisions, and a local API (lapi) which the first two connect to, and tracks the decisions and pulls from public block lists
* I realized I could also get external hosts involved, and also wait Crowdsec can parse logs from an aggregator, in this case Loki
* Awesome, step one, get logs into Loki. This lead to a whole chain of events that caused me to deploy Grafana/Alloy to collect those logs
* At this point I realized that shit, the remote nodes need auth and I'd need to copy around tokens everywhere
* Right, tokens everywhere, on remote nodes, etc. but wait, both alloy and Crowdsec support mTLS, all I need is client certs

record scratch

* Right so this would be easy if it wasn't for the pesky external nodes
* This lead me to setting up smallstep's step-ca with an ACME provider
* I got rsyslog setting logs to a central log server via mTLS! Even without the rest of this the log collection is a win.
* (Aside, I also got ssh certs working)
* And I got the Traefik bouncer plus agent to lapi connections working over mTLS but there was a little bit of strangeness there
* Crowdsec's components do not understand cert lifespans,and will not reload certs if they're renewed, hilarious. Fine they get certs with a lifespan measured in "eh, I'll probably reboot a node before then"

Ok and here we are caught up with current day. The very last part is getting the various non cluster nodes connected so their ssh is covered by the block lists. I go to edit the config, and...

nothing

In the logs of the lapi there is a bad cert error. After some browsing of the issue tracker I see mention of and allowed OU setting. Huh. Yeah. The certs created by the helm chart have an OU setting.

Ok but can I ask for a specific OU via ACME?

Whelp.

​:neocat_flop:​

@homelab@fedigroups.social
#Homelab #Suffering #PKI #Grafana #Crowdsec

Cursed homelab update:

Because I've been sitting on this for long enough.

TL;DR: Qotom micro-server blew up so my gaming machine is now being the hardware underneath server #3. So I need both a new gaming machine and a replacement server-class thing.

Part 1: The Qotomihilation

So the cheap Qotom server I purchased a while back blew up. Or rather something in the power circuitry developed a dead short (ish) which caused the PSU to cut power to protect itself. I initially thought the PSU was bad so I spent far too much on buying a replacement retail (same brand, higher rating) and all that did was confirm that the problem was the board not the PSU. (I also purchased fans to add to the server as I'm 90% sure this failure was heat related. They're going in the fan box.)

There's no obvious damage so no possibility to repair it without sending it back to the manufacturer, and if I'm going to do that, I might as well buy a replacement.

The obvious solution here is to replace it with something better - buying this server was a sensible decision at the time, but with hindsight, it was a bad solution to my problems.

Part 2: Rage Swap 2: Swap Harder

So the gaming machine got pressed back into server service. This is the one that had bad RAM, the one I was nervous about because it was still having problems after the known-bad RAM stick got pulled out.

And the one that, after I installed Bazzite on it, has been rock solid and survived multiple days of uptime without issue.

The biggest problem here was that the Qotom box had a mini-SAS port on the back for bulk storage, 9 Ethernet ports, and two M.2 slots. So the simplest solution for bulk storage was to plug in the QNAP card I got with the external box, hook it up with the cable QNAP supplied and have done. Finding two more Ethernet ports was as simple as finding the two PCI-e gigabit Ethernet cards I purchased back when I had dreams of running a high-availability router. Dealing with the other 2 Ethernet ports I was using was as simple as connecting the NBN box directly to server 3, and using the now-spare gigabit switch for the "IoT" network.

Which left the problem of the two M.2 slots. Bazzite was installed on an M.2 drive in one of them, but the other was missing the standoff and screw, probably because they had never been taken out of the baggie of screws that came with that motherboard.

So where was it?

Part 3: The curse of the rage-search

It was in the box for my gaming rig, obviously.

I didn't know that at the time, so I tore up the stratified pile of misc computer junk on (and in) the beautiful Silverstone desktop case Server 1 used to be in. No dice.

I then sat down and realised that it wasn't that the boxes "weren't there", it was that I couldn't see them, so I found the right box, right screw installed it, and it booted up first time. (and just to underscore this, there was no boot shenanigans required at all)

Part 4: 64GB of server in 24GB of RAM

The great thing about having space is that you can put stuff in it. The crap thing about space is that when it's gone, you can't fit everything in there anymore.

So I aggressively hacked at Kubernetes, Ceph and Elasticsearch to get everything to fit on that server without it running out of RAM (it deliberately has no swap) and it seems to now be stable.

Interesting fact: Linux seems to break IPv6 forwarding over virtual bridges when it OOMs.

Part 5: Next steps

I cleaned up the room (I now have a fan bag. Yay.) lay the computer down on it's wrong side (tower case) and it's been stable since I figured out what was eating all it's RAM.

So now all I need is a new server (thanks @decryption for pointing me to https://www.bargainhardware.co.uk - Australia's server options are nonexistent and I've been primarily looking at retail options) and a new gaming rig to go in that beautiful Silverstone desktop case, and I'm having to do all of this way sooner than I'd planned to.

But needs must.

#homelab #cursedhomelab #tech #it #linux

Người dùng muốn chuyển từ Truenas Scale sang Proxmox để tận dụng phần cứng mạnh hơn (HP Z440) và chạy thêm dịch vụ như Immich, Home Assistant. Hiện tại hệ thống cũ (HP Pavilion) không hỗ trợ GPU transcoding và cần hướng dẫn: (1) Cách tốt nhất di chuyển, (2) Có thể dùng Truenas dưới dạng VM trên Proxmox không (lo máy chủ không có controller SSD), (3) Giải pháp thay thế Truenas? #Proxmox #Truenas #Selfhosted #NAS #VietnamTech #HobbyIT #Docker #Migrate #HPC #Storage #HomeLab #VM #VietnamIT #LưuTrữ

I also upgraded dovecot from version 2.3 to 2.4 as part of an OS upgrade.

Immediate breakage all of the place. Apparently the config files between versions are incompatible in both syntax and variable names, so ive just spent a few hours rebuilding my config from scratch.

Thanks Ubuntu!

Yet another nudge for me to move my dovecot capability to an enterprise Linux host and star managing its config via ansible.

#sysadmin #homelab #dovecot #ubuntu

Just took a look at my PFSense box, and saw that my secondary OpenVPN connection was down.
Looked at the provider's site, and it turns out that they deprecated OpenVPN a while back!

So, tonight has consisted of me learning about and setting up: Wireguard

I'm back in action now, and looking forward to seeing whether this new VPN protocol will have any speed benefits for me!

#homelab #sysadmin #openvpn #wireguard #pfsense

Thanks to everyone who joined for the MOS livestream! Had a blast checking out the new OS. AND we hit 100 concurrent viewers during the stream!! That's a new record!

Check out the replay!
https://youtube.com/live/ALbteAwDfrQ?feature=share

#homelab #selfhosting

I just wrote a new blog post about my migration from NGINX Ingress Controller to Envoy Gateway with Cloudflared. It goes into some pretty good detail, so it might be useful for anyone that has yet to make the jump!

https://blog.jameswynn.com/posts/2026/01/migrating-to-gatewayapi/

#kubernetes #selfhosting #selfhosted #homelab

Today I learned how to use socat to pipe IPv4 traffic to my #homelab that's accessible only over IPv6 (thanks 1&1 and DS-Lite).

Set the DNS A record to a VPS I already have, spin up a reverse proxy on the VPS via docker, point the right domains to a socat docker container which TCP6's the IPv4 traffic the right way. Don't forget to add an IPv6 network to the socat container. Done.

I found good offer for a PC I could use to finally build proper NAS, bonus: has 32 GB RAM, so I could use half to buy small island

Now the question is: should I buy it?

#homelab #selfhosted

Does it make sense to run a #homelab on mobile internet? Please note that routing will be via #VPS with #VPN - #pangolin

We're LIVE and taking an early look at MOS - Modular Operating System for Servers and Homelabs.

Join here: https://youtube.com/live/ALbteAwDfrQ

10am Mountain (5pm UTC)

Be sure to stop by and say hi!

#homelab #selfhosting

Welp, looks like my K3s cluster is kaput. Guess I'm rebuilding that from scratch.

I lost the server node while upgrading Ubuntu, so I attempted to restore it from backup (as in: put a blank disk in the machine, reinstalled the OS and K3s, and then restored the important bits from the old disk, per the "backup and restore" documentation).

But it's just not coming up cleanly.

I'm seeing a bunch of errors from Longhorn on the server node -- can't attach the CSI drivers, and there are also some weird-ass flannel errors in the logs.

There's nothing important on the cluster, so I figure I'll just nuke it back down to the bare OS and build it back up again.

Here's hoping that I actually wrote everything down correctly the first time.

#homelab

Finally found the motivation to migrate #Keycloak from Bitnami to Codecentric Helm chart today, using the official upstream Keycloak image from quay.io (also mirrored on Docker Hub, if you prefer).

https://github.com/codecentric/helm-charts/tree/master/charts/keycloakx

#homelab #kubernetes #k8s

Efter at have kørt Immich herhjemme i mit homelab i et års tid eller mere, så har jeg nu købt et server licens til €115. Det er virkelig et godt produkt, som jeg har planer om, at rulle ud til hele husholdningen stille og roligt.

#Immich #selfhosting #homelab

https://immich.app/

In 3 hours we're gonna take an early look at MOS - Modular Operating System for Servers and Homelabs.

Join here: https://youtube.com/live/ALbteAwDfrQ

10am Mountain (5pm UTC)

Be sure to stop by and say hi!

#homelab #selfhosting

Mal eben den RAM beim NUC upgraden…

Die Chromebox3, die andauernd neu startet hat RAM, die ein NUC gut gebrauchen kann.
Der Plan: NUC drainen, System-Updates einspielen, herunterfahren, RAM einbauen, hochfahren, fertig.

Realität:
NUC drainen, System-Updates einspielen, herunterfahren, RAM einbauen, feststellen: och, die NVME-SSD ist ja viel kleiner als die der Chromebox3 …

Also auch die SSD umbauen und das System „mal eben“ neu installieren.
— dann sagte das BIOS: CMOS Error. —> Die Batterie ist leer

Dafür hat man ja passende CR2032 zu Hause. Dumm nur, dass hier die Batterie per Molex-Stecker auf dem Mainboard sitzt.
An einer Batterie wollte ich auch nicht rumlöten.

Also sitzt der NUC jetzt im Wohnzimmer, wg. LAN und Monitor und spielt dort so lange bis die Austauschbatterien da sind.

Die Neuinstallation habe ich trotzdem gemacht und dann kam es zum "Waiting for other members to finish joining etcd cluster: etcdserver: unhealthy cluster“.

Also erst einmal den etcd vom k3s angeschaut... und da gab es einen (laufenden Node 2x) …

Was für eine Freude. Okay, es läuft nun alles wieder.

Die Lektion aus der Geschichte:
- seltener verwendete Kommandos wie „irgendwas mit etcdctl“ dokumentieren
- ab und zu einmal Geräte vom Strom trennen um defekte Batterien zu finden

#homelab #yakshaving #nuc #k3s

I wish #mikrotik had a 2-port SPF+ router. Like a smaller CRS305-1G-4S+IN.

Preferably with a CPU with a bit more oomph that can also run a couple of Docker containers.

I would buy one of those in a heartbeat, especially if they made it at the price point of the CRS305.

Finding a small, decently-powerful, and decently-priced, 10Gbps router is... Not easy...

#homelab

So,
Ich mach dann auch mal mit bei der digtalen Unabhängigkeit. Ich bin schon recht weit mit Linux und Co, daher dann über die letzten Tage den Technitium DNS Server zuhause aufgesetzt. Damit werden Domainnamen nun komplett lokal rekursiv aufgelöst. Noch ein paar Wochen Testbetrieb, bis ich auch den Rest der Familie darauf umstelle. Werbung und Tracking werden natürlich geblockt. Danke an das Blocklistproject. #dutgemacht #diday #technitium #dns #homelab #BlocklistProject