Catalogue of Kubernetes issues

https://github.com/vijay2181/k8s-500-prod-issues/blob/main/README.md

#kh8s k8s #Kubernetes

@cafkafk *whack* `kubectl scale deployment --relicas 0`
*thunk* `kubectl delete pv`
(distant screaming)

#kh8s

#Kubernetes #AWS #EKS #DevOps #PlatformEngineering #k8s #kh8s

This morning's *absolute* WTF moment:

Pod network traffic 100% packet loss outbound UNTIL I `tcpdump` it on the node, then it starts working fine.

#k8s #Kubernetes #Cilium #kh8s #Networking #HomeLab

Man Prometheus is a pain to recover once its data store is in any way out of shape. Did NOT help that it was buried inside Kubernetes inside a PVC.

Thankfully it was only Dev environment today but if this ever pages on Prod we're losing data as it stands.

I'll write something up for a run book but eesh.

#Prometheus #Kubernetes #k8s #kh8s #Observability #o11y

YAML.

No I do not count anchors.

#YAML #yaml #yamlCult #kh8s #shitpost #k8s #kubernetes

cAdvisor's `container_oom_events_total` metric has been known broken since late 2021.

Just FYI. Not like we were relying on that or anything...

https://github.com/google/cadvisor/issues/3015

#k8s #Kubernetes #kh8s

#kh8s #k8s #kubernetes #shitpost

kubespray, kubeadm init, kind create etc

#Kubernetes #k8s #kh8s #DevOps #PlatformEngineering

Kubernetes is actually the specification, what you're referring to is Kubernetes' monster

#kh8s #k8s #kubernetes

Aaaaand we have a revelation. Dug through containerd source code and found the check, it's bailing cause loopback provides no IPs on the default interface which is eth0.

Not really sure how to fix it yet but it's a breakthrough when it was looking pretty stale.

So....is it even possible to run a pod with ONLY local loopback? No east-west, no access from the host, no vEth pairs just alone.

#CNI #Containerd #kh8s #k8s #Kubernetes

We've progressed from hacked-up shell scripts dumping context and writing Bash test harnesses.

We're up to compiling GoLang debug CNI plugins with static linking so we can dump context Even Betterer ™️

It's been a long day.

#kh8s #k8s #Containerd #Kubernetes

I have started writing a Kubernetes CNI debugging post - partially distraction, partially reference for myself, partially to ensure I really understand this area....

but mostly out of anger.

#kh8s #k8s #Kubernetes

What? How even?!

#Cilium #Kubernetes #kh8s #k8s

Fixed the auth issue at least but MY GOD

#kh8s #k8s #Kubernetes #Cloud #DevOps #PlatformEngineering #YAML

> Instead, everything is orchestrated, and it's feels a bit like operating a light switch with a broom stick.

~ @stephen

https://s-n.me/differently-complicated-hosting-the-before

#Kubernetes #K8s #kh8s

I used to think I didn't understand controller manager, now I see it's just a logic rat's nest tied together by their tails.

#Kubernetes #k8s #kh8s