This weekend I did something very funny and disastrous in my setup of #talos #kubernetes cluster. I got up and running with my first node and various services running and saw that I was using about 5GB of RAM just for infrastructure stuff - #longhorn, #openobserve, etc. So, I decided to add another node with my #netcup provider and add VLAN, which isn't something that they advertise well.

Anyway, I purchased an identical VPS (10 arm vcpu, 16GB ram, 512GB storage), copied the machine config and patched the names, and added it to the new VPS after installing talos. It came online fine and attached to the cluster. Then I wanted to add the VLAN, so I attached that to the VMs and restarted n1(?) first - I kinda forget the order. What happened then was that I didn't quite have the right networking configuration for the vlan interface. Despite configuring dhcp: false, talos was trying to get #dhcp off of the new interface and failing, causing apid to not start, so I couldn't access the node. I was totally locked out. Eventually the same thing happened to n1, but what else had happened was that when I restarted the node to apply the vlan interface, the cluster lost quorum because, guess what? 50% is not >50%. Woops.

So, the cluster was down and I was totally locked out. With the way the interfaces work, I wound up wiping the disks and reinstalling talos on n2 until I could find the right magic.

I found a solution, but I noticed that external-dns was trying to use the internal IP and kubelet didn't know about the external id. I got around that by using explicit IP addresses for external-dns annotations for now, and also adding nodeIp: .... in the configs. Here's the final version. Notice that eth0 no longer works, I had to use enps70.

networking config

machine:
  network:
    hostname: n2
    interfaces:
      - dhcp: true
        interface: enp7s0
        addresses:
        - <my external node ip>/22 # /22 is how it's reported in netcup
      - dhcp: false
        interface: enp9s0
        addresses:
        - 10.132.0.20/24

machine:
  kubelet:
    extraArgs:
      node-ip: "<my external node ip>"

#selfhosting

I been messing around trying self hosted options for logs. Mostly to scratch an itch, but also to know what is available in the market.

#openObserve is nice, but feels pretty clunky for what I want. Found this thing called #seq, which is kind of brilliant. But right now, I've settled with #victorialogs from #victoriametrics.

It can ingest #elasticsearch formatted logs. But you get the ease that #loki was trying to do. I have to say, I'm impressed. 😄

https://docs.victoriametrics.com/victorialogs/logsql/

During the #SharkBytes session at #SharkFest conference I had an opportunity to present a lightning talk about my pet project called IDS Lab.
It is a lab infrastructure deployable as docker containers, which simulates the small company network.

The IDS Lab consists of web webserver with #Wordpress, #MySQL database, #Linux desktop with RDP, the #WireGuard VPN for "remote" workers and for connecting another virtual or physical machines into the lab network.
This part of infrastructure can be used for attack simulations.

There are additional components for playing with logs and detections, too: #Fluentbit, #Suricata and #OpenObserve as lightweight SIEM.

In the #SIEM we already have preconfgured dashboards for alerts, netflows, web logs and logs from windows machines, if present.

Using the provided setup script, the whole lab can be up and running in up to 5 minutes. For more info, please check my GitHub repository with the IDS Lab:

https://github.com/SecurityDungeon/ids-lab/

#sf24eu #wireshark @wireshark

Lol. Their docs can't decide on the port they use. #openobserve

Since morning I am searching for a nice free log analyzer, I used #splunk around 12 years just wanted to search quickly on some application logs, most probably log4j or log4net logs. I tried
- #ELK<-too hard to install configure
- #graylog<-too complex or non working docs
- #jaeger<-wanted json format
- #openobserve<-does not have simple log upload or file path provider, needs fluentd or kubectl

I did not know splunk is this good, now I am convinced it is super product. Feel free to tell if you have a good suggestion and boost please for reach.

I created an addon for #homeassistant last night that allows you to ship logs from your #hassio instance to somewhere else via #fluentbit: https://github.com/ablyler/ha-addon-fluent-bit

I am personally using this to send my logs to a local #openobserve instance: https://openobserve.ai

577 - El cron lo carga el diablo
Realizar copias de #seguridad en #Linux y #Docker utilizando #cron o #systemd y como monitorizar la actividad con herramientas como #OpenObserve

El día que se me ocurrió la idea de levantar OpenObserve para controlar los contenedores Docker y otros procesos en mi VPS principal, me tenía que haber dado un premio. Con el paso del tiempo esta herramienta se ha convertido en una fuente increíble de reso
#atareaoConLinux
https://www.youtube.com/watch?v=vYJRmsWuGGc

The February #syslog_ng newsletter is now available:

- #OpenObserve #JSON API support

- Syslog-ng PE can now send logs to #Google #BigQuery

- syslog-ng can now do a full configuration check

- How build services make life easier for upstream developers

https://www.syslog-ng.com/community/b/blog/posts/the-syslog-ng-insider-2024-02-openobserve-configuration-check-build-services

I remember someone mentioning #OpenObserve here sometime ago and decided to give it a go in the #homelab it's a binary which is easy enough to get going. Documentation is garbage tho. It was easy enough to start ingesting logs and it uses an OTEL collector which I guess is in the spirit and all that.

The December syslog-ng newsletter is now out:

- Compressing HTTP traffic

- Why is a feature not available in the #syslog_ng package?

- Sending logs to #OpenObserve

- Removing duplicate messages with syslog-ng in a redundant logging environment

https://www.syslog-ng.com/community/b/blog/posts/the-syslog-ng-insider-2023-12-compressed-http-packages-openobserve-duplicates

Version 4.5.0 of syslog-ng is now available with #OpenObserve JSON API support:

https://www.syslog-ng.com/community/b/blog/posts/version-4-5-0-of-syslog-ng-is-now-available-with-openobserve-json-api-support

#LogManagement

Version 4.5.0 of #syslog_ng is now available with #OpenObserve #JSON API support, and many other smaller features. My blog shows you how to get up-to-date installers, and a sample syslog-ng configuration for OpenObserve.

https://www.syslog-ng.com/community/b/blog/posts/version-4-5-0-of-syslog-ng-is-now-available-with-openobserve-json-api-support

#LogManagement

#OpenObserve has an #Elasticsearch compatible API for log ingestion, but syslog-ng is not mentioned in the documentation. Luckily, as it turned out, OpenObserve has a ready to use #syslog_ng configuration example in the web UI.

https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-openobserve-using-syslog-ng

#LogManagement

🌖 GitHub - openobserve/openobserve: 🚀 10倍簡單，🚀 140倍低儲存成本，🚀 高效能，🚀 PB級 - Elasticsearch/Splunk/Datadog的替代方案🚀（日誌，指標，跟踪）。
➤ 一個簡單易用的日誌、指標、跟踪和分析的雲原生觀測平台，設計用於PB級的規模。
✤ https://github.com/openobserve/openobserve
openobserve是一個雲原生的觀測平台，專門為日誌、指標、跟踪和分析而建，設計用於PB級的規模。它是一個簡單易用的替代方案，可以降低日誌存儲成本，並提供高效能和高可用性。它可以作為Elasticsearch的替代方案，並提供自己的UI，不需要單獨安裝Kibana。它還提供了許多高級功能，如增強、遮蔽、日誌減少、合規性等。
+ 這個平台看起來非常有用，尤其是對於需要處理大量日誌和指標的企業來說。我很想試試看它的高級功能。
+ 我喜歡
#GitHub #openobserve #Elasticsearch #Splunk #Datadog #日誌 #指標 #跟踪 #雲原生 #觀測平台