Given the ludicrous costs of observability vendors and complexity of the OTEL SDK, I'm convinced the best ROI in observability is https://github.com/go-graphite with https://github.com/statsite/statsite and a self-hosted ElasticSearch cluster for logs.
I know it's blasphemous, but I'm not sure tracing and metrics/logs with attributes really give anyone a competitive advantage over what you can achieve leveraging statsd and a Perl script to parse logs to an OpenSearch or ElasticSearch cluster. What I witnessed at Booking.com with just statsd and web access logs parsed to self-hosted ElasticSearch was staggering compared to what I've seen from OTEL and observability vendors.
I think there's a general trend in the industry to migrate to advanced, complex solutions without fully utilizing existing simple, boring solutions.
#observability #blasphemy #otel #datadog #perl #elasticsearch