Have you ever though "boy I love distributed tracing, but I wish I could add kernel produced spans to my traces"?
Well, now you can! I just published ebpf_exporter v2.4.0 and it can now create kernel spans that are integrated in your normal traces.
See: https://github.com/cloudflare/ebpf_exporter/releases/tag/v2.4.0
The most interesting example is producing network related spans, like TCP retransmits or SKB drops. Here's a screenshot of a whole bunch of them attached to a userspace trace.
It's been a while since the last ebpf_exporter release, so I cut a new one just now. Lots of changes in v2.3.0.
You can now start testing with just docker:
```
docker run --rm -it --privileged -p 9435:9435 ghcr.io/cloudflare/ebpf_exporter:v2.3.0 --config.dir=examples --config.names=timers
```
* https://github.com/cloudflare/ebpf_exporter/releases/tag/v2.3.0
Empty ebpf getpid() tracepoint overhead: 15ns 🏎️, fentry is 24ns 🚗
Empty ebpf uprobe overhead: 1600ns 🐌
Benchmark: https://github.com/cloudflare/ebpf_exporter/pull/331
Is there a better way if I want arbitrary userspace programs to pass something to my ebpf code? Giving a ringbuf fd via some library seems convoluted.
New cool example for ebpf_exporter: CFS delay histogram. In addition to knowing overall CFS throttlig delay from cgroups in cpu.state, now you can have a histogram of individual throttling durations in prometheus.
* https://github.com/cloudflare/ebpf_exporter/pull/311
As a bonus, you get a bpftrace command to observe these.
I hacked together opentelemetry distributed tracing support for ebpf_exporter: https://github.com/cloudflare/ebpf_exporter/pull/297
So far I managed to add some block i/o tracing via tracepoints, but it's unclear how to tie this together to userspace traces, since there's no way for userspace to pass the trace id.
Are there any other kernel areas that people are interested in having integrated with distributed tracing? Sockets? Scheduling? Something else?
ebpf_exporter now comes with a docker image with examples if you couldn't be bothered to build it yourself:
* https://github.com/cloudflare/ebpf_exporter/pkgs/container/ebpf_exporter
docker run --rm -it --privileged -p 9435:9435 ghcr.io/cloudflare/ebpf_exporter:master --config.dir=examples --config.names=timers
