If we are running any eBPF workload or writing eBPF code, we want to measure its performance impact, and in this post we will demonstrate an example of how to do it.
In this example, our goal is to measure the performance of file open operations, one of the most critical functions in the OS. Our code used file open hooks in eBPF, we wanted to measure the performance overhead introduced by adding this hook.
To identify likely bottlenecks, we need a simple C test harness with the fewest dependencies, designed to measure file open performance.
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <time.h>
#include <stdint.h>
#include <stdlib.h>
#include <unistd.h>
#include <sched.h>
#include <sys/mman.h>
#include <sys/syscall.h>
static inline uint64_t now_ns(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts); /* VDSO, no syscall */
return (uint64_t)ts.tv_sec * 1000000000ull + ts.tv_nsec;
}
int main(int argc, char **argv) {
const char *path = argv[1];
uint64_t n = strtoull(argv[2], NULL, 10);
uint64_t warm = n / 10;
/* preallocate + prefault + lock: no page faults in the loop */
uint32_t *d = mmap(NULL, n * sizeof(uint32_t), PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE, -1, 0);
mlock(d, n * sizeof(uint32_t));
for (uint64_t i = 0; i < n; i++) d[i] = 0; /* fault everything in */
for (uint64_t i = 0; i < n; i++) {
uint64_t t0 = now_ns();
long fd = syscall(SYS_openat, AT_FDCWD, path, O_RDONLY); /* raw, no libc wrapper */
uint64_t t1 = now_ns();
if (fd >= 0) close(fd);
d[i] = (uint32_t)(t1 - t0);
}
/* dump after the loop only */
for (uint64_t i = warm; i < n; i++) printf("%u\n", d[i]);
return 0;
}
The goal of the code example above is to keep it simple and reopen the same file under warm cache conditions, minimizing unrelated filesystem and disk I/O variability and helping identify the p50/p99. The code invokes syscall(SYS_openat, …) instead of the libc openat() wrapper, and the first 10% of the results are discarded as a warmup period.
This test harness would produce results of opening a file x number of times and how long it took to open. So we could use this to measure the before and after of when the eBPF hook attached.
Setup
When profiling the eBPF code, we want perf tool to be able to resolve symbols so we can analyze where the issue is in our code. To do that, we have to run these commands.
sudo sysctl -w net.core.bpf_jit_enable=1
sudo sysctl -w net.core.bpf_jit_kallsyms=1
The above commands enable jit and expose jit-compiled BPF symbols so the perf report can display program names instead of unknown addresses.
To check whether the symbols appear in the perf tool, run your eBPF code and use a command like this.
sudo bpftool prog show | rg -A4 ' lsm '
sudo rg 'bpf_prog_[0-9a-f]+_ '/proc/kallsyms | rg 'security|path|file|open'
In the above rg command, we are checking for lsm as we are measuring LSM hooks.
Also, we are using a custom kernel version, so perf for that kernel version is not in the standard path, and we have it installed in our example: PERF=/usr/lib/linux-tools/6.8.0-134-generic/perf
Measuring
Now that we have set up all the necessary tools, the first step is to measure without the eBPF code running, and this is where using the above C code can help. We measure the code by opening the file /etc/hostname and piping the results to a file so that we can calculate the p50/p99.
sudo taskset -c 3 chrt -f 99 ./bench /etc/hostname 100000 > /tmp/samples.txt
The taskset -c 3 pins execution to CPU 3, reducing CPU migration noise, and chrt -f 99 gives the benchmark extremely high CPU priority. It runs before almost all normal programs and keeps running until it finishes, blocks, or is interrupted. The C code discards the first 10%; the file should contain 90,000 samples.txt.
Next, run the eBPF code and execute something like this.
sudo $PERF record \
-g \
--call-graph fp \
-e cycles:k \
-F 997 \
-o ~/perf.data \
-- \
taskset -c 3 \
chrt -f 99 \
./bench /etc/hostname 200000 \
> /tmp/samples.txt
The -g records the call stacks, --call-graph fp unwinds stacks using frame pointers, and -e samples CPU cycles in kernel mode only, which includes syscall, VFS, LSM, and eBPF execution and not userspace benchmark work. The -F 997 requests 997 samples per second, and the non-round frequency helps avoid periodic alignment.
After running the above, run this command to sort the data.
sudo "$PERF" report -i ~/perf.data --stdio --sort comm,dso,symbol > perf.txt
Flamegraph of the same perf.data, generated with Inferno. The stack of interest here is bpf_lsm_file_open and everything above it.
Here is an example output from perf.txt, which shows that the time being spent on bpf_lsm_file_open and its tail calls is where the performance bottleneck is. This turned out to be in a hot path, which meant every allocation-to-CPU cycle shaving will make a significant impact on the performance of the system.
|
| | | | | |–90.52%–do_dentry_open
| | | | | | |
| | | | | | --89.78%–bpf_lsm_file_open
| | | | | | |
| | | | | | --89.30%–0xffffffffc0288c18
| | | | | | |
| | | | | | |–87.40%–bpf_prog_b06f413955402a4b_tail_call_security_check
| | | | | | | |
| | | | | | | |–77.57%–bpf_prog_934361d723613c1c_enforce_access_policy
| | | | | | | | |
| | | | | | | | |–57.87%–bpf_prog_a0f18f4b0b140d77_path_check_callback
| | | | | | | | | |
| | | | | | | | | |–29.94%–bpf_probe_read_kernel
| | | | | | | | | | |
| | | | | | | | | | |–18.88%–copy_from_kernel_nofault
| | | | | | | | | | |
| | | | | | | | | | --4.99%–copy_from_kernel_nofault_allowed
| | | | | | | | | |
| | | | | | | | | |–4.88%–htab_map_hash
| | | | | | | | | |
| | | | | | | | | --2.29%–copy_from_kernel_nofault
This post focuses on the profiling method rather than a specific result, and the overhead you’ll see depends heavily on what your hook actually does, so we’re leaving the numbers out and focusing on how to get them yourself.
Now, from the above, we can start analyzing where the time is being spent and perf-tune the code along with the p50/p99 of the C code with eBPF running.
By doing this, we can clearly identify the perf impact of the eBPF code and likely pinpoint where optimizations are required. It can be as simple as caching something or coming up with a better algorithm based on where the problem is.