ebpf-observability

eBPF 可观测性

eBPF（扩展伯克利包过滤器）允许你在 Linux 内核中运行沙箱化程序，而无需修改内核源代码或加载内核模块。本技能介绍如何使用 eBPF 对云原生基础设施进行深度可观测、网络监控和安全防护。

---

1. 适用场景

在以下情况下，使用基于 eBPF 的可观测性：

• 深度性能调试 —— 跟踪内核级延迟、系统调用开销和调度延迟，这些问题无法通过应用层指标发现。
• 无需 sidecar 的网络可观测性 —— 直接从内核捕获 L3/L4/L7 流量、DNS 查询和 TCP 状态转换，消除 sidecar 代理带来的 CPU 和内存开销。
• 内核边界安全监控 —— 实时检测容器逃逸、意外的进程执行、敏感文件访问和异常的系统调用模式。
• 生产环境持续剖析 —— 以极低开销（通常低于 1% CPU）生成 CPU 火焰图和内存分配剖析结果。
• 替换或增强服务网格 —— Cilium 可以替换 kube-proxy，并提供在内核级执行的身份感知型网络策略。

当内核版本低于 4.19、使用限制 BPF 功能的托管平台，或者应用层跟踪已能完全满足调试需求时，请避免使用 eBPF。

---

2. 前置条件

内核版本要求

功能	最低内核版本	建议内核版本
基本 BPF 映射和探针	4.9	5.10+
BPF CO-RE（BTF 支持）	5.2	5.10+
BPF 环形缓冲区	5.8	5.10+
BPF LSM 挂钩	5.7	5.15+
Cilium 完整功能	4.19	5.10+
Tetragon	4.19	5.13+

验证内核支持

# Check kernel version
uname -r

# Verify BTF (BPF Type Format) is enabled -- required for CO-RE
ls /sys/kernel/btf/vmlinux

# Check BPF filesystem is mounted
mount | grep bpf

# If not mounted, mount it
sudo mount -t bpf bpf /sys/fs/bpf

# Verify BPF JIT is enabled
cat /proc/sys/net/core/bpf_jit_enable
# Should return 1; if not:
sudo sysctl net.core.bpf_jit_enable=1

安装工具链

# Ubuntu/Debian -- install bpftrace, bcc tools, and libbpf
sudo apt-get update
sudo apt-get install -y bpftrace bpfcc-tools libbpf-dev linux-headers-$(uname -r)

# Fedora/RHEL
sudo dnf install -y bpftrace bcc-tools libbpf-devel kernel-devel

# Verify bpftrace works
sudo bpftrace -e 'BEGIN { printf("eBPF is working\n"); exit(); }'

---

3. Cilium 配置

Cilium 使用基于 eBPF 的网络替换 kube-proxy，并通过 Hubble 提供身份感知型安全和深度网络可观测性。

在 Kubernetes 上安装 Cilium

# Add the Cilium Helm repo
helm repo add cilium https://helm.cilium.io/
helm repo update

# Install Cilium with Hubble enabled
helm install cilium cilium/cilium --version 1.16.4 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost="${API_SERVER_IP}" \
  --set k8sServicePort="${API_SERVER_PORT}" \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.metrics.enableOpenMetrics=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload}"

# Wait for Cilium to be ready
cilium status --wait

安装 Cilium CLI 和 Hubble CLI

# Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --remote-name "https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz"
sudo tar xzvf cilium-linux-amd64.tar.gz -C /usr/local/bin
rm cilium-linux-amd64.tar.gz

# Hubble CLI
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --remote-name "https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-amd64.tar.gz"
sudo tar xzvf hubble-linux-amd64.tar.gz -C /usr/local/bin
rm hubble-linux-amd64.tar.gz

Hubble 网络可观测性

# Port-forward the Hubble Relay
cilium hubble port-forward &

# Observe all flows in real time
hubble observe --follow

# Filter flows by namespace
hubble observe --namespace production --follow

# Filter by verdict (dropped traffic)
hubble observe --verdict DROPPED --follow

# Filter by DNS queries
hubble observe --protocol DNS --follow

# Filter HTTP traffic to a specific service
hubble observe --to-label "app=api-server" --protocol HTTP --follow

# Export flows as JSON for ingestion into SIEM
hubble observe --output json --last 1000 > flows.json

访问 Hubble UI

# Port-forward the Hubble UI
kubectl port-forward -n kube-system svc/hubble-ui 12000:80

# Access at http://localhost:12000 -- provides a real-time service dependency map

---

4. 使用 Tetragon 进行安全防护

Tetragon 是 Cilium 的运行时安全防护引擎。它使用 eBPF 在内核级观察并执行安全策略，无需对应用进行任何更改。

安装 Tetragon

helm repo add cilium https://helm.cilium.io/
helm repo update

helm install tetragon cilium/tetragon \
  --namespace kube-system \
  --set tetragon.grpc.enabled=true \
  --set tetragon.exportFilename=/var/run/cilium/tetragon/tetragon.log

# Install the tetra CLI
curl -LO "https://github.com/cilium/tetragon/releases/latest/download/tetra-linux-amd64.tar.gz"
sudo tar xzvf tetra-linux-amd64.tar.gz -C /usr/local/bin
rm tetra-linux-amd64.tar.gz

进程执行监控

# process-monitor.yaml -- TracingPolicy to monitor all process executions
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: process-execution-monitor
spec:
  kprobes: []
  tracepoints: []
  uprobes: []
  enforcers: []
  # process_exec and process_exit events are always emitted by default
  # Use tetra CLI to observe them:

# Watch all process executions cluster-wide
kubectl exec -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --process-exec

# Filter to a specific namespace
kubectl exec -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact \
  --namespace production

文件访问跟踪

# file-access-policy.yaml -- detect reads/writes to sensitive files
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: sensitive-file-access
spec:
  kprobes:
    - call: "security_file_open"
      syscall: false
      args:
        - index: 0
          type: "file"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Prefix"
              values:
                - "/etc/shadow"
                - "/etc/passwd"
                - "/etc/kubernetes/pki"
                - "/var/run/secrets/kubernetes.io"
                - "/root/.ssh"

kubectl apply -f file-access-policy.yaml

# Observe file access events
kubectl exec -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact \
  | grep "sensitive-file-access"

网络连接防护

# restrict-egress.yaml -- block unexpected outbound connections
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: restrict-egress-connections
spec:
  kprobes:
    - call: "tcp_connect"
      syscall: false
      args:
        - index: 0
          type: "sock"
      selectors:
        - matchArgs:
            - index: 0
              operator: "DAddr"
              values:
                - "169.254.169.254"  # Block IMDS access
          matchActions:
            - action: Sigkill
        - matchNamespaces:
            - namespace: Mnt
              operator: NotIn
              values:
                - "host_mnt"

kubectl apply -f restrict-egress.yaml

检测权限提升

# detect-privilege-escalation.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-privilege-escalation
spec:
  kprobes:
    - call: "__x64_sys_setuid"
      syscall: true
      args:
        - index: 0
          type: "int"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Equal"
              values:
                - "0"
          matchActions:
            - action: Post
              rateLimit: "1m"
    - call: "__x64_sys_setns"
      syscall: true
      args:
        - index: 1
          type: "int"
      selectors:
        - matchActions:
            - action: Post

kubectl apply -f detect-privilege-escalation.yaml

---

5. bpftrace 单行命令

以下是实用的 bpftrace 命令，可直接在生产环境中运行，用于针对性调试。

系统调用延迟

# Trace read() syscall latency distribution (microseconds)
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
  tracepoint:syscalls:sys_exit_read /@start[tid]/ {
    @usecs = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
  }'

# Top 10 slowest syscalls by total time
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @start[tid] = nsecs; }
  tracepoint:raw_syscalls:sys_exit /@start[tid]/ {
    @ns[probe] = sum(nsecs - @start[tid]);
    delete(@start[tid]);
  } END { print(@ns, 10); }'

DNS 跟踪

# Trace DNS queries via UDP port 53 sends
sudo bpftrace -e 'kprobe:udp_sendmsg {
    $sk = (struct sock *)arg0;
    $dport = ($sk->__sk_common.skc_dport >> 8) | (($sk->__sk_common.skc_dport & 0xff) << 8);
    if ($dport == 53) {
      printf("%-8d %-16s DNS query to %s\n", pid, comm,
        ntop($sk->__sk_common.skc_daddr));
    }
  }'

# Count DNS queries by source process
sudo bpftrace -e 'kprobe:udp_sendmsg {
    $sk = (struct sock *)arg0;
    $dport = ($sk->__sk_common.skc_dport >> 8) | (($sk->__sk_common.skc_dport & 0xff) << 8);
    if ($dport == 53) { @dns[comm] = count(); }
  }'

TCP 重传

# Trace TCP retransmits with source/destination
sudo bpftrace -e 'kprobe:tcp_retransmit_skb {
    $sk = (struct sock *)arg0;
    $daddr = ntop($sk->__sk_common.skc_daddr);
    $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
    $dport = ($sk->__sk_common.skc_dport >> 8) | (($sk->__sk_common.skc_dport & 0xff) << 8);
    $sport = $sk->__sk_common.skc_num;
    printf("%-20s %-6d -> %-20s %-6d (%s)\n", $saddr, $sport, $daddr, $dport, comm);
  }'

磁盘 I/O 延迟

# Block I/O latency histogram by device
sudo bpftrace -e 'tracepoint:block:block_rq_issue { @start[args->dev, args->sector] = nsecs; }
  tracepoint:block:block_rq_complete /@start[args->dev, args->sector]/ {
    @usecs[args->dev] = hist((nsecs - @start[args->dev, args->sector]) / 1000);
    delete(@start[args->dev, args->sector]);
  }'

# Top processes by disk I/O bytes
sudo bpftrace -e 'tracepoint:block:block_rq_issue {
    @bytes[comm] = sum(args->bytes);
  } interval:s:5 { print(@bytes, 10); clear(@bytes); }'

容器感知跟踪

# Trace process exec inside containers (cgroup-filtered)
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve {
    printf("%-8d %-8d %-16s %s\n", pid, cgroup, comm, str(args->filename));
  }'

# Memory allocation hotspots per container
sudo bpftrace -e 'kprobe:__alloc_pages { @pages[cgroup] = count(); }
  interval:s:10 { print(@pages, 10); clear(@pages); }'

---

6. Prometheus 集成

Hubble 的 Prometheus 指标

在 Cilium Helm 安装配置中启用后，Hubble 会自动公开 Prometheus 指标。验证指标端点：

# Check that Hubble metrics are being served
kubectl exec -n kube-system ds/cilium -- curl -s http://localhost:9965/metrics | head -50

为 Prometheus Operator 创建 ServiceMonitor：

# hubble-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hubble-metrics
  namespace: kube-system
  labels:
    app: cilium
spec:
  selector:
    matchLabels:
      k8s-app: cilium
  endpoints:
    - port: hubble-metrics
      interval: 15s
      path: /metrics

用于自定义内核指标的 eBPF Exporter

# Deploy cloudflare/ebpf_exporter for custom kernel metrics
helm repo add ebpf-exporter https://cloudflare.github.io/ebpf_exporter
helm install ebpf-exporter ebpf-exporter/ebpf-exporter \
  --namespace monitoring \
  --set config.programs[0].name=oom_kills \
  --set config.programs[0].metrics.counters[0].name=oom_kill_total \
  --set config.programs[0].metrics.counters[0].help="Total number of OOM kills"

用于跟踪 OOM 终止和运行队列延迟的 ebpf_exporter 配置示例：

# ebpf-exporter-config.yaml
programs:
  - name: oom_kills
    metrics:
      counters:
        - name: oom_kill_total
          help: "Total number of OOM kills"
          labels:
            - name: cgroup
              size: 128
              decoders:
                - name: string
    kprobes:
      oom_kill_process: count_oom
  - name: runqlat
    metrics:
      histograms:
        - name: run_queue_latency_seconds
          help: "Run queue latency histogram in seconds"
          bucket_type: exp2
          bucket_min: 0
          bucket_max: 26
          bucket_multiplier: 0.000000001
    tracepoints:
      sched:sched_wakeup: trace_wakeup
      sched:sched_switch: trace_switch

Grafana 仪表板

导入以下社区仪表板以查看 eBPF 指标：

# Hubble dashboard -- Grafana dashboard ID 16611
# Cilium Agent dashboard -- Grafana dashboard ID 16612
# Cilium Operator dashboard -- Grafana dashboard ID 16613

# Or create a ConfigMap for automatic provisioning
kubectl create configmap grafana-cilium-dashboard \
  --from-file=cilium-dashboard.json \
  --namespace monitoring \
  -o yaml --dry-run=client | \
  kubectl label --local -f - grafana_dashboard=1 -o yaml | \
  kubectl apply -f -

用于 eBPF 来源指标的关键 Prometheus 查询：

# Dropped packets rate by reason
rate(hubble_drop_total[5m])

# DNS error rate by query type
sum(rate(hubble_dns_responses_total{rcode!="No Error"}[5m])) by (rcode, qtypes)

# HTTP request latency (p99) from Hubble L7 visibility
histogram_quantile(0.99, sum(rate(hubble_http_request_duration_seconds_bucket[5m])) by (le, destination))

# TCP retransmit rate from eBPF exporter
rate(tcp_retransmits_total[5m])

# Run queue latency p99
histogram_quantile(0.99, sum(rate(run_queue_latency_seconds_bucket[5m])) by (le))

---

7. 网络可观测性

L3/L4 流量日志

# Log all TCP connections with Hubble
hubble observe --type l3/l4 --protocol TCP --follow

# Filter SYN packets only (new connections)
hubble observe --type trace:to-endpoint --tcp-flags SYN --follow

# Export flows to a file for batch analysis
hubble observe --output json --since 1h > network-flows.json

# Count flows by destination service over the last hour
hubble observe --output json --since 1h | \
  jq -r '.destination.labels[] | select(startswith("k8s:app="))' | \
  sort | uniq -c | sort -rn | head -20

L7 协议可见性

使用目标 Pod 上的 Cilium 注解启用 L7 可见性：

# Annotate a namespace for HTTP visibility
apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    policy.cilium.io/proxy-visibility: "<Egress/53/UDP/DNS>,<Ingress/80/TCP/HTTP>,<Ingress/443/TCP/HTTP>"

# Observe L7 HTTP flows
hubble observe --type l7 --protocol HTTP --follow

# Filter by HTTP status code (5xx errors)
hubble observe --type l7 --http-status "500+" --follow

# Filter by HTTP method and path
hubble observe --type l7 --http-method GET --http-path "/api/v1/.*" --follow

DNS 监控

# All DNS queries and responses
hubble observe --type l7 --protocol DNS --follow

# DNS queries that returned NXDOMAIN
hubble observe --type l7 --protocol DNS --dns-rcode NXDOMAIN --follow

# DNS latency analysis with bpftrace
sudo bpftrace -e 'kprobe:dns_resolve { @start[tid] = nsecs; }
  kretprobe:dns_resolve /@start[tid]/ {
    @dns_latency_us = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
  }'

服务依赖关系图生成

Hubble UI 会自动生成服务地图。如需通过编程方式访问：

# Get a service map via Hubble Relay API
hubble observe --output json --since 24h | \
  jq '{src: .source.labels, dst: .destination.labels, verdict: .verdict}' | \
  jq -s 'group_by(.src, .dst) | map({
    source: .[0].src,
    destination: .[0].dst,
    flow_count: length,
    verdicts: [.[].verdict] | group_by(.) | map({(.[0]): length}) | add
  })' > service-map.json

---

8. 安全监控

检测容器逃逸

# container-escape-detection.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-container-escape
spec:
  kprobes:
    - call: "__x64_sys_unshare"
      syscall: true
      args:
        - index: 0
          type: "int"
      selectors:
        - matchActions:
            - action: Post
    - call: "__x64_sys_mount"
      syscall: true
      args:
        - index: 0
          type: "string"
        - index: 1
          type: "string"
        - index: 2
          type: "string"
      selectors:
        - matchArgs:
            - index: 2
              operator: "Equal"
              values:
                - "proc"
                - "sysfs"
                - "cgroup"
          matchActions:
            - action: Post
    - call: "__x64_sys_ptrace"
      syscall: true
      args:
        - index: 0
          type: "int"
      selectors:
        - matchActions:
            - action: Post

kubectl apply -f container-escape-detection.yaml

检测异常系统调用

# unexpected-syscalls.yaml -- alert on dangerous syscalls
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: unexpected-syscalls
spec:
  kprobes:
    - call: "__x64_sys_bpf"
      syscall: true
      args:
        - index: 0
          type: "int"
      selectors:
        - matchNamespaces:
            - namespace: Pid
              operator: NotIn
              values:
                - "host_ns"
          matchActions:
            - action: Post
    - call: "__x64_sys_perf_event_open"
      syscall: true
      selectors:
        - matchNamespaces:
            - namespace: Pid
              operator: NotIn
              values:
                - "host_ns"
          matchActions:
            - action: Post
    - call: "__x64_sys_init_module"
      syscall: true
      selectors:
        - matchActions:
            - action: Sigkill

文件完整性监控

# file-integrity-monitor.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: file-integrity-monitor
spec:
  kprobes:
    - call: "security_file_open"
      syscall: false
      args:
        - index: 0
          type: "file"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Prefix"
              values:
                - "/etc/"
                - "/usr/bin/"
                - "/usr/sbin/"
                - "/usr/lib/"
          matchActions:
            - action: Post
              rateLimit: "1m"
    - call: "security_inode_rename"
      syscall: false
      args:
        - index: 0
          type: "path"
        - index: 1
          type: "path"
      selectors:
        - matchActions:
            - action: Post

kubectl apply -f file-integrity-monitor.yaml

# Stream events to your SIEM
kubectl logs -n kube-system ds/tetragon -c export-stdout -f | \
  jq 'select(.process_kprobe.policy_name == "file-integrity-monitor")' | \
  tee /dev/stderr | \
  curl -X POST -H "Content-Type: application/json" -d @- https://siem.internal/api/events

---

9. 性能分析

使用 Parca 进行持续分析

Parca 使用 eBPF 持续收集 CPU 性能分析数据，开销极低。

# Install Parca Agent via Helm
helm repo add parca https://parca-dev.github.io/helm-charts
helm repo update

helm install parca-agent parca/parca-agent \
  --namespace parca \
  --create-namespace \
  --set config.node=true \
  --set config.store.address="parca-server.parca.svc:7070" \
  --set config.store.insecure=true \
  --set config.debuginfo.strip=true \
  --set config.debuginfo.upload.enabled=true

使用 Pyroscope 进行持续分析

# Install Grafana Pyroscope with eBPF profiling
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install pyroscope grafana/pyroscope \
  --namespace pyroscope \
  --create-namespace \
  --set ebpf.enabled=true \
  --set agent.mode=ebpf

使用 bpftrace 生成 CPU 火焰图

# Sample kernel and user stacks at 99Hz for 30 seconds
sudo bpftrace -e 'profile:hz:99 { @[kstack, ustack, comm] = count(); }' \
  -d 30 > stacks.out

# Using perf with BPF for flame graphs
sudo perf record -F 99 -a -g -- sleep 30
sudo perf script > perf.stacks

# Convert to flame graph (using Brendan Gregg's tools)
git clone https://github.com/brendangregg/FlameGraph.git
./FlameGraph/stackcollapse-perf.pl perf.stacks | \
  ./FlameGraph/flamegraph.pl > flamegraph.svg

进程外分析

# Trace off-CPU time to find where threads are blocked
sudo bpftrace -e '
  kprobe:finish_task_switch {
    $prev = (struct task_struct *)arg0;
    if ($prev->__state != 0) {
      @block_start[$prev->pid] = nsecs;
    }
    if (@block_start[tid]) {
      @off_cpu_us[kstack, comm] = sum((nsecs - @block_start[tid]) / 1000);
      delete(@block_start[tid]);
    }
  }
  END { print(@off_cpu_us, 20); }'

检测内存泄漏

# Track memory allocations not freed
sudo bpftrace -e '
  kprobe:kmalloc { @allocs[kstack] = count(); @bytes[kstack] = sum(arg0); }
  kprobe:kfree { @frees = count(); }
  interval:s:10 { print(@bytes, 10); }
'

# Per-process heap growth tracking
sudo bpftrace -e '
  uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc { @size[comm, tid] = sum(arg0); }
  interval:s:5 { print(@size, 10); clear(@size); }
'

---

10. 故障排除

常见的 eBPF 问题

BPF 验证器拒绝程序：

# Get verbose verifier output
sudo bpftrace -d -e 'your_program_here' 2>&1 | tail -50

# Common causes:
# - Unbounded loops (BPF requires bounded loops or unrolled iterations)
# - Stack size exceeds 512 bytes
# - Accessing memory without null checks
# - Back-edges in control flow (pre-5.3 kernels)

BTF 不可用：

# Check if BTF is compiled into the kernel
cat /boot/config-$(uname -r) | grep CONFIG_DEBUG_INFO_BTF

# If not, install BTF data from btfhub
# https://github.com/aquasecurity/btfhub
wget "https://github.com/aquasecurity/btfhub-archive/raw/main/ubuntu/22.04/x86_64/$(uname -r).btf.tar.xz"
tar xvf "$(uname -r).btf.tar.xz"

权限被拒绝：

# BPF requires CAP_BPF (or CAP_SYS_ADMIN on older kernels)
# For containers, add to securityContext:
# securityContext:
#   capabilities:
#     add: ["BPF", "PERFMON", "SYS_RESOURCE"]

# Check current capabilities
cat /proc/self/status | grep Cap
capsh --decode=$(cat /proc/self/status | grep CapEff | awk '{print $2}')

Cilium Pod 无法启动：

# Check Cilium agent logs
kubectl logs -n kube-system -l k8s-app=cilium --tail=100

# Verify BPF filesystem
kubectl exec -n kube-system ds/cilium -- mount | grep bpf

# Check for conflicting CNIs
ls /etc/cni/net.d/

# Run Cilium connectivity test
cilium connectivity test

缺少 Tetragon 事件：

# Verify TracingPolicy is loaded
kubectl get tracingpolicies

# Check Tetragon agent logs for verifier errors
kubectl logs -n kube-system ds/tetragon -c tetragon --tail=200 | grep -i error

# Verify the kprobe is attached
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
  cat /sys/kernel/debug/kprobes/list | grep your_function

eBPF 程序开销过高：

# List all loaded BPF programs and their run time
sudo bpftool prog show
sudo bpftool prog profile id <PROG_ID> duration 5

# Check map memory usage
sudo bpftool map show
sudo bpftool map dump id <MAP_ID> | wc -l

# If a program is consuming too much CPU, check its run count and time
sudo bpftool prog show id <PROG_ID> --json | jq '{run_cnt, run_time_ns}'

# Detach a misbehaving program
sudo bpftool prog detach id <PROG_ID> type <ATTACH_TYPE>

内核兼容性矩阵

# Quick check: which eBPF features your kernel supports
sudo bpftool feature probe kernel

# Check specific program types
sudo bpftool feature probe kernel | grep program_type

# Check available map types
sudo bpftool feature probe kernel | grep map_type

# Check available helper functions
sudo bpftool feature probe kernel | grep helper