hodgesds / perf_exporter

A perf exporter for prometheus

perf_exporter

perf_exporter is a Prometheus exporter that exposes metrics from the perf subsystem in Linux. It can read any kernel tracepoints and expose them as Prometheus compatible metrics. NOTE this code is largely been merged upstream into node_exporter, which you can use the --collector.perf.tracepointflag (ex: --collector.perf.tracepoint="sched:sched_process_exec") to accomplish the same type of monitoring.

Configuration

The configuration format allows you to specify counters at the subsytem level. For each subsytem individual events can be configured. Note that configuring a subsystem event for a specific processor isn't supported as of now. To find available events for your system you can use the perf tooling (i.e. perf list) or you can read directly from tracefs available_events in combination with the tools/tracepoint2yaml script. Here is a rough example of a configuration file to get started (note this is highly system specific).

kmem:
  events:
    - mm_page_alloc_extfrag
    - mm_page_pcpu_drain
    - mm_page_alloc_zone_locked
    - mm_page_alloc
    - mm_page_free_batched
    - mm_page_free
    - kmem_cache_free
    - kfree
    - kmem_cache_alloc_node
    - kmalloc_node
    - kmem_cache_alloc
    - kmalloc
net:
  events:
    - netif_rx_ni_entry
    - netif_rx_entry
    - netif_receive_skb_entry
    - napi_gro_receive_entry
    - napi_gro_frags_entry
    - netif_rx
    - netif_receive_skb
    - net_dev_queue
    - net_dev_xmit
    - net_dev_start_xmit

Note that the proper value for perf_event_paranoid should be set, in this case it should be set to 0 becuase the exporter runs on all processors. For more info see man perf_event_open.

Building

This repo uses make for the build system, to build the binary just type make. It is assumed that you are using go 1.11+.

Example

Here is an example of some of the events that can be exposed:

FAQ

• How is perf being used? You may want to see this library which is where most of the perf related utilities are.
• I don't see values for my perf events, is the collector broken? This is difficult to debug due to a large number of factors at play. Everything from the way your kernel was configured to debugfs mount points can cause an issue, please file an issue so that datapoints can be collected.
• Is there a max number of events that can be profiled? Yes, it is dependent on your kernel configuration, originally there was a --yolo flag to trace everything but that didn't work out so well.

perf_exporter's Issues

which grafana dashboard and perf hard event

Hello, please help to answer two questions about perf related, thank you

1. Which grafana dashboard id is used by perf export？
2. How to display perf hardware events, such as cache-misses events

my environment：

1. cat /proc/sys/kernel/perf_event_paranoid
   0
2. uname -a
   Linux ubuntu 4.15.18 #2 SMP Thu Feb 24 14:27:34 HKT 2022 aarch64 aarch64 aarch64 GNU/Linux

5.mount | grep -E '(debugfs|tracefs)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)

6. cat /boot/config-$(uname -r) | grep -iE '(perf|debugfs)

CONFIG_GENERIC_IRQ_DEBUGFS is not set

CONFIG_CGROUP_PERF=y
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
CONFIG_HAVE_PERF_EVENTS=y

Kernel Performance Events And Counters

CONFIG_PERF_EVENTS=y

CONFIG_DEBUG_PERF_USE_VMALLOC is not set

CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y

CONFIG_PCIEASPM_PERFORMANCE is not set

CONFIG_HW_PERF_EVENTS=y

CONFIG_CMA_DEBUGFS is not set

CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_L2TP_DEBUGFS=m

CONFIG_6LOWPAN_DEBUGFS is not set

CONFIG_CLS_U32_PERF is not set

CONFIG_BATMAN_ADV_DEBUGFS is not set

CONFIG_BT_DEBUGFS=y
CONFIG_CFG80211_DEBUGFS=y
CONFIG_MAC80211_DEBUGFS=y
CONFIG_ATH9K_DEBUGFS=y
CONFIG_ATH9K_HTC_DEBUGFS=y

CONFIG_CARL9170_DEBUGFS is not set

CONFIG_WIL6210_DEBUGFS=y
CONFIG_ATH10K_DEBUGFS=y

CONFIG_WCN36XX_DEBUGFS is not set

CONFIG_IWLEGACY_DEBUGFS=y
CONFIG_IWLWIFI_DEBUGFS=y

CONFIG_RT2X00_LIB_DEBUGFS is not set

CONFIG_RSI_DEBUGFS is not set

CONFIG_IEEE802154_AT86RF230_DEBUGFS=y
CONFIG_IEEE802154_CA8210_DEBUGFS=y
CONFIG_SMS_SIANO_DEBUGFS=y

CONFIG_DRM_AMDGPU_GART_DEBUGFS is not set

CONFIG_DEVFREQ_GOV_PERFORMANCE=y
CONFIG_NTB_PERF=m

Performance monitor support

CONFIG_ACPI_EC_DEBUGFS=m
CONFIG_RCU_PERF_TEST=m

CONFIG_ARM64_PTDUMP_DEBUGFS is not set

CONFIG_SECURITY_PERF_EVENTS_RESTRICT=y

7. perf_exporter -h
   Prometheus exporter for perf events
   Usage:
   perf_exporter [flags]

Flags:
-c, --config string config file (default is $HOME/.perf_exporter.yaml)
-l, --listen-address string Server listen address (default "0.0.0.0:8585")
--metrics-path string Metrics endpoint (default "/metrics")
-p, --pid int PID (default all) (default -1)

Build Timestamp:

Go Version:
devel go1.19-da2773fe3e Sun Mar 6 20:47:39 2022 +0000