[Bug]: False-positive inbound_packets_dropped_ratio alert on Linux bridge (vmbr0)

#24126 · open · 1 comments

View on GitHub ↗

cbeckstein

### Bug description <html> <body> <!--StartFragment--><html><head></head><body><p><span>Netdata Agent v2.12.0 repeatedly reports an elevated </span><code><span>inbound_packets_dropped_ratio</span></code><span> on the Linux bridge </span><code><span>vmbr0</span></code><span> of a Proxmox VE host.</span></p><p><span>The underlying physical interface (</span><code><span>nic0</span></code><span>, Intel I226-LM) shows no corresponding hardware RX missed or FIFO errors during the measurements.</span></p><p><span>Kernel tracing using </span><code><span>skb:kfree_skb</span></code><span> identified frequent </span><code><span>UNHANDLED_PROTO</span></code><span> events. Packet captures showed Layer-2 control/discovery frames, including:</span></p><ul><li><p><span>RLDP (EtherType 0x8899)</span></p></li><li><p><span>HomePlug AV (0x88e1)</span></p></li><li><p><span>EtherType 0x6970 (not conclusively identified)</span></p></li></ul><p><span>A controlled A/B/A test produced:</span></p> Phase | Duration | vmbr0 RX drops -- | -- | -- A1: Without tcpdump | 20 seconds | 56 B: With tcpdump on vmbr0 | 20 seconds | 0 A2: Without tcpdump | 20 seconds | 56 <p><span>The result demonstrates that opening an AF_PACKET socket on the bridge changes the RX-drop accounting.</span></p><p><span>The observed drops are consistent with locally unhandled Layer-2 frames rather than necessarily indicating loss of forwarded IP traffic. The exact contribution of individual EtherTypes has not been established.</span></p><p><span>The default health rule can therefore produce a misleading network packet-loss warning for this configuration.</span></p></body></html><!--EndFragment--> </body> </html> ### Expected behavior The default `inbound_packets_dropped_ratio` health check should account for the semantics of Linux bridge RX-drop counters. Locally unhandled Layer-2 control frames should not automatically be interpreted as evidence of actual network packet loss. Please consider bridge-aware alert handling, more selective default filtering, or documentation explaining this limitation, while preserving monitoring of physical interfaces. ### Steps to reproduce 1. Run Netdata v2.12.0 on a Proxmox VE host with a Linux bridge (`vmbr0`) connected to a LAN carrying non-IP Layer-2 control/discovery traffic. 2. Enable the stock `inbound_packets_dropped_ratio` health template. 3. Observe a continuously increasing `/sys/class/net/vmbr0/statistics/rx_dropped` counter. 4. Measure the counter for 20 seconds without tcpdump: 56 drops. 5. Run `timeout -s INT 20 tcpdump -p -ni vmbr0 -q -w /dev/null 'ether proto 0x6970 or ether proto 0x8899 or ether proto 0x88e1'`. 6. Measure the counter during tcpdump: 0 drops. 7. Repeat the 20-second measurement without tcpdump: 56 drops. All three measurements were performed consecutively on October 2, 2026. ### Installation method other ### System info ```shell Linux asus-nuc 7.0.14-19-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.14-19 (2026-09-18T07:31Z) x86_64 GNU/Linux PRETTY_NAME="Debian GNU/Linux 13 (trixie)" VERSION_ID="13" VERSION_CODENAME=trixie Proxmox VE 9.2.20 ``` ### Netdata build info ```shell Packaging: Netdata Version ____________________________________________ : v2.12.0 Installation Type __________________________________________ : binpkg-deb Package Architecture _______________________________________ : x86_64 Package Distro _____________________________________________ : debian 13 Configure Options __________________________________________ : cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_STANDARD=11 -DCMAKE_CXX_STANDARD=17 -DBUILD_SHARED_LIBS=OFF -DCMAKE_C_FLAGS=' -fexceptions -fno-omit-frame-pointer -funwind-tables -fasynchronous-unwind-tables' -DCMAKE_CXX_FLAGS=' -fexceptions -fno-omit-frame-pointer -funwind-tables -fasynchronous-unwind-tables' -DCMAKE_COMPILE_DEFINITIONS='_GNU_SOURCE' -DCMAKE_EXE_LINKER_FLAGS=' -fexceptions -rdynamic' -DCMAKE_SHARED_LINKER_FLAGS='' Default Directories: User Configurations ________________________________________ : /etc/netdata Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d Stock Data Files ___________________________________________ : /usr/share/netdata Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata Permanent Databases ________________________________________ : /var/lib/netdata Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d Static Web Files ___________________________________________ : /usr/share/netdata/web Log Files __________________________________________________ : /var/log/netdata Lock Files _________________________________________________ : /var/lib/netdata/lock Home _______________________________________________________ : /var/lib/netdata Operating System: Kernel _____________________________________________________ : Linux Kernel Version _____________________________________________ : 7.0.14-19-pve Operating System ___________________________________________ : Proxmox VE Operating System ID ________________________________________ : proxmox Operating System ID Like ___________________________________ : proxmox Operating System Version ___________________________________ : 9.2.20 Operating System Version ID ________________________________ : 9 Detection __________________________________________________ : pveversion Hardware: CPU Cores __________________________________________________ : 16 CPU Frequency ______________________________________________ : 2200000000 RAM Bytes __________________________________________________ : 66695446528 Disk Capacity ______________________________________________ : 4000797868032 CPU Architecture ___________________________________________ : x86_64 Virtualization Technology __________________________________ : none Virtualization Detection ___________________________________ : systemd-detect-virt Container: Container __________________________________________________ : none Container Detection ________________________________________ : systemd-detect-virt Container Orchestrator _____________________________________ : none Container Operating System _________________________________ : none Container Operating System ID ______________________________ : none Container Operating System ID Like _________________________ : none Container Operating System Version _________________________ : none Container Operating System Version ID ______________________ : none Container Operating System Detection _______________________ : none Features: Built For __________________________________________________ : Linux Netdata Cloud ______________________________________________ : YES Health (trigger alerts and send notifications) _____________ : YES Streaming (stream metrics to parent Netdata servers) _______ : YES Back-filling (of higher database tiers) ____________________ : YES Replication (fill the gaps of parent Netdata servers) ______ : YES Streaming and Replication Compression ______________________ : YES (zstd lz4 gzip brotli) Contexts (index all active and archived metrics) ___________ : YES Tiering (multiple dbs with different metrics resolution) ___ : YES (5) Machine Learning ___________________________________________ : YES Memory Allocator ___________________________________________ : system Database Engines: sqlite _____________________________________________________ : 3.53.4 dbengine (compression) _____________________________________ : YES (zstd lz4) alloc ______________________________________________________ : YES ram ________________________________________________________ : YES none _______________________________________________________ : YES Connectivity Capabilities: ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES static (Netdata internal web server) _______________________ : YES WebRTC (experimental) ______________________________________ : NO Native HTTPS (TLS Support) _________________________________ : YES TLS Host Verification ______________________________________ : YES Libraries: LZ4 (extremely fast lossless compression algorithm) ________ : YES ZSTD (fast, lossless compression algorithm) ________________ : YES zlib (lossless data-compression library) ___________________ : YES Brotli (generic-purpose lossless compression algorithm) ____ : YES protobuf (platform-neutral data serialization protocol) ____ : YES (system) OpenSSL (cryptography) _____________________________________ : YES libdatachannel (stand-alone WebRTC data channels) __________ : NO JSON-C (lightweight JSON manipulation) _____________________ : YES libcap (Linux capabilities system operations) ______________ : YES libcrypto (cryptographic functions) ________________________ : YES libyaml (library for parsing and emitting YAML) ____________ : YES libmnl (library for working with netfilter) ________________ : YES stacktraces (library for getting stack traces) _____________ : libbacktrace (mmap, threads, data) Plugins: apps (monitor processes) ___________________________________ : YES cgroups (monitor containers and VMs) _______________________ : YES cgroup-network (associate interfaces to CGROUPS) ___________ : YES proc (monitor Linux systems) _______________________________ : YES tc (monitor Linux network QoS) _____________________________ : YES diskspace (monitor Linux mount points) _____________________ : YES freebsd (monitor FreeBSD systems) __________________________ : NO macos (monitor MacOS systems) ______________________________ : NO macos-logs (monitor macOS unified logs) ____________________ : NO windows (monitor Windows systems) __________________________ : NO statsd (collect custom application metrics) ________________ : YES timex (check system clock synchronization) _________________ : YES idlejitter (check system latency and jitter) _______________ : YES bash (support shell data collection jobs - charts.d) _______ : YES debugfs (kernel debugging metrics) _________________________ : YES cups (monitor printers and print jobs) _____________________ : YES ebpf (monitor system calls) ________________________________ : YES freeipmi (monitor enterprise server H/W) ___________________ : YES network-viewer (monitor TCP/UDP IPv4/6 sockets) ____________ : YES systemd-journal (monitor journal logs) _____________________ : YES windows-events (monitor Windows events) ____________________ : NO nfacct (gather netfilter accounting) _______________________ : YES perf (collect kernel performance events) ___________________ : YES slabinfo (monitor kernel object caching) ___________________ : YES Xen ________________________________________________________ : YES Xen VBD Error Tracking _____________________________________ : NO Exporters: AWS Kinesis ________________________________________________ : NO GCP PubSub _________________________________________________ : NO MongoDB ____________________________________________________ : YES Prometheus (OpenMetrics) Exporter __________________________ : YES Prometheus Remote Write ____________________________________ : YES Graphite ___________________________________________________ : YES Graphite HTTP / HTTPS ______________________________________ : YES JSON _______________________________________________________ : YES JSON HTTP / HTTPS __________________________________________ : YES OpenTSDB ___________________________________________________ : YES OpenTSDB HTTP / HTTPS ______________________________________ : YES All Metrics API ____________________________________________ : YES Shell (use metrics in shell scripts) _______________________ : YES Debug/Developer Features: Trace All Netdata Allocations (with charts) ________________ : NO Developer Mode (more runtime checks, slower) _______________ : NO Runtime Information: Profile ____________________________________________________ : standalone Stream Parent (accept data from Children) __________________ : NO Stream Child (send data to a Parent) _______________________ : NO Total System Memory ________________________________________ : 66695446528 Available System Memory ____________________________________ : 40499445760 root@asus-nuc:~# ``` ### Additional info Verified workaround: A local override in `/etc/netdata/health.d/net.conf` modifies only the `inbound_packets_dropped_ratio` template: ```diff -chart labels: device=!wl* * +chart labels: device=!wl* !vmbr0 * ``` After executing `netdatacli reload-health`, the alert instance for `vmbr0` disappeared. The alert instances for `nic0` and `tailscale0` remained present and in CLEAR state. No changes to the physical network, Linux bridge configuration, IGMP, RSTP, or switch settings were required.

Comments

luckman212

Thanks for filing this - I'm in the exact same boat and was wondering if something was wrong with my node. **edit**: apparently this also causes a false positive alert email to be sent from cloud, e.g. <img width="1620" height="1164" alt="Image" src="https://github.com/user-attachments/assets/837d94a9-0f82-4902-881c-d6ef0425c62a" />