twitter / rezolus

Systems performance telemetry

Rezolus

Rezolus is a tool for collecting detailed systems performance telemetry and exposing burst patterns through high-resolution telemetry. Rezolus provides instrumentation of basic systems metrics, performance counters, and support for eBPF (extended Berkeley Packet Filter) telemetry. Measurement is the first step toward improved performance.

Per-metric documentation can be found in the METRICS documentation.

Overview

Rezolus collects telemetry from several different sources. Currently, Rezolus collects telemetry from traditional sources (procfs, sysfs), the perf_events subsystem, and from BPF. Each sampler implements a consistent set of functions so that new ones can be easily added to further extend the capabilities of Rezolus.

Each telemetry source is oversampled so that we can build a histogram across a time interval. This histogram allows us to capture variations which will appear in the far upper and lower percentiles. This oversampling approach is one of the key differentiators of Rezolus when compared to other telemetry agents.

With its support for BPF as well as more common telemetry sources, Rezolus is a very sophisticated tool for capturing performance anomalies, profiling systems performance, and conducting performance diagnostics.

More detailed information about the underlying metrics library and sampler design can be found in the DESIGN documentation.

Features

• traditional telemetry sources (procfs, sysfs, ...)
• perf_events support for hardware performance counters
• BPF support to instrument kernel and user space activities
• oversampling and percentile metrics to capture bursts

Traditional Telemetry Sources

Rezolus collects metrics from traditional sources (procfs, sysfs) to provide basic telemetry for CPU, disk, and network. Rezolus exports CPU utilization, disk bandwidth, disk IOPs, network bandwidth, network packet rate, network errors, as well as TCP and UDP protocol counters.

These basic telemetry sources, when coupled with the approach of oversampling to capture their bursts, often provide a high-level view of systems performance and may readily indicate areas where resources are saturated or errors are occurring.

Perf Events

Perf Events allow us to report on both hardware and software events. Typical software events are things like page faults, context switches, and CPU migrations. Typical hardware events are things like CPU cycles, instructions retired, cache hits, cache misses, and a variety of other detailed metrics about how a workload is running on the underlying hardware.

These metrics are typically used for advanced performance debugging, as well as for tuning and optimization efforts.

BPF

There is an expansive amount of performance information that can be exposed through BPF, which allows us to have the Linux Kernel perform telemetry capture and aggregation at very fine-grained levels.

Rezolus comes with samplers that capture block IO size distribution, EXT4 and XFS operation latency distribution, and scheduler run queue latency distribution. You'll see that here we are mainly exposing distributions of sizes and latencies The kernel is recording the appropriate value for each operation into a histogram. Rezolus then accesses this histogram from user-space and transfers the values over to its own internal storage where it is then exposed to external aggregators.

By collecting telemetry in-kernel, we're able to gather data about events that happen at extremely high rates - e.g., task scheduling - with minimal performance overhead for collecting the telemetry. The BPF samplers can be used to both capture runtime performance anomalies as well as characterize workloads.

Sampling rate and resolution

In order to accurately reflect the intensity of a burst, the sampling rate must be at least twice the duration of the shortest burst to record accurately. This ensures that at least 1 sample completely overlaps the burst section of the event. With a traditional minutely time series, this means that a spike must least 120 seconds or more to be accurately recorded in terms of intensity. Rezolus allows for sampling rate to be configured, allowing us to make a trade-off between resolution and resource consumption. At 10Hz sampling, 200ms or more of consecutive burst is enough to be accurately reflected in the pMax. Contrast that with minutely metrics requiring 120_000ms, or secondly requiring 2000ms of consecutive burst to be accurately recorded.

Getting Started

Building

Rezolus is built with the standard Rust toolchain which can be installed and managed via rustup or by following the directions on the Rust website.

The rest of the guide assumes you've chosen to install the toolchain via rustup.

NOTE: Rezolus is intended to be built and deployed on Linux systems but has some very limited support for MacOS to test the overall framework. It is focused on providing systems telemetry for Linux systems. To get the best experience and develop new samplers, you should build and run on Linux.

Clone and build Rezolus from source

git clone https://github.com/twitter/rezolus
cd rezolus

# create an unoptimized development build
cargo build

# run the unoptimized binary and display help
cargo run -- --help

# create an optimized release build
cargo build --release

# run the optimized binary and display help
cargo run --release -- --help

# run the optimized binary with the example config (needs sudo for perf_events)
cargo build --release && \
sudo target/release/rezolus --config configs/example.toml

# metrics can be viewed in human-readable form with curl
curl --silent http://localhost:4242/vars

Building with BPF Support

By default, BPF support is not compiled in. If you wish to produce a build with BPF support enabled, follow the steps below:

Prerequisites

BPF support requires that we link against the BPF Compiler Collection. You may either use the version provided by your distribution, or can build BCC and install from source. It is critical to know which version of BCC you have installed. Rezolus supports multiple versions by utilizing different feature flags at build time.

Our current policy is to support BCC versions back to the version in the distribution repository for Debian Stable or CentOS 7 (whichever is older) to the most recent version of BCC.

This policy provides coverage for current stable and testing versions of Debian, Ubuntu, CentOS, Fedora, Arch, and Gentoo. Users of other distributions may need to build a supported version of BCC from source. See BCC Installation Guide for details.

Building

As mentioned above, we provide different feature flags to map to various supported BCC versions. The bpf feature will map to the newest version supported by the rust-bcc project. For most users, this will be the right flag to use. However, if you must link against an older BCC version, you will need to use a more specific form of the feature flag. These version-specific flags take the form of bpf_v0_10_0 with the BCC version being reflected in the name of the feature. You can find a complete list of the feature flags in the cargo manifest for this project.

# create an optimized release build with BPF support
cargo build --release --features bpf
sudo target/release/rezolus --config configs/example.toml

# metrics can be viewed in human-readable form with curl
curl --silent http://localhost:4242/vars

HTTP Exposition

Rezolus exposes metrics over HTTP, with different paths corresponding to different exposition formats.

• human-readable: /vars
• JSON: /vars.json, /metrics.json, /admin/metrics.json
• Prometheus: /metrics

NOTE: currently, JSON exposition is provided by default for any other path. This behavior may change in the future and should not be relied on.

Additionally, you can get the running version on the root-level path /

Support

Create a new issue on GitHub.

Contributing

We feel that a welcoming community is important and we ask that you follow Twitter's Open Source Code of Conduct in all interactions with the community.

Authors

• Brian Martin [email protected]

A full list of contributors can be found on GitHub.

Follow @TwitterOSS on Twitter for updates.

License

Copyright 2019 Twitter, Inc.

Licensed under the Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0

Security Issues?

Please report sensitive security issues via Twitter's bug-bounty program (https://hackerone.com/twitter) rather than GitHub.

rezolus's Issues

nividia: change power/energy units

Currently power and energy units are mW and mJ respectively. It would be better to consolidate and use the SI units W and J for each of those measurements.

Basic usage documentation

Brief usage documentation explaining on how to run rezolus. After running the binary I know I have to provide a config file but not sure how that is formatted.

        --config <FILE>    TOML config file

Expected behavior

Documentation for usage to exist

Actual behavior

Not that I could find from the readme file

Steps to reproduce the behavior

Look at the readme

BCC Perf samplers does not report correct counters

BPF perf counters do not return any counters at all.

There are few issues I want to address:

1

Setting key = 0 does not expose the counter to user space. Not sure whether its a BPF constraint or a bug with rust-bcc.

BPF_HASH(counter, u32);
int do_perf_cnt(struct bpf_perf_event_data *ctx) {
   u32 key = 0;
   counter.increment(key); // <--- Does not work, need to have something else (pid works)
   return 0;
}

2

Simply incrementing the sampler inside the bpf context is incorrect.
Need to change count.increment(key) -> count.increment(key, ctx->sample_period). See llcstat

3

Sample frequency seems too low. We convert the sampling interval to hertz. By default 1000 milliseconds = 1Hz which seems a bit low as a sampling rate. (needs more investigation)

runqueue latency bpf fails on newer kernels

Starting from kernel v5.14 the runqueue latency bpf sampling fails due to the
state field of the task_struct being renamed.

To Reproduce
Steps to reproduce the behavior:

1. On a system with kernel >= 5.14 with Rezolus installed
2. Configure Rezolus to enable scheduler bpf sampling
3. Run Rezolus
4. See error

Consider using bcc for perf events

x86 crate is sometimes segfaulting on compile and requires use of nightly rust. Look into using bcc for handling perf events. If we can use stable rust and drop the x86 dependency, it might make sense to make the switch.

Disk BPF sampler failing on Ubuntu Focal

Expected behavior

Disk BPF sampler should be able to load probes on Ubuntu Focal

Actual behavior

log level: Info
2020-07-24 20:38:45.494 INFO  [rezolus] ----------
2020-07-24 20:38:45.494 INFO  [rezolus] rezolus 2.4.1-alpha.0
2020-07-24 20:38:45.494 INFO  [rezolus] ----------
cannot attach kprobe, probe entry may not exist
2020-07-24 20:38:46.771 ERROR [rezolus] failed to initialize disk sampler

Steps to reproduce the behavior

Run Rezolus on Ubuntu Focal (eg in GH workflow) with BPF enabled for disk sampler

docs: readme mentions nightly rust

Readme still references a requirement of nightly rust. This has not been accurate for some time now.

Reduce boilerplate involved in adding new samplers

Right now, there's a lot of boilerplate involved in creating new samplers. Ideally, the only things that people would need to provide would be name, initialization, and sample methods.

feature: add page cache hit/miss

Add statistics around page cache hit/miss so that hitrate can be calculated. An example standalone tool is: https://github.com/brendangregg/perf-tools/blob/master/fs/cachestat

perf event collection overhaul

Perf events sampler needs more advanced scheduling. We may want to limit the number of event counters to avoid multiplexing on the PMUs. Extending support to per-cgroup as requested in #19 requires that we do something clever to avoid performance penalties. At the same time, we would want to be able to enforce that things like Cycles and Instructions are sampled across the same intervals so CPI calculations remain valid.

example configs have invalid percentiles

The example configs still have the older pXX format percentiles shown. These need to be changed to show floating point representation.

Better scoped metrics

In #108 we added support to expose metrics for interrupts per NUMA node. There has also been interest in exposing metrics for individual disks, network interfaces, etc.

To better support the various scopes on which we may gather and summarize metrics, we should consider a more holistic approach. For example, it may be beneficial to make a more robust type of Statistic that can hold information about its scope for reporting.

We need to decide if we wish to allow fine-grained control of scoped metrics, or if we would be okay with on/off per-sampler for scoped metrics. If we need fine-grained control, a more robust scheme for serializing/deserializing, particularly with regard to specifying the metrics in the configuration, will be necessary.

x86 dependency seems to be unused

If that is the only nightly requiring dependency, you may want to update the docs regarding requiring nightly, after removing the x86 dep.

Extend usage documentation

Documentation could do with more explanation, how do we send custom events?

Collection and sampling as a separate library

Hi,

I'm interested in using rezolus as an agent in an event-driven, push-model monitoring infrastructure that is based on Riemann as an metrics stream processor.

My requirements are as follows:

• being able to send metrics gathered by rezolus to Riemann API, either directly via Riemanns Protobuf API or indirectly through a custom HTTP gateway
• for each measurement (event) specify additional metadata, eg. host of origin, tags and other user-defined attributes (eq. unit of measurement)
• method for metadata retrieval should be extensible in the same way the samplers are right now and ideally scriptable
• support additional samplers, eg. being able to sample docker runtime metrics would be nice

It seems to me, that many of these requirements are out of scope of rezolus as it is now.

The idea is to split off collectors and sampling functionality into a library that would be not be dependent on method of telemetry publication or any other additional processing (metadata retrieval).

What is your opinion on this?

Thank you.

memcache sampler causes tokio worker to panic

Looks like there's still a bug in the memcache sampler:

thread 'rezolus-worker' panicked at 'there is no reactor running, must be called from the context of Tokio runtime',

ccccccnejhkhljngdlunkljfctcvkbjbikerllldfnki

One line summary of the issue here.

Expected behavior

As concisely as possible, describe the expected behavior.

Actual behavior

As concisely as possible, describe the observed behavior.

Steps to reproduce the behavior

Please list all relevant steps to reproduce the observed behavior.

Metrics publishing

Add push support for telemetry publishing, currently we only support pull-based collection. We should extend the metrics library to allow for pluggable exposition and publishing. Targets to consider for push support are InfluxDB and Kafka

Create rust-toolchain file

Whenever rustup encounters a file named rust-toolchain, it will automatically install and use the rust version specified in that file. You can for example put nightly in there to use any nightly rustc, or nightly-2019-08-19 to use the nightly as of 2019-08-19.

https://github.com/rust-lang/rustup.rs#the-toolchain-file

Use from_ne_bytes methods to avoid the need to use unsafe

Example location:

rezolus/src/samplers/ebpf/ext4/mod.rs

Lines 133 to 137 in 2c1c618

	let mut k = [0_u8; 4];
	for (index, byte) in k.iter_mut().enumerate() {
	*byte = *key.get(index).unwrap_or(&0);
	}
	let k: u32 = unsafe { mem::transmute(k) };

The u32::from_ne_bytes method can be used to avoid unsafe: https://doc.rust-lang.org/stable/std/primitive.u32.html#method.from_ne_bytes

Simplify cargo build profiles

Reduce the cargo build profiles to the minimum set of overrides

ntp sampler fails to build on musl libc (Alpine Linux)

Describe the bug

ntp sampler fails to build on a system with musl libc (Alpine Linux):

error[E0425]: cannot find function `ntp_gettime` in crate `libc`
  --> src/samplers/ntp/mod.rs:94:37
   |
94 |         let status = unsafe { libc::ntp_gettime(&mut timeval) };
   |                                     ^^^^^^^^^^^ not found in `libc`

error[E0412]: cannot find type `ntptimeval` in crate `libc`
   --> src/samplers/ntp/mod.rs:128:34
    |
128 |   fn default_ntptimeval() -> libc::ntptimeval {
    |                                    ^^^^^^^^^^ help: a struct with a similar name exists: `timeval`
    | 
   ::: .cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.82/src/unix/mod.rs:43:1
    |
43  | / s! {
44  | |     pub struct group {
45  | |         pub gr_name: *mut ::c_char,
46  | |         pub gr_passwd: *mut ::c_char,
...   |
195 | |     }
196 | | }
    | |_- similarly named struct `timeval` defined here

error[E0422]: cannot find struct, variant or union type `ntptimeval` in crate `libc`
   --> src/samplers/ntp/mod.rs:129:11
    |
129 |       libc::ntptimeval {
    |             ^^^^^^^^^^ help: a struct with a similar name exists: `timeval`
    | 
   ::: .cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.82/src/unix/mod.rs:43:1
    |
43  | / s! {
44  | |     pub struct group {
45  | |         pub gr_name: *mut ::c_char,
46  | |         pub gr_passwd: *mut ::c_char,
...   |
195 | |     }
196 | | }
    | |_- similarly named struct `timeval` defined here

error: aborting due to 3 previous errors

To Reproduce
Steps to reproduce the behavior:

Build the project on Alpine Linux Edge.

1. apk add build-base cargo
2. wget https://github.com/twitter/rezolus/archive/v2.11.1.tar.gz && tar -xzf v2.11.1.tar.gz && cd rezolus-2.11.1
3. cargo build --release --locked

Expected behavior
The build should complete successfully.

Environment

• rust 1.51.0
• musl libc 1.2.2
• Alpine Linux Edge
• x86_64

Additional context
Add any other context about the problem here.

How to read Cstate value from the file?

Hi,

The point I would like to confirm is below.

rezolus/src/samplers/cpu/mod.rs

Line 357 in 8f726f9

if let Ok(time) = reader.read_u64().await {

The document about read_u64 is written below.

Reads an unsigned 64-bit integer in big-endian order from the underlying reader.

I think the value in the target file is recorded as microsecond and integer.
I don't know what the function read correctly, but I think it is not necessary to read the file in big-endian order.
I think the function may read information about the file itself, not the contents of the file.

I would like you to confirm whether the implementation is valid.

regards,

remove deregister checks

Once twitter/rpc-perf#262 is fixed, we will not need to check if the sampler is initialized in the deregister functions

build issue: v2.15.1 fails to build from release archive

Describe the bug
Compiling the latest version is failing.

To Reproduce
Steps to reproduce the behavior:

1. Download https://github.com/twitter/rezolus/archive/refs/tags/v2.15.1.tar.gz
2. Extract it
3. $ cargo build --release

Expected behavior
Based on the document it should be compiled successfully.

Screenshots

   .....
   Compiling wrapcenum-derive v0.4.0
   Compiling thiserror v1.0.30
   Compiling rustcommon-histogram v1.0.0 (https://github.com/twitter/rustcommon?branch=master#88e07783)
   Compiling rustcommon-streamstats v0.1.0 (https://github.com/twitter/rustcommon?branch=master#88e07783)
   Compiling bcc v0.0.32
   Compiling rustcommon-heatmap v0.1.1 (https://github.com/twitter/rustcommon?branch=master#88e07783)
   Compiling nvml-wrapper v0.7.0
   Compiling rustcommon-metrics v2.0.0-alpha.0 (https://github.com/twitter/rustcommon?branch=master#88e07783)
   Compiling rezolus v2.15.1 (/home/mort/Downloads/rezolus-2.15.1)
error: failed to run custom build command for `rezolus v2.15.1 (/home/mort/Downloads/rezolus-2.15.1)`

Caused by:
  process didn't exit successfully: `/home/mort/Downloads/rezolus-2.15.1/target/release/build/rezolus-79176f2da258cf67/build-script-build` (exit status: 1)
  --- stderr
  Error: could not find repository from '/home/mort/Downloads/rezolus-2.15.1'; class=Repository (6); code=NotFound (-3)
warning: build failed, waiting for other jobs to finish...
error: build failed

Environment
ArchLinux x86_64
Rustc 1.58.1 (Arch Linux rust 1:1.58.1-1)

Additional context
I'm not sure but probably it's related to the nightly build's note. I'm using the stable Rust version.

Some metircs cannot be displayed at exposition

① Several metrics cannot be displayed (interrupts sampler)
If /proc/interrupts does not contain the metrics implemented in Rezolus, the following metircs in /proc/interrupts cannot store.

rezolus/src/samplers/interrupt/mod.rs

Line 175 in 41d4b1d

while reader.read_line(&mut line).await? > 0 {

rezolus/src/samplers/interrupt/mod.rs

Line 256 in 41d4b1d

continue;

rezolus/src/samplers/interrupt/mod.rs

Line 260 in 41d4b1d

continue;

It is because variable line is not cleared when the keyword did not match the implemented keywords and appended next line to line.
I think that line.clear() is needed before continue.

② Active(anon), Inactive(anon) in /proc/meminfo cannot be displayed (memory sampler)

rezolus/src/samplers/memory/mod.rs

Line 118 in 41d4b1d

Regex::new(r"(?P<stat>\w+):\s+(?P<value>\d+)").expect("failed to compile regex");

Above implementation cannot match the word including parentheses.

I think it should be below:
Regex::new(r"(?P<stat>.+):\s+(?P<value>\d+)").expect("failed to compile regex");
(change \w to .)

③ The node1 metrics is also displayed in the environments that have only node0 (interrupts sampler)
I think the cause is below implementation.

rezolus/src/samplers/interrupt/mod.rs

Line 231 in 41d4b1d

if let Some(previous) =

if let Some(previous) =
    result.get_mut(&InterruptStatistic::Node1Network)
{
    *previous += node1;
} else {
    result.insert(InterruptStatistic::Node1Network, sum);
}

As above implementation, sum variable for total value is used in Node1 metircs.
So only the value acquired for the first time is recorded in node1 regardless of node.
I think it should use node1 instead of sum.

④ Cstate is not displayed correctly (cpu sampler)
In the environment using Intel Skylake CPU, Cstate name in /sys/devices/system/cpu/{}/cpuidle/{}/name can be given the suffix "-SKX".
A example is below.

cat /sys/devices/system/cpu/cpu0/cpuidle/state?/name
POLL
C1-SKX
C1E-SKX
C6-SKX

So under the environment, C1,C1E,C6 is not match the implementation.

rezolus/src/samplers/cpu/stat.rs

Line 198 in 41d4b1d

"C1" => Ok(CState::C1),

So I think it should be below:

        if s == "POLL" || s == "C0" {
            Ok(CState::C0)
        } else if s.starts_with("C1E") {
            Ok(CState::C1E)
        } else if s.starts_with("C1") {
            Ok(CState::C1)
        [..]

Please tell me about the design concept

I'm interested in this projects, I think that getting these values with low resources and high resolution is unique attraction.
But I cannot be sure of several points.
I'm understanding somehow with reading DESIGN.md > Metrics...
(I wasn't sure when I looked at the implementation with debugging... it is hard to read it because it it difficult for me.. I will take some times to read this.)

1. How to achieve oversampling
The interval of Rezolus looks like higher rate with low resources than other products ,
But Rezolus looks like get the data by configured interval ,
How should I understand oversampling in the implementation?

(Does Oversampling mean simply "how to set interval" ?
And Rezolus can achieve it with high resolution with high sampling rate configured fine interval?

And is this indicate that for example, if the burst duration is 200ms, I should set the interval to 100ms ?)

2. What does it mean "secondly rate"
I cannot be sure this word but I believe this word is very important for understanding Rezolus concept.
I think first rate is configured interval in toml, and first rate value is recorded and displayed as metrics/histogram/count (case : cpu/frequency).
But actually, does secondly rate indicate simply metrics/histogram/percentile value?

3. How to construct the histogram
Is it constructed by the data that stored within time window?
For example, under window = 10s and interval = 100ms, Is percentile value decided by 100 samples?

4. How to read the value (ex. cpu/frequency)
There are cases that the server has several cpu. And rezolus looks like take each cpu's frequency.
But the result I can see from exposition (/vars etc..), it looks like only 1 specific cpu data (or summarized value).
If configuration is above setting and the server has 4 cpus, Is the sample stored 400 samples and the result indicate specific cpu's data?

switch from chashmap to dashmap

dashmap has better performance characteristics than chashmap in benchmarking done in context of the metrics library, switch remaining uses of chashmap in Rezolus to dashmap

build failure using latest rust nightly

Instructions indicate to build with rust nightly install, but project fails to build with:
rustc 1.39.0-nightly (9af17757b 2019-09-02).

May I ask to specify nightly rustc version used to produce binaries?

Expected behavior

Build is producing binaries.
Instructions ammended with nightly version used by author.

Actual behavior

Build fails for rustc 1.39.0-nightly (9af17757b 2019-09-02) with:
Compiling x86 v0.19.0
Running CARGO_PKG_NAME=x86 CARGO_MANIFEST_DIR=/root/.cargo/registry/src/github.com-1ecc6299db9ec823/x86-0.19.0 CARGO_PKG_VERSION_MINOR=19 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION=0.19.0 CARGO_PKG_HOMEPAGE='https://github.com/gz/rust-x86' CARGO_PKG_REPOSITORY='https://github.com/gz/rust-x86' CARGO_PKG_AUTHORS='Gerd Zellweger <[email protected]>:Eric Kidd <[email protected]>:Philipp Oppermann <[email protected]>:Dan Schatzberg <[email protected]>:John Ericson <[email protected]>:Rex Lunae <[email protected]>:Brian Martin <[email protected]>:Caleb Boylan <[email protected]>' LD_LIBRARY_PATH='/root/rezolus/target/debug/deps:/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib' CARGO_PKG_VERSION_PRE= CARGO_PKG_DESCRIPTION='Library to program x86 (amd64) hardware. Contains x86 specific data structure descriptions, data-tables, as well as convenience function to call assembly instructions typically not exposed in higher level languages.' OUT_DIR=/root/rezolus/target/debug/build/x86-b4cf20a8ca132655/out CARGO=/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo CARGO_PKG_VERSION_MAJOR=0 rustc --edition=2018 --crate-name x86 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/x86-0.19.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 --cfg 'feature="csv"' --cfg 'feature="performance-counter"' --cfg 'feature="phf"' --cfg 'feature="phf_codegen"' --cfg 'feature="serde_json"' -C metadata=e495abe61bcdff7a -C extra-filename=-e495abe61bcdff7a --out-dir /root/rezolus/target/debug/deps -L dependency=/root/rezolus/target/debug/deps --extern bitflags=/root/rezolus/target/debug/deps/libbitflags-9d0eaefa633987ee.rmeta --extern phf=/root/rezolus/target/debug/deps/libphf-b29ef04220dcb0d2.rmeta --extern raw_cpuid=/root/rezolus/target/debug/deps/libraw_cpuid-93a645499f3040dd.rmeta --cap-lints warn
error: Could not compile x86.

Caused by:
process didn't exit successfully: CARGO_PKG_NAME=x86 CARGO_MANIFEST_DIR=/root/.cargo/registry/src/github.com-1ecc6299db9ec823/x86-0.19.0 CARGO_PKG_VERSION_MINOR=19 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION=0.19.0 CARGO_PKG_HOMEPAGE='https://github.com/gz/rust-x86' CARGO_PKG_REPOSITORY='https://github.com/gz/rust-x86' CARGO_PKG_AUTHORS='Gerd Zellweger <[email protected]>:Eric Kidd <[email protected]>:Philipp Oppermann <[email protected]>:Dan Schatzberg <[email protected]>:John Ericson <[email protected]>:Rex Lunae <[email protected]>:Brian Martin <[email protected]>:Caleb Boylan <[email protected]>' LD_LIBRARY_PATH='/root/rezolus/target/debug/deps:/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib' CARGO_PKG_VERSION_PRE= CARGO_PKG_DESCRIPTION='Library to program x86 (amd64) hardware. Contains x86 specific data structure descriptions, data-tables, as well as convenience function to call assembly instructions typically not exposed in higher level languages.' OUT_DIR=/root/rezolus/target/debug/build/x86-b4cf20a8ca132655/out CARGO=/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo CARGO_PKG_VERSION_MAJOR=0 rustc --edition=2018 --crate-name x86 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/x86-0.19.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 --cfg 'feature="csv"' --cfg 'feature="performance-counter"' --cfg 'feature="phf"' --cfg 'feature="phf_codegen"' --cfg 'feature="serde_json"' -C metadata=e495abe61bcdff7a -C extra-filename=-e495abe61bcdff7a --out-dir /root/rezolus/target/debug/deps -L dependency=/root/rezolus/target/debug/deps --extern bitflags=/root/rezolus/target/debug/deps/libbitflags-9d0eaefa633987ee.rmeta --extern phf=/root/rezolus/target/debug/deps/libphf-b29ef04220dcb0d2.rmeta --extern raw_cpuid=/root/rezolus/target/debug/deps/libraw_cpuid-93a645499f3040dd.rmeta --cap-lints warn (signal: 9, SIGKILL: kill)

Steps to reproduce the behavior

Follow build instructions from README, i.e. invoke:
cargo build -vv

log level in config file is ignored

Expected behavior

Setting the logging level to "debug" in the config should result in debug level logging

Actual behavior

If verbosity is not specified on the command line, logging is always "info" level.

Steps to reproduce the behavior

• Specify the logging = "debug" in the config file
• run the agent
• note that log level is still info
• add --verbose to cli arguments
• note that log level increased to debug

Cgroup support?

Is there any intent to support gathering metrics per cgroup?

Admin endpoint for dynamic reconfiguration

We would like to add an 'admin endpoint' which can allow for temporary adjustments to the configuration. Use-cases involve temporarily disabling perf event collection so that we don't interfere with profiling using tools like vTune. Other use-cases involve temporarily increasing sampling resolution and enabling additional samplers to gain additional insights.

Consider limiting number of supported bcc versions

Currently, the Cargo manifest supports building with bcc from 0.5.0 onwards. However, we're only testing from 0.10.0 onward in CI. We should consider establishing a policy to limit the number of supported bcc versions and aligning the CI configs with the established policy.

I'd propose supporting the 3 most recent versions (0.10.0, 0.11.0, 0.12.0), which should provide users of Ubuntu, Arch, Fedora all options to use bcc packages either from iovisor or from their distribution.

Debian users on buster (stable) would need to build bcc from source, as the package in the repo for buster is 0.8.0.

make sampler initialization errors fatal in CI

Currently, if a sampler has some error - for example, an eBPF sampler's BPF code fails to compile, this error is handled without the process terminating. This is great for production use-cases where we prefer to avoid process termination and would rather log an error and continue running. However, in CI we want to surface errors like this.

consider switching from tiny_http

tiny_http is being flagged for having an open rustsec advisory:

error: 1 vulnerability found!
ID:       RUSTSEC-2020-0000
Crate:    tiny_http
Version:  0.6.2
Date:     2020-06-16
URL:      https://rustsec.org/advisories/RUSTSEC-2020-0000
Title:    HTTP Request smuggling through malformed Transfer Encoding headers
Solution:  No safe upgrade is available!
Dependency tree: 
tiny_http 0.6.2
└── rezolus 2.6.1-alpha.0

It doesn't look like this exposes us to any issues - but we can probably consolidate some dependencies by switching to warp as we already use reqwest which shares a depdency on hyper

expand coverage for in-container telemetry

Currently we have some primitive support for gathering container-level telemetry from within a container. The primary use-case would be using Rezolus as a side-car to provide additional insight into container performance. Currently, we only support some basic telemetry from the cpu accounting cgroups. We should expand our coverage here to expose telemetry for other aspects of the container's runtime

Support multiple endpoints for memcache sampler

Currently the memcache sampler in Rezolus only supports passing a single Memcache endpoint to monitor and sample
https://github.com/twitter/rezolus/blob/master/src/config/samplers.rs#L93

rezolus/src/samplers/memcache/mod.rs

Lines 43 to 57 in fadf6f8

	let endpoint = common.config.samplers().memcache().endpoint().unwrap();
	let mut addrs = endpoint.to_socket_addrs().unwrap_or_else(|_| {
	fatal!("ERROR: endpoint address is malformed: {}", endpoint);
	});
	let address = addrs.next().unwrap_or_else(|| {
	fatal!("ERROR: failed to resolve address: {}", endpoint);
	});
	let sampler = Self {
	address,
	common,
	stream: None,
	};
	if sampler.sampler_config().enabled() {
	sampler.register();
	}

It would be nice if we could pass a comma separated list of host:port memcache endpoints to Rezolus memcache sampler. This would allow you to run a single Rezolus agent to monitor multiple Memcache process or containers on a single host. The config might look something like this:

# An example config that produces percentile metrics for specific Memcached stats
# while preserving the original metric names.

[general]
listen = "0.0.0.0:4242"
fault_tolerant = false
reading_suffix = ""

[samplers]
[samplers.memcache]
enabled = true
endpoint = "localhost:11211,localhost:11212,localhost:11213,localhost:11214"

An alternative would be to run a Rezolus agent per memcache process 1:1 but this creates additional overhead for end user to configure and run on host in this way.

build failures: asm macro changed on latest nightly

Using the latest nightly toolchain, build fails complaining about llvm_asm

Expected behavior

Successful build

Actual behavior

error: the legacy LLVM-style asm! syntax is no longer supported --> /home/nani/.cargo/registry/src/github.com-1ecc6299db9ec823/x86-0.19.0/src/lib.rs:96:5 | 96 | asm!("rdpid $0" : "=r"(pid)); | ----^^^^^^^^^^^^^^^^^^^^^^^^^ | | | help: replace with: llvm_asm!`
|
= note: consider migrating to the new asm! syntax specified in RFC 2873
= note: alternatively, switch to llvm_asm! to keep your code working as it is

Compiling serde_derive v1.0.104
Compiling strum_macros v0.17.1
Compiling async-trait v0.1.23
error: aborting due to 61 previous errors

error: could not compile x86.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed
`

Steps to reproduce the behavior

Please list all relevant steps to reproduce the observed behavior.
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env rustup toolchain install nightly git clone https://github.com/twitter/rezolus cd rezolus/ cargo build --release
Follow exact steps mentioned in ReadMe

add TCP Connect Latency

Add TCP Connect Latency based on https://github.com/iovisor/bcc/blob/master/tools/tcpconnlat.py