awslabs / swage

A collection of libraries useful for developing and running services.

Swage

Swage is a collection of libraries useful for developing and running services.

Currently it consists of the following Java libraries:

• Metrics
• Type Safe
• Thread Context
• Disseminating Executors

Metrics

The Swage Metrics library provides a clean, robust API for recording metric events in a Java application, as well as a core implementation for sending the instrumented metric data to AWS CloudWatch.

It consists of the following modules:

• metrics-core
• metrics-api

Type Safe

Swage Type Safe is a small library that provides type-safe solutions that are not part of the Java standard libraries. At the moment it consists soley of the TypedMap, a class for storing heterogenous key/value pairs in a a type safe way.

It lives in the type-safe module.

Thread Context

Swage Thread Context is a library to help manage and propagate task context in an application across threads. Where a ThreadLocal can help manage context within one thread, this library provides a way to manage context where a task may span many threads.

It lives in the thread-context module.

Disseminating Executors

The Disseminating Executors library solves the same basic problem as the Thread Context library, but with a different approach. Ths library is intended to allow targetd state to cross thread boundaries without invasive changes at every execution point. It uses custom task executors to capture and propagate state.

It lives in the disseminating-executors module.

swage's Issues

Support for aggregating metrics during recording

MetricDataAggregate provides a way of aggregating metrics that is specific to CloudWatch - we could extract that into a common class that could be used by any recorder implementation.

Define a naming scheme for metrics

Different users, products, communities will want to define their own names for metrics which may lead to conflicts. We should define a scheme that allows them to reserve a namespace using some well-know naming system (e.g. DNS or URIs).

Use javax.measure?

See https://github.com/unitsofmeasurement

JSR 363 added "official" support for Units of Measure. JSR 385 extends that to account for the SI revision expected in 2019 as well as updating the API for modularity (Java 9 and later).

The Unit class has a "todo" to reexamine units once JSR 363 goes mainstream. Even if we don't we should decide which units we should support, especially ones related to computing systems.

Provide Samples

Provide sample code to illustrate how metrics are used by:

1. Code emitting metircs
2. Code managing the contexts in which metrics are emmitted
3. Code recording metrics, and how to deal with aggregations

Refactor module to separate implementations

There are two metric-related modules at the moment: metrics-api and metrics-core.
Some things in core might be worth moving to the API e.g. ScopedMetrics and Timer

Core contains recorder implementations for 'file' and 'cloudwatch' - these could be moved to separate modules to control dependencies. If that is done, do we still need 'core'?

Should the JMX integration be moved to its own module?

Child Context support

Provide an API for creating a sub-context when collecting metrics.

ScopedMetrics is unsafe to use if uninitialized.

Library code cannot utilize ScopedMetrics in its current state. There is no way for a library to tell if ScopedMetrics has been initialized or a context has been opened. Attempting to open ScopedMetrics will result in an IllegalStateException. Attempting to record on an unopened metric will result in a NPE. As a library I cannot assume that all of my clients will have initialized SwageMetrics. It would be better to have ScopedMetrics handle these cases rather than expecting every library to implement this exception handling.

Review need to inject executor in metrics reporters

Any of our reporters that use executors should probably allow customers to inject one to handle it's shutdown scenarios.
See this PR for CloudWatch #25

Metric class defines arbitrary restrictions on metric names

The current Metric class restricts metric names to 250 characters which must be either alphanumeric or from the set .:/_@-.

While the comment indicates the constraints are taken from existing metrics backend systems, those systems are not identified. Further, Metric names that are compatible with the backends for both of the officially vended recorders (FileRecorder and CloudWatchRecorder) are disqualified by the restrictions in place on Metric. In particular, it was identified in this pull-request, that JMX Object Names contain spaces, which are not currently allowed in Metric names.

The metric name validation should be significantly loosened in the general-purpose Metric class, and any invalid metric label handling rightly belongs in the particular Recorders themselves.

Fate of ScopedMetrics

TODO: throw this away, make frameworks handle it?

This provides a mechanism for associating a metric Context with the current Thread so that it does not need to be passed on every call. However, it relies on the ThreadContext library to do this. This is the type of thing a framework might use to tunnel metrics Context through application code so that it is available to framework libraries the application may use.

TODO: move fewer-parameter versions to MetricRecorder?

There are two aspects to what this class is doing. On one hand, it is providing a way for an emitted (the library) to retrieve the current metric Context so it can record a measurement. On the other hand, it is providing mechanisms for binding a Context to the current Thread before calling application code.

How about moving this the API class, but splitting the aspects between the metric Context and the Recorder?

Simplify JMX sampling?

Is the JMX sampler doing more than reading the values of a set of MBean attributes?
Can we replace all the Sensor implementations with something that takes a MBean ObjectName and attribute, with an optional Function to process the current value into something to be recorded?
Can we replace the Executor with a Timer MBean?

Review constructor/builder use in CloudWatchRecorder

Most of the constructors use the builder, a few of them don't.

Review and unify if possible/reasonable.

JMX metrics should use dimensions

The metric named MemoryPoolUsed_Code Cache is pretty unfortunate, and doesn't have good support in CloudWatch. The JMX sensors should utilize dimensions:

# So instead of:
dimensions=[], metric='MemoryPoolUsed_Code Cache'
# Use:
dimensions=[('type', 'MemoryPool'), ('name', 'Code Cache')], metric='Used'

There, dimensions matches the ObjectName of the particular PlatformManagedObject, to borrow the names in the Javadocs. Currently, none of the JMX metrics use dimensions.

TypedMap.Key is unintuitive

Consider the following code

package software.amazon.swage.howto;

import software.amazon.swage.collection.TypedMap;
import software.amazon.swage.metrics.ContextData;
import software.amazon.swage.metrics.Metric;
import software.amazon.swage.metrics.MetricRecorder;

import java.time.Instant;
import java.util.Objects;

/**
 * This task shows how to write metrics using Swage.
 */
public final class WriteMetricsTask implements Runnable {
    /**
     * A Metric to write.
     */
    private static final Metric METRIC = Metric.define("Cool");
    /**
     * A value for the Metric.
     */
    private static final Number NUMBER = 42;
    /**
     * that does the recording.
     */
    private final MetricRecorder recorder;
    /**
     * the ContextData that will be recorded from.
     */
    private final ContextData data;

    /**
     * constructor.
     * @param r should not be null.
     * @param data should not be null.
     */
    WriteMetricsTask(final MetricRecorder r,
                     final ContextData data) {
        this.recorder = Objects.requireNonNull(r);
        this.data = Objects.requireNonNull(data);
    }

    @Override
    public void run() {
        final TypedMap c = data
                .with(TypedMap.key("StartTime", String.class),
                        Long.valueOf(Instant.now().toEpochMilli()).toString())
                .build();
        final MetricRecorder.Context context = recorder.context(c);
        context.record(METRIC, NUMBER, null, Instant.now());
    }
}

with a test:

package software.amazon.swage.howto;

import org.junit.Before;
import org.junit.Test;
import org.mockito.ArgumentCaptor;

import org.mockito.Captor;
import org.mockito.Mock;
import org.mockito.MockitoAnnotations;
import software.amazon.swage.collection.TypedMap;
import software.amazon.swage.metrics.ContextData;
import software.amazon.swage.metrics.MetricRecorder;

import static org.junit.Assert.*;
import static org.mockito.BDDMockito.*;

public class WriteMetricsTaskTest {
    @Mock MetricRecorder recorder;
    @Mock ContextData data;
    @Captor ArgumentCaptor<TypedMap> captor;
    WriteMetricsTask toTest;


    @Before
    public void setup() {
        MockitoAnnotations.initMocks(this);
        given(recorder.context(captor.capture())).willCallRealMethod();
        toTest = new WriteMetricsTask(recorder, data);
    }

    @Test
    public void canWriteMetrics() {
        toTest.run();
        final TypedMap captured = captor.getValue();
        verify(recorder).context(captured);
        assertTrue(captured.containsKey(TypedMap.key("StartTime", String.class)));
    }
}

That test fails. The assertion is false. Looking at the JavaDoc for TypedMap.Key seems to imply this was intentional.

Why would anyone find that usable? What would the intended usage look like here?

Should MetricsRecorder be passed Context rather than TypedMap

The context for recording a measurement is passed to the recorder as a TypedMap. We could pass up the Context instance itself and use accessors to allow the recorder to access dimensions. This would allow us to extend Context if we needed to.

Should Metric define the Unit it measures?

When recording a measurement, the caller has to specify the units of the measurement:

record(Metric label, Number value, Unit unit, Instant time)

This has the potential for different units to be used in different places for the same Metric. How about having the Metric define the canonical unit? That way, application code need not supply it, or if we allow it to, the recorder would then be able to convert it to the canonical unit for consistency.

Remove NullContext (MetricsContext)

There's no need for a null MetricsContext as the MetricsContext object shouldn't be doing any work.

The correct way to create a no op handler for metrics is to implement a no op recorder as seen in the NullRecorder.

Audit and fix use of Data, Attributes, and Dimensions.

Previously we used the term data and dimensions interchangeably. We have since moved to use Attributes when talking about the data within a context and using dimensions to refer too Cloudwatch dimensions. This change was made as part of a larger refactor #19 . In order to unblock the refactor we have merged the PR, despite there being at least one case of the incorrect usage.

We need to audit our use of these terms to ensure consistency.

Metric's Datum are aggregated by dimensions not attributes: [(https://github.com/awslabs/swage/blame/7193ce1285c7502011a12144ef71bdc1dfd3f1c5/metrics-core/src/main/java/software/amazon/swage/metrics/record/cloudwatch/CloudWatchRecorder.java#L353)]

Remove or Rename ContextData

With the change from "data" to "attributes" when talking about contexts, should we rename ContextData? Or is it even necessary as a concept? it seems to just be a holder for attribute keys. Is there any reason we need to couple that to TypedMapBuilder?