rbmhtechnology / eventuate-tools Goto Github PK

From @magnusart on July 23, 2015 7:39

Based on the discussion on Gitter I'd like to discuss the design of a tool that would allow team members to query and view event logs for support issues, debugging and testing purposes.

Problem description

Event log payloads are not available directly by querying Cassandra or other storages since all event log payloads are serialized into binary blobs. Trying to make sense of these blobs without proper deserialization support becomes an exercise in frustration. With a hex editor you can discern some tidbits of information.

Open questions

Generic deserialization

Is there is a way to deserialize java serialization, proto buf or thrift into a human readable format without knowing anything about the data apart from the protocol before hand?

I'm not familiar enough with the formats to judge above, but I'm guessing it is not possible and that we need to have the data model available to deserialize with field labels visible.

Desktop or web app

I'm leaning towards a desktop application, since it is very convenient to not rely on a server to be up to be able to troubleshoot. But if we cannot solve generic deserialization then a server application would be the most obvious choice.

Copied from original issue: RBMHTechnology/eventuate#88

The ReplicationEndpointGauge returns a structured object (ReplicationEndpointMetrics), however spring-boot for example only supports gauges returning simple numbers. So there should be an alternative set of gauges that just return numbers. As the replication partners are not necessarily know in advance this set has to adapt at runtime and thus cannot be implemented as dropwizrd's MetricSet.

From @krasserm on September 14, 2015 12:55

Copied from original issue: RBMHTechnology/eventuate#110

The actors involved in health monitoring de-register all health checks when stopped. This is fine, if they are stopped on purpose for example through a corresponding stopMonitoring call. However if they are stopped prematurely for example through unexpected termination of the ActorSystem they should rather report all monitored item as unhealthy with an unknown status. One can see unexpected ActorSystem termination for example when Eventuate's cassandra extension cannot access the database at event log startup.

An alternative metrics-lib is kamon. In addition to the support for dropwizard lib (implemented through #3) kamon should be supported with the same entries.

An eventuate application should be enabled to expose metrics about the replication state. In particular the following numbers should be exposed per replication endpoint and per log:

• current sequence number of the local log
• per remote log: sequence number in the remote log up to which events are replicated into the local log

Initial implementation is based on dropwizard's metrics library, but support for other metrics libraries can follow.

please:

https://oss.jfrog.org/oss-release-local/com/rbmhtechnology/eventuate-tools/kamon-metrics_2.11/0.8/kamon-metrics_2.11-0.8.pom

eventuate -> v 0.8.1

kamon -> v 0.6.5 (both 2.11 and 2.12)

From @krasserm on September 14, 2015 12:7

Context

An Eventuate application version 1 is replicated across multiple locations and shall be upgraded to version 2. In the following description, a single local event log per location is assumed. This can be easily extended to multiple event logs. Depending on version compatibility, several scenarios can be distinguished.

Backwards and forward compatibility

Version 2 is backwards compatible to version 1 and version 1 is forward compatible to version 2. In this case, locations can be upgraded incrementally with the following procedure at each location.

• Stop making updates to the event log of a location i.e. stop accepting commands and replicating events from other locations (= close replication connections from this location to other locations).
• Close all replication connections from other locations to this location. The result is that this location is now completely disconnected.
• Start version 2 at this location and replicate all events from version 1 to version 2 (by opening a replication connection from version 2 to version 1). This recovers application state in version 2. As an optimization, this step could also run in parallel with previous steps (with minor modifications, in order to ensure that version 2 is running read-only, i.e. only reading from but not writing to the event log, while version 1 is still running and writing. When version 2 caught up, stop version 1 and allow version 2 to write).
• From version 2, open replication connections to other locations in both directions. Locations running version 1 will be able to process events from version 2 (forward compatibility) and vice-versa (backwards compatibility).

Backwards compatibility only

Update: this section is obsolete with RBMHTechnology/eventuate#218 and the procedure described in the previous section can be applied.

Version 2 is backwards compatible to version 1 but version 1 is not forward compatible to version 2. In this case, locations can also be upgraded incrementally with the following procedure at each location.

• Steps 1-3 are identical to the procedure described in section Backwards and forward compatibility.
• From version 2, open replication connections to locations running version 2 in both directions but open replication connections to locations running version 1 only in one direction, so that events are replicated from version 1 to version 2 but not in the other direction.

Version incompatibility

Version 2 is not backwards compatible to version 1. This scenario may arise after a more complex log migration and locations should be upgraded with the following procedure.

• Stop accepting commands at each location.
• Wait for all replicas to converge.
• Close all replication connections.
• Start version 2 at each location and migrate all events from version 1 to version 2 by running a log migrator instead of a simple replicator. After completion, version 2 can recover from the migrated log. As an optimization, this step could also run in parallel with previous steps (with minor modifications).
• Update sequence numbers of replication progress metadata at each location. This might be necessary because log migrations may merge or split events which has an impact on these numbers.
• Open replication connections to other locations in both directions. Since all locations have already been upgraded to version 2, only connections between the same versions are established.

Copied from original issue: RBMHTechnology/eventuate#107

Currently LogViewer allows to select a range of events based on the event's sequence numbers. This should be extended to support content based filtering, i.e. select only those events that match a given criterion.

Note that content based filtering is applied client-side, i.e. all events are retrieved from the source log, but only those that match the criterion are printed.

Eventuate tries to establish a replication connection on activation of an ReplicationEndpoint. Un/Available messages for this replication connection are emitted after the connection is successfully established. This makes current replication health reporting appear only after the initial connection is established. If the connection cannot be established replication health is not reported at all. As a connection that cannot be established can be considered unhealthy it should be reported accordingly.

Currently the log-viewer uses a hard-coded timeout of 15s when reading events from a remote log. It should instead use eventuate.log.replication.remote-read-timeout

From @krasserm on September 14, 2015 13:4

Copied from original issue: RBMHTechnology/eventuate#111

From @krasserm on September 14, 2015 14:21

Export events and their metadata (schema, ...) for long-term storage (on HDFS, for example).

Copied from original issue: RBMHTechnology/eventuate#112

Currently the PersistenceHealthMonitor will only create a Health.Result for an event-log if the status of the event-log's CircuitBreaker has changed.

This means that even if an application is up and running and event log persistence is working with e.g. a Cassandra backend the status will not be exposed as being up nor done. A <prefix>.persistence-of.<log-id> simply is not present.

I would have expected an <prefix>.persistence-of.<log-id> entry in any case reflecting the current status.

In the first step the tool simply prints a string representation of a selection of DurableEvents to stdout. The application writing the log must be running for this and the log viewer tool must have access to the class definitions of the event's payload and corresponding serializers. This tool is the first step for a more sophisticated tool as described in eventuate issue #88