bo0mer / mozzle

Pull metrics for Cloud Foundry applications and forward them to Riemann

mozzle

Pull metrics for Cloud Foundry applications and forward them to Riemann.

This repo provides two functionalities - a mozzle Go package and a mozzle command-line tool.

Package mozzle provides an API for monitoring infrastructure metrics of Cloud Foundry applications and emitting them to a 3rd party monitoring system.

The mozzle command-line tool emits metrics for Cloud Foundry applications to a specified Riemann instance. The rest of this document describes its usage.

Before reading futher, make sure you have a running Riemann instance. If you want just to try out mozzle, refer to this guide how to setup one in a minute.

User's guide

If you want to monitor all applications under your current Cloud Foundry target, as set with the CF CLI, you can do the following.

mozzle -use-cf-cli-target

If you want to explicitly specify the monitored target, you can do that too.

# This example assumes that you have exported the necessary env variables
mozzle -api https://api.bosh-lite.com -access-token $CF_ACCESS_TOKEN -refresh-token $CF_REFRESH_TOKEN -org NASA -space rocket

If you do not want to deal with access and refresh tokens, you can provide plain username and password.

mozzle -api https://api.bosh-lite.com -username admin -password admin -org NASA -space rocket

And if your Cloud Foundry has invalid TLS certificate for some reason, you can skip its verification.

mozzle -insecure -api https://api.bosh-lite.com -username admin -password admin -org NASA -space rocket

Following is a full list of supported command-line flag arguments.

Usage of mozzle:
  -access-token string
    	Cloud Foundry OAuth2 token; either token or username and password must be provided
  -api string
    	Address of the Cloud Foundry API (default "https://api.bosh-lite.com")
  -events-queue-size int
    	Queue size for outgoing events (default 256)
  -events-ttl float
    	TTL for emitted events (in seconds) (default 30)
  -insecure
    	Please, please, don't!
  -org string
    	Cloud Foundry organization (default "NASA")
  -password string
    	Cloud Foundry password; usage is discouraged - see token option instead
  -refresh-interval duration
    	Time between polling the CF API (default 15s)
  -refresh-token string
    	Cloud Foundry OAuth2 refresh token; to be used with the token flag
  -riemann string
    	Address of the Riemann endpoint (default "127.0.0.1:5555")
  -rpc-timeout duration
    	Timeout for RPCs (default 15s)
  -space string
    	Cloud Foundry space (default "rocket")
  -use-cf-cli-target
    	Use CF CLI's current configured target
  -username string
    	Cloud Foundry user; usage is discouraged - see token option instead
  -v	Report mozzle version
  -version
    	Report mozzle version

Demo usage

This repo brings a vagrant automation that will setup a VM ready for showing your application metrics. For more info on settin it up, refer to its README file.

mozzle's Issues

Provide Emitter implementation that sends metrics to Stackdriver

Stackdriver is Google's Cloud monitoring, logging & tracing service. It provides a public HTTP API, as well as a Go client library, so this should be pretty straight forward without changing any of the mozzle's internals.

Changing the API so it gets simpler will be a huge benefit, though.

Space query filters only by organization

When querying for a particular space, the query filters only by organization guid:

(taken from https://github.com/Bo0mer/mozzle/blob/master/monitor.go#L331-L342)

	spaceQuery := ccv2.Query{
		Filter: ccv2.FilterOrganizationGUID,
		Op:     ccv2.OperatorEqual,
		Value:  orgs[0].GUID,
	}
	spaces, err := cc.Spaces(ctx, spaceQuery)
	if err != nil {
		return ccv2.Space{}, err
	}
	if len(spaces) != 1 {
		return ccv2.Space{}, fmt.Errorf("%q does not describe a single space", space)
	}

If there are multiple spaces within the organization the code will fail with <space> does not describe a single space, which is obviously a bug.

Provide both client side and server side HTTP metrics

Currently the reported HTTP metrics are regarding the communication between the Cloud Foundry router and the application instances (m2 in the picture below).

The firehose supports metrics both for the communication between the CF router and the application instances, and the end user and the CF router.

      HTTP          HTTP
user <----> router <----> application instance
       m1            m2

Provide both m1 and m2 metrics, as this will enable one to do a more detailed RCAs. E.g. to determine whether there is problem with its application instance, the overall performance of the CF router is bad, or there is problem with the network, or ....

The new metric could be called the same way as the old one (e.g. http response time_ms), but should have additional attribute that specifies its origin (m1 or m2).

Dependency missing: https://github.com/bigdatadev/goryman

The Riemann client library under github.com/bigdatadev/goryman is not present any more.

Add option to provide OAuth token instead of credentials

So one does not have to provides his credentials via the command-line.

Report CF application events as metrics

Application events are available via the events CC API[0].

Some of them are more significant and could be reported as metrics - we could start with app.crash and some subset of audit.app.update, depending on the description.

The end goal would be to have events for:

• crashes
• updates, including the context of the update (e.g. scale to 2 instances)
• starts/stops
• route creations/deletions
• maybe more?

Note that some of the events are still experimental, as described by [0].
Ideally, once they become non-experimental, it would be possible to emit events for every application lifecycle event, so one could visualize it with the other metrics.

[0] https://apidocs.cloudfoundry.org/249/events/list_events_associated_with_an_app_since_january_1,_2014.html

Overview dashboard including all apps under org/space

It would be nice to have a dashboard that shows general information about all applications under a org/space, since one might want to observe the behavior of it's applications/services as a whole, not just looking at a single application.

mozzle cli not available

hi,

i am not sure if i missed something somewhere in the instructions. after cloning the repo and successfully provision the vagrant container i am not able to find the mozzle command inside and outside the container. would you be so kind to give me a hint?

thanks!

br
gernot

Add option to use the CF CLI config.json for obtaining configuration

The api, organization, space, access token and refresh token could be read from the CF CLI configuration file, config.json

This enables ever more convenient usage of the mozzle tool to monitor the current Cloud Foundry target one's been working with.

The flag could be called use-cf-config or use-cf-cli-config. The later is more long, but more descriptive.

cmd/mozzle: Add option to specify protocol for communication to Riemann

ATM it is fixed to TCP: https://github.com/Bo0mer/mozzle/blob/master/cmd/mozzle/main.go#L117

This should come from the command line, so one is able to pick what's appropriate, depending on the infrastructure.

Possible implementations:

1. Use tcp for default, add a flag for udp (it is used rarely IMHO)
2. Derive it from the -riemann address flag, e.g. tcp://<addr:port>, or udp://<addr:port>. <addr:port> defaults to tcp.
   Snippet showing how parsing could be achieved.

u, _ := url.Parse("tcp6://1.2.3.4:8080")
fmt.Println(u.Scheme, u.Host)
// Output: tcp6 1.2.3.4:8080

Report content length of HTTP requests and responses

Report content length of HTTP requests and responses if it is possible. This will enable monitoring of the bandwidth consumed by an application.

The metrics could be named something like:

http request content_length_bytes
http response content_length_bytes

Data race condition

Stack trace:

==================
WARNING: DATA RACE
Read at 0x00c4203984b0 by goroutine 8:
  runtime.mapiterinit()
      /usr/local/go/src/runtime/hashmap.go:620 +0x0
  github.com/amir/raidman.eventToPbEvent()
      /Users/i305961/workspace/Go/src/github.com/amir/raidman/raidman.go:230 +0x142d
  github.com/amir/raidman.(*Client).SendMulti()
      /Users/i305961/workspace/Go/src/github.com/amir/raidman/raidman.go:291 +0xf5
  github.com/amir/raidman.(*Client).Send()
      /Users/i305961/workspace/Go/src/github.com/amir/raidman/raidman.go:283 +0x7c
  github.com/Bo0mer/mozzle.(*riemann).SendEvent()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/riemann.go:129 +0x5c
  github.com/Bo0mer/mozzle.(*RiemannEmitter).emitLoop()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/riemann.go:93 +0x190

Previous write at 0x00c4203984b0 by goroutine 28:
  runtime.mapassign1()
      /usr/local/go/src/runtime/hashmap.go:442 +0x0
  github.com/Bo0mer/mozzle.(*RiemannEmitter).Emit()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/riemann.go:68 +0x2ce
  github.com/Bo0mer/mozzle.applicationMetrics.EmitTo()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/appmetrics.go:32 +0x6b5
  github.com/Bo0mer/mozzle.(*AppMonitor).emitAppSummary()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/monitor.go:285 +0x3ab
  github.com/Bo0mer/mozzle.(*AppMonitor).monitorApp()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/monitor.go:234 +0x308

Goroutine 8 (running) created at:
  github.com/Bo0mer/mozzle.(*RiemannEmitter).Initialize()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/riemann.go:35 +0x210
  main.main()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/cmd/mozzle/main.go:117 +0x358

Goroutine 28 (running) created at:
  github.com/Bo0mer/mozzle.(*AppMonitor).Monitor()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/monitor.go:208 +0x863
  github.com/Bo0mer/mozzle.Monitor()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/monitor.go:152 +0xbab
  main.main()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/cmd/mozzle/main.go:124 +0x3dc
==================
==================
WARNING: DATA RACE
Read at 0x00c42033c060 by goroutine 8:
  runtime.mapiterinit()
      /usr/local/go/src/runtime/hashmap.go:620 +0x0
  github.com/amir/raidman.eventToPbEvent()
      /Users/i305961/workspace/Go/src/github.com/amir/raidman/raidman.go:230 +0x142d
  github.com/amir/raidman.(*Client).SendMulti()
      /Users/i305961/workspace/Go/src/github.com/amir/raidman/raidman.go:291 +0xf5
  github.com/amir/raidman.(*Client).Send()
      /Users/i305961/workspace/Go/src/github.com/amir/raidman/raidman.go:283 +0x7c
  github.com/Bo0mer/mozzle.(*riemann).SendEvent()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/riemann.go:129 +0x5c
  github.com/Bo0mer/mozzle.(*RiemannEmitter).emitLoop()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/riemann.go:93 +0x190

Previous write at 0x00c42033c060 by goroutine 29:
  runtime.mapassign1()
      /usr/local/go/src/runtime/hashmap.go:442 +0x0
  github.com/Bo0mer/mozzle.(*RiemannEmitter).Emit()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/riemann.go:68 +0x2ce
  github.com/Bo0mer/mozzle.containerMetrics.EmitTo()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/containermetrics.go:61 +0x126a
  github.com/Bo0mer/mozzle.(*AppMonitor).monitorFirehose()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/monitor.go:259 +0x36e

Goroutine 8 (running) created at:
  github.com/Bo0mer/mozzle.(*RiemannEmitter).Initialize()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/riemann.go:35 +0x210
  main.main()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/cmd/mozzle/main.go:117 +0x358

Goroutine 29 (running) created at:
  github.com/Bo0mer/mozzle.(*AppMonitor).monitorApp()
      /Users/i305961/workspace/Go/src/github.com/Bo0mer/mozzle/monitor.go:227 +0x12a
==================
^Cexiting...

Not able to run the mozzle binary file

While i run =>mozzle_windows_amd64 -use-cf-cli-target
ran in to below error:
mozzle: error reading CF CLI config: error open ".cf\config.json": open .cf\config.json: The system cannot find the path specified.
I have no clue what is missing. Please help me

cmd/mozzle: No way to configure RPC timeouts or refresh interval

It sticks to the default 5s which may not be sufficient. This should be configurable through the command-line. E.g.

mozzle -rpc-timeout 30s -refresh-interval 30s