Syslog drains killing loggregator in CF170 about loggregator-release

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/71945822. This repo is managed by the 'Logging' team.

[trastle]

Logs when working fine

{"timestamp":1400858570.913817883,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 1342 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858570.914405823,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.914939642,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.914915562,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.915156364,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 1235 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.915246248,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.915240049,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: New services for e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":27,"method":"loggregator/store.(*AppServiceStore).Run"}
{"timestamp":1400858570.915543795,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Got 1235 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.915626526,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.915795565,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: removing app e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":89,"method":"loggregator/store.(*AppServiceStore).removeAppFromStore"}
{"timestamp":1400858570.915943384,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.916057348,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.920450926,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 278 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858570.921004772,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.921431541,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.921369314,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.921662092,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 171 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.922039032,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.922117710,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: New services for e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":27,"method":"loggregator/store.(*AppServiceStore).Run"}
{"timestamp":1400858570.922680855,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Got 171 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.922674656,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: removing app e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":89,"method":"loggregator/store.(*AppServiceStore).removeAppFromStore"}
{"timestamp":1400858570.922883749,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 254 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858570.922844410,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.923185587,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.923261166,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.922619581,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.923821211,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 250 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858570.923916101,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.924548864,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.923816681,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.925348520,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 147 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.925594807,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.930262327,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: New services for e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":27,"method":"loggregator/store.(*AppServiceStore).Run"}
{"timestamp":1400858570.930540085,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.930679321,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.930801630,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 143 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.930938721,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.931132317,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: removing app e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":89,"method":"loggregator/store.(*AppServiceStore).removeAppFromStore"}
{"timestamp":1400858570.931239605,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: New services for e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":27,"method":"loggregator/store.(*AppServiceStore).Run"}
{"timestamp":1400858570.931340218,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: removing app e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":89,"method":"loggregator/store.(*AppServiceStore).removeAppFromStore"}
{"timestamp":1400858570.931449175,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.931567192,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Got 147 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.931630373,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 259 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858570.931707382,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.931783199,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.931841135,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.931885719,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.931987286,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Got 143 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.932079315,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.932146788,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.932192326,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.932286501,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.932331085,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 152 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.932513475,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.932623863,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.932729483,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: New services for e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":27,"method":"loggregator/store.(*AppServiceStore).Run"}
{"timestamp":1400858570.932817698,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: removing app e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":89,"method":"loggregator/store.(*AppServiceStore).removeAppFromStore"}
{"timestamp":1400858570.932966232,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Got 152 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.933085680,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.933194399,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.934039116,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 241 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858570.934518099,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.934947491,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.935219765,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 134 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.935640335,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.936088562,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.936334848,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: New services for e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":27,"method":"loggregator/store.(*AppServiceStore).Run"}
{"timestamp":1400858570.936732769,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: removing app e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":89,"method":"loggregator/store.(*AppServiceStore).removeAppFromStore"}
{"timestamp":1400858570.937188148,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Got 134 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.937973976,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.938408852,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.938893080,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.940565109,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 489 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858570.940954924,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.941341877,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.941300631,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858570.941459656,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 382 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858570.941595554,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.941745043,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858570.941901207,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Got 382 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.942076921,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858570.942211628,"process_id":9089,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38226: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858571.140211105,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 254 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858571.140768766,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858571.142080069,"process_id":9089,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858571.142142773,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858571.142271280,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 147 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858571.142343521,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: New services for 09576ca2-e004-4496-add7-8fe44894f4fa","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":27,"method":"loggregator/store.(*AppServiceStore).Run"}
{"timestamp":1400858571.142393589,"process_id":9089,"source":"loggregator","log_level":"debug","message":"AppStore: removing app 09576ca2-e004-4496-add7-8fe44894f4fa","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":89,"method":"loggregator/store.(*AppServiceStore).removeAppFromStore"}
{"timestamp":1400858571.142441988,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [09576ca2-e004-4496-add7-8fe44894f4fa].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858571.142524242,"process_id":9089,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}

[trastle]

Logs when failing

{"timestamp":1400858361.019083023,"process_id":9009,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 254 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858361.019723177,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.020492315,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.020567179,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.021011829,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 147 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.021539688,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [09576ca2-e004-4496-add7-8fe44894f4fa].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.021694660,"process_id":9009,"source":"loggregator","log_level":"debug","message":"AppStore: New services for 09576ca2-e004-4496-add7-8fe44894f4fa","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":27,"method":"loggregator/store.(*AppServiceStore).Run"}
{"timestamp":1400858361.022036552,"process_id":9009,"source":"loggregator","log_level":"debug","message":"AppStore: removing app 09576ca2-e004-4496-add7-8fe44894f4fa","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":89,"method":"loggregator/store.(*AppServiceStore).removeAppFromStore"}
{"timestamp":1400858361.022029161,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.589298010,"process_id":9009,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 1587 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858361.589900494,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.590602398,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.590539932,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.590822935,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 1480 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.590970278,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.591131687,"process_id":9009,"source":"loggregator","log_level":"info","message":"SinkManager: Sink with identifier e62e36d5-a2ab-481d-bfc3-3994b8066770 requested. Opened it.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/sinkmanager/sink_manager.go","line":119,"method":"loggregator/sinkserver/sinkmanager.(*SinkManager).RegisterSink"}
{"timestamp":1400858361.591049194,"process_id":9009,"source":"loggregator","log_level":"debug","message":"AppStore: New services for e62e36d5-a2ab-481d-bfc3-3994b8066770","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/store/app_service_store.go","line":27,"method":"loggregator/store.(*AppServiceStore).Run"}
{"timestamp":1400858361.592039824,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.592491627,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Got 1480 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.593725920,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.593818903,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.595201969,"process_id":9009,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 340 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858361.595342636,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.595453501,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.595478535,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.595655680,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 233 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.595777988,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.595899582,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.595940113,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Got 233 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.596065521,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.596193075,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.597111464,"process_id":9009,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 316 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858361.597231388,"process_id":9009,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 312 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858361.597314835,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.597436666,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.597507000,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.597457170,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.597620249,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 209 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.597749949,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.597854137,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.597914219,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.597963095,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 205 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.597917795,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.597972393,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Got 209 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.598006725,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.598092079,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.598156452,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.598325491,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.598374367,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Got 205 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.598619461,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.598673344,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.598676682,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Created for appId [e62e36d5-a2ab-481d-bfc3-3994b8066770]","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":37,"method":"loggregator/sinks/syslog.NewSyslogSink"}
{"timestamp":1400858361.598815918,"process_id":9009,"source":"loggregator","log_level":"info","message":"SinkManager: Sink with identifier syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514 requested. Opened it.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/sinkmanager/sink_manager.go","line":119,"method":"loggregator/sinkserver/sinkmanager.(*SinkManager).RegisterSink"}
{"timestamp":1400858361.598949432,"process_id":9009,"source":"loggregator","log_level":"info","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Running.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":43,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.599068642,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Starting loop. Current backoff: 0","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":53,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.599165678,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Not connected. Trying to connect.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":62,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.604358673,"process_id":9009,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 321 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858361.604526997,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.604651690,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.604670525,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.604840994,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 214 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.604948521,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.605055809,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.605088234,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Got 214 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.605240583,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.605408907,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.606214046,"process_id":9009,"source":"loggregator","log_level":"info","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: successfully connected.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":77,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.606302977,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Waiting for activity\n","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":84,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.606470823,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Got 214 bytes. Sending data\n","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":90,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.606637716,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Successfully sent data\n","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":106,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.606848240,"process_id":9009,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 303 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858361.606848955,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Starting loop. Current backoff: 0","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":53,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.606918812,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.606997252,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Waiting for activity\n","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":84,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.607065916,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.607043505,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.607112885,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 196 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.607314110,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.607462406,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Got 196 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.607545137,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.607471704,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Got 196 bytes. Sending data\n","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":90,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.607598066,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.607650042,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Successfully sent data\n","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":106,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.607784748,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Starting loop. Current backoff: 0","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":53,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.607902050,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Waiting for activity\n","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":84,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.607443094,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.614590168,"process_id":9009,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 489 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858361.614717484,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.614980936,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858361.615183115,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.615357161,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 382 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858361.615501881,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [e62e36d5-a2ab-481d-bfc3-3994b8066770].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.615614891,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858361.615762949,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Got 382 bytes. Sending data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":56,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.615897179,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Successfully sent data","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":63,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.615956306,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Websocket Sink 192.168.80.54:38216: Waiting for activity","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/websocket_sink.go","line":50,"method":"loggregator/sinks.(*WebsocketSink).Run"}
{"timestamp":1400858361.616009235,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Got 382 bytes. Sending data\n","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":90,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.616107464,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Successfully sent data\n","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":106,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.616164923,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Starting loop. Current backoff: 0","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":53,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858361.616225719,"process_id":9009,"source":"loggregator","log_level":"debug","message":"Syslog Sink syslog://logstash-logger-all.log.dev.col.tx.cpgpaas.net:1514: Waiting for activity\n","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinks/syslog/syslog_sink.go","line":84,"method":"loggregator/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":1400858366.021656036,"process_id":9009,"source":"loggregator","log_level":"debug","message":"AgentListener: Read 254 bytes from address 192.168.80.54:40687","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/github.com/cloudfoundry/loggregatorlib/agentlistener/agent_listener.go","line":50,"method":"github.com/cloudfoundry/loggregatorlib/agentlistener.(*agentListener).Start"}
{"timestamp":1400858366.022536516,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Parsing a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":26,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858366.023088217,"process_id":9009,"source":"loggregator","log_level":"debug","message":"LogMessageUnmarshaller: Sent a message","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/unmarshaller/log_message_unmarshaller.go","line":24,"method":"loggregator/sinkserver/unmarshaller.(*LogMessageUnmarshaller).Start"}
{"timestamp":1400858366.023135662,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:MessageReceived","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":43,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858366.023474216,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Received 147 bytes of data from agent listener.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":48,"method":"loggregator/sinkserver.(*MessageRouter).Start"}
{"timestamp":1400858366.023764849,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Searching for sinks with appId [09576ca2-e004-4496-add7-8fe44894f4fa].","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":81,"method":"loggregator/sinkserver.(*MessageRouter).send"}
{"timestamp":1400858366.024102926,"process_id":9009,"source":"loggregator","log_level":"debug","message":"MessageRouter:outgoingLogChan: Done sending message.","data":null,"file":"/var/vcap/data/compile/loggregator/loggregator/src/loggregator/sinkserver/message_router.go","line":83,"method":"loggregator/sinkserver.(*MessageRouter).send"}

[d]

Thank you for reporting this issue, @trastle !

@trastle it seems that

• your "working" log doesn't mention your syslog drain at all.

• your "failing" log indicates that it has successfully sent 4 messages to your fake syslog

  Are you sure you pasted the right log?

At the same time, @lcddave and @jtuchscherer Can you guys take a look if this looks familiar?

Thanks,
CF Community Pair (@d, @stupakov)

[lcddave]

This is the first time I've seen anything like this. Thank you for the detailed report. We'll try to reproduce the situation, but we might need some more info later.

[trastle]

@d that's correct.

The "working fine" log has no drain bound the the app. It's just the loggregator doing its thing and working perfectly. The logs all get through to my terminal running cf logs.

The failing log should show a few messages coming through. After these initial few, no more messages are received at either the fake drain or the terminal. The monit process watching loggregator shows that loggregator is still running but the traffic controller indicates that it has died, or at least disconnected.

@lcddave sure no worries, let me know if you need any help to reproduce. Additionally if you cannot reproduce I would be interested to understand why this is occurring in my setup.

[trastle]

Right. I have tested this bug against another CF-170 install in another OpenStack tenant today and I am not seeing the same issue. Currently trying to work out the difference between the two.

[maxbrunsfeld]

FWIW, we followed your steps but were not able to reproduce this bug on CF-171. We even killed the netcat listener and brought it back up again later, and the logs which were emitted during the downtime eventually came through.

@trastle, if you are not able to reproduce the bug on your other environment, are you ok with closing this issue?

Thanks,
CF Community Pair (@d, @maxbrunsfeld)

[jtuchscherer]

@trastle The traffic controller crashing explains why you do not see any log messages in your syslog drain (because logs from the gorouter and the DEA are routed through the traffic controller). It would be interesting to see why the traffic controller was crashing. Do you have the logs or any diagnostics from that machine?

Thanks,
@ajackson and @jtuchscherer (CF Logging team)

[trastle]

@maxbrunsfeld @d thanks for following up. I did exactly the same today in another CF installation and did not see the issue. Hence the comment earlier.

@jtuchscherer and @ajackson happy to collect some additional logs if you believe they may be useful / interesting. I need to debug this further in my failing environment anyway so happy to provide any logs you believe may be useful.

[lcddave]

At this point, this does not appear to be a bug. Will close for now until there is new information.