freeformz / logglier Goto Github PK

Would be nice if other errors (such as EOFError, Errno::ECONNRESET) that might be encountered while pushing messages via HTTP were caught with a warning printed similar to how timeouts are handled.

Now that Splunk Storm has a API for data input, it should be trivial to add support to Logglier.

Documentation for the Splunk Storm API is available at:

• http://docs.splunk.com/Documentation/Storm/latest/User/UseStormsRESTAPI
• http://docs.splunk.com/Documentation/Storm/latest/User/Stormdatainputendpoint

Ruby examples of using the Splunk Storm API are available at: https://github.com/splunk/storm-examples

It would be helpful to document how one would set the logglier log level to info for production deployments.

TODO: Alternative https implementations (Typheous, Excon, etc). May be faster?

We've done a lot of performance tests of different http clients for ruby and that rest gem makes them all behave the same way so easy to change what's being used under the covers. By default, chooses the fastest one we've found.

https://github.com/iron-io/rest

Might want to give it a try.

We're about to re-release the Storm API, so I'm picking up this issue again (see #16):

Now that Splunk Storm has a API for data input, it should be trivial to add support to Logglier.

Documentation for the Splunk Storm API is available at:

• http://docs.splunk.com/Documentation/Storm/latest/User/UseStormsRESTAPI
• http://docs.splunk.com/Documentation/Storm/latest/User/Stormdatainputendpoint

Ruby examples of using the Splunk Storm API are available at: https://github.com/splunk/storm-examples

While testing other network/connection issue handling in fog, I set up iptables to reject all traffic to port 443. This made logglier raise Errno::ECONNREFUSED when loading my app since it of course couldn't open the https connection to loggly.

Would be nice if that were handled a bit better. In async/threaded mode, perhaps it could retry with a backoff. In sync mode maybe it could try connecting every time with a very short open timeout. Either way also attempting to queue events for delivery once the connection was established would be nice.

I cannot manage to make lograge work with logglier, is that even possible?

I'm curious if anyone has found an easy workaround for this or if you would consider a PR that allows something like this.

We just signed for loggy and start to use.

We are getting the following error.

WARNING: [1/3] caught EOFError: end of file reached while attempting to deliver: 2013-03-18 04:34:58 -0500 severity=INFO, logger test

Hi!

I'm just trying out the combination of Loggly with logglier and lograge to get all my logs into Loggly. (Say that very fast three times in a row.)

I've been facing problems because I would receive the forms date and severity unparsed by Loggly, like so:

Introspecting into the code, I found that this is because logglier's logger is not appending these as JSON but rather as a string. However, I think it shouldn't do so, because my configuration specifies :json formatter:

config.logger = Logglier.new('https://logs-01.loggly.com/inputs/[redacted]/tag/rails', threaded: true, format: :json)
config.lograge.enabled = true
config.lograge.formatter = Lograge::Formatters::Json.new

(Taken from Rails Logging, Loggly documentation.)

Diving into the code, I found that logglier will only merge these values as a hash if its provided a hash already (here), while Lograge's JSON formatter will deliver a string rather than a hash (here).

As a workaround, this is my current configuration which provides everything I need to:

config.logger = Logglier.new('https://logs-01.loggly.com/inputs/[redacted]/tag/rails', threaded: true, format: :json)
config.lograge.enabled = true
config.lograge.formatter = ->(data) { data }
# alternatively: config.lograge.formatter = Lograge::Formatters::Raw.new 

However, I believe that logglier should still deliver a JSON output when the :json formatter is set, even if it does not receive a Hash.

I may look into performing these modifications and sending a PR, but I'll need some time to get accustomed to the current set of tests. Also, I'd like to ask if this is the desired direction, or if it's preferable to ask Loggly to adjust its documentation and suggest the Raw formatter.

Thanks, and great work!

Hi i have tried this gem
Am able to see log in logglier.But same log not saving into development.log/production.log as like before.
Can we keep log on both logglier and production.log

Loggly seems to require Content-Type: application/json to have json handled properly even if an input has the json box checked.

I'm working on a gem to collect a hash throughout a request cycle and push it using logglier. Ran into a nasty bug which first I thought was my fault, then I thought it was yours, then I realized it's a core bug in Loggly's JSON API. Namely, if you nest a hash inside an array it comes out in their console as just [object, Object].

Anyway, pulling your code down made me realize 0.2.6 ain't on master.

When changing the following line in development.rb in Rails 4, there's an error in activesupport/lib/active_support/core_ext/kernel/reporting.rb:82 (wrong number of arguments 0 for 1).

config.logger = Logglier.new("https://logs.loggly.com/inputs/API_KEY")

Assuming this is related to the changes in the Rails 4 logger. Any plans to fix?

Thanks!
Kapil

If you have a small project, and you add logglier to your gemfile, you get

cannot load such file -- multi_json (LoadError)

This is because multi_json is listed as a development dependency in 189059a, not as a runtime dependency.

How to replicate:

• Create a fresh Rails project (4.2.1)
• Delete all gem requirements apart from logglier, rails, and sqlite3
• Run bundle rails c

If I try to run my tests without internet connection, I always get "rake aborted! getaddrinfo: Name or service not known".

How could I handle my case? I want to "switch on/off the logglier gem, if I have no connection to the internet.

Anyone an idea?

I've started implementing this gem to log some custom events for a Rails 3 app, and it's working nicely. But I'm having difficulty stubbing out the actual HTTP request it's making when running specs. I imagined I could do something like this...

allow(Logglier::Client::HTTP::NetHTTPProxy).to receive(:deliver).and_return(true)

... but it's still hitting the web and making the request to the Loggly service. Any advice on preventing this for tests?

We have the last version of gem logglier (0.4.1) and with Ruby 2.2.3 it works correctly.
Now we have upgrade Ruby at version 2.3.0 and we get the warning:

gems/logglier-0.4.1/lib/logglier/client/http/sync.rb:11
warning: constant Logglier::Client::HTTP::NetHTTPProxy::TimeoutError is deprecated

Can we fix this warning?

Thankyou
Team Nautilus

For the HTTP input anyway ... anything other than a 200 should probably be logged.

Does this gem works in an asynchrony way?

We've had some success with Logglier and Logstasher, where we've been able to take a logstash formatted representation of our production.log and push them to Loggly by extending our LogStasher.logger with Logglier. However, when we add a second logger (in our case an event log which is currently only written to by an ActiveSupport::Notifications subscriber (which is fed from DelayedJob::Worker hooks)) we find that the performance on our Delayed Job slows significantly. Given our Delayed Job queue has far less jobs than our production.log has external hits this is unexpected. Are there any immediate problems you can see when it comes to using Logglier with more than one logger, or with ActiveSupport::Notfications?

Here's how we hook it all up currently.

Core.event_logger = Logger.new("#{Rails.root}/log/event.log")
# http://www.rubydoc.info/docs/rails/4.1.7/ActiveSupport/Logger.broadcast
def Core.broadcast(logger) # :nodoc:
  Module.new do
    define_method(:add) do |*args, &block|
      logger.add(*args, &block)
      super(*args, &block)
    end

    define_method(:<<) do |x|
      logger << x
      super(x)
    end

    define_method(:close) do
      logger.close
      super()
    end

    define_method(:progname=) do |name|
      logger.progname = name
      super(name)
    end

    define_method(:formatter=) do |formatter|
      logger.formatter = formatter
      super(formatter)
    end

    define_method(:level=) do |level|
      logger.level = level
      super(level)
    end
  end
end

{
  event: Core.event_logger,
  request: LogStasher.logger
}.each do |tag, logger|
  url = "https://logs-01.loggly.com/inputs/#{token}/tag/#{tag}"
  loggly = Logglier.new(url, threaded: true, format: :json)
  logger.extend Core.broadcast(loggly)
end

Would be useful. Maybe it really should be something separate though.

It seems pretty obvious, but if you have 4 processes on a host, it's hard to tell which one is generating which log entry. Usually you'd log pid with this, but it doesn't seem to be in the template for what is sent to loggly.

I'm trying having hostname be "#{hostname}-#{$$}" for now, but still it seems to make sense to include pid...

I'm just setting up Loggly and one major question that I can't find the answer to on my account's Ruby setup page (https://myaccount.loggly.com/sources/setup/ruby), the Loggly docs or the logglier README is how to keep logging to the default Rails log file AND log to Loggly on top of that.
It seems like this should do it but I'm not sure.

Came out yesterday, logglier doesn't install right with it, no lib/ directory is unpacked.

Run gem update --system, uninstall logglier, then reinstall it.

action@ruby-box-7827:~/server_images$ gem install logglier                                                                                            
Fetching: logglier-0.2.8.gem (100%)                                                                                                                   
Successfully installed logglier-0.2.8                                                                                                                 
1 gem installed                                                                                                                                       
action@ruby-box-7827:~/server_images$ irb                                                                                                             
2.0.0-p195 :001 > require 'logglier'                                                                                                                  
LoadError: cannot load such file -- logglier                                                                                                          
        from /home/action/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:51:in `require'                    
        from /home/action/.rvm/rubies/ruby-2.0.0-p195/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:51:in `require'                    
        from (irb):1                                                                                                                                  
        from /home/action/.rvm/rubies/ruby-2.0.0-p195/bin/irb:13:in `<main>'                 

Hi,

We are noticing that the timeout and retry mechanisms of the Logglier gem behave a bit mysteriously. We normally use the Logglier gem to log to Loggly in resque jobs in our Rails 3 app, but have isolated and reproduced the strange behavior in rails console. We've observed that when we use Logglier's default timeouts of 5 sec (which seems plenty long), we get timeout warnings from Logglier and what appears to be retries. Each log sometimes appears correctly and singly, and sometimes appears incorrectly and multiple times (maybe as result of timeouts and retries?) per Loggly's web interface. The multiple loggings concern us and make the logs unreliable as Logglier's retries appear not idempotent.

Interestingly, when we tried overriding logglier's default timeout to 10 sec, we observed the same behavior of the multiple copies for single logs, except it did take longer to timeout to timeout.

If it helps to see how we tested and observed this behavior, the spews below are copied from our rails console. We then compare these logging activities to what we see reported on Loggly's web interface.
"testing timeouts: no override 1st time" appeared 3 times per Loggly web interface.
"testing timeouts: no override 2nd time" appeared correctly once.
"testing timeouts: 10 sec timout override, 1st time" appeared correctly once.
"testing timeouts: 10 sec timout override, 2nd time" appeared twice.
"testing timeouts: 10 sec timout override, 3rd time" appeared 4 times.

######################## from rails console

1.9.3p385 :011 > loggly = Logglier.new(ENV['LOGGLY_MIXPANEL_URL'])
=> #<Logger:0x007f842eecb058 @progname=nil, @Level=0, @default_formatter=#<Logger::Formatter:0x007f842eecb008 @datetime_format=nil>, @Formatter=#Proc:0x007f842eecabf8@/Users/mwong/.rvm/gems/ruby-1.9.3-p385/gems/logglier-0.2.8/lib/logglier/client.rb:61, @logdev=#<Logger::LogDevice:0x007f842eecaf68 @shift_size=nil, @shift_age=nil, @filename=nil, @dev=#<Logglier::Client::HTTP:0x007f842ef9bf00 @input_uri=#<URI::HTTPS:0x007f842ef9b320 URL:https://logs.loggly.com/inputs/1ac73495-8ede-4366-b182-ce049ba6ad71>, @Format=nil, @Deliverer=#<Logglier::Client::HTTP::NetHTTPProxy:0x007f842ef9afb0 @input_uri=#<URI::HTTPS:0x007f842ef9b320 URL:https://logs.loggly.com/inputs/1ac73495-8ede-4366-b182-ce049ba6ad71>, @verify_mode=0, @ca_file=nil, @read_timeout=5, @open_timeout=5, @Failsafe=#IO:, @Format=nil, @headers={"Content-Type"=>"text/plain"}, @http=#<Net::HTTP logs.loggly.com:443 open=true>>>, @mutex=#<Logger::LogDevice::LogDeviceMutex:0x007f842eecaf18 @mon_owner=nil, @mon_count=0, @mon_mutex=#Mutex:0x007f842eecaea0>>>
1.9.3p385 :012 > loggly.info(['testing timeouts: no override 1st time'])
WARNING: [1/3] caught EOFError: end of file reached while attempting to deliver: 2013-07-22 16:18:31 -0700 severity=INFO, ["testing timeouts: no override 1st time"]
WARNING: [2/3] caught Timeout::Error: Timeout::Error while attempting to deliver: 2013-07-22 16:18:31 -0700 severity=INFO, ["testing timeouts: no override 1st time"]
WARNING: [3/3] caught Timeout::Error: Timeout::Error while attempting to deliver: 2013-07-22 16:18:31 -0700 severity=INFO, ["testing timeouts: no override 1st time"]
ERROR: caught Timeout::Error: Timeout::Error while attempting to deliver: 2013-07-22 16:18:31 -0700 severity=INFO, ["testing timeouts: no override 1st time"]
=> true
1.9.3p385 :016 > loggly.info(['testing timeouts: no override 2nd time'])
=> true

1.9.3p385 :023 > loggly_10 = Logglier.new(ENV['LOGGLY_MIXPANEL_URL'], :read_timeout => 10, :open_timeout => 10)
=> #<Logger:0x007f842be3bed8 @progname=nil, @Level=0, @default_formatter=#<Logger::Formatter:0x007f842be3bdc0 @datetime_format=nil>, @Formatter=#Proc:0x007f842be3b6e0@/Users/mwong/.rvm/gems/ruby-1.9.3-p385/gems/logglier-0.2.8/lib/logglier/client.rb:61, @logdev=#<Logger::LogDevice:0x007f842be3bac8 @shift_size=nil, @shift_age=nil, @filename=nil, @dev=#<Logglier::Client::HTTP:0x007f842be3d8f0 @input_uri=#<URI::HTTPS:0x007f842be3d530 URL:https://logs.loggly.com/inputs/1ac73495-8ede-4366-b182-ce049ba6ad71>, @Format=nil, @Deliverer=#<Logglier::Client::HTTP::NetHTTPProxy:0x007f842be3d378 @input_uri=#<URI::HTTPS:0x007f842be3d530 URL:https://logs.loggly.com/inputs/1ac73495-8ede-4366-b182-ce049ba6ad71>, @verify_mode=0, @ca_file=nil, @read_timeout=10, @open_timeout=10, @Failsafe=#IO:, @Format=nil, @headers={"Content-Type"=>"text/plain"}, @http=#<Net::HTTP logs.loggly.com:443 open=true>>>, @mutex=#<Logger::LogDevice::LogDeviceMutex:0x007f842be3b988 @mon_owner=nil, @mon_count=0, @mon_mutex=#Mutex:0x007f842be3b898>>>
1.9.3p385 :024 > loggly_10.info(['testing timeouts: 10 sec timout override, 1st time'])
WARNING: [1/3] caught EOFError: end of file reached while attempting to deliver: 2013-07-22 16:23:17 -0700 severity=INFO, ["testing timeouts: 10 sec timout override, 1st time"]
=> true
1.9.3p385 :025 > loggly_10.info(['testing timeouts: 10 sec timout override, 2nd time'])
WARNING: [1/3] caught EOFError: end of file reached while attempting to deliver: 2013-07-22 16:24:44 -0700 severity=INFO, ["testing timeouts: 10 sec timout override, 2nd time"]
WARNING: [2/3] caught Timeout::Error: Timeout::Error while attempting to deliver: 2013-07-22 16:24:44 -0700 severity=INFO, ["testing timeouts: 10 sec timout override, 2nd time"]
WARNING: [3/3] caught Timeout::Error: Timeout::Error while attempting to deliver: 2013-07-22 16:24:44 -0700 severity=INFO, ["testing timeouts: 10 sec timout override, 2nd time"]
ERROR: caught Timeout::Error: Timeout::Error while attempting to deliver: 2013-07-22 16:24:44 -0700 severity=INFO, ["testing timeouts: 10 sec timout override, 2nd time"]
=> true
1.9.3p385 :026 > loggly_10.info(['testing timeouts: 10 sec timout override, 3rd time'])
WARNING: [1/3] caught Timeout::Error: Timeout::Error while attempting to deliver: 2013-07-22 16:32:24 -0700 severity=INFO, ["testing timeouts: 10 sec timout override, 3rd time"]
WARNING: [2/3] caught Timeout::Error: Timeout::Error while attempting to deliver: 2013-07-22 16:32:24 -0700 severity=INFO, ["testing timeouts: 10 sec timout override, 3rd time"]
WARNING: [3/3] caught Timeout::Error: Timeout::Error while attempting to deliver: 2013-07-22 16:32:24 -0700 severity=INFO, ["testing timeouts: 10 sec timout override, 3rd time"]
=> true

Noticed messages logged near when resque workers would be exiting weren't making to loggly. Added sleep 5 to what would be the end of the main thread and now I'm seeing the messages.

I'll try to dig into it more.

We would like to add instructions for shipping logs from Ruby to our log aggregation service.
I tried the lib with a basic example and worked without any problems.

Would it be OK to add Logsene as a supported service for Logglier and add a basic example?
Thanks!