NoMethodError: undefined method `tv_sec' in outputs/elasticsearch about logstash

[colinsurprenant]

@thuck are you by any chance using a dual logstash setup, with shipper & indexer? could you please verify versions for both? This might be related to https://logstash.jira.com/browse/LOGSTASH-2092, see my last comment.

[thuck]

Yes, in fact I'm using a shipper and indexer, with redis.
When I found the issue I was using the version 1.4.0, now in the indexer I'm using the 1.4.0 nightly build from yesterday. I'm going to update it in all the machines, and let you know.

[thuck]

@colinsurprenant I updated everything for the last build (logstash-1.4.1.dev), and so far there is no errors.
Do you plan to create a fix for this? Or will update the logstash to the same version be the official fix? Just to let this clear if someone hits the same issue and need to understand how to proceed.

[electrical]

@thuck the fix is in the Master branch of LS and will be in the next release :-)
We hope to do the release ( 1.4.1 ) soon.

Cheers.

[colinsurprenant]

@thuck good stuff! as @electrical said, fix is in! 👏

Thanks for your feedback!

Closing this, will open a new one, for post 1.4.1 to evaluate if better robustness should be added to avoid crashes on invalid data.

[onnimonni]

Exactly same error happened for me (version: 1.4.2-modified). I was importing bigger log patch with shipper from other node into redis que and I guess that some faulty value was saved in redis. And the main logstash/elasticserver crashed.

deleting the value from redis: 'DEL logstash' cleared my problem.

EDIT: deleting value from redis seemed to fix it but it didn't. Here's the complete error message again.

NoMethodError: undefined method `tv_sec' for nil:NilClass
        sprintf at /opt/logstash/lib/logstash/event.rb:230
           gsub at org/jruby/RubyString.java:3041
        sprintf at /opt/logstash/lib/logstash/event.rb:216
        receive at /opt/logstash/lib/logstash/outputs/elasticsearch.rb:308
         handle at /opt/logstash/lib/logstash/outputs/base.rb:86
     initialize at (eval):307
           call at org/jruby/RubyProc.java:271
         output at /opt/logstash/lib/logstash/pipeline.rb:266
   outputworker at /opt/logstash/lib/logstash/pipeline.rb:225
  start_outputs at /opt/logstash/lib/logstash/pipeline.rb:152

This is my elasticsearch info:

{
  "status" : 200,
  "name" : "Dr. Lemuel Dorcas",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.4.1",
    "build_hash" : "89d3241d670db65f994242c8e8383b169779e2d4",
    "build_timestamp" : "2014-11-26T15:49:29Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.2"
  },
  "tagline" : "You Know, for Search"
}

And redis.
127.0.0.1:6379> info

# Server
redis_version:2.8.18
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:4b21eb49f72458c8
redis_mode:standalone
os:Linux 3.13.0-40-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.2
process_id:13446
run_id:f63ab19cf749f72cf57d9d53a6e5d2fc0029354c
tcp_port:6379
uptime_in_seconds:316
uptime_in_days:0
hz:10
lru_clock:9010809
config_file:/etc/redis/redis.conf

...

[colinsurprenant]

@onnimonni this is not related to Redis or Elasticsearch, it is a caused by your shipper logstash node sending invalid json events with a missing @timestamp field. Can you verify your Logstash shipper version?

[onnimonni]

Hey @colinsurprenant! Shipper is also logstash 1.4.2-modified

I had problems with log stash not using the timestamps from the log file. Logstash was just using current time which was wrong for me. So Im sending the lines 'as is' to main server where I'm parsing them. So I delete '@timestamp' and rereading it in the main logstash server:

/etc/logstash/50-filter-nginx.conf:

filter {
  if [type] == "nginx-access-extensive" {
    mutate {
      remove_field => [ "@timestamp" ]
    }
    grok {
      match => ["message","%{HOSTNAME:hostname} %{IP:http_client} - - \[%{HTTPDATE:time_local}\] \"%{WORD:request_method} %{URIPATHPARAM:request_path:not_analyzed} HTTP/%{NUMBER:http_version}\" %{INT:http_status:int} %{INT:body_bytes_sent:int} \"(?:%{URI:http_referer:not_analyzed}|-)\" %{QS:http_user_agent:not_analyzed} (?:%{WORD:cache_status}|-) %{NUMBER:request_time:float}"]
      tag_on_failure => []
    }
    date {
      match => [ "time_local" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    geoip {
      source => "http_client"
    }
  }
}

I removed:

    mutate {
      remove_field => [ "@timestamp" ]
    }

And it now it works. I needed this to correct my timestamps. Well this is different issue but do you know why date-block isn't overwriting timestamp?

[colinsurprenant]

@onnimonni ah, yes, I see, so the problem you are having is that by force removing the @timestamp field, if your date filter does not match you end up without any @timestamp field and this will result in the exception you are having at the output.

So, your date filter will not overwrite @timestamp if it does not match the pattern. You should check the logstash log for warnings on date parsing errors. Note that you can specify multiple patterns in the match option.

[onnimonni]

of course! Thank you so much!

[hopperd]

It appears that this same issue happens if @timestamp happens to be an array:

NoMethodError: undefined method `tv_sec' for ["2015-04-20T13:57:06.972Z", "2015-04-20T13:57:11.980Z"]:Array
        sprintf at /opt/logstash/lib/logstash/event.rb:230
           gsub at org/jruby/RubyString.java:3041
        sprintf at /opt/logstash/lib/logstash/event.rb:216
        receive at /opt/logstash/lib/logstash/outputs/elasticsearch.rb:308
         handle at /opt/logstash/lib/logstash/outputs/base.rb:86
     initialize at (eval):62
           call at org/jruby/RubyProc.java:271
         output at /opt/logstash/lib/logstash/pipeline.rb:266
   outputworker at /opt/logstash/lib/logstash/pipeline.rb:225
  start_outputs at /opt/logstash/lib/logstash/pipeline.rb:152