logstash-plugins / logstash-filter-http Goto Github PK

View Code? Open in Web Editor NEW

This project forked from lucashenning/logstash-filter-rest

12.0 7.0 29.0 133 KB

HTTP Filter Plugin for Logstash

License: Apache License 2.0

Ruby 100.00%

logstash-filter-http's Issues

Need clarification on sending data as json

Tell us about the issue

Description:
User has confusion about how to send data in json format as the document misses to point out the requirement of body and body_format.

body_format => "json" works only when body is an array or a hash

URL:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-http.html#plugins-filters-http-body
https://www.elastic.co/guide/en/logstash/current/plugins-filters-http.html#plugins-filters-http-body_format

Anything else?

Unhelpful error message when body is hash and body_format is text

In 7.3.2, if the body option is a hash and the body_format is set to text then the resulting error message is not particularly helpful.

http { body => { "foo" => "bar" } body_format => "text" url => "http://google.com/" verb=> "GET" }

results in

[ERROR][logstash.filters.http    ] error during HTTP request {:url=>"http://google.com/", :body=>{"foo"=>"bar"}, :client_error=>"undefined method `encoding' for {\"foo\"=>\"bar\"}:Hash"}

HTTP filter doesn't have ssl_enabled setting

It should have support to the ssl_enabled option, enabling and disabling the use of other SSL configurations. So the plugin complies with the Logstash SSL Standardisation.

remove core classes monkeypatching

Authentication username and password are not interpolated

Logstash 7.6.2
Plugin version logstash-filter-http (1.0.2)

Summary

The http filter is not interpolating the variables for the parameters username / password.

How to reproduce

input {
    exec {
        command => 'echo "myusername mypassword something else as message"'
        interval => 30
    }
}
filter {
    grok {
        match => {
            "message" => "(?<username>.*?) (?<password>.*?) %{GREEDYDATA:message}"
        }
        overwrite => [ "message" ]
    }
    http {
        url => "https://postman-echo.com/post"
        verb => "POST"
        body_format => text
        body => "%{[message]}"
        headers => {
            "mytest" => "logstash"
        }
        user => "%{[username]}"
        password => "%{[password]}"
    }
}
output {
    stdout {
        codec => rubydebug
    }
}

Expected

The header should contain Basic bXl1c2VybmFtZTpteXBhc3N3b3Jk

Output

The header contains Basic JXtbdXNlcm5hbWVdfTole1twYXNzd29yZF19 (represents %{[username]}:%{[password]}).

Workaround

    ruby {
        init => "require 'base64'"
        code => "event.set('auth_header', 'Basic ' + Base64.strict_encode64(event.get('username')+':'+event.get('password')))"
    }
    http {
        url => "https://postman-echo.com/post"
        verb => "POST"
        body_format => text
        body => "%{[message]}"
        headers => {
            "authorization" => "%{[auth_header]}"
            "mytest" => "logstash"
        }
        remove_field => [ "auth_header"]
    }

From https://discuss.elastic.co/t/http-filter-plugin-authorization-as-parameter-values/232085/6

Unable to parse Json with one field or none + needed clarity on target_body parameter

Logstash 7.4

When using the http filter with Zammad API https://docs.zammad.org/en/latest/api-tags.html#list I keep getting a "_jsonparsefailure" even with a valid json in response.

If I try using the input plugin "http_poller" I get no failures.

Add ability to disable SSL certificate validation

Currently if remote certificate is not trusted eg self signed cert the http request would fail. Add something like ssl_verify_mode congratulation parameter.

Documentation Mismatch

The documentation for this plugin (which appears to be incorrect) does not match the plugin's README (which does appear to be correct).

Since (it it would seem) the documentation in this repo "builds" the documentation on the Elastic "guide page" for this plugin, the documentation there is also incorrect.

This looks to be an artifact of #3 (renaming the plugin)

tag_on_request_failure and tag_on_json_failure are not documented

The plugin appears to have a default tag_on_request_failure option of '_httprequestfailure', and a default tag_on_json_failure option as well. The documentation does not mention these.

Unable to add event field to http body

I would like to reference a value from an event field into the http body.
It doesn't seem to be possible, but it states it is possible with headers? Is it also possible with the http body?

Doesn't respect quoted characters

Logstash information:

Please include the following information:

Logstash version (e.g. bin/logstash --version) 8.5.1 but also tested with 7.17.8
Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
How was the Logstash Plugin installed

JVM (e.g. java -version):

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

JVM version (java -version)
JVM installation source (e.g. from the Operating System's package manager, from source, etc).
Value of the JAVA_HOME environment variable if set.

OS version (uname -a if on a Unix-like system): any

Description of the problem including expected versus actual behavior:
When an field of the event contains quoted characters, like \" the payload sent to the server should also contains the same quoting.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

Run an http mock server

wget https://repo1.maven.org/maven2/com/github/tomakehurst/wiremock-jre8-standalone/2.35.0/wiremock-jre8-standalone-2.35.0.jar

java -jar wiremock-jre8-standalone-2.35.0.jar --verbose

run the following pipeline

input {
generator {
  count => 1
  message => '{"Field1": "22222wwwww\"www\"wwwwww"}'
  codec => "json"
}
}

filter {
http {
  target_body => "response_body"
  verb => "POST"
  url => "http://localhost:8080"
  body => "%{Field1}"
}
}

output {
 stdout{
   codec => rubydebug
 }
}

check the WireMock console
It receives

22222wwwww"www"wwwwww

while we would have expected

22222wwwww\"www\"wwwwww

Provide logs (if relevant):

Apache http client wire logs

[2023-02-28T16:09:20,469][DEBUG][org.apache.http.headers  ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> POST / HTTP/1.1
[2023-02-28T16:09:20,469][DEBUG][org.apache.http.headers  ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> Connection: Keep-Alive
[2023-02-28T16:09:20,469][DEBUG][org.apache.http.headers  ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> content-type: text/plain
[2023-02-28T16:09:20,469][DEBUG][org.apache.http.headers  ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> Content-Length: 21
[2023-02-28T16:09:20,469][DEBUG][org.apache.http.headers  ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> Host: localhost:8080
[2023-02-28T16:09:20,469][DEBUG][org.apache.http.headers  ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> User-Agent: Manticore 0.9.1
[2023-02-28T16:09:20,469][DEBUG][org.apache.http.headers  ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> Accept-Encoding: gzip,deflate
[2023-02-28T16:09:20,469][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> "POST / HTTP/1.1[\r][\n]"
[2023-02-28T16:09:20,469][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
[2023-02-28T16:09:20,470][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> "content-type: text/plain[\r][\n]"
[2023-02-28T16:09:20,470][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> "Content-Length: 21[\r][\n]"
[2023-02-28T16:09:20,470][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> "Host: localhost:8080[\r][\n]"
[2023-02-28T16:09:20,470][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> "User-Agent: Manticore 0.9.1[\r][\n]"
[2023-02-28T16:09:20,470][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
[2023-02-28T16:09:20,470][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> "[\r][\n]"
[2023-02-28T16:09:20,470][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 >> "22222wwwww"www"wwwwww"
[2023-02-28T16:09:20,475][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
[2023-02-28T16:09:20,475][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 << "Content-Type: text/plain[\r][\n]"
[2023-02-28T16:09:20,476][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
[2023-02-28T16:09:20,476][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 << "[\r][\n]"
[2023-02-28T16:09:20,476][DEBUG][org.apache.http.wire     ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 << "54[\r][\n]"
[2023-02-28T16:09:20,477][DEBUG][org.apache.http.headers  ][main][2cc5221ef0bcc203d671c6ad3ff26a8083380f8f89498dbedddea98d891870d2] http-outgoing-0 << HTTP/1.1 404 Not Found

WireMock console

2023-02-28 16:09:20.472 Request received:
127.0.0.1 - POST /

Connection: [keep-alive]
Content-Type: [text/plain]
Content-Length: [21]
Host: [localhost:8080]
User-Agent: [Manticore 0.9.1]
Accept-Encoding: [gzip,deflate]
22222wwwww"www"wwwwww


Matched response definition:
(no response definition configured)

Response:
HTTP/1.1 404
(no headers)

Implement native caching for higher scale lookups

There have already been some demand for native caching for HTTP lookups with this plugin. This would help enable higher throughput without the need for usage with conjunction with third-party caching systems like Memcached.

Please feel free to +1 if you are interested in this feature.

Plugin crashes on empty response body

If a web service responds with HTTP/1.0 204 No Content and an empty response body the plugin crashes. This is due to the invocation of body.strip on this line.

[2020-10-15T09:39:51,324][DEBUG][logstash.filters.http    ][main][bccba3b61c91111734d8fbaddeef151ae2e9d3ff77dae9376279dd322eb0bac6] success received {:code=>204, :body=>nil}
[2020-10-15T09:39:51,352][ERROR][logstash.javapipeline    ][main] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"(NoMethodError) undefined method `strip' for nil:NilClass", :exception=>Java::OrgJrubyExceptions::NoMethodError, :backtrace=>["usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_http_minus_1_dot_0_dot_2.lib.logstash.filters.http.process_response(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-http-1.0.2/lib/logstash/filters/http.rb:116)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_http_minus_1_dot_0_dot_2.lib.logstash.filters.http.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-http-1.0.2/lib/logstash/filters/http.rb:76)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1809)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175)", "org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:293)"], :thread=>"#<Thread:0x30e761f3 sleep>"}
[2020-10-15T09:39:51,513][DEBUG][logstash.filters.http    ][main] Closing {:plugin=>"LogStash::Filters::Http"}
[2020-10-15T09:39:51,531][DEBUG][logstash.filters.http    ][main] Closing {:plugin=>"LogStash::Filters::Http"}
[2020-10-15T09:39:51,532][DEBUG][logstash.filters.http    ][main] Closing {:plugin=>"LogStash::Filters::Http"}
[2020-10-15T09:39:51,939][INFO ][logstash.runner          ] Logstash shut down.

Plugin crashes when it receives Array type of content-type header.

Logstash information:

Please include the following information:

Logstash version (e.g. bin/logstash --version)
Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
How was the Logstash Plugin installed

JVM (e.g. java -version):

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

JVM version (java -version)
JVM installation source (e.g. from the Operating System's package manager, from source, etc).
Value of the JAVA_HOME environment variable if set.

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:

Steps to reproduce:

Create and run the server which returns list of content-type headers, example nodejs source

var http = require("http");

http.createServer(function (request, response) {
   response.writeHead(200, {'Content-Type': ['text/plain', 'logstash/custom-type']} );
   response.end('Hello World\n');
}).listen(8081);

console.log('Server running at http://127.0.0.1:8081/');

Config Logstash with filter-http which calls and receives list of content-type headers

filter {
    http {
        url => "http://localhost:8081"
        target_body => "body"
        target_headers => "headers"
    }
}

Run Logstash and observe the error

Provide logs (if relevant):

[2023-03-15T11:36:25,372][ERROR][logstash.javapipeline    ][main] Pipeline worker error, the pipeline will be stopped 
{:pipeline_id=>"main", :error=>"(NoMethodError) undefined method `split' for [\"text/plain\", \"logstash/custom-type\"]:Array", 
:exception=>Java::OrgJrubyExceptions::NoMethodError, :backtrace=>
["RUBY.process_response(/Users/mashhur/Dev/elastic/logstash/logstash-plugins/logstash-filter-
http/lib/logstash/filters/http.rb:136)", "RUBY.filter(/Users/mashhur/Dev/elastic/logstash/logstash-plugins/logstash-filter-
http/lib/logstash/filters/http.rb:96)", "RUBY.do_filter(/Users/mashhur/Dev/elastic/logstash/logstash-
core/lib/logstash/filters/base.rb:159)", "RUBY.multi_filter(/Users/mashhur/Dev/elastic/logstash/logstash-
core/lib/logstash/filters/base.rb:178)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", 
"RUBY.multi_filter(/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/filters/base.rb:175)", 
"org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.ja
va:133)", "RUBY.start_workers(/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:304)"], 
:thread=>"#<Thread:0x7fc1e9a8@/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}

Handle Empty body in HTTP response

When I request an API with an empty body response, the filter crashes and the pipeline is interrupted

Here the error message :

| [2022-04-20T07:46:02,164][ERROR][logstash.javapipeline ] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"Test", :error=>"(NoMethodError) undefined method `strip' for nil:NilClass", :exception=>Java::OrgJrubyExceptions::NoMethodError, :backtrace=>["usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_http_minus_1_dot_0_dot_2.lib.logstash.filters.http.process_response(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-http-1.0.2/lib/logstash/filters/http.rb:116)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_http_minus_1_dot_0_dot_2.lib.logstash.filters.http.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-http-1.0.2/lib/logstash/filters/http.rb:76)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1809)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175)", "org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:293)"], :thread=>"#<Thread:0x610d43eb sleep>"}

To solve this issue, I have modified the code in lib\logstash\filters\http.rb and added theses lines :

if body.nil?
   body = ""
end

After the line 136 in http.rb (after "event.set(@target_headers, headers)" before "if content_type == "application/json"").

Maybe other users have the same issue, so it could be a good idea to add this code to the plugin, or any other method to manage this case

If you what to reproduce, the Kibana API to delete a case doesn't reponse with a body.
Doc:
https://www.elastic.co/guide/en/security/current/cases-api-delete-case.html

Add support to extract a value from the response.

One API I tested with is https://www.neutrinoapi.com/api/api-basics/
It returns the request parameters as part of the response along with the actual response in the result key of the JSON object returned.
I think there should be support for extracting this value into a field and discarding the rest.

Implement ECS-Compatibility Mode

As a part of the effort to make plugins able to run in an ECS-Compatible manner
by default in an upcoming release of Logstash, this plugin needs to implement
an ECS-Compatibility mode that does not implicitly use fields that conflict
with ECS.

Absent target_body and target_headers directives, this plugin uses root-level
fields body and headers respectively, which are undefined in ECS and
therefore have the possiblity of conflicting with future versions of ECS. The
plugin may benefit from an ECS-Compatibility mode that requires these directives
to be set explicitly or otherwise reduces the risk of future conflict.

Debug message causes Fatal error

I have a very simple pipeline using the following http filter, but I continue to get errors (see here: https://gist.github.com/defensivedepth/7b8d22aeec6dedac6438d8adeaed4b1d 1)

I have tried different sites, different settings, but always come back to this error. Logstash 6.6.1

Appears to be a bug - https://discuss.elastic.co/t/http-filter-undefined-local-variable-or-method-parameters-sprintfed/169905?

input {
file {
path => "/input*.json"
codec => json
}

}

filter {
http {
url => "https://site.com/%{identifier}"
}

}

output {
stdout {}
}

Cannot send Content-Type header

When assigning a header, for example: "Content-Type" => "application/x-www-form-urlencoded", filter sends header, but also sends an additional "content-type: text/plain".

This breaks the API call I am attempting to make, as the end-point complains about the dual headers.

This is because the source provides a binary choice between JSON or text as content type. There are hundred of other possibilities, and this restriction severely limits the ability to make HTTP calls.

An improvement would allow any string as a value for body_format, and can still default to text/plain. Will submit a PR.

rename plugin to logstash-filter-http

add documentation

Is it possible to do PUT requests from this plugin

Hiya,

Is it possible to do PUT requests from this plugin, From the documentation it doesn't look like it can however unless I'm mistaken the only thing stopping the plugin from doing it is PUT requests aren't in the VALID_VERBS list?

Thanks

Some APIs allow for batch processing, how can we provide batch support?

It might not be possible to generalise this.

Readers, if your API can process batch queries, please add a comment below showing details of how the batch is encoded into the request.

Friendlier handling of non 200 response codes

Currently anything that's not a 200 means the plugin won't process headers or the body.
In many cases this isn't ideal as these are still valuable for debugging / observability purposes.

A way to improve this would be to always process the body and headers, but only apply the filter_matched method on successful requests (200 status code)