Comments (7)
@shinsaka Just tested updated version, thanks a lot, it works great!
Despite all fields are now present, is it possible to allow preserving entire log line "as is"? Like this:
"@message": "http 2019-10-30T15:39:10.664331Z app/xxx-lb/xxxxxxx x.x.x.x:58463 172.62.5.25:80 0.000 0.901 0.000 500 500 239 212 \"GET http://app.domain.com:80/ HTTP/1.1\" \"PostmanRuntime/7.15.2\" - - arn:aws:elasticloadbalancing:us-west-2:xxxxxxxxxxxx:targetgroup/xxxxxxxxxxxx/xxxxxxxxxx \"Root=....\" \"-\" \"-\" 5 2019-10-30T15:39:09.763000Z \"forward\" \"-\" \"-\" \"172.x.x.x:80\" \"200\""
from fluent-plugin-elb-log.
Hi @shinsaka, thanks a lot for implementing my requests.
I've tested v1.3.1, but it fails to process alb logs from s3, no matter if include_all_message
option is set or not.
Here is an error message:
fluentd_logger_1 | 2019-11-08 16:12:59 +0000 [warn]: #0 error occurred: wrong number of arguments (given 2, expected 1)
from fluent-plugin-elb-log.
@NikitaGl Thank you for request.
I changed REGEX, and released version 1.2.0. 😃
from fluent-plugin-elb-log.
@NikitaGl
I added include_all_message
setting on config.
When set true then output all_message
.
Try version 1.3.1.
Thank you!
from fluent-plugin-elb-log.
@NikitaGl sorry, v1.3.1 does not work on ruby version 2.5.
Fixed to work on ruby version 2.5.
Try version v1.3.2 🙇
from fluent-plugin-elb-log.
Yup, seems to be working now.
I'm still not understand the reason for optionN
fields in output record:
"option1": null,
"option2": null,
"option3": "- \"Root=****\" \"x.x.x.x\" \"arn:aws:acm:*****:******:certificate/****-****-****-****-*****\" 0 2019-11-07T19:52:22.680000Z \"fixed-response\" \"-\" \"-\" \"-\" \"-\"",
As you see, sometimes it catches values, that corresponds to other keys. In my tests it always happens to logs of requests that did not passed through ALB and been answered with fixed response.
Here's and full example of such log. I made it with include_all_message
disabled, but it makes no difference
{
"_index": "xxxxxxxxxx-2019.11.11",
"_type": "_doc",
"_id": "xxxxxxxxxxxxxxxxxxxxxx",
"_version": 1,
"_score": null,
"_source": {
"account_id": "xxxxxxxxxxxxx",
"region": "xxx-xxxxx-xx",
"logfile_date": "2019/11/11",
"logfile_elb_name": "xxx.xxxxxxxxxxxx-lb.1xxxxxxxxxxxx",
"elb_ip_address": "xx.xx.xx.xx",
"logfile_hash": "xxxxxxxx",
"elb_timestamp": "20191111T0530Z",
"key": "xxxxxxxxx/AWSLogs/xxxxxxxxxxx/elasticloadbalancing/xx-xxxx-xx/2019/11/11/xxxxxxxxxx_elasticloadbalancing_xx-xxxxxxx-xx_app.xxxxxx-lb.xxxxxxxx_20191111T0530Z_xx.xx.xx.xx_xxxxx.log.gz",
"prefix": "xxxxxx",
"elb_timestamp_unixtime": 1573450200,
"s3_last_modified_unixtime": 1573450208,
"time": "2019-11-11T05:28:32.708475+0000",
"elb": "app/xxxxxx-lb/xxxxxxxxxxxxxxx",
"client": "xx.xx.xx.xx",
"client_port": "xxx",
"backend": "-",
"backend_port": null,
"request_processing_time": -1,
"backend_processing_time": -1,
"response_processing_time": -1,
"elb_status_code": "404",
"backend_status_code": "-",
"received_bytes": 202,
"sent_bytes": 178,
"request_method": "GET",
"request_uri": "https://xx.xx.xx.xx:443/",
"request_protocol": "HTTP/1.1",
"user_agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36",
"ssl_cipher": "ECDHE-RSA-AES128-GCM-SHA256",
"ssl_protocol": "TLSv1.2",
"type": "https",
"target_group_arn": null,
"trace_id": null,
"domain_name": null,
"chosen_cert_arn": null,
"matched_rule_priority": null,
"request_creation_time": null,
"actions_executed": null,
"redirect_url": null,
"error_reason": null,
"option1": null,
"option2": null,
"option3": "- \"Root=1-xxxxx-xxxxxxxxxxxxxxx\" \"-\" \"arn:aws:acm:xx-xx-x:xxxxxxxxxxxx:certificate/xxxxxx-xxxx-xxxx-xxxx-xxxxxxx\" 0 2019-11-11T05:28:32.708000Z \"fixed-response\" \"-\" \"-\" \"-\" \"-\"",
"@timestamp": "2019-11-11T05:28:32.000000000+00:00"
},
"fields": {
"logfile_date": [
"2019-11-11T00:00:00.000Z"
],
"@timestamp": [
"2019-11-11T05:28:32.000Z"
],
"time": [
"2019-11-11T05:28:32.708Z"
]
},
"sort": [
1573450112000
]
}
from fluent-plugin-elb-log.
Consider issue to be resolved, thanks.
from fluent-plugin-elb-log.
Related Issues (20)
- Cannot remove non-necessary fields in output HOT 4
- Failed to execute: uninitialized constant Fluent::Elb_LogInput::AWS HOT 3
- unable to make this work HOT 3
- Wrong handling of ipv6 clients HOT 2
- ELB logs not reflecting in ElasticSearch for a custom index. HOT 3
- Change option name proxy_uri => http_proxy
- Plugin does not work with application load balancers HOT 2
- s3bucket_is_ok incompatible with cross account setups HOT 2
- Question: Why only 100 objects are retrieved from the AWS API? HOT 2
- Use AWS-SDK v3
- Delete processed objects HOT 1
- emit transaction failed: error_class=NoMethodError error="undefined method `to_msgpack' HOT 1
- Unknown input plugin 'elb_log' HOT 3
- Release request HOT 2
- Generating lots of binary files during runtime, without cleaning them up HOT 5
- Lots of "nomatch log found" errors
- Out of memory error
- plugin loads but there are no logs
- Plugin is not pulling logs from S3, fails without error log
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluent-plugin-elb-log.