Comments (2)
GeorgiMateev
commented on August 17, 2024
1
Hi,
Yes, it's requesting 100 items and that's exactly the issue: there could be many more than 100 items in the bucket.
I haven't seen any logic in the code to traverse all objects by using continuation token return by list_objects.
Can you please point me where in the code you are traversing all available objects in the bucket?
Thanks.
from fluent-plugin-elb-log.
shinsaka
commented on August 17, 2024
Hello @GeorgiMateev,
Thank you for your question.
This program code can all item-key in bucket.
"100" means get 100 contents records with one network request.
see testing code below:
https://gist.github.com/shinsaka/c0a17f5859eea8b64c3d3b85fb426b6a
memo:
http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html#list_objects-instance_method
Best,
from fluent-plugin-elb-log.
Related Issues (20)
- unable to make this work HOT 3
- Wrong handling of ipv6 clients HOT 2
- ELB logs not reflecting in ElasticSearch for a custom index. HOT 3
- Change option name proxy_uri => http_proxy
- Plugin does not work with application load balancers HOT 2
- s3bucket_is_ok incompatible with cross account setups HOT 2
- Use AWS-SDK v3
- Delete processed objects HOT 1
- emit transaction failed: error_class=NoMethodError error="undefined method `to_msgpack' HOT 1
- Unknown input plugin 'elb_log' HOT 3
- Release request HOT 2
- Generating lots of binary files during runtime, without cleaning them up HOT 5
- Some log lines are absent from ALB access log HOT 7
- Lots of "nomatch log found" errors
- Out of memory error
- plugin loads but there are no logs
- Plugin is not pulling logs from S3, fails without error log
- Cannot remove non-necessary fields in output HOT 4
- Failed to execute: uninitialized constant Fluent::Elb_LogInput::AWS HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluent-plugin-elb-log.