Syed-Shahidh-Ilhan
### Describe the bug When using `@type file` buffers with a slow or failing output (S3, Kafka, forward, etc.), Fluentd accumulates many buffer chunk files. Each chunk keeps one or two file descriptors open while it sits in **stage** or **queue**, and `buf_file#resume` re-opens every persisted chunk on startup. Once open FD count approaches the process limit (often 65536 on Linux), Fluentd enters a **death spiral**: 1. Output flush fails with `Errno::EMFILE` (cannot open TCP socket / DNS / temp files) 2. Queued chunks are not purged and their FDs are not released 3. New chunks cannot be created (`BufferOverflowError`) 4. Even after downstream recovers, Fluentd may remain stuck because resume/flush paths need more FDs than available This is related to long-standing reports ([#1612](https://github.com/fluent/fluentd/issues/1612), [#3040](https://github.com/fluent/fluentd/issues/3040), [#4012](https://github.com/fluent/fluentd/issues/4012), [#3993](https://github.com/fluent/fluentd/issues/3993)), but those issues focus on symptoms/workarounds (raise ulimit, tune buffer size). This report points at a specific **FD lifecycle design** in current `buf_file` / `FileChunk` that makes recovery impossible at scale without deleting buffers or raising limits to match chunk count. ### Root cause in code (v1.x) **1. `enqueue_chunk` closes only empty chunks — non-empty queued chunks stay open** ```ruby # lib/fluent/plugin/buffer.rb if chunk.empty? chunk.close else @queue << chunk chunk.enqueued! # renames files but keeps @chunk and @meta open end ``` **2. `FileChunk#enqueued!` keeps both `@chunk` and `@meta` handles open after rename** After staging, each chunk uses **2 FDs** (`@chunk` + `@meta`). After enqueue, both handles remain open through flush (see `file_rename` callbacks in `lib/fluent/plugin/buffer/file_chunk.rb`). The code even notes: > "Too many open files" should be fixed by proper buffer configuration and system setting. **3. `buf_file#resume` opens every buffer file during startup** ```ruby # lib/fluent/plugin/buf_file.rb Dir.glob(escaped_patterns(patterns)) do |path| chunk = Fluent::Plugin::Buffer::FileChunk.new(m, path, mode, ...) queue << chunk # File.open in load_existing_enqueued_chunk / load_existing_staged_chunk end ``` With tens of thousands of backlog files, resume alone can exhaust FDs **before any flush thread runs** ([#3040](https://github.com/fluent/fluentd/issues/3040)). **4. FDs are only released on buffer shutdown** `Buffer#close` closes dequeued, queued, and staged chunks — not when a chunk moves from stage → queue. PR [#1468](https://github.com/fluent/fluentd/pull/1468) (v0.14.12) handles EMFILE on **chunk create** by raising `BufferOverflowError`, but does not close existing queued chunk FDs or enable recovery. ### To Reproduce 1. Configure a file buffer with relatively small `timekey` and large `chunk_limit_size`, e.g. S3 output: ```apache <match **> @type s3 # ... aws / bucket config ... <buffer tag,time> @type file path /var/log/fluent/buffer timekey 5 chunk_limit_size 100MB flush_mode interval flush_interval 5 flush_thread_count 15 # note: no total_limit_size — backlog can grow by chunk count </buffer> </match> ``` 2. Block or fail the output path (wrong credentials, network partition, unreachable S3 endpoint) for long enough to accumulate **~30k+ chunks** (≈ 60k+ FDs with chunk+meta open, or ~32k chunks at 65536 limit). 3. Observe logs: ``` failed to flush the buffer. retry_times=... error="Too many open files" ... Failed to open TCP connection to ...s3.amazonaws.com:443 can't create buffer metadata for ... error = Too many open files @ rb_sysopen ``` 4. Restore downstream — Fluentd often **does not recover** without manual intervention (delete buffer dir or raise `LimitNOFILE` above chunk count). 5. Restart Fluentd with large backlog — `restoring buffer file` storm in logs; process hits EMFILE during `resume` before flushing ([#3040](https://github.com/fluent/fluentd/issues/3040)). ### Expected behavior File buffer chunks should not require a permanently open FD for every queued chunk. Suggested approaches (any of these would help): 1. **Close on enqueue**: after `enqueued!`, call `chunk.close` (or equivalent) so queued chunks hold **0 FDs** until dequeued for flush. 2. **Lazy open on flush**: `FileChunk#open` / `read` reopen paths on demand; keep metadata in memory or mmap only `.meta` briefly. 3. **Lazy resume**: do not `File.open` all chunks in `Dir.glob`; index paths from filesystem and open in batches bounded by `(flush_thread_count * 2)` or configurable `resume_open_limit`. 4. **Document FD estimation**: `queue_limit_length`, `total_limit_size`, and chunk count vs `LimitNOFILE` (as suggested in [#3040](https://github.com/fluent/fluentd/issues/3040#issuecomment)). At minimum, Fluentd should remain able to **drain an existing backlog** after downstream recovery without requiring FD count ≥ chunk count. ### Actual behavior - ~1–2 FDs per staged/queued chunk for the lifetime of the chunk in memory - Resume opens all chunks eagerly - EMFILE on output flush → permanent stall until buffer deletion or ulimit >> chunk count - `BufferOverflowError` on create stops ingestion but does not free queued FDs ### Your Environment ```markdown - Fluentd: 1.16+ / td-agent 4.x (behavior present since v0.14 file buffer redesign) - OS: Linux (Kubernetes), `LimitNOFILE` commonly 65536 - Buffer: `@type file` with timekey slicing, S3 or other slow output - Observed at production scale: ~65k FD limit reached with ~32k queued chunks (chunk + meta open) ``` ### Your Error Log ``` failed to flush the buffer. retry_times=0 next_retry_time=... error="Too many open files" error_class=Errno::EMFILE Failed to open TCP connection to <bucket>.s3.amazonaws.com:443 ``` ``` emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="can't create buffer metadata for /var/log/fluent/buffer/buffer.*.log. Stop creating buffer files: error = Too many open files @ rb_sysopen - ...log.meta" ``` ### Additional context - Community has reported this pattern since 2017 ([#1612](https://github.com/fluent/fluentd/issues/1612): 131034 files, 65516 `.log` + 65515 `.meta`, exactly at 65536 ulimit). - [#3040](https://github.com/fluent/fluentd/issues/3040) (still open) requests batch open/close on resume — no upstream fix yet. - Workarounds today: raise ulimit, delete buffer files (data loss), or cap backlog via `total_limit_size` / `overflow_action` — none fix FD retention on queued chunks. - Willing to contribute a PR for close-on-enqueue + lazy reopen if maintainers agree on direction.