Xavierando
### Self Checks - [x] I have searched for existing issues, including closed ones. - [x] I confirm that I am using English to submit this report. - [x] Non-english title submissions will be closed directly. - [x] Please do not modify this template :) and fill in all the required fields. ### RAGFlow workspace code commit ID 98b48a08 ### RAGFlow image version Go backend built from 98b48a08 (Elasticsearch doc engine) ### Other environment information - OS type: Linux x86_64, Docker - Doc engine: Elasticsearch 8.11.3 ### Actual behavior Follow-up to #16832 (point C), split out of #16833 / #20532. Related: #19366 makes the same point for an explicit `metadata_condition` with an empty `conditions` list: no conditions should mean no metadata filtering. With `meta_data_filter.method` set to `auto` or `semi_auto`, `ApplyMetaDataFilter` (`internal/service/metadata_filter.go`) returns `(nil, true)` whenever its result is empty. The result is empty in three different situations, and the flag is the same for all of them: | what the LLM returned | result | |---|---| | no conditions (`{"conditions": [], ...}`) | `nil, true` | | a reply that cannot be parsed (`GenMetaFilter` falls back to no conditions) | `nil, true` | | conditions that match no document | `nil, true` | The "no conditions" case is common: the filter prompt (`rag/prompts/meta_filter.md`, rule 5d) tells the model to skip attributes and values that are not in the metadata, so a question that does not mention any metadata gets no conditions. It ends up as `(nil, true)` because `ApplyMetaFilter` returns an empty list for zero conditions. The callers then do opposite things with that flag: | caller | what it does with `(nil, true)` | |---|---| | chat completion (`internal/service/chat_pipeline.go`, meta_data_filter step) | ignores the flag, keeps the unfiltered document list | | search app retrieval test (`internal/service/chunk/chunk.go`, `RetrievalTest`) | uses the `nil` list, so no document restriction | | `POST /api/v1/retrieval` with `meta_data_filter` (`internal/service/dataset/search.go`, `selectMetadataFilteredDocIDs`) | keeps the unfiltered list unless `metadata_condition` was also sent | | agent Retrieval tool (`internal/agent/retrievalbridge/enhancer.go`, `FilterDocuments`) | returns `NoMatchDocIDSentinel`, so retrieval returns nothing | (`manual` is consistent: it returns the sentinel itself when its conditions match nothing.) So: 1. In an agent whose Retrieval tool uses `auto`/`semi_auto`, a question that does not mention any metadata (or an LLM reply that fails to parse) retrieves **nothing**, although no filter was produced. 2. In chat and `/api/v1/retrieval`, a filter that **did** produce conditions and matched nothing is dropped silently, and the answer is built from documents that fail the user's constraint. Nothing in the response says the filter was ignored. Example with `POST /api/v1/retrieval` (dataset with a `date` metadata field, ~13 000 documents): ```json {"dataset_ids": ["<id>"], "question": "<question>", "page_size": 64, "meta_data_filter": {"method": "semi_auto", "semi_auto": [{"key": "date"}]}} ``` | question | `total` | `date` of the returned documents | |---|---|---| | "What was discussed in the council session of 22 April 2026?" | 64 | all `2026-04-22` (6 documents) | | "What was discussed in the council session of 3 January 1990?" (no such date) | 64 | 2021-02-11 … 2026-05-21 (unrelated sessions) | The server log is at `warn`, so I can't tell whether the model emitted a condition on `1990-01-03` or no condition. From the caller's side the two are indistinguishable, which is the point of this issue. ### Expected behavior `ApplyMetaDataFilter` should tell "no filter was produced" apart from "the filter matched nothing", and all callers should treat each case the same way: - **no conditions / unparseable reply** → no metadata restriction (the base document scope), everywhere, including agents; - **conditions that match nothing** → one agreed behaviour on every path. The second point is a product decision. Options: - (a) restrict to nothing (sentinel), as the agent tool and `manual` already do. The answer then relies on the empty-retrieval reply instead of unrelated documents. - (b) fall back to unrestricted retrieval, as chat does today, but make it visible, at least in the log and ideally in the response. I'd suggest (a) for `auto`/`semi_auto` in chat, the retrieval test and the agent tool. For `/api/v1/retrieval` I'd keep the current fallback unless you prefer (a) there too. A related note on the prompt: rule 5d makes the model drop a condition whose value is not in the metadata. With (a), a question about a date that doesn't exist would still turn into "no conditions" and be answered unfiltered. If (a) is chosen, should rule 5d keep conditions on known keys even when the value is unknown? ### Steps to reproduce 1. Dataset with documents that have a `date` metadata field (`YYYY-MM-DD`). 2. Agent with a Retrieval tool on that dataset, metadata filter `auto`. Ask a question that does not mention any metadata → no chunks retrieved. 3. Chat on the same dataset with metadata filter `semi_auto` on `date`. Ask about a date no document has → the answer uses documents from other dates. 4. Same with `POST /api/v1/retrieval` and `meta_data_filter` as in the example above. I ran step 4 on a live deployment. Steps 2 and 3 follow from the code paths in the table. The `(nil, true)` result for each of the three LLM replies was checked with a stub model in a local test. ### Additional information I can send a PR once the behaviour for "conditions matched nothing" is agreed. The change would be: a distinct result from `ApplyMetaDataFilter` for "no conditions"; the chat pipeline and the retrieval test handling the no-match flag the way the agent tool does; tests for the three LLM replies on each caller.