timebankspace
### Self Checks - [x] I have searched for existing issues [search for existing issues](https://github.com/infiniflow/ragflow/issues), including closed ones. - [x] I confirm that I am using English to submit this report ([Language Policy](https://github.com/infiniflow/ragflow/issues/5910)). - [x] Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) ([Language Policy](https://github.com/infiniflow/ragflow/issues/5910)). - [x] Please do not modify this template :) and fill in all the required fields. ### RAGFlow workspace code commit ID 14bb02ee (tag `v1.0.0-rc1`). The code paths cited below are unchanged on `main` at 519e7d98 (2026-10-01). ### RAGFlow image version infiniflow/ragflow:v1.0.0-rc1 ### Other environment information - Hardware parameters: x86_64, AMD Ryzen 7 8845HS (8C/16T), 23 GiB RAM, CPU only - OS type: Ubuntu 22.04.5 LTS, Docker with docker-compose v2.26.1, CPU profile - Others: `DOC_ENGINE=infinity` with `infiniflow/infinity:v0.7.3-x64-v3` (the image pinned in `docker/docker-compose-base.yml`); embedding model bge-m3 (1024-d) via OpenAI-API-Compatible; no rerank model ### Actual behavior With `DOC_ENGINE=infinity`, the full-text leg of hybrid retrieval contributes nothing for Chinese natural-language questions. The returned `similarity` is ≈ `vector_similarity_weight × cosine`, which is ≈ 0.2 with the default parameters (`vector_similarity_weight` 0.3, `similarity_threshold` 0.2). Relevant chunks therefore sit right at the default threshold, and anything with a cosine below ~0.67 is filtered out. Short keyword queries are not affected. Top-1 `similarity` from `POST /api/v1/retrieval` against a dataset that holds only the 3-chunk Markdown file from the steps below (`similarity_threshold: 0`; `*` = the top-1 chunk contains the answer): | question | vw = 0 (keyword only) | vw = 0.3 (default) | vw = 0.79 | vw = 0.8 | vw = 1 | |---|---|---|---|---|---| | 青柚七号充一次电能跑多久？ | 0.000* | 0.210* | 0.553* | 0.752* | 0.700* | | 故障码E-17怎么处理 | 0.000* | 0.204* | 0.538* | 0.740* | 0.681* | | 甲烷报警阈值是多少 | 0.000* | 0.205* | 0.540* | 0.742* | 0.684* | | 红外热像仪 (a single term) | 0.960* | 0.877* | 0.743* | 0.740* | 0.685* | | LEL | 0.906* | 0.824* | 0.691* | 0.689* | 0.634* | - With vw = 0, every question scores 0.000: the full-text match returns nothing for them. - The jump between vw = 0.79 and vw = 0.8 (0.553 → 0.752) comes from [`minMatch()`](https://github.com/infiniflow/ragflow/blob/14bb02ee4584c1ab18f0c722c48beecd6bbf9860/internal/service/nlp/retrieval.go#L645-L650). It sets `minimum_should_match` to 0 when vw ≥ 0.8, and the keyword leg immediately starts matching again. - The same deployment with `DOC_ENGINE=elasticsearch` returned 0.377 / 0.418 / 0.480 for the first three questions with default parameters, using the same document and the same embedding model. - On Infinity, the `vector_similarity` and `term_similarity` returned with each chunk are identical to `similarity` (`return sim, sim, sim, nil`, see below). This is what #12776 reports. ### Expected behavior Chinese questions should get a meaningful full-text score on Infinity, as they do on Elasticsearch, where the Go side computes term similarity with its own tokenizer. With default parameters, relevant chunks should not hover right at the default similarity threshold. ### Steps to reproduce 1. Set `DOC_ENGINE=infinity` in `docker/.env` and start the stack. 2. Register a user, add an embedding model (bge-m3 here), and create a dataset with the default General template. 3. Upload and parse this Markdown file. It produces 3 chunks: ```markdown # 青柚七号巡检机器人 技术说明 ## 概述 青柚七号是一款面向地下管廊的轨道式巡检机器人，由闽江北岸实验室于 2025 年研制。 整机重量 38 公斤，最高巡检速度每秒 1.6 米，单次充电续航 9 小时。 ## 传感器 机器人搭载一台 640×512 分辨率的红外热像仪，用于发现电缆接头过热； 另配一套激光甲烷检测仪，报警阈值设定为 1.5% LEL。 ## 维护 轨道滑轮每运行 400 公里需要更换一次；电池组设计寿命为 1200 次充放电循环。 故障代码 E-17 表示红外热像仪通信中断，处理方法是重启云台控制板。 ``` 4. Retrieve with the keyword leg only: ```bash curl -s -X POST http://<host>/api/v1/retrieval \ -H "Authorization: Bearer <api_key>" -H "Content-Type: application/json" \ -d '{"question":"甲烷报警阈值是多少","dataset_ids":["<dataset_id>"],"similarity_threshold":0,"vector_similarity_weight":0}' ``` The top-1 `similarity` is 0. Repeat with `vector_similarity_weight` 0.3 (≈ 0.205) and 0.8 (≈ 0.742). 5. (Optional) To see the generated query, call `PUT /api/v1/system/config/log` with `{"level":"debug"}`, repeat step 4, and run `docker logs <ragflow container> 2>&1 | grep -A4 'MatchTextExpr:'`. ### Additional information **Why the full-text leg is empty** 1. In Infinity mode, `Tokenize()` and `FineGrainedTokenize()` return their input unchanged ([tokenizer.go#L459](https://github.com/infiniflow/ragflow/blob/14bb02ee4584c1ab18f0c722c48beecd6bbf9860/internal/tokenizer/tokenizer.go#L459), [#L513](https://github.com/infiniflow/ragflow/blob/14bb02ee4584c1ab18f0c722c48beecd6bbf9860/internal/tokenizer/tokenizer.go#L513)). `Question()` therefore treats the whole Chinese clause as a single term. It builds `X OR "X" OR ("X"~2)^0.5` ([query_builder.go#L528](https://github.com/infiniflow/ragflow/blob/14bb02ee4584c1ab18f0c722c48beecd6bbf9860/internal/service/nlp/query_builder.go#L528)) with `minimum_should_match` ([#L599](https://github.com/infiniflow/ragflow/blob/14bb02ee4584c1ab18f0c722c48beecd6bbf9860/internal/service/nlp/query_builder.go#L599)), and the result is passed to Infinity's `MatchText` ([chunk.go#L1092](https://github.com/infiniflow/ragflow/blob/14bb02ee4584c1ab18f0c722c48beecd6bbf9860/internal/engine/infinity/chunk.go#L1092), [#L1113](https://github.com/infiniflow/ragflow/blob/14bb02ee4584c1ab18f0c722c48beecd6bbf9860/internal/engine/infinity/chunk.go#L1113)). Debug log for 甲烷报警阈值是多少: ``` matching_text=((甲烷报警阈值 OR "甲烷报警阈值" OR ("甲烷报警阈值"~2)^0.5)^1) fields=docnm@ft_docnm_rag_coarse^10,docnm@ft_docnm_rag_fine^5,important_keywords@ft_important_keywords_rag_coarse^30,important_keywords@ft_important_keywords_rag_fine^20,questions@ft_questions_rag_fine^20,content@ft_content_rag_coarse^2,content@ft_content_rag_fine extra_options=map[filter:available_int=1 AND available_int=1 minimum_should_match:30% original_query:甲烷报警阈值是多少] ``` 2. Infinity itself analyzes the bare term fine: `甲烷报警阈值` alone matches the chunk (for example, score 3.28 on `content@ft_content_rag_coarse`). What rejects the chunk is `minimum_should_match=30%` applied over this expression and the 7 fields. In the text, 甲烷 and 报警阈值 are not adjacent ("激光甲烷检测仪，报警阈值设定为…"), so both phrase branches miss. I reproduced this directly against Infinity's HTTP API (`GET /databases/default_db/tables/<chunk table>/docs` with a `match_method: text` search, `minimum_should_match` passed in `params`): | matching_text | fields | minimum_should_match | hits | |---|---|---|---| | `甲烷报警阈值` | the 7 fields above | 30% | 1 | | the full expression above | the 7 fields above | (not set) | 1 | | the full expression above | the 7 fields above | 10% / 20% / 25% / 30% / 40% | 1 / 1 / **0** / **0** / **0** | | the full expression above | `content@ft_content_rag_coarse^2,content@ft_content_rag_fine` | 30% | 1 | For terms whose words are adjacent in the text, the full expression still matches at 30% and only fails at 40%. Examples are `红外热像仪` and `报警阈值`. Natural-language questions almost never occur verbatim in a chunk, so in practice the keyword leg is lost for Chinese questions. 3. On Infinity, the hybrid score is Infinity's fused `_score` ([retrieval.go#L474-L488](https://github.com/infiniflow/ragflow/blob/14bb02ee4584c1ab18f0c722c48beecd6bbf9860/internal/service/nlp/retrieval.go#L474-L488)), fused with weights `1-vw,vw` ([buildInfinityFusionExpr](https://github.com/infiniflow/ragflow/blob/14bb02ee4584c1ab18f0c722c48beecd6bbf9860/internal/service/nlp/retrieval.go#L601-L615)). It is not re-scored on the Go side, so a zero text leg leaves `0.3 × cosine` with the defaults. The dense leg is filtered by `filter_fulltext(<fields>, <matching_text>)` ([chunk.go#L1153-L1167](https://github.com/infiniflow/ragflow/blob/14bb02ee4584c1ab18f0c722c48beecd6bbf9860/internal/engine/infinity/chunk.go#L1153-L1167)), which is the mechanism described in #17041. Here it still lets the candidates through, because `filter_fulltext` carries no `minimum_should_match`. Only the full-text score is lost. **Possible directions** (not tested): - Segment the question with the Go rag tokenizer when building the full-text query for Infinity. Infinity can keep analyzing the indexed text itself. - Avoid whole-clause phrase branches, or relax `minimum_should_match`, when the clause could not be segmented. - Compute the hybrid similarity on the Go side for Infinity, as is already done for Elasticsearch. **Workaround**: `vector_similarity_weight >= 0.8`. `minimum_should_match` then becomes 0, and the keyword leg comes back with weight 0.2. Related: #17041 (dense candidates restricted by the full-text predicate), #12776 (identical hybrid/term/vector similarity values).