RAVine (a Reality-Aligned eValuation framework for agentic LLMs with search), is a comprehensive evaluation system for agentic search, encompassing the web environment, benchmark datasets, and a novel evaluation method, serving as a full-process, reproducible, and goal-aligned evaluation sandbox.

• 🎯More Precise: RAVine provides a more precise and attributable method for nuggets (claim-level ground truth) extraction, the final generated report evaluation is more accurate.
• ⚙️More Comprehensive: RAVine not only focuses on end-to-end result evaluation, but also designs detailed search process performance indicators.
• 💰Lower Cost: RAVine provides local search APIs and reduces the cost of calling LLM-Judge(Gemini-2.5-Flash) to ~0.01$ per evaluation data.
• 🚀Full Sandbox: We have packaged the search tools and evaluation framework. You only need to provide an LLM to run or evaluate the agent search on RAVine.

We have uploaded the running evaluation data, nuggets, corpus index, etc. to hugging face. You can access them and modify the corresponding path variables in the running script:

• Queries & Nuggets: https://huggingface.co/datasets/sapphirex/RAVine-nuggets
• Raw Qrels: https://huggingface.co/datasets/sapphirex/RAVine-qrels
• Dense Index: https://huggingface.co/datasets/sapphirex/RAVine-dense-index
• URL-Docid Mapper: https://huggingface.co/datasets/sapphirex/RAVine-mapper
• Running logs (for reproduction): https://huggingface.co/datasets/sapphirex/RAVine-logs

First, install the operating environment. This project mainly uses two environments, and we recommend using uv to install the following two environments separately:

• Env for vllm: the vllm version we use is 0.9.0.1, run pip install vllm==0.9.0.1.
• Env for /src: the operating environment of our main program has been exported to requirements_agent.txt. Run uv pip install -r requirements_agent.txt.
  • The JDK version we use is 21.0.7, and we recommend that you use this version to support running the BM25 index based on PySerini.

Second, write the configuration file, which is related to the selection and setting of the model, operating environment, index, and file path. You can find examples at configs/ and write your own config here.

Third, run the vllm service and then run the main program. Example instructions are as follows:

bash scripts/server/vllm.sh your_config_file # run the server of agentic llm
bash scripts/evaluation/run.sh your_config_file

If you need to run multiple configurations at once, we provide the corresponding scripts. Please write the configurations to be run into /scripts/evaluation/run_all_eval.sh and then run it.

For more detailed steps, see Evaluation_and_reproduction.

Table Description:

• "Rate" denotes the Task Completion Rate.
• "Comp." refers to the score of Task Completeness.
• "Rec." and "Prec." represent Recall and Precision, respectively.
• "URL Err." denotes the URL Error.
• Latency is measured in seconds, and cost is measured in dollars.
• Symbols (↑) and (↓) indicate that higher or lower values are preferred, respectively.
• Models not marked with (Thinking) either run without thinking or lack support for the thinking mode.
• Bold values indicate the best performance for each corresponding metric in the column.

Evaluation results on RAVine, with a maximum context length of 32k and the index built by gte-modernbert-base:

Evaluation results on RAVine, with a maximum context length of 128k and the index built by gte-modernbert-base:

Evaluation results on RAVine, with a maximum context length of 256k and the index built by gte-modernbert-base:

Evaluation results on RAVine, with a maximum context length of 32k and the index built by BM25:

Evaluation results on RAVine, with a maximum context length of 128k and the index built by BM25:

Evaluation results on RAVine, with a maximum context length of 256k and the index built by BM25:

If you find RAVine useful in your research, please cite our paper:

@article{xu2025ravine,
  title={RAVine: Reality-Aligned Evaluation for Agentic Search},
  author={Xu, Yilong and Long, Xiang and Zheng, Zhi and Gao, Jinhua},
  journal={arXiv preprint arXiv:2507.16725},
  year={2025},
  url={https://arxiv.org/abs/2507.16725}
}