ViGOS is a release-oriented training and evaluation package for visually grounded on-policy self-distillation of multimodal large language models. The released code supports LoRA post-training and evaluation for Qwen2.5-VL and Qwen3-VL style checkpoints. The paper experiments use Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct.

Vanilla multimodal OPSD can leak the privileged reference answer into dense token supervision before the student has grounded its response in the image. ViGOS changes the supervision path: the student first writes a visual description, then reasons, then gives a boxed final answer.

• Released Checkpoints
• Environment
• Data
• Training
• Evaluation
• Results
• Project Page
• Acknowledgements
• License
• Citation

HuggingFace model repositories are reserved at the following placeholders:

Once public, the checkpoints can be used directly for evaluation:

MODEL_PATH=OedoSoldier/ViGOS-3B bash scripts/eval_vigos.sh
MODEL_PATH=OedoSoldier/ViGOS-7B bash scripts/eval_vigos.sh

Or downloaded locally:

huggingface-cli download OedoSoldier/ViGOS-3B --local-dir checkpoints/ViGOS-3B
huggingface-cli download OedoSoldier/ViGOS-7B --local-dir checkpoints/ViGOS-7B

The student output format is:

<description> image description here </description>
<think> reasoning process here </think>
\boxed{FINAL ANSWER here}

ViGOS uses three teacher roles during training:

At inference time, all teachers and reference answers are discarded. The final model receives only the image, the question, and the ViGOS student prompt.

git clone https://github.com/OedoSoldier/ViGOS.git
cd ViGOS
uv sync --python 3.11

The pinned environment includes PyTorch 2.8.0, Transformers 4.57.1, TRL 0.26.0, PEFT 0.17.1, Accelerate 1.11.0, DeepSpeed 0.18.2, and vLLM 0.11.0. Paper training used 8 NVIDIA A100 GPUs with bf16 and ZeRO-2.

If a HuggingFace dataset or model is gated:

uv run huggingface-cli login

Training uses the Vision-SR1-47K dataset on HuggingFace:

export DATASET_NAME=LMMs-Lab-Turtle/Vision-SR1-47K
export DATASET_SPLIT=train

Dataset link: LMMs-Lab-Turtle/Vision-SR1-47K

The HuggingFace dataset card reports the default subset with 47.6k training rows and Apache-2.0 license. The fields used by this release are:

Other dataset fields such as solution, options, path, and data_source may be present but are not required by the default shell scripts.

Tiny or invalid images can be filtered at load time:

FILTER_TINY_IMAGES=true MIN_IMAGE_SIZE=3 \
DATASET_NAME=LMMs-Lab-Turtle/Vision-SR1-47K \
bash scripts/train_vigos_qwen25_3b.sh

The shell launchers use Accelerate + DeepSpeed with configs/accelerate_zero2_lora_gpu_8.yaml.

Training curves and run logs are available in the W&B Training Logs of ViGOS report.

DATASET_NAME=LMMs-Lab-Turtle/Vision-SR1-47K \
bash scripts/train_vigos_qwen25_3b.sh

DATASET_NAME=LMMs-Lab-Turtle/Vision-SR1-47K \
bash scripts/train_vigos_qwen25_7b.sh

DATASET_NAME=LMMs-Lab-Turtle/Vision-SR1-47K \
MAX_STEPS=500 \
SAVE_STEPS=100 \
REPORT_TO=none \
bash scripts/train_vigos_qwen25_3b.sh

Training saves PEFT LoRA adapters under runs/. Merge an adapter into the base model before standalone evaluation:

uv run python scripts/merge_lora.py \
  --adapter runs/vigos_qwen25_3b_YYYYMMDD-HHMMSS/checkpoint-500 \
  --output runs/vigos_qwen25_3b_merged \
  --overwrite

If --base-model is omitted, the merge script reads base_model_name_or_path from the adapter config.

Evaluation uses the ViGOS student prompt with assistant-side <description> prefill. The default wrapper uses vLLM for generation and an OpenAI-compatible judge when judging is enabled.

export DEEPSEEK_API_KEY=your_api_key

MODEL_PATH=runs/vigos_qwen25_3b_merged \
bash scripts/eval_vigos.sh

Include CV-Bench:

MODEL_PATH=runs/vigos_qwen25_3b_merged \
EVAL_BENCHMARKS=vilp-f,vilp-p,cv-bench \
bash scripts/eval_vigos.sh

Generate responses without judging:

MODEL_PATH=runs/vigos_qwen25_3b_merged \
SKIP_JUDGE=true \
bash scripts/eval_vigos.sh

Outputs are written under eval_outputs/vigos_${RUN_ID}/:

Pass@5 counts an example correct if any of five samples is correct. Avg@5 averages correctness across the five samples.

Higher Score indicates better image-grounded reasoning under prior conflict. Higher Prior indicates retained prior-aligned knowledge.

The static GitHub project page lives under docs/ and can be deployed through GitHub Pages by selecting the repository branch and /docs folder.

docs/
+-- index.html
+-- static/
    +-- css/
    +-- images/
    +-- js/

We thank the authors and maintainers of zli12321/Vision-SR1 for the Vision-SR1 project and training data foundation. This release also builds on Qwen2.5-VL/Qwen3-VL model support, vLLM, Transformers, TRL, PEFT, Accelerate, DeepSpeed, and the benchmark suites used in the paper: MM-Vet, MMMU, MMMU-Pro, MathVerse, MathVista, MMSI, RealWorldQA, CV-Bench, and ViLP.

The project page is adapted from the Academic Project Page Template, which credits the Nerfies project page.

The ViGOS implementation code, including source files and configuration or launcher files under vigos/, scripts/, and configs/, is released under the Apache License 2.0, unless otherwise noted. See NOTICE for scope and attribution notes.

The ViGOS paper, paper PDF, paper text, paper figures, and project-page research content are released under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0), unless otherwise noted.

Model weights, third-party datasets, benchmark assets, external dependencies, and the project-page template are subject to their own licenses. The project-page template license is documented in docs/README.md.

@misc{wang2026seeing,
  title={Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation},
  author={Wang, Sihan and Liu, Xiyao and Liu, Lianqing and Han, Zhi},
  year={2026},
  eprint={2606.19120},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2606.19120}
}