From fixed video-to-label mappings to flexible, question-driven sign language understanding.

• 2026-08-20: Project page and paper PDF released.
• 2026-07-30: SignQA-2026 annotations released on Hugging Face.
• 2026-07-30: Paper released on arXiv.

Existing sign language understanding tasks usually learn a fixed mapping from a video to a sign class, gloss sequence, or spoken-language translation. These objectives reveal only part of what a model understands and cannot flexibly respond to different user questions about the same video.

We introduce Sign Language Question Answering (SLQA): given a sign video and a natural-language question, a model must generate the corresponding answer. SLQA evaluates fine-grained visual retrieval, temporal and structural reasoning, gloss recognition, and sentence-level semantic understanding in one unified framework.

SignQA-2026 is constructed from the gloss and translation annotations of PHOENIX-2014T and CSL-Daily. It covers five complementary question categories, ranging from low-level sequence reasoning to high-level language understanding.

1. M1 - Position Reasoning: identify a gloss at a specified position.
2. M2 - Structural Reasoning: reason about neighboring signs and subsequences.
3. M3 - Visual Search: locate a target gloss or identify non-manual markers.
4. M4 - Gloss Recognition: recover the complete gloss sequence.
5. M5 - Translation Understanding: understand the spoken-language meaning.

Our baseline, SLQAM (Sign2Answer), contains three major components:

• a visual backbone for frame-level sign representations;
• Question-Conditioned Modulated Temporal Downsampling (QCMTD) for selecting question-relevant temporal information;
• a language model for answer generation.

We further introduce a three-stage in-domain knowledge transfer pipeline: CSLR pre-training → SLT training → SLQA fine-tuning. This progressively transfers sign perception, temporal alignment, and language generation knowledge to the new SLQA task.

SLQAM consistently outperforms general-purpose video-language models and the cascaded Sign2Text2Answer baseline on both benchmarks. The table below reports overall test-set results from the paper.

RL denotes ROUGE-L. Higher is better for all metrics shown. See Tables 5-8 in the paper for complete per-task and BLEU results.

The annotations are available at hulala/SignQA-2026.

from datasets import load_dataset

# Chinese benchmark
csl_daily = load_dataset("hulala/SignQA-2026", "csl-daily-qa")

# German benchmark
phoenix14t = load_dataset("hulala/SignQA-2026", "phoenix14t-qa")

Each annotation contains the source video_id, split, dataset, module, question-template ID, question, answer, and language.

• Release the paper
• Release SignQA-2026 annotations
• Release the project page
• Release training and evaluation code
• Release pretrained checkpoints
• Release detailed preprocessing instructions

If you find this work useful, please cite:

@misc{gan2026signlanguagequestionanswering,
  title         = {Sign Language Question Answering: A New Task, Benchmark,
                   and Baseline for Sign Language Understanding},
  author        = {Shiwei Gan and Lichen Wang and Xiao Liu and Yafeng Yin and
                   Kuizhuang Liu and Sanglu Lu and Lei Xie},
  year          = {2026},
  eprint        = {2607.27826},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  doi           = {10.48550/arXiv.2607.27826},
  url           = {https://arxiv.org/abs/2607.27826}
}

We thank the creators of PHOENIX-2014T and CSL-Daily for making their resources available to the sign language research community.