Star 历史趋势
数据来源: GitHub API · 生成自 Stargazers.cn
README.md

VLX-VR logo

VLX-VR

An Agentic-Aware Video Reasoning Model

English | 中文

X YouTube Discord
arXiv Overview video Try VLX

Overview video: VLX-VR for agentic-aware video reasoning

VLX-VR overview video

📺 HD version: Watch on YouTube

VLX-VR is an agentic-aware video reasoning model trained inside a video reasoning framework defined by a Think–Memory–Observation loop. It targets real-world video analysis where visual, audio, textual, and temporal evidence is scattered across short and long videos, and where fixed-context, single-pass VideoQA is not enough.

Instead of locking the video context before reasoning begins, VLX-VR learns to decide what evidence is needed, invoke read_memory or write_memory, incorporate the returned Observation, and decide whether to continue or produce the task output.

[!TIP]

🚀 Try VLX and explore how Om AI models understand, reason over, and interact with the multimodal world.

Community

Join the VLX community to connect with developers, explore applications, share feedback, and shape the future of multimodal AI.

Official WeChat
Discord Community
VLX official WeChat QR code VLX Discord community QR code

For technical support, partnerships, and community inquiries, contact us at marketing@hzlh.com.

Updates

Overview

Think–Memory–Observation loop of VLX-VR

Figure 1. Framework-defined Think–Memory–Observation loop used to train and run VLX-VR.

Modern video large language models are strong at describing clips and answering Visual QA from a fixed sampled context. Real-world event analysis is harder:

  • Adaptive evidence acquisition: revisit earlier moments, seek missing cues, or resolve conflicts after an initial observation.
  • Multimodal grounding: combine frames, audio, OCR/text overlays, and timestamps rather than relying on a single modality.
  • State maintenance: keep intermediate hypotheses and observations across multi-step reasoning.
  • Termination control: stop when evidence is sufficient, instead of looping forever or answering too early.

Many pipelines still follow:

video + question -> fixed sampled context -> single-pass answer

VLX-VR changes the task to:

video + instruction -> Think -> Memory(read/write) -> Observation -> continue or answer

This makes evidence acquisition, memory use, and termination part of the model's learned reasoning policy, not only an external prompt wrapper.

Highlights

  • Agentic-aware video reasoning: trained to operate inside a Think–Memory–Observation loop rather than only sitting behind an external agent controller.
  • Direct multimodal memory access: Memory exposes read_memory / write_memory; VLX-VR selects the operation from the current reasoning state.
  • Strong MINERVA result: 78.79% accuracy among models in our comparison.
  • Stable across duration bins: 76.70% / 78.73% / 80.92% under MINERVA's original three duration groups; CDAV = 2.97 pp².
  • Checkable process: on correctly answered samples, 96.20% of reasoning traces agree with MINERVA reference traces and the evidence they describe; about 75.80% of all samples satisfy both answer correctness and this evidence-grounded criterion.

Problem Setting

People understand videos by locating relevant moments, comparing states before and after an event, reading text in a scene, and relating spoken content to visual actions. Supporting such reasoning requires more than recognizing objects or isolated frames.

In many standard VideoQA pipelines, models such as Video-LLaVA, Qwen3-VL, and VideoLLaMA 3 receive sampled frames or clips as a fixed input and produce an output in one pass. Because evidence is determined before reasoning begins, the model cannot adaptively acquire new evidence after detecting missing, ambiguous, or conflicting information.

Existing video agents move closer through iterative gathering, temporal localization, and memory queries. However, placing a general-purpose VLM inside an external loop without training it for that role can still fail: the model may request redundant evidence, forget intermediate states, or terminate too early.

VLX-VR focuses on a core question:

Can a video reasoning model be trained not merely to participate in an agentic pipeline, but to learn evidence acquisition, memory use, and termination as integral parts of its reasoning process?

VLX-VR's answer is:

Train an agentic-aware model inside a Think–Memory–Observation loop with direct multimodal-memory access.

Think–Memory–Observation Loop

At reasoning step t, VLX-VR maintains a state st that includes the task instruction, observed evidence, multimodal content stored in memory, current output hypotheses, and unresolved uncertainty.

The loop has three stages:

  1. Think: interpret the task, propose the next evidence need, and estimate whether current evidence is sufficient.
  2. Memory: invoke read_memory or write_memory to retrieve multimodal evidence or retain intermediate state.
  3. Observation: return the result of the Memory operation to VLX-VR for the next decision.

When evidence is sufficient, VLX-VR produces the final output with supporting evidence. When evidence is insufficient or conflicting, it returns to Think and starts another Memory step.

s_0 = initialize(video, instruction)
while not stop(s_t):
    think_t, call_t = VLX-VR(s_t)
    obs_t = execute(call_t)          # call_t ∈ {read_memory, write_memory}
    s_{t+1} = update(s_t, obs_t)
output = VLX-VR(s_t)

Multimodal memory is an external and inspectable reasoning state rather than only a cache of textual summaries. Video, audio, supporting evidence, Observations, and intermediate states can be read, written, recorded, and reused across steps.

Evaluation: MINERVA

We evaluate primarily on MINERVA because it combines broad temporal coverage with human-annotated reasoning traces. Each question includes a video, a question, five choices, and a reference reasoning trace, making it suitable for scoring both answer accuracy and evidence-grounded process quality.

Overall Accuracy

ModelAccuracy (%)Δ vs. VLX-VR (pp)Source
VLX-VR78.79—This work
Seed2.1 Pro70.70-8.09Seed2.1
Gemini 3.5 Flash68.60-10.19Seed2.1
Gemini 2.5 Pro Thinking66.20-12.59MINERVA
Seed2.1 Turbo65.90-12.89Seed2.1
Gemini 3.1 Pro63.50-15.29Seed2.1
GPT-4.153.99-24.80MINERVA
GPT-4o45.54-33.25MINERVA
OpenAI o143.48-35.31MINERVA
Claude 3.5 Sonnet v231.28-47.51MINERVA
Human92.54+13.75MINERVA
Random20.00-58.79MINERVA

Accuracy across Video Durations

MINERVA accuracy by duration

Figure 2. Accuracy across MINERVA duration groups. VLX-VR stays strong as duration increases under the original three-bin grouping.

ModelBelow 5 min5–15 minAbove 15 min
VLX-VR76.7078.7380.92
Gemini 2.5 Pro Thinking68.8766.8457.97
GPT-4.158.8454.7947.25
OpenAI o148.2841.4540.38
Claude 3.5 Sonnet v240.9033.6828.30

Under the original MINERVA grouping, VLX-VR has mean accuracy 78.78%, CDAV 2.97 pp², std 1.72 pp, and range 4.22 pp. Public baselines decline with duration; VLX-VR does not under this grouping.

A refined split of the long-video bin shows the Above 15 min result is driven mainly by 15–30 min (83.40%), while Above 30 min is 74.75%. Performance remains broadly stable rather than collapsing on longer videos.

Skill Profile

SkillAccuracy (%)
Reading90.76
Situational awareness90.32
Temporal reasoning85.95
Numerical reasoning85.71
Event occurrence82.81
Object recognition82.01
Goal reasoning76.92
Listening75.19
Counterfactual reasoning74.19
Spatial perception73.47
Cause and effect72.73
State changes69.23
Counting66.67

VLX-VR is strongest on reading, situational awareness, temporal reasoning, and numerical reasoning. Counting, state changes, causal reasoning, and spatial perception remain challenging.

Reference-trace Agreement

On correctly answered samples, 96.20% of VLX-VR reasoning traces are consistent with MINERVA reference traces and the evidence they describe. Across the full set:

78.79% × 96.20% ≈ 75.80%

samples satisfy both answer correctness and this evidence-grounded trace criterion. This joint rate is still higher than Seed2.1 Pro's answer-only accuracy (70.70%), though the criteria differ.

Case Demos

Three real side-by-side comparisons of VLX-VR and Gemini 3.1 Pro on the same user prompts and public web footage, focusing on how each model grounds evidence, timestamps, and final answers.

Case 1: hockey counterfactual score
Case 1 · Counterfactual score
If the player in green had made his shot on the goal at 02:22 in the video, what would the score have been? (0) 1-0. (1) 2-0. (2) 3-0. (3) 2-1. (4) 2-2.
▶️ Watch MP4
Case 2: chess temporal counting
Case 2 · Temporal counting
How many turns does white play between the first time they are put in check and the end of the game? (0) 4. (1) 2. (2) 0. (3) 3. (4) 1.
▶️ Watch MP4
Case 3: basketball anomaly timestamps
Case 3 · Anomaly + timestamps
看看视频里有哪些和我们日常操作反常的内容,时间点发我下
▶️ Watch MP4
What each case highlights
  • Case 1 (hockey / counterfactual): both models can reach the correct option, but VLX-VR keeps an evidence trail (scoreboard OCR → jersey–team binding → shot at 02:22 → hypothetical 2-1). Gemini 3.1 Pro often collapses this into a short correct sentence without showing how team names and colors were verified.
  • Case 2 (chess / temporal): VLX-VR counts white’s two post-check moves with aligned timestamps (~215.7s first check → ~223.6s / ~228.1s white turns → ~229.7s mate). Missing a single intervening move flips the MCQ answer; Gemini 3.1 Pro is prone to drop one turn.
  • Case 3 (basketball court / anomaly): the hard part is listing soccer-like actions on a basketball court person-by-person with second-level times, not inventing a global story (e.g. “video played in reverse”). VLX-VR reports grounded intervals; Gemini 3.1 Pro may over-commit to a false global hypothesis.

Files live under assets/demos/. Test clips are public-web footage.

Notebook

See notebooks/vlx_vr.ipynb.

Open-Source Status

ItemStatus
PaperReleased (arXiv:2609.09985)
Overview videoReleased (YouTube)
Case demosReleased (assets/demos/)
API cookbook notebookReleased (notebooks/vlx_vr.ipynb)
README + notebook packageThis repository

Why VLX-VR

  • Compared with a fixed-context VideoQA model, VLX-VR can adaptively acquire and reassess evidence during reasoning.
  • Compared with an untrained VLM inside an external agent loop, VLX-VR learns evidence acquisition, memory use, and termination as policy behaviors.
  • Compared with answer-only VideoQA evaluation, MINERVA-style traces let us check whether correct answers are also evidence-grounded.
  • Compared with public long-video curves that drop with duration, VLX-VR remains strong under MINERVA's original three duration groups.

Technology Lineage

Our team has spent years building multimodal perception and reasoning systems, with open-source projects such as OmDet, VLM-R1, VLX-Seek, VLX-Flow, and VLX-Go. VLX-VR extends this line from perception and streaming understanding toward agentic-aware video reasoning with explicit multimodal memory control.

Citation

@article{li2026vlx,
  title={VLX-VR: An Agentic-Aware Video Reasoning Model},
  author={Li, Sheng and Liu, Peng and Zhang, Qianqian and Zhao, Tiancheng},
  journal={arXiv preprint arXiv:2609.09985},
  year={2026}
}

License

This project is released under the Apache License 2.0. See the license file for terms covering code and model weights.

关于 About

VLX-VR: An Agentic-Aware Video Reasoning Model

语言 Languages

Jupyter Notebook100.0%

提交活跃度 Commit Activity

代码提交热力图
过去 52 周的开发活跃度
16
Total Commits
峰值: 16次/周
Less
More

核心贡献者 Contributors