Detection · Vision-Language Models · Fusion

TianWen 天问

A Universal Training Framework for Detection-VLM Fusion

Peifu Zhang ¹

¹ Xidian University

Paper Soon
arXiv Soon
Code
Video Soon
BibTeX

Apache 2.0 Python 3.9 – 3.12 PyTorch Lightning Hydra

One framework, three fusion strategies — pluggable detectors and VLMs.

Abstract

One framework for Detection × VLM

TianWen is a modular, extensible framework for combining object detection models with Vision-Language Models (VLMs) to improve detection performance through configurable fusion strategies. It treats detectors and VLMs as interchangeable building blocks: any registered detector — YOLOv8, YOLOv11, RT-DETR, RF-DETR, or Grounding-DINO — can be paired with any registered VLM — Qwen2-VL or InternVL3 — under a single training pipeline.

Three fusion strategies cover the practical design space: knowledge distillation (VLM-as-teacher), feature fusion (VLM features injected into the detector's backbone, neck, or head), and decision fusion (VLM verifies and refines detector outputs). Configuration is fully hierarchical via Hydra, training runs on PyTorch Lightning with multi-GPU support, and every component is registered through a plug-in registry so new models or fusion modes drop in with a single decorator.

Approach

Three fusion strategies

TianWen treats detector–VLM coupling as a design space. Pick the strategy that fits your latency budget, dataset size, and the modality of supervision you can afford.

01 Strategy 1

Knowledge Distillation

VLM as teacher, detector as student

The VLM emits soft targets — feature maps, logits, or response distributions — that supervise a lightweight detector. The student inherits semantic richness without paying VLM inference cost at deployment.

featurelogitresponse

fusion:
  type: distillation
  distill_mode: feature
  temperature: 4.0
  alpha: 0.5

02 Strategy 2

Feature Fusion

Inject VLM features into the detector

VLM visual features merge into the detector’s feature pyramid at the backbone, neck, or head via cross-attention, an adapter, or simple concatenation. The detector keeps its speed; the VLM lends its semantics.

backboneneckhead

fusion:
  type: feature_fusion
  fusion_level: neck
  fusion_type: cross_attention

03 Strategy 3

Decision Fusion

VLM verifies and refines detections

The detector proposes candidates; the VLM acts as a verifier — accepting, rescoring, or reclassifying each box. Useful when the detector is fast and the VLM is precise on edge cases.

binaryconfidencereclassify

fusion:
  type: decision_fusion
  verification_mode: confidence
  score_adjustment: 0.3

Pluggable models

Detectors and VLMs supported out of the box

Every model is registered through a decorator; adding a new one means writing a wrapper and one line of registration code.

Detectors

YOLOv8

Real-time

Anchor-free
YOLOv11

Real-time

Latest Ultralytics
RT-DETR

Transformer

End-to-end
RF-DETR

Transformer

Roboflow
Grounding-DINO

Open-vocab

Zero-shot

Vision-Language Models

Qwen2-VL

Alibaba

Multimodal
InternVL3

OpenGVLab

High-res visual

Quick start

From clone to first training run

bash install

# Clone & install (editable mode)
git clone https://github.com/Hollis36/TianWen.git
cd TianWen
pip install -e .

# Optional extras
pip install -e ".[dev]"      # development tools
pip install -e ".[rfdetr]"   # RF-DETR support

bash train.sh

# Train with a pre-defined experiment
python tools/train.py experiment=yolov8_qwen_distill

# Compose strategies from the command line
python tools/train.py \
    detector=yolov8 \
    vlm=qwen_vl \
    fusion=distillation \
    train.learning_rate=1e-4 \
    train.batch_size=16

bash infer.sh

# Inference on a single image
python tools/demo.py \
    --checkpoint path/to/checkpoint.ckpt \
    --image path/to/image.jpg \
    --output result.jpg

# Quick benchmark
python tools/benchmark.py --quick

Architecture

Configs in, trained models out

Hierarchical configs flow into a Lightning trainer that dispatches to plug-in modules.

Results

Benchmarks

Benchmark results — coming soon

Comparison tables across COCO and domain-specific benchmarks — pure detectors vs. each fusion strategy, latency-vs-accuracy curves, and ablations — will land here once final runs complete.

Method	mAP@50	mAP@50:95	FPS
Detector only	— pending —
+ Distillation	— pending —
+ Feature Fusion	— pending —
+ Decision Fusion	— pending —

Citation

If you use TianWen in your work

A formal preprint and BibTeX will be published with the upcoming paper. In the meantime, please cite the project page:

BibTeX

@misc{zhang2026tianwen,
  title  = {TianWen: A Universal Training Framework for Detection-VLM Fusion},
  author = {Zhang, Peifu},
  year   = {2026},
  note   = {Project page: https://hollis36.github.io/tianwen-project-page/},
  url    = {https://github.com/Hollis36/TianWen}
}