Visual Grounding with Region-Text Alignment, Multitask SFT, and Verifier-Guided Self-Training

This project studies visual grounding through two complementary components. The first freezes GroundingDINO as a proposal generator and trains a lightweight feature-fusion reranker over cached region candidates. The second uses the same grounding data engine to build a three-task Qwen2.5-VL-3B instruction-tuning dataset, then performs rollout generation, IoU-verifier filtering, and rejection-sampling-augmented SFT.

Headline Results

Frozen Candidate Reranking

All reranker results below use fixed N=500 subsets of refcoco_unc_val, refcoco_unc_testA, and refcoco_unc_testB. The trainable rerankers reuse the same frozen candidate pools as the adaptive rule baseline, so candidate recall@5 is unchanged and the measured gains come from candidate ordering.

Method	Val	TestA	TestB
Adaptive rule reranker	65.4	63.6	59.8
Trainable multimodal fusion reranker	71.0	69.2	63.8
Improvement	+5.6pp	+5.6pp	+4.0pp

Paired improvement is statistically significant on all three splits under McNemar's test.

Split	Win	Loss	Tie	Net gain	McNemar p-value
Val	38	10	452	+28	9.73e-5
TestA	31	3	466	+28	3.65e-6
TestB	29	9	462	+20	2.05e-3

Results Visualization

Qwen2.5-VL Multitask SFT and Rejection-Sampling Augmentation

The generative model is evaluated separately on candidate selection, direct box prediction, and candidate verification. Task-specific sample counts differ from the fixed N=500 reranker subsets, so absolute values across the two result blocks should not be compared directly.

Rejection-sampling augmentation consistently improves candidate-level grounding over the first-round SFT model:

Split	N	Candidate Acc: SFT	Candidate Acc: RS	Delta	Selected-box Acc@0.5: SFT	Selected-box Acc@0.5: RS	Delta
Val	452	52.21	59.07	+6.86pp	57.52	64.38	+6.86pp
TestA	448	51.56	59.15	+7.59pp	54.46	61.16	+6.70pp
TestB	404	56.19	59.65	+3.47pp	60.40	65.10	+4.70pp

Paired McNemar tests show significant gains in selected_box_acc@0.5 on all three splits. Candidate-index accuracy is significant on Val and TestA and remains positive but not significant on TestB.

Split	Metric	Wins	Losses	Ties	McNemar p-value
Val	Candidate accuracy	47	16	389	1.17055e-4
Val	Selected-box Acc@0.5	50	19	383	2.44397e-4
TestA	Candidate accuracy	54	20	374	9.61299e-5
TestA	Selected-box Acc@0.5	53	23	372	7.64582e-4
TestB	Candidate accuracy	35	21	348	8.14268e-2
TestB	Selected-box Acc@0.5	33	14	357	7.94273e-3

Qwen Experiment Visualization

Qwen Qualitative Method Effects

Green denotes the ground-truth box. For direct box prediction, red denotes the Base output and blue denotes the SFT output. For candidate selection, red denotes the first-round SFT selection and blue denotes the RS-augmented SFT selection. All examples are selected from held-out RefCOCO validation or test splits.

Scope and Non-Goals

The implemented system contains:

frozen GroundingDINO proposal generation
cached region-candidate construction
candidate-level multimodal feature fusion
supervised listwise ranking
IoU-derived hard-negative and pairwise-ranking ablations
Qwen2.5-VL-3B multitask grounding SFT
model rollout generation and IoU-verifier filtering
rejection-sampling-augmented SFT
held-out evaluation with paired significance testing

The current implementation does not perform image-text pretraining, standard DPO/RLHF, or end-to-end visual-backbone fine-tuning.

Problem Formulation

The first part of the project starts from a fixed candidate pool. With union proposals, the correct box is often already present among the top candidates; the remaining failure mode is that it is not ranked first.

This motivates the first research question:

Given a frozen proposal generator and a fixed candidate pool, can a lightweight multimodal reranker learn better region-text alignment than a hand-tuned adaptive rule system?

The second part asks whether the grounding system can be extended from candidate scoring to multimodal instruction tuning:

Can a vision-language model learn candidate selection, direct box prediction, and candidate verification jointly, and can IoU-verifier-guided rejection sampling improve held-out grounding performance?

System Overview

Image + referring expression
        |
        v
Frozen GroundingDINO proposal generator
        |
        v
Union candidate pool + cached grounding metadata
        |                                \
        |                                 \
        v                                  v
Trainable feature-fusion MLP       Qwen2.5-VL multitask SFT
        |                                  |
        v                                  v
Candidate reranking               Model rollouts
                                           |
                                           v
                                  IoU / format verifier
                                           |
                                           v
                                  Accepted pseudo-labels
                                           |
                                           v
                                  RS-augmented SFT

Frozen Candidate Reranker

Candidate-Level Representation

Each candidate is represented by a fused feature vector combining:

region-text semantic score
detector confidence
spatial relation score
box geometry
expression semantic attributes
query-source indicators

Concretely, the trainable reranker uses:

detector score
local region-text CLIP similarity
spatial score
adaptive reranker score
normalized box coordinates
center, width, height, area, aspect ratio, distance-to-center
expression length
has_spatial, has_color, has_size, has_relation
query type: full expression vs head-like vs other

The current reported model does not use a cached global image-text feature. A reserved slot remains inactive and is recorded as unavailable in the training manifest.

Model and Objectives

The trainable reranker is a lightweight feature-level fusion head. The reported best model includes the cached adaptive score among its inputs, so it learns to reweight and combine the existing signals rather than discarding the rule baseline entirely:

Candidate feature vector
    -> Linear(input_dim, 128)
    -> GELU
    -> Dropout(0.1)
    -> Linear(128, 1)
    -> Candidate ranking scores

The training objectives explored in this repository are:

A. Feature Fusion

Masked listwise cross-entropy over all candidates for the same referring expression.

B. Same-Image Hard Negatives

Adds a harder training subset by selecting same-image false candidates with:

high detector confidence
high local text similarity
low IoU to the target
optional label / head-noun consistency cues when available

C. Pairwise Preference Ranking

Adds an IoU-verifier-derived pairwise hinge loss on top of the listwise objective. This is preference-style supervision, but it is not standard DPO or RLHF.

Candidate Training Data

Training supervision is built automatically from cached candidates:

Positive: highest-IoU candidate with IoU >= 0.5
Hard negatives: same expression, same image, IoU <= 0.3, positive-negative IoU gap >= 0.3
Up to 3 hard negatives per sample

Samples without a candidate reaching the positive IoU threshold are excluded from reranker training but remain available for proposal-recall analysis.

Qwen2.5-VL Multitask Grounding

Task Definitions

The instruction-tuning dataset contains three tasks in a unified JSONL format:

candidate_selection: select the correct numbered candidate.
box_prediction: output a direct bounding box in <box>x1,y1,x2,y2</box> format.
candidate_verification: answer yes or no for a highlighted candidate.

Each record contains:

source_id, split, task_type, image_id, image_path,
expression, prompt, answer, metadata

The metadata stores the ground-truth box, candidate boxes, candidate IoUs, correct candidate index, and verification label required by training and evaluation.

First-Round SFT

The first-round SFT model is trained on 2,482 examples:

Task	Train rows	Validation rows
Box prediction	745	271
Candidate selection	1,241	452
Candidate verification	496	181
Total	2,482	904

Training uses LoRA on Qwen/Qwen2.5-VL-3B-Instruct with:

trainable parameters: 29,933,568
total parameters: 2,075,387,904
trainable ratio: 1.4423%
one training epoch

The logged validation loss from the original run is kept for reproducibility, but the main comparisons use task-specific generation metrics rather than loss values.

Rollout Filtering and Rejection Sampling

The first-round SFT model generates 7,944 rollouts from the training prompts. The verifier applies task-specific rules:

candidate_selection: the generated index must be valid and its candidate IoU must be at least 0.5.
box_prediction: the box must be parseable and its IoU must be at least 0.5.
for repeated rollouts of the same prompt, only the accepted output with the highest IoU is retained.

The filtering summary is:

Statistic	Value
Total rollouts	7,944
Format-valid rate	92.47%
Accepted samples	989
Acceptance rate	12.45%
Mean accepted IoU	0.8875
Maximum accepted IoU	1.0000

Rejection reasons:

Reason	Count
Low IoU	4,783
Deduplicated lower-IoU accepted rollout	2,563
Invalid candidate index	572
Invalid box format	26

The accepted pseudo-labels are mixed with the original SFT data to form a 3,471-row training set:

Task	Mixed-train rows
Box prediction	1,005
Candidate selection	1,970
Candidate verification	496
Total	3,471

The RS-augmented model is trained from the same Qwen base checkpoint on this mixed dataset rather than by continuing from the first SFT adapter.

Evaluation Protocol

Candidate Reranker

The reranker experiments use fixed N=500 subsets from:

refcoco_unc_val
refcoco_unc_testA
refcoco_unc_testB

Metrics:

Acc@0.5
mean IoU
candidate recall@5
paired win / loss / tie
McNemar significance test

The proposal pool is identical across rerankers, so changes in Acc@0.5 measure candidate-ordering quality rather than proposal recall.

Qwen Grounding

The Qwen evaluation uses the following task-specific sample counts:

Split	Candidate selection	Box prediction	Candidate verification
Val	452	271	181
TestA	448	269	179
TestB	404	242	162

Metrics:

Candidate selection: parse-valid rate, candidate accuracy, selected-box Acc@0.5, selected-box mean IoU
Box prediction: format-valid rate, Acc@0.5, mean IoU
Candidate verification: accuracy, precision, recall, F1

For direct box prediction, generated pixel-space coordinates are converted using the original image width and height into the 0-1000 coordinate system before IoU is computed against gt_box_1000.

Qualitative Examples

Green denotes the ground-truth box, red denotes the adaptive reranker prediction, and blue denotes the trainable feature-fusion prediction. All examples are selected from held-out RefCOCO testA/testB samples.

Main Findings

1. Trainable feature fusion improves ranking with a fixed proposal pool

Method	Val	TestA	TestB
Adaptive rule reranker	65.4	63.6	59.8
MLP feature fusion	71.0	69.2	63.8
MLP + hard negatives	68.6	67.0	62.4
MLP + hard negatives + pairwise loss	69.4	68.4	62.6

The best-performing recipe is the simple listwise feature-fusion model. Hard-negative and pairwise variants remain above the adaptive baseline but do not exceed feature fusion.

2. Multitask SFT strongly improves direct box prediction

Split	Base Acc@0.5	SFT Acc@0.5	RS Acc@0.5	Base mean IoU	SFT mean IoU	RS mean IoU
Val	70.11	87.82	88.56	62.33	78.85	76.76
TestA	70.26	91.82	90.33	63.47	81.48	78.63
TestB	62.40	77.27	74.79	58.40	70.89	67.75

SFT also raises box-format validity from 75.65/76.21/78.51% for the base model to 100.00/100.00/99.59% on Val/TestA/TestB.

3. Rejection-sampling augmentation improves candidate-level grounding

Candidate accuracy improves from 52.21/51.56/56.19% to 59.07/59.15/59.65% on Val/TestA/TestB. Selected-box Acc@0.5 improves from 57.52/54.46/60.40% to 64.38/61.16/65.10%.

The selected-box gains are significant under paired McNemar tests on all three splits. Candidate-index accuracy is significant on Val and TestA, while TestB shows a positive +3.47pp change with p=0.0814.

4. Self-training produces a task-specific trade-off

The RS-augmented model improves candidate selection on every split and raises candidate-verification F1:

Split	Base F1	SFT F1	RS F1
Val	6.38	29.57	30.77
TestA	10.10	28.10	33.33
TestB	13.64	31.78	43.20

However, direct box-prediction mean IoU decreases from SFT to RS by 2.09pp, 2.85pp, and 3.14pp on Val, TestA, and TestB. The result is therefore a task-specific trade-off rather than a uniform improvement across all objectives.

5. Reranker gains are strongest on hard grounding subproblems

Feature fusion improves most on spatial expressions, same-category multi-object disambiguation, and color / attribute-heavy expressions on Val and TestA.

Split	Spatial Delta Acc	Color / Attribute Delta Acc	Same-Category Multi-Object Delta Acc
Val	+7.98pp	+9.30pp	+6.28pp
TestA	+4.44pp	+3.85pp	+5.91pp
TestB	+5.01pp	+0.99pp	+4.60pp

Limitations

The MLP reranker operates on cached scores and metadata rather than token-level cross-attention over raw visual and text tokens.
The best MLP includes the adaptive score as an input feature, so it should be interpreted as learned stacking over baseline signals rather than a complete replacement of the rule system.
Proposal recall remains an upper bound for candidate-based methods.
The reranker headline results use fixed N=500 subsets rather than the complete RefCOCO splits.
The Qwen task datasets contain different numbers of candidate-selection, box-prediction, and verification examples; their absolute metrics are not directly interchangeable.
The RS-augmented model uses both more training rows and more optimization steps than the first-round SFT model (3,471 rows / 434 steps versus 2,482 rows / 311 steps). A matched-step control is not reported.
Rejection-sampling augmentation improves candidate-level tasks but slightly reduces direct box-prediction mean IoU.
Standard DPO/RLHF is not implemented; pairwise reranker supervision is a hinge-ranking ablation rather than DPO.

Repository Layout

Key candidate-reranking components:

src/grounding.py: frozen GroundingDINO proposal generation
src/rerank.py: rule-based adaptive reranking
src/mlp_reranker.py: trainable feature-fusion head and ranking objectives
scripts/run_clip_rerank.py: candidate generation and adaptive reranking
scripts/run_mlp_reranker.py: MLP training and evaluation
scripts/compare_reranker_upgrade.py: reranker comparison and paired reporting
experiments/run_lightweight_reranker_upgrade.sh: reproducible reranker pipeline

Key Qwen grounding components:

configs/qwen25vl_grounding_sft.yaml: Qwen grounding training configuration
src/vlm_grounding.py: data, parsing, coordinate conversion, and grounding utilities
scripts/build_grounding_sft_data.py: multitask data construction
scripts/train_grounding_sft.py: LoRA SFT training
scripts/generate_grounding_rollouts.py: rollout generation
scripts/build_rejection_sampling_data.py: IoU-verifier filtering and mixed-data construction
scripts/eval_grounding_vlm.py: task-specific grounding evaluation
scripts/build_qwen_grounding_comparison.py: Base/SFT/RS comparison tables
scripts/paired_candidate_stats.py: paired McNemar statistics for candidate selection
experiments/run_qwen_grounding_sft_selftrain.sh: end-to-end Qwen training and evaluation workflow

Reproducing the Candidate-Reranker Result

1. Install

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

If you are on a Blackwell GPU, use:

bash scripts/install_torch_blackwell.sh

2. Prepare or reuse cached candidates

python -m scripts.prepare_refcoco_subset \
  --config configs/default.yaml \
  --split refcoco_unc_train \
  --limit 2000 \
  --output data/outputs/cache_refcoco_unc_train_2000/subset.jsonl

python -m scripts.run_clip_rerank \
  --config configs/default.yaml \
  --input data/outputs/cache_refcoco_unc_train_2000/subset.jsonl \
  --candidate_source union \
  --rerank_mode adaptive \
  --output data/outputs/cache_refcoco_unc_train_2000/union_adaptive_rerank_predictions.jsonl

3. Train the best-performing reranker

python -m scripts.run_mlp_reranker \
  --recipe feature_fusion \
  --train_predictions data/outputs/cache_refcoco_unc_train_2000/union_adaptive_rerank_predictions.jsonl \
  --val_predictions data/outputs/rerank_adaptive_500.jsonl \
  --checkpoint data/outputs/lightweight_reranker_upgrade/feature_fusion/checkpoint.pt \
  --last_checkpoint data/outputs/lightweight_reranker_upgrade/feature_fusion/last_checkpoint.pt \
  --val_output data/outputs/lightweight_reranker_upgrade/feature_fusion/val_predictions.jsonl

4. Evaluate on held-out splits

python -m scripts.run_mlp_reranker \
  --eval_only \
  --recipe feature_fusion \
  --checkpoint data/outputs/lightweight_reranker_upgrade/feature_fusion/checkpoint.pt \
  --val_predictions data/outputs/rerank_adaptive_500.jsonl \
  --val_output data/outputs/lightweight_reranker_upgrade/feature_fusion/val_reload_predictions.jsonl \
  --test_output_dir data/outputs/lightweight_reranker_upgrade/feature_fusion \
  --test_predictions \
    data/outputs/holdout_refcoco_unc_testA/union_adaptive_rerank_predictions.jsonl \
    data/outputs/holdout_refcoco_unc_testB/union_adaptive_rerank_predictions.jsonl

5. Generate the reranker comparison

python -m scripts.compare_reranker_upgrade \
  --adaptive_val data/outputs/rerank_adaptive_500.jsonl \
  --mlp_val data/outputs/lightweight_reranker_upgrade/feature_fusion/val_predictions.jsonl \
  --adaptive_testA data/outputs/holdout_refcoco_unc_testA/union_adaptive_rerank_predictions.jsonl \
  --mlp_testA data/outputs/lightweight_reranker_upgrade/feature_fusion/holdout_refcoco_unc_testA_union_mlp_feature_fusion_predictions.jsonl \
  --adaptive_testB data/outputs/holdout_refcoco_unc_testB/union_adaptive_rerank_predictions.jsonl \
  --mlp_testB data/outputs/lightweight_reranker_upgrade/feature_fusion/holdout_refcoco_unc_testB_union_mlp_feature_fusion_predictions.jsonl \
  --output data/outputs/lightweight_reranker_upgrade/comparison.json

Reproducing the Qwen SFT and Rejection-Sampling Pipeline

1. Build the multitask dataset

python -m scripts.build_grounding_sft_data \
  --train_cache data/outputs/cache_refcoco_unc_train_2000/union_adaptive_rerank_predictions.jsonl \
  --val_cache data/outputs/rerank_adaptive_500.jsonl \
  --testA_cache data/outputs/holdout_refcoco_unc_testA/union_adaptive_rerank_predictions.jsonl \
  --testB_cache data/outputs/holdout_refcoco_unc_testB/union_adaptive_rerank_predictions.jsonl \
  --output_dir data/processed/grounding_sft

2. Train the first-round SFT model

python -m scripts.train_grounding_sft \
  --config configs/qwen25vl_grounding_sft.yaml \
  --output_dir outputs/checkpoints/qwen25vl_grounding_sft

3. Generate rollouts

python -m scripts.generate_grounding_rollouts \
  --dataset data/processed/grounding_sft/train.jsonl \
  --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \
  --adapter_path outputs/checkpoints/qwen25vl_grounding_sft \
  --output_jsonl data/processed/grounding_selftrain/raw_rollouts.jsonl \
  --manifest_json data/processed/grounding_selftrain/raw_rollouts_manifest.json

4. Build the rejection-sampling dataset

python -m scripts.build_rejection_sampling_data \
  --train_jsonl data/processed/grounding_sft/train.jsonl \
  --rollouts_jsonl data/processed/grounding_selftrain/raw_rollouts.jsonl \
  --output_dir data/processed/grounding_selftrain

5. Train the RS-augmented model

python -m scripts.train_grounding_sft \
  --config configs/qwen25vl_grounding_sft.yaml \
  --train_jsonl data/processed/grounding_selftrain/mixed_train.jsonl \
  --output_dir outputs/checkpoints/qwen25vl_grounding_selftrain

6. Evaluate Base, SFT, and RS-Augmented SFT

mkdir -p outputs/evals/qwen_grounding/{base,sft,rs_augmented_sft}

for split in val testA testB
do
  dataset="data/processed/grounding_sft/${split}.jsonl"

  python -m scripts.eval_grounding_vlm \
    --dataset "$dataset" \
    --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \
    --variant_name "base_${split}" \
    --output_dir "outputs/evals/qwen_grounding/base/${split}"

  python -m scripts.eval_grounding_vlm \
    --dataset "$dataset" \
    --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \
    --adapter_path outputs/checkpoints/qwen25vl_grounding_sft \
    --variant_name "sft_${split}" \
    --output_dir "outputs/evals/qwen_grounding/sft/${split}"

  python -m scripts.eval_grounding_vlm \
    --dataset "$dataset" \
    --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \
    --adapter_path outputs/checkpoints/qwen25vl_grounding_selftrain \
    --variant_name "rs_augmented_sft_${split}" \
    --output_dir "outputs/evals/qwen_grounding/rs_augmented_sft/${split}"
done

7. Build comparison and paired statistics

python -m scripts.build_qwen_grounding_comparison \
  --base_val outputs/evals/qwen_grounding/base/val/base_val_metrics.json \
  --sft_val outputs/evals/qwen_grounding/sft/val/sft_val_metrics.json \
  --selftrain_val outputs/evals/qwen_grounding/rs_augmented_sft/val/rs_augmented_sft_val_metrics.json \
  --base_testA outputs/evals/qwen_grounding/base/testA/base_testA_metrics.json \
  --sft_testA outputs/evals/qwen_grounding/sft/testA/sft_testA_metrics.json \
  --selftrain_testA outputs/evals/qwen_grounding/rs_augmented_sft/testA/rs_augmented_sft_testA_metrics.json \
  --base_testB outputs/evals/qwen_grounding/base/testB/base_testB_metrics.json \
  --sft_testB outputs/evals/qwen_grounding/sft/testB/sft_testB_metrics.json \
  --selftrain_testB outputs/evals/qwen_grounding/rs_augmented_sft/testB/rs_augmented_sft_testB_metrics.json \
  --output_dir outputs/comparisons/qwen_grounding_sft_selftrain

python -m scripts.paired_candidate_stats

Reproducing the Figures

python -m scripts.build_readme_figures \
  --comparison data/outputs/lightweight_reranker_upgrade/comparison.json \
  --adaptive_testA data/outputs/holdout_refcoco_unc_testA/union_adaptive_rerank_predictions.jsonl \
  --mlp_testA data/outputs/lightweight_reranker_upgrade/feature_fusion/holdout_refcoco_unc_testA_union_mlp_feature_fusion_predictions.jsonl \
  --adaptive_testB data/outputs/holdout_refcoco_unc_testB/union_adaptive_rerank_predictions.jsonl \
  --mlp_testB data/outputs/lightweight_reranker_upgrade/feature_fusion/holdout_refcoco_unc_testB_union_mlp_feature_fusion_predictions.jsonl \
  --output_dir assets/readme

python -m scripts.build_qwen_readme_figures \
  --comparison outputs/comparisons/qwen_grounding_sft_selftrain/comparison.json \
  --paired_stats outputs/comparisons/qwen_grounding_sft_selftrain/paired_candidate_stats.json \
  --rollout_manifest data/processed/grounding_selftrain/rollout_manifest.json \
  --sft_manifest outputs/checkpoints/qwen25vl_grounding_sft/train_manifest.json \
  --output_dir assets/readme

python -m scripts.build_qwen_effect_examples \
  --base_dir outputs/evals/qwen_grounding/base \
  --sft_dir outputs/evals/qwen_grounding/sft \
  --rs_dir outputs/evals/qwen_grounding/rs_augmented_sft \
  --output_dir assets/readme

Conclusion

With the proposal generator and candidate pools held fixed, the trainable feature-fusion reranker improves Acc@0.5 by +5.6pp, +5.6pp, and +4.0pp on the reported Val, TestA, and TestB subsets. Multitask SFT substantially improves Qwen direct box prediction and output validity, while IoU-verifier-guided rejection-sampling augmentation raises candidate accuracy by +6.86pp, +7.59pp, and +3.47pp and selected-box Acc@0.5 by +6.86pp, +6.70pp, and +4.70pp on Val, TestA, and TestB. The selected-box improvements are significant on all three splits under paired McNemar tests. The RS-augmented model slightly reduces direct box-prediction mean IoU, indicating a task-specific trade-off rather than uniform improvement across every objective.

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
assets/readme		assets/readme
configs		configs
data		data
experiments		experiments
scripts		scripts
src		src
tests		tests
.DS_Store		.DS_Store
.gitignore		.gitignore
Last.code-workspace		Last.code-workspace
README.md		README.md
pip.conf.example		pip.conf.example
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation