The implementation is now structurally compatible with:
- training on image-caption pairs from VRSBench,
- caption inference from a saved checkpoint.
Before these fixes, there were blockers that could degrade/disable proper captioning behavior (notably missing decoder masking behavior and missing inference entrypoint).
-
Model wiring
- BLIP forward path provides ITC, ITM, and caption logits.
- Caption decoder cross-attends to image tokens.
-
Dataset compatibility for VRSBench
- Reads annotation JSON files and resolves corresponding image files.
- Uses BERT tokenizer with fixed sequence length for batching.
-
Training script flow
- Teacher forcing for captioning (shifted decoder input/targets).
- ITM positive/negative construction via within-batch permutation.
- Mixed precision behavior now gated correctly for CUDA-only GradScaler.
-
Inference flow
- Added autoregressive decode loop that starts from
[CLS]and stops at[SEP].
- Added autoregressive decode loop that starts from
- Replaced custom attention internals with standard multi-head attention projections and explicit support for attention masks.
- Ensured decoder self-attention receives and applies a causal mask.
- Removed import-time debug execution from the attention module.
- Added configurable CLI arguments for training and checkpoint output.
- Added an inference CLI script for caption generation.
- Improved VRSBench dataset loader robustness (split validation, JSON filtering, max length parameter).
- Updated caption loss to ignore padding token index by default.
- This is a BLIP-like educational implementation, not a full reproduction of official BLIP pretraining recipes.
- No beam search/top-k/top-p decoding is implemented yet (current inference is greedy).
- True production-grade BLIP training typically benefits from larger-scale data pipelines and more complete masking/token handling.
In this environment, PyTorch runtime was unavailable for executing full forward/backward tests. Static Python compilation checks passed.