vrlm-trajectories

dataset for creating RLM trajectories on video understanding tasks

LongVideoBench local smoke test

Print one normalized validation example. This downloads only the small LongVideoBench annotation files into data/raw/longvideobench.

uv run vrlm-trajectories longvideobench-sample --index 0

The full video archive is much larger and is an explicit opt-in:

uv run vrlm-trajectories longvideobench-download --archives
uv run vrlm-trajectories longvideobench-extract --subtitles --videos
uv run vrlm-trajectories longvideobench-sample --index 0 --require-video

For a smaller trajectory-generation subset, download source videos for the first 100 usable validation examples plus the official subtitle archive:

uv run vrlm-trajectories longvideobench-download-subset --target 100

Successful examples are written to a manifest such as data/raw/longvideobench/subsets/validation_100.jsonl. The default yt-dlp format avoids separate audio/video streams so it does not require ffmpeg. If ffmpeg is installed and you want higher-quality merged streams, pass a custom --format.

OpenAI backend

Set OPENAI_API_KEY in the process environment before running trajectory jobs. Optional machine-specific API settings can be exported the same way:

export OPENAI_API_KEY=sk-...
export OPENAI_BASE_URL=
export OPENAI_TIMEOUT_SECONDS=60

Model choices and RLM behavior live in YAML config files. The default is configs/default.yaml, with separate settings for the orchestrator model and the sub model used by generated-code helper calls.

from vrlm_trajectories.config import load_run_config

config = load_run_config("configs/default.yaml")
orchestrator, sub_model = config.create_openai_backends()

code = orchestrator.complete_text("Write Python code that answers the question.")
summary = sub_model.complete_multimodal(
    "Describe these frames.",
    frames,
    detail=config.sub_model.multimodal_detail,
)
orchestrator.write_call_log("data/outputs/openai_calls.jsonl")
sub_model.write_call_log("data/outputs/openai_calls.jsonl")