Code and data for analysing narrative coherence in visually grounded stories from the Visual Writing Prompts (VWP) corpus. We compare human-written narratives with stories generated by five vision-language models (VLMs) using a set of metrics combined into a Narrative Coherence Score (NCS).
See portability-and-arrhenius.md.
The pipeline has six stages: (1) data preparation, (2) story generation, (3) post-processing, (4) coreference resolution, (5) metric computation, and (6) analysis. The sections below describe the first two stages and the human collection setup that supports them. Later stages will be documented separately.
We sampled 60 visual story sequences from VWP. Download the full story images and character images:
cd data
python download_data.py --csv-file ./vwp-acl2025-subset.csv --output-dir ./sampled_60Multiple prompt conditions are used. They differ in whether a target word count is specified and, when it is, whether the prompt asks for an approximate target or only an upper bound.
No explicit target length
- Short prompt (
data/prompts/prompt-original-w-names.txt): follows the original VWP data collection instructions. Human stories come from the VWP corpus, model stories are newly generated. Lower and upper bounds are specified. - Long prompt (
data/prompts/prompt-large-w-names.txt): provides more explicit guidance in prompt, focusing on tellability, coherence, and character consistency. Model stories are generated with this prompt, additional human stories were collected via Amazon Mechanical Turk. Only the lower bound is specified.
Target length specified
- Length-matched short prompt (
data/prompts/prompt-original-target-w-names.txt): extends the short prompt by specifying a sequence-specific target length (N) instead of the min/max number of words to be generated. For each visual sequence, (N) is the word count of the corresponding human-authored VWP story. Models are instructed to generate approximately (N) words. - Length-matched long prompt (
data/prompts/prompt-large-target-w-names.txt): extends the long prompt by specifying a sequence-specific target length (N) instead of the min/max number of words to be generated. Because three human-authored stories were collected for each visual sequence, (N) is defined as the median word count of those three stories. Models are instructed to generate approximately (N) words.
Upper bound specified
- Upper-bound long prompt (
data/prompts/prompt-large-upper-bound-w-names.txt): extends the long prompt by imposing a sequence-specific maximum length (N). As in the length-matched long condition, (N) is defined as the median word count of the three human-authored stories for the corresponding visual sequence. Models are instructed not to exceed (N) words. Unlike the length-matched condition, it does not require the generated story to be close to (N) words.
Stories were generated for all 60 sequences using five VLMs under all prompt conditions. Each model received the story image sequence together with character images and names.
| Model | Type | Instructions |
|---|---|---|
| InternVL3-78B | Open-source | models/internvl3/README.md |
| Qwen3-VL-235B | Open-source | models/qwen3vl/README.md |
| Llama-4-Scout | Open-source | models/llama4scout/README.md |
| Claude 4.5 Sonnet | Proprietary | models/claude/claude45.ipynb |
| GPT-4o | Proprietary | models/gpt/gpt4o.ipynb |
All models used temperature=0.6 and max_tokens=4096.
Long-prompt human stories were collected via AMT (3 descriptions per sequence, 180 total). See mturk/README.md for the full recruitment criteria, workflow, and payment details.
Further documentation for stages 3 onward will be added later.