LLM Figurative Understanding

A comprehensive evaluation framework for assessing Large Language Models' understanding and pragmatic use of figurative language across various cultural contexts in Arabic and English.

Setup

1. Clone the LM Evaluation Harness

In the root directory, clone the lm-eval-harness fork to access new tasks:

git clone https://github.com/menaattia/lm-evaluation-harness

2. Create Conda Environment

Create and activate the conda environment:

conda create -n vllm-eval python=3.10 -y
conda activate vllm-eval

3. Install Dependencies

pip install lm-eval==0.4.8
pip install --no-cache-dir vllm==0.9.0.1
pip install -U huggingface_hub
pip install transformers==4.52.4
pip install hf_transfer

4. Install LM Eval Harness in Development Mode

Navigate to the cloned lm-evaluation-harness directory and install in editable mode:

cd lm-evaluation-harness
pip install -e .
cd ..

5. Install API Libraries (for Claude and Gemini)

If you plan to evaluate Claude or Gemini models, install their respective API libraries:

pip install anthropic
pip install google-generativeai

6. Configure Hugging Face

Set up a writable Hugging Face cache directory:

export HF_HOME=/workspace/hf_cache
mkdir -p "$HF_HOME"

Login to Hugging Face:

huggingface-cli login
huggingface-cli whoami || true

Evaluation Tasks

All tasks are located in lm-evaluation-harness/lm-eval/tasks/jawaher/

Available Tasks

1. Pragmatic Use

   • prag_use
   • prag_use_proverbs

2. MCQ Understanding (MAPS Dataset)

   • maps_mcq - Without context
   • maps_mcq_context - With context

3. MCQ Understanding (Random Option Order with General Prompt)

   • figmcq_rand
   • figmcq_rand_idioms

4. MCQ Contextual Understanding

   • figmcq_rand_idioms_context

5. MCQ Understanding (Random Option Order with Semantic-Role-Labeling (SRL) Prompt)

   • figmcq_rand_srl
   • figmcq_rand_idioms_srl

6. Negation

   • negation_proverb
   • negation_idiom

7. Completion Tasks

   • complete - Jawaher proverbs
   • complete_maps - MAPS proverbs

8. Explanation Generation

   • figgen - Jawaher Arabic proverbs
   • figgen_idiom - Kinayat Arabic idioms

9. Connotation Labeling

   • connotation_idiom - Kinayat Arabic idioms
   • connotation_proverb - Jawaher Arabic proverbs

Note: For closed-source models that don't provide log-likelihoods, use the version of the task that contains gen in the name. For example, use prag_use_gen instead of prag_use. Check the jawaher folder for all available task files.

Evaluation Scripts

The repository includes three evaluation scripts for different model types:

• eval.sh - Evaluation of open-source models
• gpt_eval.sh - Evaluation of GPT models
• api_eval.sh - Evaluation of Claude and Gemini models

Paper

Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural Processing of Figurative Language

📄 Read on arXiv

Citation

If you use this work in your research, please cite:

@misc{attia2025understandingevaluatingpragmaticgap,
  title={Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural Processing of Figurative Language},
  author={Mena Attia and Aashiq Muhamed and Mai Alkhamissi and Thamar Solorio and Mona Diab},
  year={2025},
  eprint={2510.23828},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2510.23828}
}

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact

For questions, issues, or contributions, please contact the authors via email: [mena.attia@mbzuai.ac.ae]