IllusionIQ
Evaluating Multimodal LLMs on Optical Illusions – generation, benchmarking, and insights.
Spring 2025PythonPyTorchDiffusion ModelsHugging FaceOpenAI API+3 more
Overview
A CSE576 (NLP) course project exploring whether state-of-the-art multimodal LLMs can recognize and interpret optical illusions. We built a reproducible pipeline to generate 447 illusion pairs across six categories using diffusion models, then benchmarked GPT-4o, GPT-4.1, o4-mini, and Gemini 2.0 Flash with a standardized question set.
447
Illusion Pairs
6
Categories
4
Models Benchmarked
CSE524 (NLP)
Course
Illusion Categories
Color Hybrids
49
Flips
48
Jigsaw
35
Multi-Object Hybrids
49
Rotations
186
Text-Blend Hybrids
80
Sample Gallery
Benchmark Results
Correct mappings remain < 4% across models; high false-affirmation rates indicate overconfidence without precise transformation reasoning.
| Model | Total Images | Correct Mappings | False Affirmations | Q1 Negatives |
|---|---|---|---|---|
| Gemini 2.0 Flash | 447 | 12 (2.7%) | 410 (91.7%) | 25 (5.6%) |
| GPT-4o | 447 | 11 (2.5%) | 412 (92.2%) | 24 (5.4%) |
| o4-mini | 447 | 11 (2.5%) | 415 (92.9%) | 21 (4.7%) |
| GPT-4.1 | 447 | 17 (3.8%) | 403 (90.2%) | 27 (6.0%) |
Generation Pipeline
- Visual Anagrams & Factorized Diffusion (DeepFloyd IF) for anagrams / hybrids
- Illusion-Diffusion for text-blend hybrids
- Prompt-pair batching, negative prompts, guidance & noise tuning
- Manual curation and metadata for reproducibility
Evaluation Protocol
- Paired views per illusion (canonical + transformed)
- 5-question template probing ambiguity, category, and transformation mapping
- Metrics: correct mappings, false affirmations, Q1 negatives