HallusionBench

A diagnostic benchmark for language hallucination and visual illusion.

POPE

Object perception benchmark for probing object hallucination.

MME

A comprehensive multimodal evaluation suite with perception tasks used for hallucination analysis.

CHAIR

Classic object hallucination metric and benchmark setup.

AMBER

A benchmark for multimodal hallucination evaluation across tasks.

Visually Dehallucinative Instruction Generation: Know What You Don't Know

Evaluates idk (ik) with Dis tasks using Acc metrics.

HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning

Evaluates synthetic images / vqa with Gen tasks using Acc metrics.

HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models

Evaluates speech/sound/music with Dis & Gen tasks using Acc / Hallucination Rate metrics.

Detecting and Evaluating Medical Hallucinations in Large Vision Language Models

Evaluates medical context with Dis & Gen tasks using MediHall Score metrics.

VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models

Evaluates temporal/extrinsic with Dis & Gen tasks using Acc metrics.

MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context

Evaluates medical context with Dis & Gen tasks using Characterization Score metrics.

EventHallusion: Diagnosing Event Hallucinations in Video LLMs

Evaluates video event with Dis & Gen tasks using Accuracy metrics.

Mitigating Dialogue Hallucination for Large Vision Language Models via Adversarial Instruction Tuning

Evaluates dialogue hallucination with Dis & Gen tasks using Acc metrics.

Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMs

Evaluates grounded medical with Gen tasks using Acc / Loc metrics.

MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models

Evaluates multi-image reasoning with Dis & Gen tasks using Acc metrics.

The Instinctive Bias: Spurious Images lead to Illusion in MLLMs

Evaluates spurious images with Dis tasks using Acc metrics.

Explore the Hallucination on Low-level Perception for MLLMs

Evaluates low-level perception with Gen tasks using Acc metrics.

JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images

Evaluates imaginary vqa with Dis & Gen tasks using Acc metrics.

Detecting and Preventing Hallucinations in Large Vision Language Models

Evaluates fine-grained with Dis tasks using Reward Score metrics.

Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models

Evaluates negative pronoun with Dis tasks using Acc/METEOR metrics.

THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models

Evaluates free-form generations with Gen tasks using P/R/F metrics.

Quantity Matters: Towards Assessing and Mitigating Number Hallucination in Large Vision-Language Models

Evaluates object counting with Dis tasks using Acc / Consistency metrics.

CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning

Evaluates multimodal hallucination with Dis tasks using Acc/P/R/F1/Specificity metrics.

Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models

Evaluates visual grounding with Dis tasks using Accuracy metrics.

Unified Hallucination Detection for Multimodal Large Language Models

Evaluates t2i with Gen tasks using Acc/P/R/F metrics.

Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models

Evaluates event hallucination with Dis & Gen tasks using Acc/Score metrics.

VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models

Evaluates multimodal hallucination with Gen tasks using Faithfulness & Coverage metrics.

AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models

Evaluates multimodal hallucination with Dis tasks using ASR/MASR/CASR metrics.

ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models

Evaluates open-set with Dis & Gen tasks using AMBER/Acc metrics.

Measuring the Measurers: Quality Evaluation of Hallucination Benchmarks for Large Vision-Language Models

Evaluates ocr/action/counting with Dis & Gen tasks using Hallucination Rate metrics.

FIHA: Autonomous Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphs

Evaluates eval method with Dis & Gen tasks using Acc/P/R/F1 metrics.

TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questions

Evaluates unanswerable qs with Dis tasks using Acc metrics.

MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Models

Evaluates manipulation/ooc/veracity with Dis & Gen tasks using Acc/F1 metrics.

Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs

Evaluates perturbed inputs with Dis tasks using Acc metrics.

CAST: Cross-modal Alignment Similarity Test for Vision Language Models

Evaluates self-consistency with Dis & Gen tasks using Similarity metrics.

FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models

Evaluates obj. counting with Gen tasks using FaithScore metrics.

Mitigating Open-Vocabulary Caption Hallucinations

Evaluates open-vocabulary with Gen tasks using OpenCHAIR metrics.

HallE-Control: Controlling Object Hallucination in Large Multimodal Models

Evaluates multimodal hallucination with Gen tasks using CHAIR/Coverage metrics.

ALOHa: A New Measure for Hallucination in Captioning Models

Evaluates captioning metric with Gen tasks using ALOHa metrics.

Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?

Evaluates metric method with Gen tasks using CHAIR-MEN / FaithScore metrics.

Aligning Large Multimodal Models with Factually Augmented RLHF

Evaluates model bias with Gen tasks using LLM Rating metrics.

Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

Evaluates env/action/comp with Gen tasks using LLM Rating metrics.

Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites

Evaluates behaviour with Dis tasks using Acc/P/R/F metrics.

Visual Hallucinations of Multi-modal Large Language Models

Evaluates visual hallucination with Dis & Gen tasks using Acc metrics.

PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset

Evaluates sentiment with Dis tasks using PhD Index metrics.

Automated Multi-level Preference for MLLMs

Evaluates multi-round conv with Gen tasks using Preference Score metrics.

Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models

Evaluates multimodal hallucination with Dis tasks using Acc/P/R/F1 metrics.

Multi-Object Hallucination in Vision-Language Models

Evaluates multi-object with Gen tasks using Accuracy metrics.

BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models

Evaluates before-after changes with Dis tasks using TU/IG/SB/ID metrics.

Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models

Evaluates perceptive / cognitive with Dis & Gen tasks using R-score metrics.

Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception

Evaluates cp-bench mitigation with Gen tasks using Acc metrics.

EH-Benchmark: Ophthalmic hallucination benchmark and agent-driven top-down traceable reasoning workflow

Evaluates ophthalmology with Dis & Gen tasks using N/A metrics.

Evaluation and Analysis of Hallucination in Large Vision-Language Models

Evaluates multimodal hallucination with Gen tasks using LLM Rating metrics.

Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges

Evaluates model bias with Gen tasks using Human Assessment metrics.

MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity

Evaluates instruction tuning with Gen tasks using Acc metrics.

MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation

Evaluates decoding method with Gen tasks using Acc metrics.

Mitigating Hallucination in Visual Language Models with Visual Supervision

Evaluates multimodal hallucination with Dis tasks using False Positive Rate metrics.

ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling

Evaluates grounding with Gen tasks using Fine-Grained Reward metrics.

Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Models

Evaluates unsolvable (upd) with Dis tasks using Acc metrics.

Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models

Evaluates generated distractions with Dis tasks using Acc metrics.

LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences

Evaluates societal bias/pref with Dis tasks using Alignment/Bias metrics.

How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts

Evaluates deceptive prompts with Gen tasks using Acc metrics.

VLind-Bench: Measuring Language Priors in Large Vision-Language Models

Evaluates language priors with Dis tasks using Acc (Y/N) metrics.

MMRel: Benchmarking Relation Understanding in Multi-Modal Large Language Models

Evaluates relation understanding with Dis & Gen tasks using Acc metrics.

Understanding Multimodal Hallucination with Parameter-Free Representation Alignment

Evaluates alignment method with Dis tasks using Pfram metrics.