Skip to content

OpenHallu

Multimodal Hallucination Hub

Home›Quantification›Overview
📊

Quantification Methods

Metrics and scorecards for multimodal hallucination research.

6Curated entries
4Categories
11Verified links
6Dated papers

Definitions, mappings, and short notes.

6 curated entries

⌕

Metric Categories

Curated Metrics

1

CHAIR

Classic object hallucination metric for captioning outputs.

captioningobject labelshallucination rate
Hallucination Rate
vision-language

CVPR 2018

First posted Sep 6, 2018

object hallucination
papergithub
Authors: Anna Rohrbach, Lisa Anne Hendricks, Kaylee BurnsCorresponding: not specifiedAffiliation: UC Berkeley; Boston University
2

POPE Score

Probe-based evaluation score for object perception hallucination.

object existencebinary probingF1 score
Faithfulness
vision-language

EMNLP 2023

First posted May 17, 2023

object perception probing
papergithub
Authors: Yifan Li, Yifan Du, Kun ZhouCorresponding: Wayne Xin ZhaoAffiliation: Renmin University of China; Meituan Group
3

FaithScore

Reference-free faithfulness score over verified atomic image facts.

atomic factsimage groundingreference-free
Faithfulness
vision-language

Findings of EMNLP 2024

First posted Nov 2, 2023

atomic fact precision
papergithub
Authors: Liqiang Jing, Ruosen Li, Yunmo ChenCorresponding: not specifiedAffiliation: University of Texas at Dallas; Johns Hopkins University
4

HaELM

A trained evaluator that scores hallucination in LVLM responses.

LLM evaluatorlocal evaluationhallucination scoring
Faithfulness
vision-language

arXiv 2023

First posted Aug 29, 2023

model-based scoring
papergithub
Authors: Junyang Wang, Yiyang Zhou, Guohai XuCorresponding: not specifiedAffiliation: Shandong University; Beijing Jiaotong University; Xi'an Jiaotong University
5

AMBER Score

A composite score over generative and discriminative hallucination dimensions.

existenceattributesrelations
Composite Score
vision-language

arXiv 2023

First posted Nov 13, 2023

existence / attribute / relation
papergithub
Authors: Junyang Wang, Yuhang Wang, Guohai XuCorresponding: not specifiedAffiliation: Beijing Jiaotong University; Alibaba Group
6

Uncertainty Estimation in Autoregressive Structured Prediction

A general ensemble-based framework for token-level and sequence-level uncertainty estimation in autoregressive structured prediction.

ensemble uncertaintytoken-level estimatessequence-level estimates
Calibration
vision-language

ICLR 2021

First posted Feb 18, 2020

autoregressive uncertainty
paper
Authors: Andrey Malinin, Mark GalesCorresponding: not specifiedAffiliation: Yandex; Higher School of Economics; University of Cambridge