EvalCards: A Framework for Standardized Evaluation Reporting

Ruchira Dhar; Danae Sanchez Villegas; Antonia  Karamolegkou; Alice  Schiavone; Yifei  Yuan; Xinyi  Chen; Jiaang  Li; Stella  Frank; Laura  De Grazia; Monorama Swain; Stephanie  Brandl; Daniel  Hershcovich; Anders  Søgaard; Desmond  Elliott

doi:10.48550/arXiv.2511.21695

EvalCards: A Framework for Standardized Evaluation Reporting

Ruchira Dhar
, Danae Sanchez Villegas
, Antonia Karamolegkou
, Alice Schiavone
, Yifei Yuan
, Xinyi Chen
, Jiaang Li
, Stella Frank
, Laura De Grazia
, Monorama Swain
, Stephanie Brandl
, Daniel Hershcovich
, Anders Søgaard
, Desmond Elliott

Institute of Computational Perception

Research output: Working paper and reports › Preprint

Abstract

Evaluation has long been a central concern in
NLP, and transparent reporting practices are
more critical than ever in today’s landscape of
rapidly released open-access models. Draw-
ing on a survey of recent work on evaluation
and documentation, we identify three persis-
tent shortcomings in current reporting prac-
tices: reproducibility, accessibility, and gover-
nance. We argue that existing standardization
efforts remain insufficient and introduce Evalu-
ation Disclosure Cards (EvalCards) as a path
forward. EvalCards are designed to enhance
transparency for both researchers and practi-
tioners while providing a practical foundation
to meet emerging governance requirements.

Original language	English
DOIs	https://doi.org/10.48550/arXiv.2511.21695
Publication status	Published - 2025

Fields of science

102003 Image processing
202002 Audiovisual media
102001 Artificial intelligence
102015 Information systems
102 Computer Sciences

JKU Focus areas

Digital Transformation

Access to Document

10.48550/arXiv.2511.21695

Cite this

@techreport{8667a30f98f74902bc24e7cda5d4b1ad,

title = "EvalCards: A Framework for Standardized Evaluation Reporting",

abstract = "Evaluation has long been a central concern in NLP, and transparent reporting practices are more critical than ever in today{\textquoteright}s landscape of rapidly released open-access models. Draw- ing on a survey of recent work on evaluation and documentation, we identify three persis- tent shortcomings in current reporting prac- tices: reproducibility, accessibility, and gover- nance. We argue that existing standardization efforts remain insufficient and introduce Evalu- ation Disclosure Cards (EvalCards) as a path forward. EvalCards are designed to enhance transparency for both researchers and practi- tioners while providing a practical foundation to meet emerging governance requirements.",

author = "Ruchira Dhar and Villegas, \{Danae Sanchez\} and Antonia Karamolegkou and Alice Schiavone and Yifei Yuan and Xinyi Chen and Jiaang Li and Stella Frank and \{De Grazia\}, Laura and Monorama Swain and Stephanie Brandl and Daniel Hershcovich and Anders S{\o}gaard and Desmond Elliott",

year = "2025",

doi = "10.48550/arXiv.2511.21695",

language = "English",

type = "WorkingPaper",

}

TY - UNPB

T1 - EvalCards: A Framework for Standardized Evaluation Reporting

AU - Dhar, Ruchira

AU - Villegas, Danae Sanchez

AU - Karamolegkou, Antonia

AU - Schiavone, Alice

AU - Yuan, Yifei

AU - Chen, Xinyi

AU - Li, Jiaang

AU - Frank, Stella

AU - De Grazia, Laura

AU - Swain, Monorama

AU - Brandl, Stephanie

AU - Hershcovich, Daniel

AU - Søgaard, Anders

AU - Elliott, Desmond

PY - 2025

Y1 - 2025

N2 - Evaluation has long been a central concern in NLP, and transparent reporting practices are more critical than ever in today’s landscape of rapidly released open-access models. Draw- ing on a survey of recent work on evaluation and documentation, we identify three persis- tent shortcomings in current reporting prac- tices: reproducibility, accessibility, and gover- nance. We argue that existing standardization efforts remain insufficient and introduce Evalu- ation Disclosure Cards (EvalCards) as a path forward. EvalCards are designed to enhance transparency for both researchers and practi- tioners while providing a practical foundation to meet emerging governance requirements.

AB - Evaluation has long been a central concern in NLP, and transparent reporting practices are more critical than ever in today’s landscape of rapidly released open-access models. Draw- ing on a survey of recent work on evaluation and documentation, we identify three persis- tent shortcomings in current reporting prac- tices: reproducibility, accessibility, and gover- nance. We argue that existing standardization efforts remain insufficient and introduce Evalu- ation Disclosure Cards (EvalCards) as a path forward. EvalCards are designed to enhance transparency for both researchers and practi- tioners while providing a practical foundation to meet emerging governance requirements.

U2 - 10.48550/arXiv.2511.21695

DO - 10.48550/arXiv.2511.21695

M3 - Preprint

BT - EvalCards: A Framework for Standardized Evaluation Reporting

ER -