1<!doctype html><!-- This site was created with Hugo Blox. https://hugoblox.com --><!-- Last Published: September 8, 2026 --><html lang=en-us><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><meta http-equiv=X-UA-Compatible content="IE=edge"><meta name=generator content="Hugo Blox Builder 5.9.7"><link rel=preconnect href=https://fonts.gstatic.com crossorigin><link rel=preload as=style href="https://fonts.googleapis.com/css2?family=Montserrat:wght@400;700&family=Roboto+Mono&family=Roboto:wght@400;700&display=swap"><link rel=stylesheet href="https://fonts.googleapis.com/css2?family=Montserrat:wght@400;700&family=Roboto+Mono&family=Roboto:wght@400;700&display=swap" media=print onload='this.media="all"'><link rel=stylesheet href=/css/vendor-bundle.min.26c458e6907dc03073573976b7f4044e.css media=print onload='this.media="all"'><link rel=stylesheet href=https://cdn.jsdelivr.net/gh/jpswalsh/[email protected]/css/academicons.min.css integrity="sha512-IW0nhlW5MgNydsXJO40En2EoCkTTjZhI3yuODrZIc8cQ4h1XcF53PsqDHa09NqnkXuIe0Oiyyj171BqZFwISBw==" crossorigin=anonymous media=print onload='this.media="all"'><link rel=stylesheet href=https://cdn.jsdelivr.net/npm/[email protected]/dist/leaflet.min.css integrity crossorigin=anonymous media=print onload='this.media="all"'><link rel=stylesheet href=/css/wowchemy.5691485ce8e88de823038b8e3d90ec13.css><link rel=stylesheet href=/css/libs/chroma/github-light.min.css title=hl-light media=print onload='this.media="all"'><link rel=stylesheet href=/css/libs/chroma/dracula.min.css title=hl-dark media=print onload='this.media="all"' disabled>
vendor: 63 bytes, line 1
1<script async src="https://www.googletagmanager.com/gtag/js?id=
1G-M1JBEB5F0H
vendor: 11 bytes, line 1
1"></script>
1<script>window.dataLayer=window.dataLayer||[];function gtag(){dataLayer.push(arguments)}function trackOutboundLink(e,t){gtag("event","click",{event_category:"outbound",event_label:e,transport_type:"beacon",event_callback:function(){t!=="_blank"&&(document.location=e)}}),console.debug("Outbound link clicked: "+e)}function onClickCallback(e){if(e.target.tagName!=="A"||e.target.host===window.location.host)return;trackOutboundLink(e.target,e.target.getAttribute("target"))}gtag("js",new Date),gtag("config","G-M1JBEB5F0H",{}),gtag("set",{cookie_flags:"SameSite=None;Secure"}),document.addEventListener("click",onClickCallback,!1)</script>
1<meta name=description content="Explainability in recommender systems (RS) remains a pivotal challenge. Counterfactual explanations have emerged as a particularly actionable paradigm, offering intuitive âwhat-ifâ reasoning. However, their evaluation lacks principled standards. Current metrics primarily assess whether explanations change the top-ranked recommendation, overlooking two fundamental aspects of explanation quality. First, evaluation results are frequently inconsistent, as metrics are tightly coupled to the underlying recommenderâs performance. Second, explanations are rarely assessed for compactnessâwhether changes are sufficiently small to remain interpretable. Oversized counterfactuals may technically succeed but fail to provide practical insight. In this work, we advocate for a holistic evaluation perspective centered on consistency and compactness. We systematically analyze how extending evaluation beyond top-1 to top-k recommendations improves metric stability and reduces dependence on recommender fluctuations. In parallel, we introduce compactness-aware evaluation criteria that quantify the minimality of counterfactual modifications. Through extensive experiments across multiple datasets and models, we demonstrate that jointly considering these dimensions yields more reliable assessments. Our findings expose key factors driving metric instability, highlight the trade-off between effectiveness and explanation size, and provide practical guidelines toward standardized, faithful, and compact evaluation of counterfactual explanations in recommender systems."><link rel=alternate hreflang=en-us href=https://evazangerle.at/publication/mohammadi-tors-2026/><link rel=canonical href=https://evazangerle.at/publication/mohammadi-tors-2026/><link rel=manifest href=/manifest.webmanifest><link rel=icon type=image/png href=/media/icon_hu15160663797846276485.png><link rel=apple-touch-icon type=image/png href=/media/icon_hu12043829570639752632.png><meta name=theme-color content="#1565c0"><meta property="twitter:card" content="summary"><meta property="twitter:image" content="https://evazangerle.at/media/icon_hu9002677103972937553.png"><meta property="og:type" content="article"><meta property="og:site_name" content="Eva Zangerle"><meta property="og:url" content="https://evazangerle.at/publication/mohammadi-tors-2026/"><meta property="og:title" content="Measuring What Matters: Consistency and Compactness in Evaluation of Counterfactual Explanations | Eva Zangerle"><meta property="og:description" content="Explainability in recommender systems (RS) remains a pivotal challenge. Counterfactual explanations have emerged as a particularly actionable paradigm, offering intuitive âwhat-ifâ reasoning. However, their evaluation lacks principled standards. Current metrics primarily assess whether explanations change the top-ranked recommendation, overlooking two fundamental aspects of explanation quality. First, evaluation results are frequently inconsistent, as metrics are tightly coupled to the underlying recommenderâs performance. Second, explanations are rarely assessed for compactnessâwhether changes are sufficiently small to remain interpretable. Oversized counterfactuals may technically succeed but fail to provide practical insight. In this work, we advocate for a holistic evaluation perspective centered on consistency and compactness. We systematically analyze how extending evaluation beyond top-1 to top-k recommendations improves metric stability and reduces dependence on recommender fluctuations. In parallel, we introduce compactness-aware evaluation criteria that quantify the minimality of counterfactual modifications. Through extensive experiments across multiple datasets and models, we demonstrate that jointly considering these dimensions yields more reliable assessments. Our findings expose key factors driving metric instability, highlight the trade-off between effectiveness and explanation size, and provide practical guidelines toward standardized, faithful, and compact evaluation of counterfactual explanations in recommender systems."><meta property="og:image" content="https://evazangerle.at/media/icon_hu9002677103972937553.png"><meta property="og:locale" content="en-us"><meta property="article:published_time" content="2026-09-08T05:08:55+00:00"><meta property="article:modified_time" content="2026-08-01T00:00:00+00:00">
1<script type=application/ld+json>{"@context":"https://schema.org","@type":"Article","mainEntityOfPage":{"@type":"WebPage","@id":"https://evazangerle.at/publication/mohammadi-tors-2026/"},"headline":"Measuring What Matters: Consistency and Compactness in Evaluation of Counterfactual Explanations","datePublished":"2026-09-08T05:08:55Z","dateModified":"2026-08-01T00:00:00Z","author":{"@type":"Person","name":"Amir Reza Mohammadi"},"publisher":{"@type":"Organization","name":"Eva Zangerle","logo":{"@type":"ImageObject","url":"https://evazangerle.at/media/icon_hu18335020596408119049.png"}},"description":"Explainability in recommender systems (RS) remains a pivotal challenge. Counterfactual explanations have emerged as a particularly actionable paradigm, offering intuitive âwhat-ifâ reasoning. However, their evaluation lacks principled standards. Current metrics primarily assess whether explanations change the top-ranked recommendation, overlooking two fundamental aspects of explanation quality. First, evaluation results are frequently inconsistent, as metrics are tightly coupled to the underlying recommenderâs performance. Second, explanations are rarely assessed for compactnessâwhether changes are sufficiently small to remain interpretable. Oversized counterfactuals may technically succeed but fail to provide practical insight. In this work, we advocate for a holistic evaluation perspective centered on consistency and compactness. We systematically analyze how extending evaluation beyond top-1 to top-k recommendations improves metric stability and reduces dependence on recommender fluctuations. In parallel, we introduce compactness-aware evaluation criteria that quantify the minimality of counterfactual modifications. Through extensive experiments across multiple datasets and models, we demonstrate that jointly considering these dimensions yields more reliable assessments. Our findings expose key factors driving metric instability, highlight the trade-off between effectiveness and explanation size, and provide practical guidelines toward standardized, faithful, and compact evaluation of counterfactual explanations in recommender systems."}</script>
1<title>Measuring What Matters: Consistency and Compactness in Evaluation of Counterfactual Explanations | Eva Zangerle</title></head><body id=top data-spy=scroll data-offset=70 data-target=#TableOfContents class=page-wrapper data-wc-page-id=3f6b52c43e609118584946ae8a0e33dc>
1<script src=/js/wowchemy-init.min.9e4214442a7711d35691acd58f6f6361.js></script>
1<aside class=search-modal id=search><div class=container><section class=search-header><div class="row no-gutters justify-content-between mb-3"><div class=col-6><h1>Search</h1></div><div class="col-6 col-search-close"><a class=js-search href=# aria-label=Close><i class="fas fa-times-circle text-muted" aria-hidden=true></i></a></div></div><div id=search-box><input name=q id=search-query placeholder=Search... autocapitalize=off autocomplete=off autocorrect=off spellcheck=false type=search class=form-control aria-label=Search...></div></section><section class=section-search-results><div id=search-hits></div></section></div></aside><div class="page-header header--fixed"><header><nav class="navbar navbar-expand-lg navbar-light compensate-for-scrollbar" id=navbar-main><div class=container-xl><div class="d-none d-lg-inline-flex"><a class=navbar-brand href=/>Eva Zangerle</a></div><button type=button class=navbar-toggler data-toggle=collapse data-target=#navbar-content aria-controls=navbar-content aria-expanded=false aria-label="Toggle navigation"> 2<span><i class="fas fa-bars"></i></span></button><div class="navbar-brand-mobile-wrapper d-inline-flex d-lg-none"><a class=navbar-brand href=/>Eva Zangerle</a></div><div class="navbar-collapse main-menu-item collapse justify-content-start" id=navbar-content><ul class="navbar-nav d-md-inline-flex"><li class=nav-item><a class=nav-link href=/#about><span>Home</span></a></li><li class=nav-item><a class=nav-link href=/#news><span>News</span></a></li><li class=nav-item><a class=nav-link href=/#publications><span>Publications</span></a></li><li class=nav-item><a class=nav-link href=/#projects><span>Projects</span></a></li><li class=nav-item><a class=nav-link href=/#service><span>Service</span></a></li><li class=nav-item><a class=nav-link href=/#contact><span>Contact</span></a></li></ul></div><ul class="nav-icons navbar-nav flex-row ml-auto d-flex pl-md-2"><li class=nav-item><a class="nav-link js-search" href=# aria-label=Search><i class="fas fa-search" aria-hidden=true></i></a></li><li class="nav-item dropdown theme-dropdown"><a href=# class=nav-link data-toggle=dropdown aria-haspopup=true aria-label="Display preferences"><i class="fas fa-moon" aria-hidden=true></i></a><div class=dropdown-menu><a href=# class="dropdown-item js-set-theme-light"><span>Light</span> 3</a><a href=# class="dropdown-item js-set-theme-dark"><span>Dark</span> 4</a><a href=# class="dropdown-item js-set-theme-auto"><span>Automatic</span></a></div></li></ul></div></nav></header></div><div class=page-body><div class=pub><div class="article-container pt-3"><h1>Measuring What Matters: Consistency and Compactness in Evaluation of Counterfactual Explanations</h1><div class=article-metadata><div><span>Amir Reza Mohammadi</span>, <span>Andreas Peintner</span>, <span>Michael Mueller</span>, <span>Eva Zangerle</span></div><span class=article-date>August, 2026</span></div><div class="btn-links mb-3"><a href=# class="btn btn-outline-primary btn-page-header js-cite-modal" data-filename=/publication/mohammadi-tors-2026/cite.bib>Cite 5</a><a class="btn btn-outline-primary btn-page-header" href=https://doi.org/10.1145/3839564 target=_blank rel=noopener>DOI 6</a><a class="btn btn-outline-primary btn-page-header" href=https://doi.org/10.1145/3839564 target=_blank rel=noopener>URL</a></div></div><div class=article-container><h3>Abstract</h3><p class=pub-abstract>Explainability in recommender systems (RS) remains a pivotal challenge. Counterfactual explanations have emerged as a particularly actionable paradigm, offering intuitive âwhat-ifâ reasoning. However, their evaluation lacks principled standards. Current metrics primarily assess whether explanations change the top-ranked recommendation, overlooking two fundamental aspects of explanation quality. First, evaluation results are frequently inconsistent, as metrics are tightly coupled to the underlying recommenderâs performance. Second, explanations are rarely assessed for compactnessâwhether changes are sufficiently small to remain interpretable. Oversized counterfactuals may technically succeed but fail to provide practical insight. In this work, we advocate for a holistic evaluation perspective centered on consistency and compactness. We systematically analyze how extending evaluation beyond top-1 to top-k recommendations improves metric stability and reduces depen
6dence on recommender fluctuations. In parallel, we introduce compactness-aware evaluation criteria that quantify the minimality of counterfactual modifications. Through extensive experiments across multiple datasets and models, we demonstrate that jointly considering these dimensions yields more reliable assessments. Our findings expose key factors driving metric instability, highlight the trade-off between effectiveness and explanation size, and provide practical guidelines toward standardized, faithful, and compact evaluation of counterfactual explanations in recommender systems.</p><div class=row><div class=col-md-1></div><div class=col-md-10><div class=row><div class="col-12 col-md-3 pub-row-heading">Type</div><div class="col-12 col-md-9"><a href=/publication/#article-journal>Journal article</a></div></div></div><div class=col-md-1></div></div><div class="d-md-none space-below"></div><div class=row><div class=col-md-1></div><div class=col-md-10><div class=row><div class="col-12 col-md-3 pub-row-heading">Publication</div><div class="col-12 col-md-9"><em>ACM Trans. Recomm. Syst.</em></div></div></div><div class=col-md-1></div></div><div class="d-md-none space-below"></div><div class=space-below></div><div class=article-style></div><div class=article-tags><a class="badge badge-light" href=/tag/recommender-systems/>Recommender Systems</a> 7<a class="badge badge-light" href=/tag/counterfactual-explanations/>Counterfactual Explanations</a> 8<a class="badge badge-light" href=/tag/evaluation/>Evaluation</a> 9<a class="badge badge-light" href=/tag/consistency/>Consistency</a> 10<a class="badge badge-light" href=/tag/compactness/>Compactness</a></div><div class=share-box><ul class=share><li><a href="https://twitter.com/intent/tweet?url=https%3A%2F%2Fevazangerle.at%2Fpublication%2Fmohammadi-tors-2026%2F&text=Measuring+What+Matters%3A+Consistency+and+Compactness+in+Evaluation+of+Counterfactual+Explanations" target=_blank rel=noopener class=share-btn-twitter aria-label=twitter><i class="fab fa-twitter"></i></a></li><li><a href="https://www.facebook.com/sharer.php?u=https%3A%2F%2Fevazangerle.at%2Fpublication%2Fmohammadi-tors-2026%2F&t=Measuring+What+Matters%3A+Consistency+and+Compactness+in+Evaluation+of+Counterfactual+Explanations" target=_blank rel=noopener class=share-btn-facebook aria-label=facebook><i class="fab fa-facebook"></i></a></li><li><a href="mailto:?subject=Measuring%20What%20Matters%3A%20Consistency%20and%20Compactness%20in%20Evaluation%20of%20Counterfactual%20Explanations&body=https%3A%2F%2Fevazangerle.at%2Fpublication%2Fmohammadi-tors-2026%2F" target=_blank rel=noopener class=share-btn-email aria-label=envelope><i class="fas fa-envelope"></i></a></li><li><a href="https://www.linkedin.com/shareArticle?url=https%3A%2F%2Fevazangerle.at%2Fpublication%2Fmohammadi-tors-2026%2F&title=Measuring+What+Matters%3A+Consistency+and+Compactness+in+Evaluation+of+Counterfactual+Explanations" target=_blank rel=noopener class=share-btn-l
10inkedin aria-label=linkedin-in><i class="fab fa-linkedin-in"></i></a></li><li><a href="whatsapp://send?text=Measuring+What+Matters%3A+Consistency+and+Compactness+in+Evaluation+of+Counterfactual+Explanations%20https%3A%2F%2Fevazangerle.at%2Fpublication%2Fmohammadi-tors-2026%2F" target=_blank rel=noopener class=share-btn-whatsapp aria-label=whatsapp><i class="fab fa-whatsapp"></i></a></li><li><a href="https://service.weibo.com/share/share.php?url=https%3A%2F%2Fevazangerle.at%2Fpublication%2Fmohammadi-tors-2026%2F&title=Measuring+What+Matters%3A+Consistency+and+Compactness+in+Evaluation+of+Counterfactual+Explanations" target=_blank rel=noopener class=share-btn-weibo aria-label=weibo><i class="fab fa-weibo"></i></a></li></ul></div></div></div></div><div class=page-footer><div class=container><footer class=site-footer><p class="powered-by copyright-license-text">© 2025 Eva Zangerle. This work is licensed under <a href=https://creativecommons.org/licenses/by-nc-nd/4.0 rel="noopener noreferrer" target=_blank>CC BY NC ND 4.0</a></p><p class="powered-by footer-license-icons"><a href=https://creativecommons.org/licenses/by-nc-nd/4.0 rel="noopener noreferrer" target=_blank aria-label="Creative Commons"><i class="fab fa-creative-commons fa-2x" aria-hidden=true></i> 11<i class="fab fa-creative-commons-by fa-2x" aria-hidden=true></i> 12<i class="fab fa-creative-commons-nc fa-2x" aria-hidden=true></i> 13<i class="fab fa-creative-commons-nd fa-2x" aria-hidden=true></i></a></p><p class=powered-by>Published with <a href="https://hugoblox.com/?utm_campaign=poweredby" target=_blank rel=noopener>Hugo Blox Builder</a> â the free, <a href=https://github.com/HugoBlox/hugo-blox-builder target=_blank rel=noopener>open source</a> website builder that empowers creators.</p></footer></div></div>
13<script src=/js/vendor-bundle.min.50933d940896e49f984a778650d5f7f5.js></script>
13<script src=https://cdn.jsdelivr.net/npm/[email protected]/dist/leaflet.min.js integrity crossorigin=anonymous></script>
13<script id=search-hit-fuse-template type=text/x-template> 14 <div class="search-hit" id="summary-{{key}}"> 15 <div class="search-hit-content"> 16 <div class="search-hit-name"> 17 <a href="{{relpermalink}}">{{title}}</a> 18 <div class="article-metadata search-hit-type">{{type}}</div> 19 <p class="search-hit-description">{{snippet}}</p> 20 </div> 21 </div> 22 </div> 23 </script>
23<script src=https://cdn.jsdelivr.net/gh/krisk/[email protected]/dist/fuse.min.js integrity="sha512-o38bmzBGX+hD3JHWUFCDA09btWaqrNmoJ3RXLlrysA7PP01Kgs4UlE4MhelE1v5dJR3+cxlR4qQlotsW7jKsnw==" crossorigin=anonymous></script>
23<script src=https://cdn.jsdelivr.net/gh/julmot/[email protected]/dist/jquery.mark.min.js integrity="sha512-mhbv5DqBMgrWL+32MmsDOt/OAvqr/cHimk6B8y/bx/xS88MVkYGPiVv2ixKVrkywF2qHplNRUvFsAHUdxZ3Krg==" crossorigin=anonymous></script>
23<script id=page-data type=application/json>{"use_headroom":true}</script>
23<script src=/js/wowchemy-headroom.db4755770454eb63685f8de785c0a172.js type=module></script>
23<script src=/en/js/wowchemy.min.abfe36939207af9cb82df4fe2604ba17.js></script>
23<script src=/js/wowchemy-map.a26e9d2f7238ba5b868384f1c5bc6477.js type=module></script>
23<div id=modal class="modal fade" role=dialog><div class=modal-dialog><div class=modal-content><div class=modal-header><h5 class=modal-title>Cite</h5><button type=button class=close data-dismiss=modal aria-label=Close> 24<span aria-hidden=true>×</span></button></div><div class=modal-body><pre><code></code></pre></div><div class=modal-footer><a class="btn btn-outline-primary my-1 js-copy-cite" href=# target=_blank><i class="fas fa-copy"></i> Copy 25</a><a class="btn btn-outline-primary my-1 js-download-cite" href=# target=_blank><i class="fas fa-download"></i> Download</a><div id=modal-error></div></div></div></div></div>
25<script src=/js/wowchemy-publication.9c0e895144aef5a693008b5c5d450147.js type=module></script>
25</body></html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.