EXPLAINS 2025 Abstracts

Area 1 - Social and Legal Issues
Short Papers
Area 2 - Technology
Full Papers Short Papers
Area 3 - Applications
Full Papers Short Papers

Area 1 - Social and Legal Issues

Short Papers
Paper Nr: 115
Title:

Analyzing Accuracy and Consistency of GPT 4o Mini in Trivial Pursuit, and the Implications for its Use in Professional Contexts
Authors:

Keilen Smith and Rich Maclin
Abstract: Large language models such as OpenAI's GPT-4 have recently become a subject of interest for their use in education and information exchange. Such applications are hampered by the LLM's capacity to produce superficially plausible but ultimately incorrect information, called hallucinations. Using a bank of trivia questions sourced from the 1981 Genus Edition of Trivial Pursuit, we conducted a series of trials on GPT-4o Mini to measure the accuracy and consistency of the model and the characteristics of its hallucinations. The model demonstrated impressive accuracy across trials, averaging approximately 85%. Our results reinforced prior research, showing that the model's output is effectively non-deterministic and that its answers to identical questions can vary across sessions, meaning correct information cannot always be trusted to persist. We find evidence that GPT's demonstrated ability to improve incorrect answers when prompted is a consequence of random chance, that the model is more likely to break correct answers than fix incorrect ones, and that the strength of wording affects how likely this change is. We show that the model's factual knowledge is not discrete but probabilistic, with some questions being more prone to hallucination and the model's consistency being a reasonable metric of confidence in the information provided. Finally, we use the results to recommend best practices for future research examining the model's performance in professional and educational environments.

Area 2 - Technology

Full Papers
Paper Nr: 18
Title:

AutoCausalAIME: A CMA-ES-Driven Framework for Parametric Penalty Tuning in Causal Inverse Explanations
Authors:

Takafumi Nakanishi
Abstract: In recent years, the importance of artificial intelligence (AI) and machine learning model explainability has led to growing interest in Explainable AI (XAI). Specifically, global feature importance (GFI), which identifies the key explanatory variables (features) and their contributions to the target variable, plays a central role from two perspectives: (1) understanding the overall model behavior and (2) discovering true associations. Previous methods, such as LIME and SHapley Additive exPlanations, have primarily addressed the first perspective. To handle both, we introduced approximate inverse model explanations (AIME), which derive GFI in a data-driven manner using algebraic operations centered on the target variable. However, AIME was not fully robust to feature interdependencies, prompting us to develop CausalAIME, which integrates causal structure estimation (via the Peter–Clark algorithm) and the penalty-based suppression of multicollinearity. In this paper, we propose “AutoCausalAIME,” a method that eliminates the need for manual adjustment of CausalAIME’s hyperparameters—namely the penalty strength (λ) and causal ratio (α)—by leveraging covariance matrix adaptation evolution strategy (CMA-ES). We compare AIME, AutoCausalAIME, and “worst case” CausalAIME. While the Wilcoxon signed-rank test does not reveal statistically significant differences among them, comparisons across six metrics (descriptive accuracy, sparsity, stability, efficiency, robustness, and completeness) show that AutoCausalAIME achieves high explanatory accuracy, suppressed multicollinearity, and sufficient robustness. Thus, we demonstrate that AutoCausalAIME (1) eliminates the need for manual trial and error, (2) is applicable to large-scale and diverse tasks, and (3) derives interpretable GFI through a causal-structure-based difference penalty.
Paper Nr: 26
Title:

Uncertainty in Deep Model Performance for Radiology: A Case Study of Classifying Maxillary Sinus Appearance
Authors:

Fara Aninha Fernandes, Martin Gerdes, Georgi Chaltikyan and Christian W. Omlin
Abstract: Predictions from deep learning models need to be perfect in radiology, where incorrect predictions challenge their trustworthiness. As models of different architectures enter the field of image analysis, it is important to uncover the source and quantify the extent of the uncertainty in the models’ predictions. In this study, uncertainty methods are applied to a downstream task in classifying maxillary sinus images using three trained models: convolutional neural network (CNN), vision transformer (ViT), and gated multilayer perceptron (gMLP). The uncertainty is explored through probability distribution, expected calibration error, calibration curves, predictive entropy, and second-order representation. Of the three models, the ViT was the most uncertain in its predictions. Images that affected model confidence were identified. Regardless of the model architecture, most ‘clear’ sinus images were identified with certainty, while considerable uncertainty in predicting ‘opaque’ and ‘thick’ radiographic appearances was noted. These findings represent a novel direction towards the explainability of deep learning models as uncertainty estimation is a viable method for scrutinizing their inner workings.
Paper Nr: 29
Title:

Explain to Gain: Optimising Performance Through Explainable Reinforcement Learning Parameter Investigation
Authors:

Patrick Capaldo, Varniethan Ketheeswaran, Santiago Quintana-Amate, Delaney Stevens and Hall Mark
Abstract: Building upon the foundational work of "Explain to Gain: Introspective Reinforcement Learning for Enhanced Performance," this article further investigates methods for leveraging explainable reinforcement learning (XRL) knowledge to enhance the performance of reinforcement learning (RL) agents. While our initial work demonstrated the potential of XRL approaches to guide and optimise RL agent training beyond merely improving interpretability and user trust, this paper extends that exploration. We expand upon the previously introduced introspective analysis framework by incorporating an additional XRL metric parameter into the search space of XRL parameter configurations within the training pipelines of model-free RL algorithms. This refined integration allows for even more nuanced dynamic adjustments of algorithm-specific parameters based on real-time feedback from a broader set of XRL metrics. The proposed methodology is validated across diverse OpenAI Gym environments (CartPole and Taxi). By evaluating both on-policy and off-policy approaches with this expanded parameter space, we demonstrate that incorporating these additional XRL insights leads to further significant improvements in agent performance. The analysis of the results highlights the enhanced benefits of deepened explainability and more finely tuned decision-making. This work contributes to the XRL research area by continuing to align interpretability with actionable performance gains, advancing the development of more reliable, transparent, and effective RL systems for complex, real-world applications.
Paper Nr: 81
Title:

Attention Maps in 3D Shape Classification for Dental Stage Estimation with Class Node Graph Attention Networks
Authors:

Barkin Buyukcakir, Rocharles Cavalcante Fontenele, Reinhilde Jacobs, Jannick De Tobel, Patrick Thevissen, Dirk Vandermeulen and Peter Claes
Abstract: Deep learning offers a promising avenue for automating many recognition tasks in fields such as medicine and forensics. However, the "black box" nature of these models hinders their adoption in high-stakes applications where trust and accountability are required. For 3D shape recognition tasks in particular, this paper introduces the Class Node Graph Attention Network (CGAT) architecture to address this need. Applied to 3D meshes of third molars derived from CBCT images, for Demirjian stage allocation, CGAT utilizes graph attention convolutions and an inherent attention mechanism, visualized via attention rollout, to explain its decision-making process. We evaluated the local mean curvature and distance to centroid node features, both individually and in combination, as well as model depth, finding that models incorporating directed edges to a global CLS node produced more intuitive attention maps, while also yielding desirable classification performance. We analyzed the attention-based explanations of the models, and their predictive performances to propose optimal settings for the CGAT. The combination of local mean curvature and distance to centroid as node features yielded a slight performance increase with a 0.76 weighted F1 score, and more comprehensive attention visualizations. The CGAT architecture’s ability to generate human-understandable attention maps can enhance trust and facilitate expert validation of model decisions. While demonstrated on dental data, CGAT is broadly applicable to graph-based classification and regression tasks, promoting wider adoption of transparent and competitive deep learning models in high-stakes environments.
Paper Nr: 105
Title:

Mechanistic Interpretability for Transformer-based Time Series Classification
Authors:

Matiss Kalnare, Sofoklis Kitharidis, Thomas Bäck and Niki van Stein
Abstract: Transformer-based models have become state-of-the-art tools in various machine learning tasks, including time series classification, yet their complexity makes understanding their internal decision-making challenging. Existing explainability methods often focus on input-output attributions, leaving the internal mechanisms largely opaque. This paper addresses this gap by adapting various mechanistic interpretability techniques; activation patching, attention saliency, and sparse autoencoders, from NLP to transformer architectures designed explicitly for time series classification. We systematically probe the internal causal roles of individual attention heads and timesteps, revealing causal structures within these models. Through experimentation on a benchmark time series dataset, we construct causal graphs illustrating how information propagates internally, highlighting key attention heads and temporal positions driving correct classifications. Additionally, we demonstrate the potential of sparse autoencoders for uncovering interpretable latent features. Our findings provide both methodological contributions to transformer interpretability and novel insights into the functional mechanics underlying transformer performance in time series classification tasks.
Paper Nr: 138
Title:

Contrasting Human and Emergent Concepts in Image Classifiers
Authors:

Tamara Bíla and Igor Farkaš
Abstract: In the age of AI becoming an everyday partner and support tool in both professional and private domains, users and stakeholders are increasingly confronted with the question of interpretability. This work contributes to the search for answers by exploring hidden meanings in the internal layers of convolutional neural networks trained on image classification tasks. Combining supervised concept-based and unsupervised learning paradigms, the goal is to discover semantically meaningful representations that can be contrasted with human-defined concepts (defined once per dataset). More specifically, layer-wise network-inherent clusters were extracted using hierarchical agglomerative clustering. To evaluate their semantic fidelity, an auxiliary classifier was trained on both concept and cluster memberships and evaluated across all layers. Our experiments reveal a higher classification accuracy for clusters extracted from each layer compared to human-defined concepts, indicating better separability and suggesting that clusters may capture patterns beyond human labels. Additionally, the classification accuracy increases for both clusters and concepts toward the output layer. Beyond quantitative evaluation, qualitative insights were provided using visualization techniques such as UMAP projections and Concept Localization Maps. Our findings highlight the potential of hybrid approaches for post hoc explainability and point to promising directions in uncovering emergent structures within deep neural networks.
Paper Nr: 140
Title:

An Explainable Multi-Domain Document Summarization Framework using Domain-Aware Fine-Tuned Large Language Models
Authors:

Donggyu Kim, Kunyoung Kim and Mye Sohn
Abstract: Large Language Model (LLM)-based text summarization has significantly advanced the generation of coherent and semantically rich summaries. However, these models still face critical limitations in real-world applications, particularly in multi-domain scenarios. Moreover, lack of explainability hinders the trust and reliability of generated summaries. To address these issues, we propose a novel eXplainable Multi-domAin document Summarization (X-MAS) framework that enhances both the performance and explainability of multi-domain text summarization. X-MAS utilizes semantic clustering of documents using BERT embeddings and HDBSCAN to discover the domains of each document in the corpus. Based on the domains, X-MAS utilizes domain-specific fine-tunned LLMs to generate the summary. Finally, X-MAS utilizes keyword matching and BERT to map summary content back to its source documents. We evaluate X-MAS on a real-world multi-domain document dataset and demonstrate that it outperforms existing methods in both summary quality and explainability.
Paper Nr: 149
Title:

SPAX: A Shapley-Based Point Attribution eXplanation for Interpreting 3D Point Cloud Classification
Authors:

Marc F. Harinck, Muhammad Shoaib Sarwar, Bram Ton and Faizan Ahmed
Abstract: In this paper, we propose an Explainable Artificial Intelligence methodology to interpret point cloud classification models. Our approach uses Shapley values from cooperative game theory which assigns contribution values to individual points towards a classification. Point cloud data contains rich spatial informa- tion, therefore it requires sophisticated deep learning models to make accurate predictions. However, these models often lack interpretability. Our methodology, Shapley-based Point Attribution eXplenations (SPAX), addresses this issue by quantifyingandvisualisingthecontributionofeachpointinapointcloudtowards a prediction. To make it computationally feasible, a Monte Carlo approxima- tion is used to estimate the Shapely values. To evaluate the method, a PointNet classifier trained on the ModelNet10 dataset is used to determine the Shapley values. These values are then coloured according to their magnitude and used to visually interpret their impact on the classification. The results show how spa- tial features become more interpretable because they identify key components to determine classification results. This study creates foundational principles to enable better point cloud interpretation, which produces opportunities to boost real-world deployments of transparent algorithmic systems.
Short Papers
Paper Nr: 20
Title:

Leveraging Large Language Models for Generating and Evaluating Natural Language Explanations in XAI: A Comparative Study
Authors:

Renato Okabayashi Miyaji and Pedro Luiz Pizzigatti Correa
Abstract: Making machine learning (ML) model predictions understandable to diverse users is a critical challenge in Explainable Artificial Intelligence (XAI). While XAI techniques such as SHAP and LIME provide feature attributions, their outputs often require technical expertise to interpret. This study investigates the use of Large Language Models (LLMs) to bridge this gap by generating accessible natural language explanations from technical XAI outputs and to evaluate the quality of these explanations. We conduct a comparative analysis employing two distinct LLMs (OpenAI’s GPT-4o and o4-mini) and two prompting strategies (Zero-Shot and Few-Shot learning) for explanation generation using the TEXEN dataset. The generated explanations are evaluated using traditional NLP metrics (BLEU, ROUGE, METEOR, BERTScore) against human-authored references, and through an LLM-as-a-Judge approach (Prometheus 2) assessing Soundness, Completeness, Fluency, and Context-Awareness. Results indicate that while GPT4o with Few-Shot prompting achieved higher scores on traditional NLP metrics (e.g., BERTScore F1 of 0.858), the LLM-as-a-Judge evaluation revealed that the smaller o4-mini model, particularly with Few-Shot prompting, often matched or surpassed GPT-4o in human-aligned qualitative criteria, achieving, for instance, a fluency score of 4.30 and completeness of 4.10. These findings highlight the nuanced capabilities of different LLMs and the importance of multifaceted evaluation frameworks for developing truly interpretable XAI systems.
Paper Nr: 46
Title:

Quantifying Prototype Stability in ProtoPNet Without Manual Part Annotations
Authors:

Iker Sancho, Ruben Naranjo, Nerea Aranjuelo and Itsaso Rodríguez-Moreno
Abstract: Prototype-based networks such as ProtoPNet offer intrinsically interpretable predictions by matching regions of the inferenced image to learned prototypical patches. However, existing stability metrics rely on expensive manual part annotations and are limited to narrow perturbation types. In this work, we introduce the Structural Stability Score (Sss), a scalable, annotation-free metric that quantifies prototype stability under a diverse set of visual transformations by comparing prototype activation maps. We evaluate Sss on two ProtoPNet variants (using VGG-19 and Resnet-34 backbones) trained on the CUB-200-2011 dataset, and assess stability across six distinct perturbations. Our results reveal clear differences in robustness both between models and among different transformations. These findings demonstrate that Sss is a practical tool for highlighting stability variations within and across prototype-based networks, guiding model selection and interpretability analysis.
Paper Nr: 47
Title:

Interpretable Railway Object Classification Using Part-Prototype Networks
Authors:

Ruben Naranjo, Iker Sancho, Nerea Aranjuelo, Itsaso Rodríguez-Moreno and Marcos Nieto
Abstract: Enhancing safety and operational efficiency in railway systems benefits from robust AI-powered perception, particularly for reliable obstacle and pedestrian detection. However, the prevalent black-box nature of contemporary deep learning models presents significant challenges for verification and trust, especially within safety-critical railway environments characterised by dynamic weather, illumination changes, and high speed, which create undesirable effects such as cluttered backgrounds, and motion blur, which may hinder the performance of computer vision approaches. This paper addresses the need for more transparent models by proposing the application of Prototypical Part Networks (ProtoPNet) for interpretable obstacle and pedestrian classification within the railway domain. Experiments with the OSDaR23 dataset demonstrate that training with a careful selection of data augmentation processes enhances key metrics such as precision, recall and F1-score while yielding transparent results with visually robust prototypes.
Paper Nr: 50
Title:

Efficient Construction of Interpretable Oblique Decision Trees
Authors:

Vladimir Estivill-Castro and Nuru Nabuuso
Abstract: Human oversight of AI demands transparent models, especially in high-stakes contexts. While interpretable models like decision trees offer clarity, traditional axis-aligned trees struggle with complex features, and oblique trees, though more expressive, are hard to understand. This work introduces a human-centred framework using parallel coordinates (PCs) to visualise and construct oblique decision trees interactively. By enabling the visualisation of 3D hyperplanes and allowing users to manipulate them directly, the method enhances explainability without compromising accuracy. We present an efficient method for finding hyperplanes of up to 3 dimensions. We show experimentally by comparing several methods that our oblique trees do not sacrifice accuracy, but the explainability of the trees (due to their simplicity) is radically enhanced. This bridges the gap between model complexity and interpretability, supporting human oversight in AI decision-making.
Paper Nr: 67
Title:

Extracting Deterministic Finite Automata from RNNs via Hyperplane Partitioning and Learning
Authors:

Sandamali Yashodhara Wickramasinghe, Jacob M. Howe and Laure Daviaud
Abstract: Recurrent Neural Networks (RNNs) have achieved remarkable success in handling sequential data. However, they lack interpretability; understanding their decision-making process is challenging, leading to their characterisation as `black boxes.' Extracting Deterministic Finite Automata (DFAs) from black-box models enhances interpretability and provides insight into their decision-making processes. This research focuses on extracting DFAs from RNNs trained on regular languages using an exact learning framework. The proposed approach employs the L* algorithm to learn a DFA, and it demonstrates how a hyperplane-based method can be used to partition the RNN state space when evaluating equivalence queries.
Paper Nr: 71
Title:

Profiling German Text Simplification with Model-Fingerprints
Authors:

Lars Klöser, Mika Elias Beele and Bodo Kraft
Abstract: While Large Language Models (LLMs) produce highly nuanced text simplifications, developers currently lack tools for a holistic, efficient, and reproducible diagnosis of their behavior. This paper introduces the Simplification Profiler, a diagnostic toolkit that generates a multidimensional, interpretable fingerprint of simplified texts. Multiple aggregated simplifications of a model result in a model's fingerprint. This novel evaluation paradigm is particularly vital for languages, where the data scarcity problem is magnified when creating flexible models for diverse target groups rather than a single, fixed simplification style. We propose that measuring a model's unique behavioral signature is more relevant in this context as an alternative to correlating metrics with human preferences. We operationalize this with a practical meta-evaluation of our fingerprints' descriptive power, which bypasses the need for large, human-rated datasets. This test measures if a simple linear classifier can reliably identify various model configurations by their created simplifications, confirming that our metrics are sensitive to a model's specific characteristics. The Profiler can distinguish high-level behavioral variations between prompting strategies and fine-grained changes from prompt engineering, including few-shot examples. Our complete feature set achieves classification F1-scores up to 71.9\%, improving upon simple baselines by over 48 percentage points. The Simplification Profiler thus offers developers a granular, actionable analysis to build more effective and truly adaptive text simplification systems.
Paper Nr: 92
Title:

SemantriX: An Explainable Hybrid Model for Aligning Vector Similarity and Semantic Relevance
Authors:

Antony Medeiros, Claudio Cavalcante, Edward Hermann Hauesler, Daniel Schwabe and Sergio Lifschitz
Abstract: Retrieval-Augmented Generation (RAG) systems combine large language models with retrieval mechanisms to generate contextually relevant answers. However, while vector similarity effectively retrieves geometrically close documents, it often lacks the semantic depth required to fully address user queries. This paper investigates the gap between vector similarity and semantic relevance through mathematical formulations and real-world examples. We propose SemantriX, an explainable hybrid retrieval strategy that integrates metadata enrichment and cross-encoder-based reranking. The model is evaluated within a question-answering system applied to the domain of contract management. Experimental results show significant improvements in precision, recall, and F1-score. By aligning retrieval with semantic relevance, our approach enhances the performance and explainability of RAG systems in real-world decision-support scenarios.
Paper Nr: 99
Title:

How Prompting Shapes Decisions: Analyzing LLM Behavior in XAI-Augmented Decision Support Systems
Authors:

Finn Schwall, Maximilian Becker, Anmol Ashri and Jürgen Beyerer
Abstract: Large Language Models (LLMs) become increasingly prevalent in downstream tasks and user facing applications. As explainable AI (XAI) strives to become more end user-friendly, utilization of LLMs in XAI increases. However, the consequences of this are unclear. Do LLMs really improve user experience, or are there hidden problems that may limit their applicability? In this paper, we present results of experiments on the decision-making process of LLMs with the goal of evaluating their usefulness for such applications. By providing the LLM with different information and applying different metrics to evaluate their decisions, we present findings that should inform the applicability of LLMs. By analyzing nearly 300,000 prompts we found that the LLM’s decisions are only minimally influenced by XAI data. Secondly, the LLM’s behavior can be changed significantly through prompting. This suggests that LLM behavior is more sensitive to presentation than to underlying model reliability, raising concerns about its role as a rational arbiter. If our results hold true, we advise for caution when utilizing LLMs, especially when facing laypeople.
Paper Nr: 109
Title:

User Fairness in Recommender Systems using Beyond-Accuracy Basket Quality Metrics
Authors:

Debarati Bhaumik, Diptish Dey, Milan Bril and Vanessa Stoica
Abstract: Recommender systems (RecSys) have conventionally assessed user fairness through accuracy-based metrics, such as Root Mean Squared Error and Hit Rate. These metrics are limited in their ability to evaluate fairness of user experiences, as they do not consider beyond-accuracy perspectives of recommendation basket quality. Key elements of beyond-accuracy perspectives - including novelty, relevancy, unexpectedness, coverage, serendipity, ranking, diversity, and recency - collectively enhance existing frameworks. They do so by enabling evaluation of user satisfaction and user fairness across different demographic groups. Expanding upon established fairness metrics in the literature, this work introduces novel beyond-accuracy basket quality perspectives, including novelty-relevancy (NovRel) and recency, along with complementary metrics for computing relevancy and unexpectedness at individual and group levels. While leveraging existing accuracy-based fairness measures, our approach evaluates user fairness through the lens of beyond-accuracy basket quality measures and provides a comprehensive assessment of whether RecSys deliver comparable experiences to similar users and ensure equitable treatment across different sensitive groups. Methods to calculate these metrics are demonstrated using common algorithms, SVD++, Q-SVD++, and MLP-SVD++, which are trained on two widely used benchmark datasets. The trade-off between accuracy metrics and beyond-accuracy fairness metrics is presented. Beyond-accuracy metrics provide valuable insights into how these algorithms differ in user fairness due to their inherent design considerations.
Paper Nr: 131
Title:

Unsupervised Hierarchical Growing Neural Architecture for Sensorimotor Map Learning
Authors:

Abu Abu and Andrew Starkey
Abstract: The ability to build cognitive maps of unknown environments in a continuous, unsupervised manner is an important capability for autonomous agents. Deep-reinforcement neural networks, despite demonstrating impressive capabilities across diverse domains, fail to rival mammalian proficiency in this critical navigation task due to their lack of explainability, sample inefficiency, and limited capacity to generalize to new environments. This paper presents a Modular and Incremental Network with Enhanced Representation and Vertical Abstraction (MINERVA), a bioinspired and explainable architecture for sensorimotor map learning that aims to extend the capabilities of growing neural architectures by incorporating principles observed in mammalian spatial cognition, including distributed and hierarchical processing of inputs and sparse coding mechanisms. The algorithm is compared with the Temporospatial Merge Grow When Required (TMGWR) network, which was previously demonstrated in a maze navigation context to be superior to algorithms such as growing neural gas (GNG), Grow When Required (GWR) and time GNG (TGNG) in terms of disambiguation performance, sensorial representation accuracy, and sensorimotor-link error. From the experiments conducted, MINERVA demonstrated more robust performance in these metrics with better multi-sensorial processing capabilities, which can be leveraged in solving more complex challenges in more difficult sensorial environments.
Paper Nr: 134
Title:

Rule Extraction from Fake News Classifiers
Authors:

Fatima Iqbal and Jacob Howe
Abstract: This study explores the decision-making processes of machine learning models for text classification problems. Using a fake news dataset as a test case, the study compares neural networks and machine learning approaches to fake news detection. This text-based problem has a feature space of 12,569 dimensions, and the necessity of the full dimensionality of the data is investigated. In addition to developing effective classifiers, this study aims to investigate neural network interpretability by applying an explainable AI framework to extract human-understandable rules from trained models. The rule extraction process, taking a pedagogical approach, investigates the decision making of models. A Boolean function based model was developed, and the extent to which this rule-based system over a reduced feature set is successful is evaluated.
Paper Nr: 155
Title:

Exposing Shortcuts in Image Classification by Aggregating Counterfactuals
Authors:

James Hinns and David Martens
Abstract: Deep learning has achieved remarkable success in image classification, yet models often rely on misleading 'shortcuts', that do not generalise beyond lab-settings, such as watermarks or background cues. It has been proposed that instance-level explanations in explainable AI (XAI) may help reveal such shortcuts without external data, but doing so typically requires examining many individual explanations, making the process labour-intensive and often infeasible. We introduce Counterfactual Frequency (CoF) tables, a novel method that aggregates local explanations into global insights and efficiently exposes shortcuts. This requires semantically meaningful image segments, with appropiate labels which we obtain through pre-trained foundation segmentation models. We demonstrate the effectiveness of CoF tables across multiple datasets, including real-world datasets such as ImageNet, toy datasets constructed for this study, and Biased Action Recognition (BAR), highlighting shortcuts like background reliance and watermarks learned by both models we train and well-studied pre-trained classifiers.
Paper Nr: 88
Title:

Extensibility, Model Interpretability and Explainability, and Automation in ML.NET: A Comprehensive Analysis
Authors:

Robin Nunkesser
Abstract: This paper presents an in-depth analysis of the extensibility, model interpretability and explainability, and automation capabilities of ML.NET, Microsoft's open-source machine learning framework for .NET developers. We identify and address the challenges faced by third-party developers, particularly due to ML.NET's restricted internal APIs and reliance on friend assemblies. We propose practical approaches for implementing custom estimators and evaluation metrics, enabling greater flexibility for external contributors. Furthermore, we examine the interpretability and explainability of models produced by ML.NET and its AutoML component, demonstrating both black-box and white-box strategies. Finally, we introduce an automated methodology for fair benchmarking and comparison of ML.NET's AutoML results with alternative frameworks, leveraging JSON-based configuration and reproducible evaluation pipelines. All proposed methods are supported by open-source code and validated through experiments on standard benchmark datasets. Our findings highlight both the strengths and current limitations of ML.NET, providing actionable guidance for practitioners and researchers seeking to extend and analyze machine learning workflows within the .NET ecosystem.
Paper Nr: 121
Title:

Interpretable Explainable AI: Comparing Bayesian Structural Equation Modelling with Other Algorithms
Authors:

Corina Ilinca
Abstract: This research study compares various machine learning algorithms for predicting self-assessments of cognitive functioning, specifically memory, using data from the Health and Retirement Study involving individuals over 50 years old. The study reveals that while different algorithms yield only minor differences in predictive performance, the theoretical and social context of the data plays a significant role in model construction. Study Objective: The aim is to identify which machine learning algorithm best predicts memory assessments using a dataset from the Health and Retirement Study. Algorithms Compared: The study examines several algorithms including ordinal regression, linear regression, neural networks, and Bayesian approaches, evaluating their effectiveness in predicting memory outcomes. Predictors Used: Key predictors in the models include self-rated hearing, age, education, work status, gender, marital status, and life satisfaction, derived from a sample of 15,408 respondents. Results Summary: The results indicate moderate levels of variance explained by the models, with significant predictors identified across all algorithms, particularly highlighting the importance of hearing and education. Model Performance: The goodness of fit measures show that the models provide similar levels of predictive accuracy, indicating modest explainability. Conclusions: The study concludes that while predictive performance varies slightly among algorithms, understanding the underlying theoretical connections is crucial for developing explainable machine learning models in social sciences.
Paper Nr: 150
Title:

A Privacy-Preserving and Explainable Approach for Anomaly Detection in Substation Networks
Authors:

Paul Tavolato, Oliver Eigner, Philipp Kreimel-Haindl, Patrizia Agnello, Marta Petyx, Antonella Santone, Fabio Martinelli and Francesco Mercaldo
Abstract: Electrical substations play a crucial role in managing electrical energy, making them critical targets cyber-attacks on these systems could severely impact the general population, hospitals, and both critical and non-critical infrastructure. Several research papers, from both academic than industrial world, propose anomaly detection in substation networks but currently there is a lacking of explainability on the reasons why an anomaly is detected. Moreover, privacy is also an issue, as a matter of fact current methods require that network logs are sent to a centralized server for model training, exposing sensitive and confidential network traces outside the electrical substation network infrastructure. To overcome both of these limitations, in this paper, we present an explainable and privacy-preserving approach for detecting possible anomalies within electrical substations. The proposed method analyzes network logs to identify potential anomalies in substation networks. We convert a network trace into an image and we consider a Vision Transformer model for anomaly detection. We also consider prediction explainability by highlighting specific areas in the image generated from the network trace that the classifier identifies as indicative of an anomaly, with the aim to visually show which area of the image is symptomatic of a possible anomaly.
Paper Nr: 160
Title:

On Explainable Disease Progression Forecasting with Transformer Models
Authors:

Umair Mirza, Faizan Ahmed and Faryal Siddique
Abstract: Accurate prediction of disease progression is essential to enable timely clinical interventions and support personalized treatment strategies in both chronic and acute care settings. While recurrent neural networks (RNNs) have traditionally dominated time series modeling in healthcare, transformer-based architectures have recently shown considerable promise. This study evaluates the applicability of the Informer architecture on longitudinal clinical data from the Parkinson's Progression Markers Initiative (PPMI) dataset. Multiple model variations are explored, including different temporal embedding techniques and multi-task learning configurations. To enhance clinical interpretability, explainability methods such as self-attention visualizations, permutation importance, and GradientSHAP are employed. The results show that the Informer architecture can be viable for forecasting tasks within clinical health domains.

Area 3 - Applications

Full Papers
Paper Nr: 96
Title:

Explainable Knowledge Access: Recursive and Rerank-Based RAG for Interpretable QA
Authors:

Melchiorre Cosseddu, Alessandro Giuliani, Marco Manolo Manca, Marco Pilloni, Alessandro Sebastian Podda and Sandro Gabriele Tiddia
Abstract: In response to the crucial need for more transparent and reliable tools based on Large Language Models (LLMs), this work introduces a novel and explainable architectural enhancement to the Retrieval-Augmented Generation (RAG) pipeline. As organizations increasingly rely on LLMs to navigate and interpret vast, unstructured datasets, ensuring the factual accuracy and traceability of generated content has become essential. Our key contribution is designing a system that formally anchors all produced output in a formally verified knowledge base, thereby significantly alleviating challenges related to hallucination and verbosity. This focus on transparency is achieved by providing exact citations to the information used, enabling users to trace the source of generated content. Our work demonstrates significant improvements to chunk retrieval accuracy, achieved using a multi-granularity recursive chunking strategy and re-ranking with maximum information exposure to the LLM. We also provide extensive comparative analysis of various combinations of LLM and embedding models under this enhanced system. Our contributions lay the groundwork for developing more responsible and reliable AI solutions across knowledge-intensive domains, focusing on enhanced information retrieval and transparent response generation.
Short Papers
Paper Nr: 107
Title:

XAI-Driven Solutions to Enhance Safety for Limited-Mobility Road Users
Authors:

Gianmarco Cherchi, Nicola Floris, Alessandro Sebastian Podda, Livio Pompianu, Roberto Saia and Riccardo Scateni
Abstract: The large availability of urban surveillance video, combined with recent advances in deep learning, allows us to exploit them to support road safety by analyzing pedestrian behavior in real-time. This work proposes an XAI-driven approach for monitoring crosswalks with limited visibility, where the risk to Vulnerable Road Users (VRUs) is particularly high. We focus specifically on Limited-Mobility VRUs (LM-VRUs), including wheelchair users or adults with strollers, who may have a reduced ability to respond quickly to dangerous situations. The system processes live video streams from fixed surveillance cameras using deep learning-based computer vision to detect and track LM-VRUs in both crossing and waiting areas. Unlike data-intensive predictive models, our approach utilizes a lightweight, rule-based module that infers pedestrian crossing intent through human-understandable spatiotemporal heuristics. This explainable component ensures that the decision-making process remains transparent and auditable. Upon detecting a potentially dangerous crossing scenario, the system immediately activates acoustic and visual warnings for approaching drivers, improving safety, including for visually impaired pedestrians. Beyond its technical contribution, this work explores the social impact of AI technologies designed to protect mobility-impaired individuals in urban environments. Our goal is to reduce traffic-related accidents and contribute to more inclusive, intelligent city infrastructures.