A Robust Hybrid Ensemble Framework for Anomaly Detection in Financial Transactions

Abstract

Detecting fraudulent transactions in financial systems remains a critical challenge, owing to the highly imbalanced distribution of legitimate versus fraudulent activities and the continuously evolving sophistication of fraud strategies. Conventional supervised models, which are effective at learning known patterns, often struggle to identify rare anomalies, leading to low recall or increased false positives. Conversely, unsupervised methods, although capable of discovering novel patterns, frequently suffer from reduced precision and model instability. To address these limitations, this study proposes XRAI, which is a robust hybrid ensemble framework for anomaly detection in financial transactions. The framework integrates supervised learners (XGBoost and Random Forest) with unsupervised detectors (Autoencoder and Isolation Forest), combining their complementary strengths to enhance the overall detection capability. The outputs of these models were aggregated through an optimized weighted scoring and thresholding mechanism, ensuring a balance between sensitivity and specificity. Furthermore, explainable AI (XAI) techniques, specifically SHAP (SHapley Additive exPlanations), were employed to interpret model decisions, highlight influential features, and improve transparency for regulatory and operational auditability. Experimental evaluation using the creditcard.csv dataset demonstrated superior results, with a precision of 0.9569, recall of 0.9250, F1-score of 0.9407, MCC of 0.9407, and accuracy of 0.9998, outperforming the established benchmarks. The findings confirm XRAI’s capability to deliver a scalable, interpretable, and reproducible fraud detection framework adaptable to cross-domain anomaly detection applications and aligned with emerging AI governance and transparency requirements.