AI-Driven Fraud Detection in Digital Banking: A Hybrid Approach using Deep Learning and Anomaly Detection

Abstract

The rapid digital transformation in the banking sector has introduced new opportunities for efficiency and customer convenience but has also amplified the risks of financial fraud. Traditional fraud detection mechanisms, often reliant on static rule-based systems, struggle to keep pace with the dynamic, evolving nature of fraudulent activities. This paper proposes a novel hybrid framework that integrates deep learning models with anomaly detection techniques to enhance the accuracy, robustness, and adaptability of fraud detection in digital banking. The proposed approach leverages a deep neural network (DNN) architecture trained under supervised learning to capture complex transactional patterns and combines it with autoencoder-based unsupervised anomaly detection to uncover previously unseen fraud strategies. Extensive experiments on benchmark financial datasets demonstrate that the hybrid system significantly outperforms state-of-the-art methods in terms of precision, recall, and false-positive reduction. Furthermore, the study highlights the scalability of the approach for real-time banking applications and its potential for multi-institutional deployment, enabling secure inter-bank fraud intelligence sharing without compromising data privacy. Extensive experiments on benchmark financial datasets demonstrate that the hybrid system significantly outperforms state-of-the-art methods in terms of precision, recall, and false-positive reduction. Furthermore, the study highlights the scalability of the approach for real-time banking applications. This work contributes to the growing field of AI-driven financial security by addressing both detection performance and adaptability to emerging fraud behaviors.