DAD Notes III: Applications of Deep Anomaly Detection
Anomaly detection is applicable in a very large number and variety of domains, such as fraud detection, fault detection, system health monitoring, event detection in sensor networks, detecting ecosystem disturbances, defect detection in images using machine vision, medical diagnosis and law enforcement.
In this post, we will discuss several applications of deep anomaly detection. For each application domain, we discuss the following four aspects:
- The notion of an anomaly
- Nature of the data
- Challenges associated with detection anomalies
- Existing deep anomaly detection techniques
1. Applications of Deep Anomaly Detection
The intrusion detection system (IDS) refers to identifying malicious activity in a computer-related system. IDS may be deployed at a single computer known as Host Intrusion Detection (HIDS) to large networks Network Intrusion Detection (NIDS). The classification of deep anomaly detection techniques for intrusion detection is in the figure below.
Depending on the different detection methods, IDS can be divided into signature-based and anomaly-based. Signature-based IDS is not efficient to detect new attacks, for which no specific signature pattern is available, hence anomaly-based detection methods are dominated in this field.
1.1 Host-Based Intrusion Detection Systems (HIDS)
Such systems are installed software programs that monitor a single computer for malicious activity or policy violations by listening to system calls or events occurring within that host. The system call logs could be generated by programs or by user interaction resulting in logs as shown in the figure below.
Malicious interactions lead to the execution of these system calls in different sequences. HIDS may also monitor the state of a system, its stored information, in Random Access Memory (RAM), in the file system, log files or elsewhere for a valid sequence. DAD techniques applied for HIDS are required to handle the variable length and sequential nature of data. They have to either model the sequence data or compute the similarity between sequences. Here we list some DAD techniques for HIDS.
| Techniques | Medel Architecture |
|---|---|
| Discriminative | LSTM, CNN-LSTM-GRU, DNN |
| Hybrid | GAN |
| Generative | AE, SPN |
Many deep learning algorithms have well-known abbreviations, here we give their full names, in the future, we will only use abbreviations to represent these algorithms.
- AE: Autoencoders
- CVAE: Convolutional Variational Autoencoder
- CNN: Convolution Neural Networks
- DBN: Deep Belief Networks
- DCA: Dilated Convolution Autoencoders
- DNN: Deep Neural Networks
- GAN: Generative Adversarial Networks
- GRU: Gated Recurrent Unit
- LSTM: Long Short Term Memory Networks
- RBM: Restricted Boltzmann Machines
- RNN: Recurrent Neural Networks
- SAE: Stacked Autoencoders
- SPN: Sum Product Networks
- VAE: Variational Autoencoders
1.2 Network Intrusion Detection Systems (NIDS)
NIDS deals with monitoring the entire network for suspicious traffic by examining each and every network packet. Owing to real-time streaming behavior, the nature of data is synonymous with big data with high volume, velocity, variety. The network data also has a temporal aspect associated with it. Here we list some of the successful DAD techniques for NIDS.
| Techniques | Model Architecture |
|---|---|
| Generative | DCA, SAE, RBM, DBN, CVAE |
| Hybrid | GAN |
| Discriminative | RNN, LSTM, CNN |
2. Fraud Detection
Fraud is a deliberate act of deception to access valuable resources. The Pricewaterhouse-Coopers (PwC) global economic crime survey of 2018 found that half of the 7,200 companies they surveyed had experienced fraud of some nature. Fraud detection refers to the detection of unlawful activities across various industries, illustrated in the figure below.
Fraud in telecommunications, insurance claims, banking represents significant problems in both governments and private businesses. Detecting and preventing fraud is not a simple task since fraud is an adaptive crime. Therefore, the challenge associated with detecting fraud is that it requires real-time detection and prevention. Let's take a look at some main types of fraud detection.
2.1 Banking Fraud
The credit card has been a popular payment method for a long time. Credit card fraud involves the theft of payment card details and using it as a fraudulent source of funds in a transaction. The challenge in credit card fraud detection is that frauds have no consistent patterns. The typical approach in credit card fraud detection is to maintain a usage profile for each user and monitor the user profiles to detect any deviations. Since there are billions of credit card users this technique of user profile approach is not very scalable. In this case, DAD techniques are gaining broad adoption in credit card fraud detection owing to their scalable nature.
2.2 Mobile Cellular Network Fraud
In recent years, mobile cellular networks have witnessed rapid deployment and evolution supporting billions of users and a vastly diverse array of mobile devices. Due to this broad adoption and low mobile cellular service rates, mobile cellular networks is now faced with frauds such as voice scams targeted to steal customer private information, and messaging-related scams to extort money from customers. Detecting such fraud is of paramount interest and not an easy task due to the volume and velocity of the mobile cellular network. Traditional machine learning methods with static feature engineering techniques fail to adapt to the nature of evolving fraud. DAD techniques usually perform better in this field.
2.3 Insurance Fraud
Although several traditional machine learning methods have been applied successfully to detect fraud in insurance claims. The traditional approach for fraud detection is based on features extraction, which is highly manual expertise depended. Another challenge of insurance fraud is that the incidence of frauds is far less than the total number of claims, and also each fraud is unique in its way. DAD techniques are proven to be effective to overcome these limitations.
2.4 Healthcare Fraud
Healthcare is an integral component in people's lives. Waste, abuse, and fraud drive up costs in healthcare by tens of billions of dollars each year. Healthcare insurance claims fraud is a significant contributor to increased healthcare costs, but its impact can be mitigated through fraud detection. DAD methods for healthcare fraud have been investigated in recent years.
Now we listed all DAD techniques used in different fraud detection across various application domains in the table below.
| Application Domain | Technique Used |
|---|---|
| Credit Card Fraud Detection | AE, RBM, DBN, VAE, GAN, DNN, LSTM, RNN, GRU, CNN |
| Mobiel Cellar Network Fraud Detection | CNN, SAE, DBN, DNN, GAN |
| Insurance Fraud Detection | DBN, VAE, GAN, DNN, CNN |
| Healthcare Fraud Detection | RBM, GAN, CNN |
3. Malware Detection
Malware, short for Malicious Software. In order to protect legitimate users from malware, machine learning based efficient malware detection methods are proposed. In classical machine learning methods, the process of malware detection is usually divided into two stages: feature extraction and classification/clustering. As usual, the performance of traditional malware detection approaches critically depends on the extracted features and the methods for classification/clustering. The challenge associated with malware detection problems is the sheer scale of data, for instance considering data as bytes a specific sequence classification problem could be of the order of two million time steps. Furthermore, the malware is very adaptive in nature, wherein the attackers would use advanced techniques to hide the malicious behavior.
DAD techniques used for malware detection include AE, CNN, DNN, DBN. LSTM, CNN-BiLSTM, GAN, AE-CNN, AE-DBN, RNN, etc.
4. Medical Anomaly Detection
Deep learning in medical and bioinformatics. Finding rate events (anomalies) in areas such as medical image analysis, clinical electroencephalograph (EEG) records, enable to diagnose and provide preventive treatments for a variety of medical conditions. Deep learning based architectures are employed with great success to detect medical anomalies. The vast amount of imbalanced data in the medical domain presents significant challenges to detecting outliers. Deep learning models increase their interpretability while producing outstanding performance.
DAD methods used for medical anomaly detection include AE, DBN, RBM, VAE, GAN, LSTM, RNN, GRU, CNN, AE+KNN, etc.
5. Deep Learning for Anomaly Detection in Social Networks
In recent years, online social networks have become more and more important in people's daily life. Anomalies in a social network are irregular often unlawful behavior patterns of individuals within a social network; such individuals may be identified as spammers, sexual predators, online fraudsters, fake users, or rumor-mongers. Detecting these irregular patterns is of prime importance since if not detected, the act of such individuals can have a serious social impact. The heterogeneous and dynamic nature of data presents significant challenges to DAD techniques. Despite these changes, several DAD techniques are shown to outperform state-of-the-art methods. Those techniques include AE, DAE, CNN-LSTM, DNN, CNN-LSTM-SVM, etc.
6. Log Anomaly Detection
Anomaly detection in log files aims to find text, which can indicate the reasons and the nature of the failure of a system. Most commonly, a domain-specific regular expression is constructed from past experience with finding new faults by pattern matching. The limitation of such approaches is that newer messages of failures are easily failed to be detected. The unstructured and diversity in both format and semantics of log data pose significant challenges to log anomaly detection. Anomaly detection techniques should adapt to the concurrent set of log data generated and detect outliers in real-time. Following the success of deep neural networks in real-time text analysis, several DAD techniques model the log data as a natural language sequence are shown very effective in detecting outliers, such as LSTM, AE, RNN, DAE, CNN, etc.
7. Internet of Things Big Data Anomaly Detection
IoT is identified as a network of devices that are interconnected with soft-wares, servers, sensors and etc. In the field of the Internet of things (IoT), data generated by weather stations, Radia-frequency identification (RFID) tags, IT infrastructure components, and some other sensors are mostly time-series sequential data. Anomaly detection in these IoT networks identifies fraudulent, faulty behavior of these massive scales of interconnected devices. The challenge associated with outlier detection is that heterogeneous devices are interconnected which renders the system more complex. DAD techniques used in IoT include AE, DBN, LSTM, etc.
8. Industrial Anomalies Detection
Industrial systems consisting of wind turbines, power plants, high-temperature energy systems, and storage devices are exposed to enormous stress on a day-to-day basis. Damage to these types of systems not only causes economic loss but also a loss of reputation, therefore detecting and repairing them early is of utmost importance. Despite several publications on deep learning methods for detecting early industrial damage showing great promise, owing to the rareness of the events, DAD techniques still have challenges in industrial anomalies detection. Some of the DAD techniques employed across various industries are LSTM, AE, DNN, CNN, SDAE, DAE, RNN, DNN-SVM, etc.
9. Anomaly Detection in Time Series
Data recorded sequentially over duration is known as time series. Time series data can be broadly classified into univariate and multivariate time series. In the case of univariate time series, only a single variable varies over time. A multivariate time series consists of several variables which change over time. In the literature, types of anomalies in univariate and multivariate time series are categorized into the following groups:
- Point Anomalies.
- Contextual Anomalies.
- Collective Anomalies.
In recent years, many deep learning models have been proposed for detecting anomalies within univariate and multivariate time series data respectively. Some of the challenges to detecting anomalies in time series using deep learning models data are:
- Lack of defined pattern in which an anomaly is occurring may be defined.
- Noise within the input data seriously affects the performance of algorithms.
- As the length of the time series data increases the computational complexity also increases.
- Time series data is usually non-stationary, non-linear and dynamically evolving.
All these challenges seem to be dwarfed, as deep learning techniques are used to detect time series anomalies.
| Type of Time Series | DAD Techniques |
|---|---|
| Univariate Time Series | LSTM, AE, LSTM-AE, CNN-AE, GRU-AE, RNN, CNN, CNN-LSTM, LSTM-VAE, DNN, GAN |
| Multivariate Time Series | LSTM, AE, LSTM-AE, CNN-AE, GRU-AE, CNN, CNN-LSTM, LSTM-VAE, GAN, DNN-RNN |
10. Video Surveillance
Video surveillance also popular known as Closed-circuit television (CCTV) involves monitoring designated areas of interest in order to ensure security. In videos surveillance applications unlabelled data is available in large amounts, this is a significant challenge for supervised machine learning and deep learning methods. Hence video surveillance works have studied the state-of-the-art deep models for video anomaly detection and have classified them based on the type of model and criteria of detection. The lack of an explicit definition of an anomaly in real-life video surveillance is a significant issue that hampers the performance of DAD methods as well. DAD techniques used in video surveillance include CNN, SAE, AE, CAE-OCSVM, LSTM-AE, STN, RBM, LSTM, RNN, AAE, etc.
