Introduction
Malware evolution and challenges in abstracting malware semantics affect the performance of learning-based algorithms over time, due to non-stationarity and concept drift. Therefore, it has become common practice to mitigate concept drift by retraining or updating models whenever new ground-truth labels become available. The quality of new ground-truth labels can be challenged by label instability, the use of unvetted datasets, and the intensive costs of labeling whenever pseudo-labels cannot be trusted. A questionable ground-truth may, unfortunately, facilitate data poisoning.
Current defenses aim to identify poisoned models, or trace data poisoning attacks in specific settings under a forensic lens. However, understanding the temporal effects of concept drift mitigation strategies on data poisoning attacks in non-stationary contexts remains largely unexplored.
This paper investigates whether and how concept drift mitigation strategies help to passively recover model compromise induced by availability data poisoning attacks. We propose the formalization of “recovery” as the convergence of the poisoned model performance with that of the hypothetical performance of the same model if it were not poisoned (which we refer to as the hypothetical model). We distinguish passive recovery as the recovery process that results as a by-product of the complete classification system, as opposed to active recovery that specifically seeks to mitigate the impact of poisoning through directed methods. We quantify passive recovery by introducing two new metrics: intercept and recovery rate. Then, we consolidate these metrics and methodologies to propose RPAL, an extensible framework to aid in the investigation of passive recovery for other ML classification systems given a set of parameters, classifiers, concept drift mitigations and poisoning strategies.
We evaluate RPAL in the context of Android malware detection to quantify passive recovery from dirty label (label-flip) and clean label (feature-flip) poisoning by using active learning (uncertainty sampling).
This paper extends our previously published poster.
Distinction between Passive and Active Recovery
We define recovery as the act of converging the performance of a poisoned model with that of the hypothetical model, which was never poisoned.
Passive recovery (this paper)
Specifically, passive recovery refers to recovery achieved by an approach which is implemented for another purpose such as uncertainty sampling active learning for concept drift mitigation in the case of this paper.
Active recovery
In contrast, the active recovery common in the existing literature refers to recovery achieved by an approach specifically designed to reduce or eliminate the effects of data poisoning.
We focus on passive recovery because the temporal effects of passive approaches on classifiers are overlooked in existing poisoning research. To our knowledge, we are the first to evaluate the passive recovery of traditional ML and DNN systems from availability data poisoning in the context of malware detection.
Threat Model
The threat in our research is data poisoning, which is the most common adversarial machine learning training-time attack and aims to affect a classifier by adding specially designed data to the training set.
Specifically, data poisoning attacks can be used to impact the availability of a model through deliberate performance degradation or the integrity of a model via the creation of backdoors. Availability data poisoning further exacerbates concept drift as it causes the divergence of training data from accurate data acting as a pre-test-time drift. Our research therefore focuses on the threat of availability data poisoning attacks as a result of its close connection with concept drift.
Attacker’s Objective & Capabilities
There can be two main attacker goals for availability data poisoning: having the classifier missing threats (False Negatives); having the classifier generating false alarms (False Positives). Without loss of generality, we examine both options by evaluating an attack which compromises the general availability of the model.
For the capability, we assume that the attacker only has access to a portion of the training data, reflected in the poisoning rate, and no additional information on the classification system utilized. Therefore, there are two potential capabilities for the attacker: either (i) they have influence over the training data labeling process, or (ii) they have the ability to modify the prepared training data (feature vectors). In addition, the poisoning attack strengths of 2%, 4%, 8%, and 16% of the training samples were used in our evaluation.
Dirty-label poisoning (label-flip)
This attack is used for the case in which the attacker has influence over the labeling process as we consider a label-flip poisoning attack for our dirty-label poisoning approach. Label-flip poisoning is chosen as it is model-agnostic, therefore requires no knowledge about the model and does not rely on assumptions or knowledge about the classifier like gradient-based attacks. We implement the attack to flip the labels of training data in an equal ratio of goodware to malware so as to maintain the original class distribution to avoid detection.
Clean-label poisoning (feature-flip)
This attack is used for the case in which the attacker can only modify the prepared training data and not the labels; therefore, we use a feature-flip attack as our clean-label poisoning approach. We designed our clean-label approach to mimic our dirty-label approach to maintain the same model-agnostic property and training data access requirements. Therefore, our clean-label poisoning attack changes the feature values of samples to recreate samples of the opposite class, thereby enabling the creation of the same poisoning set as the dirty-label attack without label modification.
We minimize the number of features required to be changed by finding goodware/malware pairs with the least number of different benign and malicious features.
| Poisoning rate | Samples | Features flipped by clean-label poisoning | |
|---|---|---|---|
| Overall | Per sample | ||
| 2% | 1,156 | 54,658 | 47 |
| 4% | 2,312 | 106,802 | 46 |
| 8% | 4,626 | 214,254 | 46 |
| 16% | 9,252 | 421,054 | 46 |
Passive Recovery Mechanism
A passive recovery mechanism is a method that allows the model to be adapted or modified over time and typically comes with its own operational costs. When deciding on a candidate passive recovery mechanism, it is important to consider a realistic labeling budget, with the cost of labeling being the number of testing objects that must be labeled periodically.
Our time-aware evaluation follows both key recommendations from TESSERACT to ensure that the results presented are not spatially or temporally biased. Spatial bias is the result of an unrealistic test-time class distribution and can be prevented in the Android malware domain by using a realistic 10% malware distribution. Temporal bias is the incorporation of future knowledge into the model and is prevented by the use of time-stamped data to ensure that training data precedes testing data.
Active Learning
The candidate passive recovery mechanism in this paper is uncertainty sampling, a type of active learning, which selects samples where the classifier is less certain, as these samples are likely the most informative to adjust the decision boundary. To measure uncertainty, we use two different methods based on the classifier. Firstly, for RF and DNN we use probabilistic uncertainty which selects the samples with the lowest probability for its predicted label. Secondly, since SVM does not have probabilities, we use the absolute distance to the hyperplane to measure the uncertainty and select the samples that are closest (most uncertain).
Evaluation Pipeline
The initial input is a time-stamped dataset consisting of a feature matrix X, labels y, and time-stamps t.
Split in time
The dataset is temporally split into a training set and testing months using the TESSERACT framework.
Rebalance and poison
First, the training dataset is rebalanced to a realistic class distribution (e.g., approximately 10% in the case of Android malware). The dataset is then poisoned based on the current iteration’s poisoning rate (starting from 0%, for the hypothetical model baseline) and is then used to train the model.
Predict
The model predicts on the current test month data.
Sample and retrain
After prediction, the passive recovery mechanism (e.g. active learning) selects new samples to be labeled, and then the model is re-trained. The train, predict, sample, and retrain process is then repeated for all test months.
Repeat and analyse
After all test months have been evaluated, the process is repeated for all poisoning and recovery rates in the evaluation settings. Once all setting evaluations have been completed, the intercept and recovery rates are extracted from the results.
Dataset and Classifiers
Dataset
The dataset considers the popular DREBIN feature representation. We use a feature selection of the top 10,000 features. In the rest of this paper, we refer to this dataset as D1418. The dataset consists of 259,230 applications covering the five-year period of 2014 to 2018. As in related works, we use data from 2014 to train the classifiers.
Classifiers
Linear Support Vector Machines, referred to as SVM, was chosen as it was the classifier initially used in DREBIN. Deep Neural Networks, referred to as DNN, are another common approach for malware classification. We use the same RF settings of 101 decision trees and a maximum depth of 64.
| Training | Testing | ||||
|---|---|---|---|---|---|
| 2014 | 2015 | 2016 | 2017 | 2018 | |
| Goodware | 52,043 | 28,169 | 36,782 | 60,000 | 55,849 |
| Malware | 5,697 | 3,065 | 3,973 | 7,201 | 6,451 |
| Total | 57,740 | 31,234 | 40,755 | 67,201 | 62,300 |
| Sampling (active learning rate) | |||||
| 2% | 625 | 815 | 1,344 | 1,256 | |
| 4% | 1,249 | 1,630 | 2,688 | 2,492 | |
| 8% | 2,499 | 3,260 | 5,376 | 4,984 | |
| 16% | 4,997 | 6,521 | 10,752 | 9,968 | |
Operational Cost
For the passive recovery evaluation, the uncertainty sampling of 2%, 4%, 8%, and 16% of the monthly samples was used. The granularity of the sampling settings was chosen because it has a higher concentration in lower, more realistic percentages, while maintaining sizable changes to allow variance in the scenarios evaluated. The consequence of using higher percentages is increased manual labeling requirements that may exceed a realistic labeling budget. Miller et al. suggested two possible labeling budgets of 10 and 80 samples per day; assuming an approximate 250 working days per year, the yearly manual labeling budgets would be 2,500 and 20,000, respectively. Given the aforementioned yearly labeling budgets and the sampling requirements for our approach, barring one year, 4% active learning fits into the 10 sample a day budget, and all fall well within the 80 sample a day budget.
Parameters and Metrics
Model updates occur periodically according to a predefined temporal granularity, with a monthly granularity being used in this research. We use F1-Score (F1 = 2×TP / (2×TP + FP + FN)) as our performance metric in this research.
Since exact recovery is unlikely, we introduce the concept of the tolerance margin, which defines how much approximation is tolerated when assuming that the model has at least the same performance as the hypothetical model. The tolerance margin represents how strict the definition of recovery is in the evaluation. A tolerance margin of 0.05 will report better recovery results, compared to a tolerance margin of 0.01, but will tolerate more errors. Visually, this means that the performance of the poisoned model must overcome a “lower version” of the performance of the hypothetical model.
The parameter δ for the experiments in this research was set to 0.05, 0.01 and 0.00, to show a varying level of strictness—including 0.00 tolerance for absolute recovery.
Definition 2.2 (Intercept)
Given a tolerance margin δ, the intercept I is a metric defined as the first period τ where P(fMP(Xτ), yτ) ≥ P(fM(Xτ), yτ) − δ. The intercept reports the month in which a model is initially deemed recovered. However, it is insufficient on its own as performance can fluctuate over time due to statistical noise, residual poisoning, and residual drift.
Definition 2.3 (Recovery Rate)
The recovery rate reports the stability of recovery, as it is the percentage of months in which the model maintained recovery after the initial intercept I.
Comparing Recovery
To compare the recovery of two models for a specific set of settings, both the intercept rate and the recovery rate should be considered. Lower intercepts are better as the model reaches the tolerance margin earlier. Higher recovery rates are better, as they imply that the model maintains recovered performance more consistently.
Superior recovery
A model’s recovery is deemed superior to another model’s recovery if either both metrics (intercept and recovery rate) are (i) strictly better than the other model’s metrics, or (ii) one metric is better and the other metric is equal to the other model’s.
Mixed result
The comparison is deemed mixed if (i) both metrics are equal, or (ii) each model is better in one of the two metrics.
To the best of our knowledge, the metrics defined in this subsection are the first temporally-aware poisoning recovery metrics for malware detection.
Evaluation
Our evaluation shows that passive recovery using active learning is capable of achieving an average intercept and recovery rate, across all settings and classifiers, of 9 months and 70%, respectively, for a tolerance margin of 5% given the evaluated threat model.
- RQ-INTERCEPT: How do active learning and poisoning rates affect the speed of passive recovery?
- RQ-RECOVERY-RATE: How stable is the passive recovery over time across different settings?
- RQ-CLASSIFIER: Does the type of classifier used affect the passive recovery of the poisoned model?
Recovery Results Table
For a general overview on more combinations, Table 3 reports tables with passive recovery results of the complete combination of active learning and poisoning rates for tolerance margins: 0.05 (5%), 0.01 (1%), 0.00 (0%). Each table reports the results of each setting via its intercept and recovery rate. The intercept is as defined in Definition 2.2 and is presented as the test month in which the poisoned model performs within the tolerance margin of the hypothetical model. The recovery rate is defined in Definition 2.3 and is presented as the percentage of months after the intercept within the tolerance margin.
| Active learning | Poisoning rate | |||
|---|---|---|---|---|
| 2% | 4% | 8% | 16% | |
| 0% (none) | 10%from month 1 | 12%from month 24 | 0%X, never reached | 0%X, never reached |
| 2% | 75%from month 1 | 67%from month 16 | 67%from month 22 | 52%from month 22 |
| 4% | 90%from month 1 | 88%from month 9 | 79%from month 15 | 67%from month 22 |
| 8% | 83%from month 1 | 74%from month 2 | 83%from month 19 | 82%from month 21 |
| 16% | 98%from month 1 | 87%from month 2 | 82%from month 9 | 96%from month 21 |
Linear SVM, tolerance margin 0.05. Cells: recovery rate, then intercept month.
Speed of Passive Recovery
When comparing the intercepts of experiments with the same active learning and poisoning rates, we can see that the intercept increases consistently as the rates increase. This shows that an increase in the poisoning rate has a larger impact than the active learning rate on the intercept. Poisoning could force substantially higher active learning rates to maintain passive recovery, as a result of two factors. Firstly, the known diminishing returns of active learning, which can be observed in the converging active learning rates in the fixed poisoning plots. Secondly, the ever increasing impact of poisoning, which can be observed by the growing separation in the poisoning rates in the fixed active learning plots.
Intercept month when active learning and poisoning rise together
Tolerance margin 0.05; active learning rate = poisoning rate; lower is better.
Show as table
| Active learning / poisoning | SVM | DNN | RF |
|---|---|---|---|
| 2% / 2% | 1 | 1 | 1 |
| 4% / 4% | 9 | 1 | 9 |
| 8% / 8% | 19 | 7 | 11 |
| 16% / 16% | 21 | 15 | 14 |
RQ-INTERCEPT. Higher poisoning rates of the training dataset result in a delayed intercept, even if the active learning rate is much higher than the poisoning rate. This also corresponds to the diminishing returns in the reduction of time to intercept for increasing active learning rates.
Stability of Passive Recovery
Throughout the three classifiers, we see comparable stability in passive recovery, with an overall average recovery rate of 55% in all classifiers, settings, and tolerance margins. For the tolerance margins 0.00, 0.01, and 0.05, the average recovery rate is 42%, 53%, and 70%, respectively. The comparable average recovery rate between classifiers under the same tolerance margin shows that the choice of classifier does not have a significant impact on the stability of recovery rate across multiple settings. The varying average recovery rate of the different tolerance margins shows the importance of considering the tolerance requirement for an application, as passive recovery may be sufficient under certain tolerance margins, but insufficient under others.
Average recovery rate by tolerance margin and classifier
Comparison of the classifiers’ recovery stability across the 20 settings separated for three tolerance margins.
Show as table
| Classifier | Margin 0.00 | Margin 0.01 | Margin 0.05 | Average |
|---|---|---|---|---|
| SVM | 45% | 52% | 65% | 54% |
| DNN | 46% | 52% | 72% | 57% |
| Random Forest | 36% | 57% | 73% | 55% |
| Average | 42% | 53% | 70% | 55% |
RQ-RECOVERY-RATE. The comparable average recovery rate between the different active learning and poisoning rates between classifiers for a specific tolerance margin shows that the recovery rate is more dependent on the active learning and poisoning rates than the choice of classifier.
Impact of Classifiers on Passive Recovery
The experiments for SVM, DNN, and RF all use the same: dataset D1418, top 10,000 features, and poisoned samples under the same poison rates, which allows the evaluation of the impact the classifier has on passive recovery. 8% is chosen for both fixed active learning and poisoning rates, as it examines a moderate strength that lies at the mean of the rates evaluated.
Across the four poisoning settings, SVM has the best performance 4% of the time, DNN has the best performance 49% of the time and RF has the best performance 47% of the time. Across the four active learning settings, SVM has the best performance 11% of the time, DNN has the best performance 50% of the time, RF has the best performance 38% of the time, with the remaining 1% tied.
DNN is the best recovering classifier with superior recovery for 24 of the settings compared to SVM and 17 of the settings compared to RF, in the three tolerance margins. RF is the second best recovery classifier with a superior recovery in 19 settings compared to SVM. Across both F1-Score and passive recovery, the best performing classifier was DNN followed by RF and then SVM.
| Row superior to column | SVM | DNN | RF |
|---|---|---|---|
| SVM | X | 4 | 8 |
| DNN | 24 | X | 17 |
| RF | 19 | 6 | X |
RQ-CLASSIFIER. The selection of classifier has a impact on passive recovery and F1-Score. Although the classifier does not have a large impact on the average recovery rate, classifiers with more flexible (nonlinear) decision boundaries (DNN & RF) outperforms the more rigid (linear) decision boundary of SVM in both F1-Score and overall recovery.
Open Research Directions
We consider a very specific domain and setting: passive recovery from dirty-label (label-flip) or clean-label (feature-flip) poisoning with uncertainty-sampling active learning, in the context of Android malware detection. Despite the very focused scenario, we were able to derive interesting conclusions and propose a generalizable evaluation that can open up future investigations on more poisoning and passive recovery strategies.
- Problem-space Attacks. For simplicity, we consider only feature-space attacks, by assuming that the attacker is able to poison a dataset by flipping the labels or feature values of existing apps. Future research to explore the impact of problem-space data poisoning attacks on passive recovery would improve the transferability of these findings to practitioners.
- Poisoning-aware Passive Recovery Strategies. The development of poisoning-aware concept drift mitigation strategies will not only improve the robustness of the system as a whole to an attack, but also improve the passive recovery capability of that system.
- Time-aware Poisoning. In this paper, we assume the attacker can poison the model only once at training time. However, it would be illuminating for future research to explore whether there are feasible scenarios in which the attacker may want to poison a model in a “low and slow” fashion (similarly to advanced persistent threats), and how that can impact the overall model performance over time.
- Relationship with Poisoning Mitigations. Here we take an orthogonal direction and investigate whether retraining over time can help in forgetting poisoning, and provide an evaluation framework and metrics for evaluating its impact. A complete classification system should contain passive and active approaches to dealing with poisoning.
- Feature Representations. This research evaluated the feature representation DREBIN as it is one of the most common feature spaces released so far in the Android malware domain. However, there are other feature extractions yet to be explored in future work.
- Recovery Strategies. Active learning with uncertainty sampling was used as a recovery strategy in this research; however, there are other potential approaches to passive recovery. CADE or Transcendent could be used as a selection strategy for active learning instead of the uncertainty sampling used in this research.
Conclusion
- We identify passive recovery as a hidden factor that would impact the performance of an availability data poisoning attack in practice on a real system.
- We discover the trend of convergence in performance between poisoned models and their hypothetical non-poisoned counterparts.
- We demonstrated that active learning, a concept drift mitigation strategy, can indeed facilitate passive recovery from availability data poisoning, and achieve convergence with its hypothetical non-poisoned model’s performance.
- The speed of passive recovery, as well as the model performance, depend on the chosen classifier and passive recovery parameters.
- Furthermore, the stability of passive recovery is consistent between the classifiers evaluated, demonstrating that they are capable of passive recovery.
We hope that the findings in this paper will encourage the development of poisoning-aware concept drift mitigation strategies, as well as broaden the perspective of adversarial machine learning to the non-stationary nature of malware classification.
We have implemented our passive recovery evaluation as a Python library. We are releasing the framework to promote future research on the evaluation of passive recovery of other systems. This can also be used to replicate all the experiments in the current paper. The repository can be found at: https://github.com/s2labres/RPAL.
Cite
@inproceedings{mcfadden2024recovery,
title={The Impact of Active Learning on Availability Data Poisoning for Android Malware Classifiers},
author={McFadden, Shae and Kan, Zeliang and Cavallaro, Lorenzo and Pierazzi, Fabio},
booktitle={Proc. of the Annual Computer Security Applications Conference Workshops ({ACSAC Workshops})},
year={2024}
}