Introduction
This paper investigates how retraining strategies help recover the model after training-time data poisoning attacks. Firstly, we propose a formalization of recovery from poisoning by introducing two new metrics: intercept and recovery rate. Finally, we propose RPAL as an extensible framework that can assess the recovery rate of an ML model for a given set of parameters, models, retraining and poisoning strategies.
We evaluate RPAL in the context of Android malware detection to quantify recovery from label-flip poisoning via active learning retraining based on uncertainty sampling using the dataset from Pendlebury et al. which contains the Drebin and MaMaDroid feature spaces. Our results demonstrate that an ML model can recover from a poisoning attack subject to the strength of the poisoning attack and retraining mechanism. Different feature abstractions under the same classifier can greatly affect the speed of recovery. We show interesting trade-offs in the defender’s diminished return in poison recovery when increasing active learning rates.
Novelty
Existing literature has indeed proposed some defenses for poisoning such as fine-pruning, however, to the best of our knowledge, we are the first to evaluate the recovery over time of a classification system from poisoning. Moreover, we are the first to consider it in the context of malware detection.
RPAL Evaluation Framework
Threat Model
In RPAL, the threat model requires defining both attackers and defenders capabilities, along with the attackers objective (e.g., increasing FNs or FPs). The attacker’s capability is defined via a poisoning strategy and the percentage of training data affected. Instead, the defender capability is defined via a recovery strategy (e.g., active learning) and the retraining budget.
A recovery strategy is a method which enables the model to adapt over time, however, recovery strategies come with their own operational costs (such as labeling cost). When deciding on a recovery strategy it is important to take into account a realistic operational budget. When choosing a poisoning strategy for evaluation it is important that it fits within the threat model chosen and that the adversarial manipulations performed are realistic.
Recovery Evaluation
For recovery evaluation, poisoned model MP’s and non-poisoned model M’s only difference should be the poisoning at initial training time. The initial input is a timestamped dataset, consisting of a feature matrix X, labels y, and time-stamps t.
Split in time
The dataset is temporally split into a training set and monthly testing sets via the Tesseract framework.
Rebalance and poison
In the time-aware evaluation, the training dataset is rebalanced to a realistic class distribution (e.g., approximately 10% in the case of Android malware). The dataset is then poisoned based on the current iteration’s poisoning rate (starting from 0%, for a clean model baseline), and is then used to train the model.
Predict
The model predicts on the current test month data.
Sample and retrain
After prediction, the recovery strategy (e.g., active learning) selects samples to retrain the model on. The process of train, predict and sample is then repeated for all testing months.
Repeat for every setting
After all test months have been evaluated then the process is repeated for all poisoning and recovery strategy rates in the evaluation settings.
Parameters and Metrics
The intercept aims to estimate when the poisoned model’s performance approximates the retrained vanilla model. Since model performance exhibits fluctuations in non-stationary context, the recovery rate aims to quantify the percentage of time that the poisoned model has a performance equal to or higher than the vanilla retrained model. We introduce the concept of tolerance margin, which defines how much approximation is tolerated when assuming a model is recovered.
Tolerance margin δ
The tolerance margin δ (δ ∈ ℝ, 0 ≤ δ ≤ 1) is an input hyper-parameter of RPAL that is subtracted to the monthly performance of a clean model M, such that the monthly performance of the poisoned MP if compared with that of M minus δ.
Intercept I
Given a tolerance margin δ, the intercept I is a metric defined as the first month τ where MP’s monthly performance is greater than or equal to M’s monthly performance minus δ.
Recovery rate R
The recovery rate R is defined as the percentage of months that MP’s performance is greater than or equal to M’s performance minus δ, after the intercept I.
To the best of our knowledge, the metrics defined in this paragraph are the first temporally-aware poisoning recovery metrics.
Evaluation
We consider two research questions: does the feature space affect the recovery rate? (RQ-Feature) and how do the active learning and poisoning rates impact the recovery time? (RQ-Intercept).
Experimental Settings
Android apps, 2014–2016
We use the Tesseract dataset by Pendlebury et al., which ranges from 2014 to 2016 and offers two feature abstractions: MaMaDroid and Drebin. We train on the 57,740 samples from 2014 and test on the 71,988 samples from 2015/2016 however, a lack of samples in month 24 of testing causes performance drop off.
Drebin and MaMaDroid features
The MaMaDroid and Drebin experiments both utilize the same underlying dataset, which allows the evaluation of the impact feature space has on recovery.
Attack: label-flip poisoning, 2–16%
We assume the attacker has the capability to fully access the training data and is utilizing label-flip poisoning to compromise the general availability of the model with granularity 2% to 16%.
Recovery: active learning, 2–16%
We assume the defender relies on active learning with uncertainty sampling as the recovery strategy for our evaluation with strength from 2% to 16%.
Random Forest classifier
We utilize the Random Forest classifier with the settings of 101 decision trees with a max depth of 64, the same as the original paper.
Tolerance margin δ = 0.02
The tolerance margin δ for the experiments in this research was set to 0.02.
Compare Different Feature Abstractions
With a constant active learning rate, the plots show a similar trend between the different rates of poisoning, however, when comparing the monthly F1 performance of the same poisoning rates between the two feature abstractions with the same fixed active learning rate as in the plots, MaMaDroid has better performance 1% of the time and Drebin has better performance 98% of the time across the four poisoning settings.
With a constant poisoning rate, the plots show a similar starting performance, however, when comparing the monthly F1 performance of the same active learning rates between the two feature abstractions with the same fixed rate as in the plots, MaMaDroid has better performance 4% of the time, Drebin has better performance 95% of the time with the remaining 1% being tied across the four active learning settings.
When comparing the two feature extractions via the tables, we can see that although MaMaDroid’s performance is worse than Drebin, it outperforms Drebin in recovery. When comparing the recovery across the 16 settings, MaMaDroid is better in 8, Drebin is better in 0, and 8 are deemed mixed results.
The feature abstraction has a significant impact on recovery, and a better-performing system does not equate to a better-recovering system. Although MaMaDroid’s performance is worse than Drebin, MaMaDroid achieves better recovery statistics on D1416.
Speed of Recovery
When comparing the intercepts of experiments with equal active learning and poisoning rates, we can see that the intercept consistently increases as the rates increase. This shows that an increase in poisoning rate has a larger impact on the intercept than the active learning rate. Poisoning could force a substantially higher active learning rate, which is the result of two factors: firstly, the known diminishing returns of active learning, which can be observed in the converging active learning rates in the fixed active learning plots; secondly, the ever increasing impact of poisoning, which can be observed by the growing separation in the poisoning rates in the fixed active learning plots.
Intercept month when active learning and poisoning rise together
Tolerance margin 0.02; Random Forest; active learning rate = poisoning rate; lower is better.
Show as table
| Active learning / poisoning | MaMaDroid | Drebin |
|---|---|---|
| 2% / 2% | 9 | 9 |
| 4% / 4% | 11 | 12 |
| 8% / 8% | 14 | 14 |
| 16% / 16% | 21 | 19 |
Higher poisoning rates of the training dataset result in a delayed intercept, even if the active learning rate is much higher than the poisoning rate. This also corresponds to a diminished return in recovery speed for increasing active learning rates.
MaMaDroid and Drebin Results
MaMaDroid Results
Table 1: MaMaDroid feature space results.
| Active learning | Poisoning rate | |||
|---|---|---|---|---|
| 2% | 4% | 8% | 16% | |
| 2% | 75%from month 9 | 64%from month 11 | 83%from month 19 | 75%from month 21 |
| 4% | 88%from month 9 | 64%from month 11 | 71%from month 11 | 67%from month 22 |
| 8% | 74%from month 2 | 50%from month 7 | 64%from month 14 | 100%from month 24 |
| 16% | 73%from month 3 | 85%from month 12 | 89%from month 16 | 75%from month 21 |
Tolerance Margin = 0.02. Each cell: recovery rate (large) and intercept month (small).
Drebin Results
Table 2: Drebin feature space results.
| Active learning | Poisoning rate | |||
|---|---|---|---|---|
| 2% | 4% | 8% | 16% | |
| 2% | 62%from month 9 | 44%from month 16 | 50%from month 21 | 0%X |
| 4% | 82%from month 8 | 62%from month 12 | 33%from month 19 | 0%X |
| 8% | 78%from month 7 | 71%from month 8 | 64%from month 14 | 0%X |
| 16% | 86%from month 4 | 80%from month 10 | 82%from month 14 | 67%from month 19 |
Tolerance Margin = 0.02. Each cell: recovery rate (large) and intercept month (small). X: no intercept within the test period.
Conclusion
- We propose new metrics and parameters for understanding recovery from poisoning over time.
- We propose an evaluation framework, RPAL, through which we demonstrate trade-offs of poisoning-recovery in the Android malware context and confirm that feature abstractions impact recovery.
- We demonstrated that drift mitigation strategies can indeed facilitate recovery of the model whereby the model forgets the poisoning effect over time.
- However, the speed of recovery heavily depends on the components of the system and data considered.
Our initial investigation paves the way for further research on this topic which in addition to exploring more scenarios, could also investigate drift mitigation strategies that take poisoning into account for optimal recoveries, as well as evaluating impact of existing poisoning defenses to time-aware recovery.
Follow-up paper: The Impact of Active Learning on Availability Data Poisoning for Android Malware Classifiers (ACSAC Workshops 2024). “This paper extends our previously published poster.”
Cite
@inproceedings{mcfadden2023rpal,
title={Poster: RPAL-Recovering Malware Classifiers from Data Poisoning using Active Learning},
author={McFadden, Shae and Kan, Zeliang and Cavallaro, Lorenzo and Pierazzi, Fabio},
booktitle={Proc. of {ACM} Conference on Computer and Communications Security ({CCS})},
year={2023}
}