SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity

Deep Reinforcement Learning (DRL) has successfully improved the speed of matrix multiplication; discovered new protein structures; and, achieved superhuman performance at game playing. Through continuous interaction with a DRL environment (often simply referred to as the environment), DRL agents learn sequential decision-making policies by optimizing the expected cumulative reward over time. Unsurprisingly, these capabilities have motivated applications in cybersecurity, where systems must make adaptive decisions in dynamic, adversarial environments. DRL applications range from detecting malicious activity to evading detection and from autonomously defending networks to discovering vulnerabilities.

However, the transition to cybersecurity introduces fundamental challenges compared to traditional domains where DRL has previously excelled, such as game playing, robotics, and computer chip design. In cybersecurity, adversaries actively adapt their strategies to evade detection, systems evolve over time, and defenders typically observe limited information about attacker intentions and system state. These characteristics violate core assumptions of Markov Decision Processes (MDPs), the model for DRL learning tasks, that presume fixed transition dynamics and complete state observability, which can cause agents to be vulnerable to adversaries. In reality, most security tasks are better characterized as Partially Observable MDPs (POMDPs), where agents must infer hidden dynamics from incomplete knowledge: an observation of the system state. Further compounding these challenges are additional task-specific requirements, from information constraints (privacy and access limitations) to practical deployment capabilities (query budgets, real-time constraints). Failing to address these can produce agents that perform well in controlled simulations, however, fail unpredictably and dangerously when deployed in operational environments.

To the best of our knowledge, ours is the first work to delineate the pitfalls that arise when applying DRL in security. In 2022, Arp et al. identified 10 pitfalls in 30 supervised learning-based security research papers, finding at least three pitfalls in each paper. More recently, Evertz et al. extended this analysis to LLM-based security research. They identified nine pitfalls, finding at least one in each of the 72 papers reviewed. While there are some parallels between previously identified pitfalls and our work, the sequential decision-making paradigm of DRL is different. This means that although similar in spirit, these parallel ideas manifest in fundamentally different ways. For example, invalid environment modeling opposed to incorrect dataset construction: they have different causes, impacts, and mitigations.

Our goal is not to discourage DRL applications in cybersecurity, as it holds genuine promise, but rather to establish methodological standards that enable the community to distinguish robust, deployable solutions from those that succeed only in idealized settings.

At its core, Reinforcement Learning (RL) problems learn optimal policies through environmental interaction formalized as MDPs. MDPs are defined by the tuple (S, A, R, T, γ), consisting of: a state space S, action space A, reward function R : S × A → ℝ, transition function T : S × A → Δ(S) (where Δ(S) denotes probability distributions over states), and discount factor γ ∈ [0, 1]. An RL ‘agent’ observes a state s_t, selects an action a_t according to the policy π, then receives a reward r_t, and transitions to the next state s_t+1. The agent then learns a policy that maximizes expected cumulative discounted reward: E[∑_t=0^∞ γ^tr_t].

Traditional RL algorithms such as Q-Learning maintain explicit tables of state-action values that do not generalize to unseen states, and become intractable for large state or action spaces, which is critical for cybersecurity applications. DRL addresses this limitation by using deep neural networks as function approximators, allowing generalization to unseen states and scalability to high-dimensional problems. The three primary paradigms of DRL are: value-based methods, which approximate the state-action values of Q-learning; policy-based methods, which directly approximate the action distribution of the policy; and, actor-critic methods that combine both paradigms.

The MDP formulation relies on the Markov property: that the future state depends only on the current state and action, not on the history of past states. This assumption is critical for DRL but can be violated in cybersecurity settings where attackers maintain a hidden state (e.g., compromised credentials, established persistence) not observable to defenders. In such cases, the problem is more accurately modeled as a POMDP, where the agent receives observations o_t from an observation function Ω : S → O rather than directly receiving states. POMDPs require agents to maintain belief states or use recurrent architectures to infer the hidden state from observation histories.

In this paper, we systematize 11 recurring methodological pitfalls that emerge when DRL is employed in cybersecurity (DRL4Sec). We organize these pitfalls across the four stages of agent development.

We conducted a systematic literature collection using IEEE Xplore, ACM DL, and Google Scholar. We performed our collection in two rounds, first in April 2025 using only Google Scholar for initial pitfall identification and second in December 2025 using all search tools to complete the corpus and update citation counts.

Significance Criteria. To ensure academic impact and quality, papers must satisfy at least one of the following criteria: (1) published in a first- or second-tier (CORE-ranked A* or A) venue in machine learning, software engineering, or security; or (2) demonstrate significant research impact by averaging more than 10 citations per year since publication from a reputable publisher.

Topic Criteria. We define the scope of papers to be included in our work as: direct applications of DRL to cybersecurity problems and have DRL as the primary methodological contribution. We then excluded several categories of related but out-of-scope work: (1) secondary studies (surveys, systematic reviews, etc.); (2) research on the security of DRL systems, i.e., non-DRL-based attacks against DRL agents (which are not applied to cybersecurity problems); (3) pure game-theoretic approaches without DRL; (4) non-deep RL methods (e.g., tabular Q-learning, which fail in complex environments); and (5) Multi-Agent RL (MARL) systems, a separate research domain (which require distinct formulations and analysis).

Review Process. We began with a pilot review of the 37 papers from the April collection to identify themes and commonalities. Using this review, we isolated the reoccurring issues in DRL4Sec papers and formalized the 11 pitfalls discussed in this work. We then evaluated the complete 66 papers for each of the identified pitfalls. Each paper was independently evaluated by two reviewers using the pitfall definitions.

Following the independent reviews, the initial prevalence scores achieved substantial agreement, according to a Cohen’s κ of 0.712 and the Landis–Koch interpretation. With a 81.5% total agreement on pitfall prevalence, the remaining disagreements (18.5%) around the severity of pitfalls for the papers reviewed were resolved through discussion between the reviewers. When evidence was ambiguous, we gave the benefit of the doubt to the paper by using the less severe pitfall classification to resolve the disagreement (e.g., marking as not present if reviews were split between not present and partially present).

Corpus Characteristics. The 66 papers in our corpus were published between 2018 and 2025 across a diverse set of cybersecurity tasks.

These domains represent the major application areas of DRL4Sec, showing concentration in areas requiring complex sequential decision-making. First, we consider Autonomous Cyber defense (ACD) as these environments are well established and often reused across subsequent works. Then, we consider adversarial malware generation and web security testing as they are some of the most prominent applications of DRL4Sec across the 66 papers studied (22.7% and 15.2%, respectively).

Through systematic analysis of 66 DRL4Sec papers, published between 2018 and 2025, we quantify the prevalence and severity of each pitfall. Our review reveals widespread issues where every paper contains at least two pitfalls, with an average of 5.8. Furthermore, across the individual pitfalls, 28.8% of papers fall victim to the least common pitfall (Underlying Assumptions), with the most common being present in 71.2% of papers (Policy Convergence).

Most commonly, we find that: 71.2% lack clear evidence of policy convergence, 60.6% fail to address partial observability, 66.7% neglect variance analysis, and 40.9% evaluate in oversimplified environments.

The gap in convergence reporting suggests that although the need for convergence may be understood, the concrete demonstration of policy convergence and stability is often under-emphasized. Such prevalence indicates that partial observability is a pervasive challenge in DRL applications to cybersecurity. While the number of papers acknowledging the issue show that some awareness exists, it does not match the high prevalence of partial observability in cybersecurity tasks.

We conducted controlled experiments across three DRL4Sec domains to demonstrate the practical impact of the identified pitfalls. For all environments, their full MDP specifications are included in our repositories. To highlight pitfalls we performed didactic case studies, these are intended as illustrative examples where pitfalls have measurable, concrete, consequences in security environments. The specific consequence of a pitfall depends on the security problem, how it is modeled, implemented, algorithm chosen, and numerous other factors.

We use the Link XSS payload generation environment to show how modeling decisions impact performance. Although the original environment contains valid Markovian transitions, its design introduces unnecessary partial observability and misalignment with the practical security task. This arises from overlapping features in the state space, whereby multiple XSS payloads can be mapped to identical DRL states. For example, a single feature is used to indicate multiple HTML tags (e.g., img, video, audio or svg tags). The unintentional pitfalls introduced by this design can be rectified through reformulation of the MDP.

Thus, we consider two formulations of the MDP for this task: (1) Original, that have merged states; (2) Distinct States, that unrolls the conflated payload features into distinct values, increasing the state dimensionality by 12.8% from 47, to 53.

Modeling Correctness & Partial Observability. Introducing Distinct States for each possible payload aligns the MDP with the underlying security task avoiding M-MC. Therefore, the agent can develop correct payloads by selecting actions and observing the distinct impact they have on the current payload. Thus, removing the unnecessary partial observability, M-PO, from the environment. This effect can be observed concretely in Table 1 where including Distinct States increases the percentage of vulnerabilities found over Original by 10.1% and the raising lower bound performance by 17.3%. Therefore, through minor changes in the state space to remove the M-MC and M-PO pitfalls, we can observe significant improvements in performance.

MDP Specification. Although originally defined clearly, the performance difference between MDP formulations in Table 1 demonstrates that when components are left unclear, MDP Specification can severely impact final performance.

To evaluate the impact of pitfalls in policy training, we use MiniCAGE. Specifically, we evaluate four hyperparameter settings for T-HR, investigate variance and convergence for T-VA and T-PC, respectively.

Impact of Hyperparameters. We examine the final performance at 2.5 million steps and see how changing a single hyperparameter results in drastically different performance, highlighting how hyperparameter transparency is needed to ensure reproducibility.

Analyzing Variance. Concretely, at 500k the upper CI bound of the ‘Alt Learning Rate’ appears to be the second best performing agent. However, we see from the variance that an agent at this upper CI bound is not representative of the (lower) mean performance. By evaluating over multiple runs and reporting 95% CI, we obtain a more accurate picture of actual performance.

Demonstrating Policy Convergence. The ‘Default HPs’ and ‘Alt Clipping’ settings show relative convergence at 2.0-2.5 million steps where plateauing performance curves and minor variance can be seen. In contrast, the ‘Alt Learning Rate’ and ‘Alt Discount Factor’ are still increasing in performance with higher variance at 2.5 million steps.

Here we consider the three pitfalls we identified in evaluating agents: first, the attribution of gains in the AutoRobust environment; second, the theoretical motivation of DRL in adversarial malware generation; third, environment complexity comparisons in the SQIRL environment for SQL injection.

Identifying the Source of Gains. To assess the impact of the E-GA pitfall, the goal in the AutoRobust environment is to disentangle the contributions of different elements in the approach. First, we train and evaluate policies with full access to the internal state of the malware detection model, referred to as MDP. Second, we consider policies without internal state access, named POMDP, to evaluate the performance of learning under a more restrictive and realistic threat model. Finally, we evaluate Random policies that sample actions at random, to isolate the intrinsic efficacy of the action space itself. There are two key takeaways. First, full observability does not guarantee better performance, as agents can struggle to learn the task even with fully observable information. Second, while trained policies (MDP & POMDP) have the potential to achieve better performance, the random baseline is already strong. This demonstrates the importance of E-GA as the action space alone provides a large portion of overall performance.

Motivating the Application. As an example of theoretical motivation, in the domain of adversarial malware creation, DRL has several key capabilities over existing gradient-based attacks. Concretely, gradient-based methods assume a fixed input size, limiting exploration to only a very small subset of the actual transformations an adversary can make. Similarly in malware, gradient-based perturbations on the feature representation do not have a consistent and reliable method for being mapped to valid changes in the problem-space, i.e., the actual program. In contrast, DRL can operate directly on malware samples, bypassing the need for feature to problem-space mapping. Furthermore, given an adversary with a well defined sets of actions, at convergence DRL can additionally confer probabilistic guarantees on the level of vulnerability/robustness.

Changing Environment Complexity. To show the effect of E-EC we use the SMB benchmark from SQIRL. Using the SMB we train two policies, one on the default SMB with a Combined sanitized and non-sanitized vulnerability set, and one policy with the same vulnerabilities without sanitization (Non-Sanitized). We evaluate both trained policies on the SMB test set of sanitized and non-sanitized vulnerabilities; Table 2 shows the percentage of vulnerabilities found across 20 runs with 95% CI.

We see both the Combined and Non-Sanitized agents are capable of discovering vulnerabilities in the simplified non-sanitized settings. However, when changing the evaluation set to the more challenging Sanitized vulnerabilities we see a 47.4% drop in vulnerabilities found for the Combined agent, with a further 11.3% for the non-sanitized agent. This demonstrates how simplified evaluation environments can inflate performance, producing promising results that fail to translate into real-world capabilities.

We evaluate the impact of pitfalls in deploying DRL agents using MiniCAGE. First, we consider Underlying Assumptions by modifying the assumed fixed action order between attacker and defender during testing, blue then red (B→R) in the original formulation. Second, we study Non-Stationarity by changing the fixed attacker strategy between training and testing.

Breakdown of Underlying Assumption. The first section of Table 3 shows the average reward of the DRL defender when varying the action order of red and blue agents in training and testing (e.g., R→B indicates red then blue and Mixed indicates random turn order). As expected, agents perform comparably when tested on the same training order (−17.2 for B→R and −15.8 for R→B). When the assumed order changes, performance degrades: minimally for R→B agent (−21.0) and drastically for the B→R agent (−72.7). These results highlight how minor design choices can degrade performance if the assumptions they rely on do not hold in practice.

Changing Attacker Strategy. Table 3 shows the average reward of defender agents under varying attacker strategies: B-Line and Meander. The agent order is kept constant as B→R. We see a degradation in performance when any stationary blue agent is evaluated against unseen or mixed attacker strategies. This degradation can be mitigated by including non-stationary attacker strategies during training as the Mixed blue agent outperforms the stationary defenders in both mixed and cross evaluations. These results show that policies can be brittle to changing conditions when trained without accounting for non-stationarity.

We encourage researchers to carefully consider the pitfalls of DRL for cybersecurity and, where applicable, apply the recommendations from this work.

The autonomous and adaptive decision-making that DRL can offer in complex environments has significant potential for cybersecurity applications. However, our review of 66 DRL4Sec papers published between 2018-2025 reveals 11 common methodological challenges (pitfalls) that undermine DRL4Sec research. Through case studies in autonomous cyber defense, adversarial malware creation, and web security testing, we demonstrate these pitfalls directly impact performance and create potentially brittle policies.

Through the identification of 11 DRL4Sec pitfalls and recommendations for their mitigation, this work establishes higher methodological standards for future applications of DRL to cybersecurity.