Introduction
Cyber attacks pose serious risks to individuals, businesses, and governments, whose daily operations all depend on networked systems. In recent history these risks have manifested as costly attacks on critical systems including nuclear facilities and power grids.
To counter such threats, agent-based Autonomous Cyber Defence (ACD) systems that can immediately monitor, adapt, and respond are key. In particular, Reinforcement Learning (RL), a subset of Machine Learning (ML) that learns through interaction with an environment has received significant interest in the literature. The unique advantage of RL, compared to agents based on pre-trained generative models, is the ability to learn how to achieve a specific goal exclusively from interacting with an environment. RL agents, therefore, do not rely on prior human knowledge or understanding. This is especially important for ACD as adversaries frequently exploit the assumptions made when systems were designed and configured.
For RL to succeed at ACD, there are at least three core challenges:
Fidelity
The environment must accurately and efficiently represent the real-world network defence problem at an appropriate level of fidelity (e.g., via a cyber environment) including, but not limited to, the network topology, attacker behaviour, and the functionality of software components.
The interface
Proper consideration must be given to the interface between the learning agent and the environment, including the identification of a suitable numerical reward mechanism.
Evaluation
A robust evaluation methodology, including appropriate measures of success and statistical significance, must be implemented.
Towards addressing the core challenges in ACD, this work makes the following main contributions:
- We propose a framework for decomposing the components and modelling choices involved in mapping between ACD environments and real systems.
- We provide guidelines on best-practice in relation to each framework component, based on the findings from our workshop, providing a valuable reference for anyone engaged in cyber environment development.
Workshop Methodology
This paper is informed by a structured online workshop bringing together 25 domain experts from across academia, industry and government with experience in RL-based cyber defence. The workshop included focused discussions and interactive activities to extract limitations, design considerations, and best-practice recommendations for ACD. These were explored from three distinct perspectives: security, RL, and environment development. Insights were gathered via collaborative tools and facilitator notes, then processed through thematic analysis into the framework and best practice recommendations presented in this work.
RL for Cyber Defence
It is the authors’ view that RL solution methods (e.g., model-free, on-policy) and algorithms that apply them (e.g., PPO, DQN) are not the main bottlenecks in RL for ACD. Instead, the core challenge of ACD remains how best to formulate a particular network defence problem within the RL framework.
Informally, the RL learning problem is one where an agent learns how to act in an environment over time to achieve a goal. At each step, the agent selects an action, and then receives an observation and a reward. Through sequential interaction with the environment, the agent learns a policy that aims to maximise the cumulative expected reward. This interaction is modelled as a Markov Decision Process (MDP) with the assumption (a.k.a the Markov property) that the future is independent of the past given the present. The horizon for future rewards is most often bounded by an episode, which starts in an initial state and ends upon arriving at a terminal state (or condition). Finally, when the state space of the environment is not able to be fully observed by the agent, the task is instead modelled as a partially observable MDP (POMDP).
Cyber Defence as an RL Task
Some famous RL tasks (e.g., chess) fit the RL paradigm without significant alteration. However, cyber defence is a complex set of real world problems that does not naturally present a single RL task. Beyond comprising multiple tasks and objectives that may change across time, cyber defence is a challenging application for RL because the state of the environment is both extremely large and almost never completely known; adversaries and users introduce non-stationary dynamics; representing cyber security goals as scalar rewards is difficult; operational constraints may be difficult to represent in the action space; and accurate simulators are necessary (e.g., for safety and efficiency) but challenging to build and interface with correctly.
Although cyber defence is not a single problem, cyber environments have thus far converged on a relatively constrained interpretation: a defensive blue agent is tasked with defending a network of hosts against one or more fixed-strategy adversaries.
Action space
The action space is usually an enumeration of high-level command and control (C2) techniques (e.g., scan, deny, restore).
Observation space
The observation space (i.e., the real network state is not completely known) is a vector of selected attributes from the underlying network (e.g., one bit per host to indicate whether a compromise has been detected).
Reward function
The reward function is typically engineered by domain experts and includes penalties and/or incentives for various network states and actions (e.g., -2 when a host is compromised, -1 for restoring a host, +0.1 for making no intervention).
In other words, defending the network becomes the task of minimising the number of compromised hosts whilst making conservative use of costly actions.
Thus, this network defence task of ACD is commonly framed as a finite horizon sequential decision-making problem in which a blue agent receives partial observations from an environment comprising a network of hosts, an adversarial red agent, and possibly benign green agents representing regular user activity. Crucially in this framing, the environment comprises everything that is not the blue agent: the network of hosts, regular users, and the adversary.
Since the real network is likely of operational significance, and the RL learning framework includes learning from “mistakes” to choose better actions over time, a virtual cyber environment is the standard way to train a network defence agent. Whether blue agents are useful for the real network defence problem therefore depends significantly on how well the cyber environment models the real environment.
A Framework for ACD Environments
The divergence between environments and real-world systems, known as the sim-to-real gap, hinders the transfer of agent performance from training to deployment. Designing high fidelity network environments can help to close the gap but is not entirely sufficient. Equally, the formulation of the RL task matters as it fundamentally constrains both what and how (i.e., learning dynamics) the agent learns. Thus, the sim-to-real gap comprises two key components: virtualisation of the environment and modelling of the task.
Virtualisation gap
If RL agents could be trained on a real production network, with real users and attackers, then there would be no virtualisation gap. The virtualisation gap can be split into two components: (1) the network and host simulation including its topology and operational characteristics, and (2) user and threat simulation.
Modelling gap
The modelling gap includes observation modelling (e.g., a vector derived asynchronously from network state information), action modelling (e.g., an enumeration of actions and how they are executed on network hosts), sequence modelling (i.e., how POMDP time steps relate to the occurrence of network events) and reward modelling (i.e., how to provide scalar rewards in alignment with real-world goals). Modelling choices significantly impact how effective learned agent policies are, irrespective of the simulation gap, but are also constrained by simulation fidelity.
Insufficiency in either simulation or modelling limit the real world performance ceiling of RL agents by creating poorly aligned and artificially challenging RL tasks.
The Virtualisation Gap
Network and Host Simulation
High-fidelity simulation of the network and hosts is both feasible, owing to widely available virtualisation software, and preferable for minimising the simulation gap. Low fidelity cyber environments offer high efficiency, making it cheaper to train RL agents, but introduce the significant risk of simulator artifacts. Agents learning from artifacts may perform very well in the simulator yet behave poorly, and unpredictably, in the real world.
User and Threat Simulation
Both regular network users and adversarial threats are an essential part of modelling a network defence problem. Within a cyber environment these are represented by green and red agents, respectively. In proportion to the realism of their behaviour both green and red agents have a large impact on the simulation gap. Since red agents model adversarial behaviour, they determine the distribution and severity of threats encountered by the blue agent in the environment. Therefore, if the strategies they simulate are not sufficiently realistic (e.g., simplistic, predictable, or overt) then the defensive policies trained will be brittle with marginal success transferring to real-world deployments. Green agents are of equal importance, as without benign user simulation the blue agent may learn policies that succeed at defence but significantly disrupt normal traffic.
The Modelling Gap
Hundreds of processes run concurrently on modern operating systems and network packets are sent, reordered, dropped and received between devices without coordinated clocks. Sequence modelling captures the temporal relationship between continuous, asynchronous network activity and the discrete-time sequence of agent actions, observations and rewards required for RL. Sequence modelling is crucial because the action, observation and reward modelling components must preserve causal ordering, allowing meaningful state transitions and reward attribution, which all depend on understanding how the network, hosts and other agents may advance between each time step.
The MDP discount factor γ, which determines an effective prior over the relevance of future rewards, and the RL task episode length when fixed, both typically assume fixed per-step wall time and therefore depend greatly on sequence modelling.
Observation Modelling
The true underlying state of the network is far too large to input to any practical algorithm (e.g., a single workstation hard disk may have 1TB of state information) and is distributed over many devices. Thus, the network state must be aggregated and parsed to provide vector observations for the RL agent.
Action Modelling
In network defence, operations are usually carried out using highly expressive graphical and command-line user interfaces. Action modelling involves constructing a set (discrete, continuous, or both) of actions that map to a useful set of choices relevant to the network defence task.
Reward Modelling
The goal of network defence is to minimise adversarial disruption while maintaining normal operations for non-malicious users. Reward modelling involves identifying scalar incentives and penalties which correspond to achieving one or more network defence goals when their combination is maximised over the discounted horizon.
Towards Better Cyber Defence Environments
This section distils best-practice insights from domain experts into actionable guidelines, designed to maximise RL task effectiveness and real-world transfer, structured around the framework. As the main focus of academic work to date, this best-practice focusses on single-agent ACD environments for network defence.
Characterising the Real-World Problem
Before any simulation or RL modelling decisions are made, it is essential to clearly define the target network defence problem. The problem scope will determine which real-world dynamics must be captured, which abstractions may be acceptable, and how problem complexity can be introduced over time.
- Specify the Problem, Constraints & Success Criteria
- The first step is to fully specify the network defence problem to be investigated. The specification should characterise the network environment (e.g., topology, scale, host details and roles), operational and business requirements (e.g., user data must have high availability), assets to be protected (e.g., critical hosts, sensitive data), adversarial assumptions (e.g., goals, tactics, techniques, and procedures), and the capabilities and constraints of defensive actors.
- Identify Areas of Uncertainty
- Uncertainty may arise from: the network environment (e.g., variations in network topology and host behaviour), the distribution of expected user and attacker profiles, and defender errors (e.g., malware misclassification). Characterising uncertainty helps to identify generalisation requirements and may improve alignment between the training, evaluation and deployment performance by allowing agents to be robust to unseen distributions of trajectories.
- Detail a Minimum Viable Problem
- The combination of a complex full-scale network defence problem, and well-known RL learning reliability challenges, is ill-suited to efficient debugging and early experimental iteration. Thus, it is important to construct a well-scoped minimum viable problem that can eventually scale to the required complexity. Once the core problem structure is understood, realism can be systematically reintroduced to recover the full-problem.
Virtualisation
The virtualisation gap constrains the behaviours, uncertainties, and causal relationships that an RL agent can learn from; setting an upper bound on the achievable real-world performance, irrespective of how sophisticated the subsequent RL modelling may be.
- Scope Existing Environments
- While a pragmatic first step is to review existing cyber environments, any suitably well-characterised (i.e., specific) network defence problem is likely to require significant customisation. If a new environment is judged necessary, then developers must decide between a simulation, emulation, or hybrid approach, balancing efficiency against the risk of simulator artefacts.
- Validate Connection to High-Fidelity
- While this may involve low-fidelity simulation initially, it is possible that the correct problem structure and causal factors are not fully understood. Thus, it is important to validate that the resulting observations, action feasibility, state transitions, and rewards correspond to those produced by realistic interactions between emulated or real-world software components, hosts, and network infrastructure. Implementing a bespoke mid-fidelity simulation should be approached with caution as there is a risk of both including unnecessary complexity and failing to match the real-world problem dynamics.
- Inform Design Choices through Problem Scale
- The problem scale should inform the choice of virtualisation technologies, and non-negotiable configurability (e.g., topology, host details, user and threat profiles), ensuring the feasibility of systematic scaling once the minimum viable problem has been satisfied. Fidelity, noise, and delay characteristics of monitoring information should reflect realistic operational constraints.
Network and Host Virtualisation
At a minimum, the virtualised network should reflect the topology, traffic flows, service dependencies, host roles, and configuration necessary for both defensive and adversarial actions to produce plausible effects. The network and host state information available to defenders should plausibly represent data produced by the suite of monitoring tools found in the real-world problem.
- Ensure Problem Structure is Preserved
- When a highly abstract network representation is justified by a simple problem structure, the resulting environment dynamics should be validated against a high-fidelity network environment, or using data from the real-world problem, to ensure that problem structure has been preserved as expected. Towards scaling from minimum viable problem to full-scale realism, network topologies, host profiles, services, and monitoring capabilities should be easily configurable.
- Examples of Best-Practice
- Current best-practice examples include Cyberwheel, which ensures network and host state information is designed to mimic real-world monitoring and detection pipelines, and NASimEmu which validates abstract red team simulation dynamics against high-fidelity emulation using a shared simulator-emulator interface.
User and Threat Virtualisation
User and threat virtualisation determine the distribution of benign and adversarial network activity encountered during training and evaluation. Both must be modelled with sufficient realism to ensure agents remain effective in the real-world, avoiding policies that depend on misrepresentative user activity or attacker behaviour.
- Move Beyond Fixed Attacker Strategies
- Fixed-strategy, deterministic red agents can be useful when implementing a minimum viable problem, as they remove a source of non-stationarity and thereby ease debugging and early experimental iteration. However, RL policies are known to be brittle under modest shifts in environment dynamics. Environments should therefore support diverse and configurable red-agent behaviour, including stochasticity, strategy switching, and adaptive decision-making. Where feasible, environments should support competitive multi-agent training; enabling red and blue agents to co-evolve, and exposing defender policies to diverse offensive strategies.
- Aim for Representative User Activity
- Legitimate user activity should be simulated with sufficient realism to reflect the diversity and variability of real-world network traffic. Where possible, real-world logs or statistically grounded models of network activity should be used to inform and validate user behaviour, rather than relying on scripted profiles.
- Consider Generative Green and Red Agents
- Alongside fixed or learning-based agents, emerging LLM-based, agentic adversaries may allow generating diverse, adaptive, goal-directed agent behaviour without requiring explicit modelling.
Sequence Modelling
Addressing sequence modelling remains an open challenge for RL in cyber defence, particularly given long-horizon dependencies, asynchronous dynamics, and persistent adversaries. However, explicitly reasoning about temporal structure, action duration, and concurrency is essential for closing the modelling gap and producing agents whose behaviour transfers to the real-world.
- Justify Episodic Structure and Horizon Length
- Many ACD environments divide agent-environment interactions into discrete episodes, either with a fixed number of time steps (i.e., fixed-horizon) or explicit terminal states (i.e., episodic). An episodic task formulation simplifies training and evaluation by ensuring that returns are finite and well-scaled, and also by providing a reset mechanism which can increase exposure to important states. However, episodic formulations impose strong assumptions about temporal structure that often do not hold in operational networks. Real-world networks are typically persistent and non-stationary, challenging the possibility of a “reset” and discrete episodes. Another difficulty is that episodic formulations impose a bounded decision horizon, eliminating the possibility of learning causal structure beyond the episode bounds and biasing RL agents towards short-term wins. Best practice therefore is to explicitly justify the temporal task modelling, based on the target problem, rather than adopting an episodic, fixed-horizon by default. If the problem is inherently continuous or long horizon, exemplified by Advanced Persistent Threat (APT) actors, who may gain access to a network months or years before executing their attack, then open-ended or infinite-horizon formulations should be considered. For example, Hammar and Stadler formulate network defence as an optimal stopping problem, where the defensive agent has two possible actions: “stop”, corresponding to a defensive intervention, and “continue”.
- Account for Temporal Variability and Action Duration
- In real networks, defensive actions are neither instantaneous nor uniform in duration. Scanning, isolating hosts, restoring services, and reconfiguring access controls may take seconds, minutes, or longer, and can overlap with other defensive or adversarial activities. Similarly, attacks unfold asynchronously, driven by independent processes and external timing constraints. In contrast, the standard RL assumption is that each time step corresponds to a fixed quantity of wall-time (i.e., each time step is weighted equally when calculating reward attributions). Thus, to support real-world transfer, it is best practice to represent temporal variability and action duration explicitly within the environment. Where possible, environments should support variable action durations, varying delays between action initiation and effect observation, overlapping and concurrent execution, and events occurring independently of agent actions. R3ACE implements continuous-time, event-driven RL modelling which is aligned with best practice, although only demonstrated on a very small problem space.
- Reconsider Turn-Based Modelling
- Many cyber environments model red, green, and blue agent interactions in a turn-based manner, implicitly enforcing an ordering of each agent’s action. This abstraction is not representative of real networks, where attackers, users, and defenders act concurrently, opportunistically, and without coordination. At baseline, it is important to be aware that turn-based agent interaction causes non-trivial impacts on RL learning outcomes in ACD. Ideally, turn-based modelling should be avoided except where the problem structure imparts an ordering distribution that can be well-determined. If turn-based structure is retained for tractability, then the sensitivity of trained policies should be evaluated.
Observation Modelling
Observations in ACD environments are often fixed-size, discrete vectors representing evidence about network status. However, workshop participants repeatedly highlighted that unrealistic “magic” observations and overly lossy encodings (e.g., minimal bit-vectors) undermine both realism and transfer, while also obscuring what a defender is assumed to know at decision time.
- Define a Realistic Observation Pipeline
- The information contained in observations should reflect what is genuinely available in practice from realistic sources such as monitoring tools, logs, and alerts. When observations are drawn directly from the raw simulator or emulator state, they risk incorporating features that could not be reliably produced in real deployments (e.g., listing if a host is compromised in the state). This creates a gap between the environment observed by the agent and real-world conditions, as these “magic” observations embed convenient but impossible information that undermine practical transferability. Equally, observations modelled directly on virtualised network state also risk containing less information than a cyber operator would typically have access to, under-representing true monitoring capability. Since magic observations cannot be reproduced outside of the virtualised environment, observation modelling must be grounded in practically available information. For example, the Cyberwheel environment demonstrates this by constructing observations directly from existing cyber detection tools.
- Consider Observation Representation
- For ACD with a single blue agent, the most common observation design is to aggregate information from all network nodes into a single observation vector, mimicking the design of most traditional RL environments. However, this approach can lead to limitations in expressive power and generalisability when using standard neural network architectures. An alternative is to factorise observations into individual network node features, and apply GNN or attention-based architectures, leveraging their permutation equivariance to produce policies that are more robust across varying network configurations.
Action Modelling
To enable learning effective strategies, agents need useful actions that correspond to real-world capabilities. Action modelling requires carefully mapping relevant capabilities into a set of actions that can be chosen from by the RL agent.
- Provide a Sufficient and Representative Action Space
- If the action space is very narrow or abstract the agent will likely be constrained to a small set of behaviours, limiting its capabilities below what may be effective in deployment. A representative action space should map to realistic operations that cyber defenders can perform on the network, preserving the temporal and causal relationships between system state, actions, and their outcomes in the environment.
- Consider the Granularity-Capability Trade-off
- Fine-grained actions can lead to large action spaces, increasing the difficulty of learning an effective policy. Whereas coarse abstractions (e.g. Mitre D3FEND or Open C2) risk removing decision-making capability and breaking causal realism. The granularity of actions should reflect the intended operator role and specified network defence problem. To deal with large combinatorial action spaces, first consider approaches from the RL action space decomposition literature.
- Mask Invalid Actions
- Allowing agents to select invalid actions can introduce undefined behaviour, and wastes training time on zero-value choices, rather than learning effective strategies. Molina-Markham et al. suggest using the Planning Domain Definition Language (PDDL) for modelling network defence tasks, including using action preconditions to determine masks.
- Ensure Effects are Reflected in Observations
- Valid actions that do not have observable consequences do not provide a useful learning signal and introduce noise into training that can impede policy learning. Furthermore, the success of a chosen action is not guaranteed in realistic deployments. Therefore, if available in practice, actions should have a discernible effect that can be observed by the agent, either directly or indirectly over subsequent states.
- Make Configuration Easy
- The interface should facilitate the addition, removal, or changing of action constraints to allow for the modelling of specific network defence subtasks. This flexibility enables experimentation with capability assumptions, promoting the reuse of principled environments and improving reproducibility.
Reward Modelling
The reward function fundamentally defines the learning objective of an RL agent, and thus the behaviour of the policy. This may be especially consequential for ACD, as an incorrectly specified reward function can lead to policies that appear successful in maximising long-term rewards, but fail to genuinely achieve defensive objectives.
- Motivate the Reward Function
- Reward functions should be motivated and justified with respect to the underlying defensive objective. The general goal in network defence is typically the sustained operation of a system that continues to meet user demands while hosts remain uncompromised.
- Align with the ACD Problem Goal
- Reward functions fundamentally assign relative value to states and actions, which can unintentionally introduce equivalencies between meaningfully different network states (e.g., restoring a user host versus an operational server). Reward misalignment can lead the agent to “game” the task by maximising rewards without producing a valid defensive policy, referred to as reward hacking.
- Simplify Instead of Over Engineering
- Highly engineered dense rewards can introduce unintended biases and impose arbitrary trade-offs between defensive actions and network states. As a result, reward functions should ideally be sparse such that only a few, but feasibly reachable, goal-aligned state-action pairs provide a reward signal. This is achievable in ACD RL tasks where episodes begin in a goal state (i.e., an uncompromised network). However, sparsity must be balanced against exploration and learning stability. Scaling from smaller, less complex networks to large networks of interest throughout training, as a form of curriculum learning, may help to mitigate some of these problems.
Evaluation
Comprehensive evaluation methods are critical for building a good RL cyber environment. Thorough evaluations in complex RL environments illuminate any unknowns about true agent performance, revealing issues in both modelling and network virtualisation that might otherwise be missed.
- Understanding the Learnt Policy
- Typical RL evaluation practices for ACD problems include reporting average episodic rewards, and variance metrics. Although, this alone is not sufficient to get a complete picture of an agent’s learnt policy, nor is it an independent measure of how correct an MDP is.
- Ensure Statistical Validity
- Sufficient analysis of variance in policy performance is frequently missing in research applying RL to cybersecurity tasks. Statistical validity is only achievable using multiple independent training runs with different random seeds to establish confidence intervals and ensure reproducibility.
- Evaluate Behaviour at the System-Level
- An RL agent will always seek to maximise the reward signal it receives and simply evaluating using average episodic rewards only shows how well the agent has optimised its behaviour with respect to that signal. Agent evaluation must move beyond only episodic rewards and extend its metrics to measure system-level activity relevant to the goal: how often are hosts attacked? Which hosts? What kind of attack? Analysis of the blue agent’s learnt policy can shed light on where in the network the blue agent is acting most and what the average action distribution looks like.
Conclusion
- Building effective RL environments for ACD requires principled decisions at every stage of design, from characterising the real-world problem to virtualisation of the network and modelling the agent interface.
- This paper provides a conceptual framework that maps the core components of ACD environments to two parts of the sim-to-real gap, virtualisation and modelling.
- Using this framework, the paper provides a systematisation of best practice guidelines.
- The central theme of these guidelines is that utility of an ACD environment is determined by how faithfully its design reflects the structure and constraints of the real-world problem; without this grounding, agents risk being optimised for performance in the training environment that does not transfer to capabilities in the operational task.
- We hope that the framework and best practice guidelines provided in this paper enable the development of more effective ACD environments.
Cite
@misc{hicks2026building,
title={Building Better Environments for Autonomous Cyber Defence},
author={Chris Hicks and Elizabeth Bates and Shae McFadden and Isaac Symes Thompson and Myles Foley and Ed Chapman and Nickolas Espinosa Dice and Ankita Samaddar and Joshua Sylvester and Himanshu Neema and Nicholas Butts and Nate Foster and Ahmad Ridley and Zoe M and Paul Jones},
booktitle={Workshop on AI for Critical National Infrastructure (AI4CNI) at AAMAS},
year={2026}
}