<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aarush Mahajan on Medium]]></title>
        <description><![CDATA[Stories by Aarush Mahajan on Medium]]></description>
        <link>https://medium.com/@aarushmahajan2008?source=rss-74b9d86eca38------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*G-Ii_u_tyQdgc6QTnWNIHg.jpeg</url>
            <title>Stories by Aarush Mahajan on Medium</title>
            <link>https://medium.com/@aarushmahajan2008?source=rss-74b9d86eca38------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:09:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aarushmahajan2008/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Is Memory the Bottleneck?]]></title>
            <link>https://pub.towardsai.net/is-memory-the-bottleneck-e1c2967246e6?source=rss-74b9d86eca38------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1c2967246e6</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[space]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[physics]]></category>
            <dc:creator><![CDATA[Aarush Mahajan]]></dc:creator>
            <pubDate>Thu, 29 Oct 2026 10:16:14 GMT</pubDate>
            <atom:updated>2026-10-30T09:54:27.006Z</atom:updated>
            <content:encoded><![CDATA[<h3>Is Memory the Bottleneck? Reward Design and Greedy Collapse in a Recurrent Collision-Avoidance Reproduction</h3><h4>Reproducing a satellite collision-avoidance paper, one reward function at a time.</h4><p><strong>By-: Aarush Mahajan, </strong>B.Tech Computer Science, Manipal University Jaipur<br>Part of my TKS Focus Project on AI for space domain awareness</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GOekPgcpeWWl4cpXcE75Pg.png" /><figcaption><em>Figure 1. Illustration of a satellite in low Earth orbit with debris fragments. [Generated image; no data.]</em></figcaption></figure><p><strong>TL;DR:</strong> Satellites must dodge debris using tracking data that is noisy and incomplete. A recurrent agent with memory seems like the natural fix. In my reproduction, under a reward I designed, the zero-thrust policy scored best of the policies I evaluated, and the recurrent agent’s greedy policy used zero fuel. Reward design and training stability look like bigger problems than memory, but one environment and one training run cannot establish that.</p><h3>Can Memory Help a Satellite Dodge Debris?</h3><p>Imagine following another car on a foggy highway. Your dashboard shows its position, updated every few seconds, but the estimate is sometimes off by a noticeable margin. A driver who brakes or swerves on every new reading will overcorrect. A careful driver keeps a rough estimate of where the car has been and where it is heading, and reacts to that estimate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IG_mgdxAy8TtFw1Evj742A.png" /><figcaption><em>Figure 2. The foggy-highway analogy. A driver who reacts to every noisy reading overcorrects; a driver who tracks a running estimate steers smoothly.</em></figcaption></figure><p>A satellite faces the same problem with debris. The debris is the car in the fog, and the satellite has to decide when to burn fuel. Bourriez, Loizeau, and Abdin (2023) argue that an agent with memory of past observations should handle this better, and they use a recurrent neural network to provide that memory. I wanted to test that claim in a simple setting, and the answer was not what I expected.</p><h3>Why This Matters</h3><p>I chose to focus on this problem because bridging the gap between simulated reinforcement learning rewards and the uncompromising physics of orbit is one of the most critical, yet under-discussed, safety barriers to scaling autonomous satellite constellations.</p><p>Low Earth orbit holds much of the infrastructure we rely on. The European Space Agency’s Annual Space Environment Report (2024 edition) estimates about 900,000 objects between 1 and 10 cm and about 128 million between 1 mm and 1 cm. Objects in the 1 to 10 cm range are too small to track reliably, and at orbital speeds even small objects can seriously damage a spacecraft (European Space Agency, 2024).</p><p>Collisions are rare but costly, and each one adds debris that raises the risk for other objects. This feedback loop is known as Kessler syndrome (Kessler and Cour-Palais, 1978).</p><p>Collision avoidance is still largely human-in-the-loop. Tracking networks issue Conjunction Data Messages (CDMs), defined in CCSDS Recommended Standard 508.0-B-1. A review of collision avoidance methods (Pavanello et al., 2025) describes CDMs as proximity alerts rather than collision warnings, which require further risk assessment. As constellations grow, this workflow becomes harder to sustain, which motivates onboard autonomy.</p><h3>Why a POMDP</h3><p>In a Markov Decision Process (MDP), an agent observes a state, chooses an action, receives a reward, and moves to a new state. The framework assumes the agent knows the state exactly. A satellite does not. Its measurements come from radar, optical sensors, and orbit predictions, and they carry two kinds of error. Measurement noise means a radar return is a cloud of possible positions, not a point. Model error means orbit predictors simplify forces such as Earth’s uneven gravity, atmospheric drag, and solar radiation pressure, so predicted positions drift over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8UW1ywkDil_OTa3XFnM4dQ.png" /><figcaption><em>Figure 3. In an MDP the agent observes the state directly. In a POMDP the agent sees only noisy observations and must maintain a belief about the hidden state.</em></figcaption></figure><p>A Partially Observable MDP (POMDP) is one way to model this situation. The agent maintains a belief about the hidden state, built from its history of observations and actions. An MDP agent does not need perfect inputs to function, but without memory it cannot tell a one-off glitch from a real trend.</p><h3>The Approach in Bourriez et al. (2023)</h3><p>Bourriez et al. frame autonomous collision avoidance as a POMDP and solve it with a Deep Recurrent Q-Network (DRQN). A 2026 preprint by Georges and Abdin applies a Transformer-based reinforcement learning approach to autonomous orbital collision avoidance in partially observable environments.</p><ul><li><strong>State and actions.</strong> The state tracks the 3D position and velocity of the spacecraft and the debris. The paper describes a discrete action space of “five possible thrust values” applied “in each of the three dimensions and at a specific time,” and gives its size as ⁵⁴ = 625. Three dimensions with five values each give ⁵³ = 125. The fourth factor must therefore be the time element, which the paper mentions but does not define as a separate choice. The paper’s description does not make clear how many independent decisions a step contains.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z0bRAZt8Ida24QSPCldnWQ.png" /><figcaption><em>Figure 4. Action space in my reproduction. Each step has four burn slots, and each slot selects one of five options (no burn, ±y, ±z). The highlighted cells show one example step.</em></figcaption></figure><ul><li><strong>Reward.</strong> The paper’s reward penalizes collision probability above 10⁻⁴, which it ties to NASA collision-risk practice. It also penalizes fuel use above a threshold and penalizes trajectory deviations in the orbital elements: a threshold of 100 for the semi-major axis (a), which is defined in meters, and 0.01 for eccentricity (e), inclination (i), longitude of the ascending node (Ω), and argument of periapsis (ω), where the angles are in radians.</li><li><strong>Recurrent memory.</strong> An LSTM is a neural network layer with a hidden state that updates at each step. In the DRQN, the LSTM takes the current observation and its previous hidden state, and passes a new hidden state to the Q-value layers. The hidden state is a learned summary of history. Nothing guarantees that it matches the Bayesian belief the POMDP framing describes.</li></ul><p><a href="https://claude.ai/chat/rl_figures/fig5_drqn_architecture.png">Figure 5. DRQN architecture. An LSTM carries a hidden state from step to step and feeds a Q-value layer that scores each of the five thrust options.</a></p><p><em>Figure 5. DRQN architecture. An LSTM carries a hidden state from step to step and feeds a Q-value layer that scores each of the five thrust options.</em></p><h3>What the Paper Reports, and What It Does Not</h3><p>The authors report that the agent performed well in terms of collision avoidance effectiveness, supported by training loss curves that converge to lower values in multi-environment setups. The paper does not report actual collision rates, fuel per episode, or held-out returns against a baseline. Its performance claims therefore rest on training behavior rather than task metrics.</p><h3>My Reproduction</h3><p><strong>Setup.</strong> I trained agents for 400 episodes in a single environment using Clohessy-Wiltshire relative-motion dynamics, a linearized model simpler than full orbit propagation. Each run used one training seed, batch size 16, discount factor 0.99, learning rate 1e-3, and 128 hidden units. I compared three learned policies: a DRQN with an LSTM, a feed-forward DQN that stacks the last four observations, and a feed-forward DQN on the current observation alone. Each observation frame consists of seven features: 3D relative position, 3D relative velocity, and a normalized time index. Stacking four frames yields a 28-input vector.</p><p><strong>Action space.</strong> Each step has four burn slots, and each slot selects one of five options: no burn, ±0.1 m/s along y, or ±0.1 m/s along z. Thrust along x is not used. Each step applies at most 0.4 m/s. Fuel is charged on the magnitude of the net Δv summed over the slots, so opposing burns in the same step cancel. The agent could in principle expend zero net fuel while issuing opposing thrusts. Per-slot actions were not individually logged, so I cannot say whether the DRQN exploited this.</p><p><strong>Reward.</strong> I designed the reward, and it differs from the paper’s. At each step the environment charges −5.0 per m/s of net Δv and a continuous risk-pressure penalty of −1.0 × min(1, p_c / 10⁻⁴), where p_c is a point-mass collision-probability proxy. The budget penalty works as follows: the 1.0 m/s budget is checked on cumulative fuel, and <strong>once cumulative fuel exceeds 1.0 m/s, a penalty of −10 is applied at every subsequent step of the episode, not once.</strong> A collision costs −100 and ends the episode. An episode that runs all 60 steps with no collision and a minimum distance above 106.7 m earns +50.</p><p><strong>Evaluation.</strong> Each policy was evaluated on a single evaluation block of 100 encounters, drawn from evaluation seeds separate from training. Training resets were not seeded, so I did not verify that no training geometry overlaps the evaluation block. “Success” means the episode ends with no collision and a minimum distance above 106.7 m. The random-action, no-maneuver, and DRQN rows come from notebook cell 20. The two feed-forward rows come from cell 33, which evaluates the same 100 encounters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y-0mSicdXf40Q3E1oL4wuw.png" /><figcaption><em>Table 1. Evaluation on 100 encounters (notebook cells 20 and 33). The DRQN row is highlighted.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*447bIFsXpmQlVimCwXjiCA.png" /><figcaption><em>Figure 6. Mean return by policy on 100 encounters. Zero thrust scores highest under this reward, while the trained agents spend the most fuel.</em></figcaption></figure><p><strong>Observations.</strong></p><p><em>First, under my reward, the zero-thrust policy scored best of the policies evaluated.</em> Its mean return of −60.33 beats random actions (−543.08) and both trained feed-forward agents (−579.82 and −606.54). This ordering is a property of the reward I designed, not evidence about the method. I hypothesize that the terminal collision penalty (−100) is small relative to the per-step fuel and budget penalties, so accepting a collision is cheaper than the trained feed-forward agents’ evasive maneuvers. The table shows this ordering; it does not test the mechanism.</p><p><em>Second, the DRQN’s greedy policy used zero fuel.</em> Training and greedy evaluation use different procedures. During training, the agent acts with ε-greedy exploration at ε = 0.05, and the training log records one result per episode. Over episodes 1–250, when exploration was still high, the DRQN succeeded in 95.2% of training episodes with no collisions. Over episodes 301–400, at ε = 0.05, it succeeded in 60.0% and collided in 3.0%. Greedy evaluation uses no exploration and is the policy that matters for deployment. Under greedy evaluation, mean fuel was exactly 0.00 m/s. Because opposing slots can cancel, zero net Δv is consistent with zero thrust but does not prove that action 0 was chosen at every step. Per-step, per-slot actions were not logged. Its summary metrics match the no-maneuver baseline exactly on every reported statistic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yi2NFHPTvtPxKkRRkI79Jg.png" /><figcaption><em>Figure 7. DRQN training. Success (teal) and collisions (red) are 20-episode rolling means; exploration ε (grey dashed) decays to 0.05 by episode 280.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kFEoUWblON3S9JDo5WF-sQ.png" /></figure><p><em>Table 3. DRQN training results by episode block, from the training log.</em></p><p><em>Third, the feed-forward networks’ loss diverged.</em> The Huber loss for the stacked-frame DQN was 6.9 at episode 1, 1.9 × ¹⁰⁵ at episode 50, about 1.5 × ¹⁰⁸ at episode 200, and 1.7 × ¹⁰⁹ at episode 400. The single-frame DQN reached 7.7 × ¹⁰⁸ at episode 400. For the DRQN, the logged loss fell from 3.8 at episode 25 to 2.3 at episode 400, with values logged every 25 episodes. I hypothesize that the feed-forward agents’ fuel overspending is tied to this instability, rather than being a deliberate evasion strategy. I have not run a controlled test to establish it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cAvEj0YvVykGuB4hFP1Qaw.png" /><figcaption><em>Figure 8. Huber loss by episode on a log scale. Feed-forward losses diverge by several orders of magnitude, while DRQN loss stays near 2–4. Feed-forward losses are plotted at the episodes printed in the notebook output.</em></figcaption></figure><p><strong>Why random actions outscore the trained agents.</strong> Random actions return −543.08, and the trained feed-forward agents return −579.82 and −606.54. The random policy has 91% success and no collisions. Its advantage comes from fuel, not from avoiding collisions better: it spent 9.49 m/s, against 13.42 and 16.97 m/s for the trained agents. The reward breakdown below shows that these fuel and budget penalties dominate the return. The comparison shows that the trained agents overspend relative to this reward. It does not show that random thrust is a good policy.</p><p><strong>Reward breakdown.</strong> The table below decomposes the returns that include a non-zero collision or success term. The collision and success terms come from the reported rates in cell 33: 95% and 97% success multiplied by +50, and 5 × mean fuel for the fuel cost. The budget penalty and the unlogged residual cannot be separated from the logged totals, so the table combines them into one derived column: the reported return minus the other terms. That combined value contains the budget penalty, the continuous risk-pressure term, and anything else not logged.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kWxGxkKhesNpXRwaF90UCA.png" /><figcaption><em>Figure 9. Return by component. Budget and fuel penalties account for almost all of the feed-forward agents’ negative returns. The budget and residual term is derived, not logged.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XWrt3fCBkMi4cYrIdaUFKA.png" /><figcaption><em>Table 2. Reward breakdown (derived). The budget and residual column is derived from reported totals.</em></figcaption></figure><p>For the two feed-forward agents, the combined column is dominated by the budget penalty. If the residual is small, dividing it by −10 gives about 56 penalized steps per episode for the stacked-frame DQN and 57 for the single-frame DQN. These counts are inferred from summary totals, not logged per episode, and they assume the residual is negligible. I have not decomposed the random-action return, because per-term values for that policy were not logged.</p><h3>Limitations</h3><p>These results come from one simplified environment, one reward I designed, one training run per architecture, and one 100-encounter evaluation block. The conclusions apply to that setup only.</p><p><strong>Compute.</strong> The DRQN performs four gradient updates per episode, which gives about 1,600 updates over 400 episodes. The feed-forward networks update once per environment step. Their total depends on episode length: at the 60-step maximum it is about 24,000 updates over 400 episodes, and it is lower for episodes that end early through collision. The 24,000 figure is an upper bound that assumes every episode runs the full 60 steps. The ratio is roughly 15-fold at that upper bound, so the architecture comparison is not compute-matched.</p><p><strong>Dynamics.</strong> Clohessy-Wiltshire is a linearized relative-motion model. It omits nonlinear perturbations and the atmospheric density changes that matter in real low orbit.</p><p><strong>Thrust model.</strong> The paper uses discrete impulsive burns. Many modern satellites use low-thrust electric propulsion, which requires extended maneuvers that discrete impulsive action sets do not represent well.</p><p><strong>Safety.</strong> Operators need concrete guarantees, and a deep network can produce surprising actions on off-nominal inputs. Safety layers, such as those that override unsafe actions, remain an active research direction.</p><h3>What I Would Test Next</h3><p><strong>First, the update-count confound.</strong> Run the feed-forward agents with the DRQN’s update schedule, and run the DRQN with a matched update count. If the DRQN’s passivity disappears when its update count matches the feed-forward agents, the rival explanation overtakes the reward hypothesis.</p><p><strong>Second, training stability.</strong> Investigate the feed-forward divergence by tuning the learning rate and clipping targets to stabilize the Huber loss. Until the baselines train stably, the stacked-versus-single comparison is not informative.</p><p><strong>Third, reward shaping.</strong> Replace the step-function budget penalty with a smooth term, or significantly reduce the −5.0 fuel penalty weight during early curriculum episodes, so the agent is not punished for early evasive burns. If the DRQN starts maneuvering greedily under a balanced reward, that supports the hypothesis that reward design is the bottleneck.</p><p>Only after agents avoid collisions reliably under a balanced reward does it make sense to ask whether recurrent memory adds value. That test would use harder, non-Gaussian, temporally correlated tracking noise, where a recurrent agent should have an advantage if memory matters.</p><h3>Takeaways</h3><ul><li>Under noisy tracking, a POMDP is one plausible model for collision avoidance. I did not test whether it is the right model.</li><li>A recurrent network learns a summary of past observations. That summary is not a verified belief.</li><li>The paper’s training curves show optimization progress, not safety or generalization. Those require held-out task metrics against baselines.</li><li>In my reproduction, random actions outscored both trained feed-forward agents on return. That gap comes mostly from fuel and budget penalties, since random actions had similar success and no collisions.</li><li>Under my reward, the zero-thrust policy scored best of the policies I evaluated, and the DRQN’s greedy policy used zero fuel. Reward design, update-count imbalance, and training stability may obscure whether recurrent memory improves collision avoidance in this setup.</li></ul><h3>References</h3><ol><li>Bourriez, N., Loizeau, A., &amp; Abdin, A. F. (2023). Spacecraft Autonomous Decision-Planning for Collision Avoidance: a Reinforcement Learning Approach. <em>74th International Astronautical Congress</em>. arXiv:2310.18966.</li><li>European Space Agency Space Debris Office. <em>ESA Space Environment Report 2024</em>. Published 19 July 2024. <a href="https://www.esa.int/Space_Safety/Space_Debris/ESA_Space_Environment_Report_2024">https://www.esa.int/Space_Safety/Space_Debris/ESA_Space_Environment_Report_2024</a></li><li>Kessler, D. J., &amp; Cour-Palais, B. G. (1978). Collision frequency of artificial satellites: The creation of a debris belt. <em>Journal of Geophysical Research</em>, 83(A6), 2637–2646. <a href="https://doi.org/10.1029/JA083iA06p02637">https://doi.org/10.1029/JA083iA06p02637</a></li><li>Pavanello, Z., De Maria, L., De Vittori, A., Maestrini, M., Di Lizia, P., &amp; Armellin, R. (2025). CAMmary: A Review of Spacecraft Collision Avoidance Manoeuvre Design Methods. arXiv:2503.22555.</li><li>Uriot, T., Izzo, D., Simões, L. F., Abay, R., Einecke, N., Rebhan, S., Martinez-Heras, J., Letizia, F., Siminski, J., &amp; Merz, K. (2022). Spacecraft collision avoidance challenge: Design and results of a machine learning competition. <em>Astrodynamics</em>, 6, 121–140. <a href="https://doi.org/10.1007/s42064-021-0101-5">https://doi.org/10.1007/s42064-021-0101-5</a></li><li>CCSDS. <em>Conjunction Data Message</em>, Recommended Standard CCSDS 508.0-B-1 (2013).</li><li>Georges, T., &amp; Abdin, A. (2026). Transformer-Based Reinforcement Learning for Autonomous Orbital Collision Avoidance in Partially Observable Environments. arXiv:2602.06088.</li><li>Mahajan, A. (2026). Code and notebook for this reproduction. Google Colab: <a href="https://colab.research.google.com/drive/1fZjUXIqCLdJBXGHxi0jSxnkVRfqMC0o8">https://colab.research.google.com/drive/1fZjUXIqCLdJBXGHxi0jSxnkVRfqMC0o8</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1c2967246e6" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/is-memory-the-bottleneck-e1c2967246e6">Is Memory the Bottleneck?</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Built a Satellite Collision Risk Classifier Using Random Forest and XGBoost on Conjunction…]]></title>
            <link>https://pub.towardsai.net/how-i-built-a-satellite-collision-risk-classifier-using-random-forest-and-xgboost-on-conjunction-b0fdc3e1e0ce?source=rss-74b9d86eca38------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0fdc3e1e0ce</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[space]]></category>
            <category><![CDATA[aerospace]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Aarush Mahajan]]></dc:creator>
            <pubDate>Mon, 26 Oct 2026 09:45:00 GMT</pubDate>
            <atom:updated>2026-10-30T09:54:28.386Z</atom:updated>
            <content:encoded><![CDATA[<h3>How I Built a Satellite Collision Risk Classifier Using Random Forest and XGBoost on Conjunction Data Messages</h3><p><strong>By Aarush Mahajan</strong> | Focus: Artificial Intelligence</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JMCkmR8Cq7SU8Y_qYX7SCg.png" /></figure><p><strong>Project Deliverables:</strong> <a href="https://www.youtube.com/watch?v=WQ0yY1r6wD4">5-Minute Video Showcase</a> · <a href="https://colab.research.google.com/drive/118LCHb4J5spUzNno8VV5N63LsO49kBTj?usp=sharing">Interactive Colab Notebook</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pHELc_OafO-e3Ql5t6NA0g.png" /><figcaption>Thumbnail for the 5 min video</figcaption></figure><h3>TL;DR</h3><p>I built a supervised pipeline that flags close approaches between satellites and debris whose collision probability exceeds the screening threshold of 10⁻⁴. I generated 5,000 synthetic Conjunction Data Message (CDM) records from a 2D Gaussian encounter model and trained Random Forest and XGBoost classifiers. XGBoost reached a test ROC-AUC of 0.98, but the labels are a known function of the inputs the models see, so that score mostly measures how well the generator can be learned. Three results matter more. Raw inputs did as well as my engineered features. Raw classifier scores are rankings, not probabilities, and isotonic calibration fixes that. But even after calibration, the model flags the high-risk region only barely, and the raw model never flags it at all.</p><h3>1. Why This Matters</h3><p>Low Earth orbit is getting crowded. The ESA Annual Space Environment Report (2024 edition) estimates about 900,000 objects between 1 and 10 cm and about 128 million between 1 mm and 1 cm. Objects in the 1 to 10 cm range are too small to track reliably, yet at orbital speeds they can seriously damage a spacecraft.</p><p>Today, collision avoidance is mostly manual. A tracking network detects a close approach and issues a Conjunction Data Message under CCSDS 508.0-B-1, typically days before the Time of Closest Approach (TCA). Flight dynamics analysts then review the message, estimate the risk, decide whether the event crosses the screening threshold, and plan any avoidance burn.</p><p>Large constellations generate hundreds of these messages per satellite each week. Automated triage could help separate routine near-misses from events that need action, so operators spend their attention and propellant where it matters.</p><h3>2. What I Built</h3><p>The project was inspired by the ESA Collision Avoidance Challenge, documented by Uriot et al. (2022). That challenge used real CDM-derived data and a different prediction target, so this project does not reproduce it.</p><p>My goal was a Python pipeline that answers two questions:</p><ol><li>Can tree models flag high-risk conjunctions from CDM-style features?</li><li>How much of any apparent skill comes from the label being a simple function of those features?</li></ol><h3>3. How the Pipeline Works</h3><pre>Generate encounter geometry: miss components (dx, dy, dz), relative speed,<br>  1-sigma position uncertainties for primary and secondary, TCA<br>        │<br>        ▼<br>Compute Pc from the 2D encounter model, then apply lognormal noise (σ = 0.3)<br>        │<br>        ▼<br>Label: high-risk if Pc &gt; 1e-4<br>        │<br>        ▼<br>Features: raw CDM columns + miss distance + covariance trace + log transforms (15 total)<br>        │<br>        ▼<br>80/20 stratified split → Random Forest (200 trees) and XGBoost<br>        │<br>        ▼<br>Evaluate: test-set metrics, 5-fold CV, calibration, sensitivity sweep, seed stability</pre><p><strong>Encounter geometry.</strong> Each record has miss components dx (along-track), dy, and dz, drawn from zero-mean normals with scales of 1,000 m, 300 m, and 200 m. Relative speed is uniform between 7 and 14 km/s. Each object has three 1-sigma position uncertainties, drawn uniformly over fixed ranges.</p><p><strong>Collision probability.</strong> The encounter-plane approximation uses a combined hard-body radius of 10 m and combined per-axis standard deviations:</p><p>σ_y = √(σ_y,pri² + σ_y,sec²), σ_z = √(σ_z,pri² + σ_z,sec²)</p><p>Pc = (HBR² / (2 σ_y σ_z)) · exp(−½ (dy²/σ_y² + dz²/σ_z²))</p><p>This decreases as the miss distance grows, as Figure 1 shows. The sample record used throughout this article (dy = 15 m, dz = 8 m) has a physics Pc of 4.02 × 10⁻².</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Em3CuI1zRl-q92NktIYlYA.png" /></figure><p><strong>Labels.</strong> The generator multiplies Pc by lognormal noise, exp(N(0, 0.3)), and labels records with the noisy value above 10⁻⁴ as high-risk (y = 1). All others are y = 0. The high-risk class is 81% of records.</p><p><strong>Features.</strong> The classifier sees the exact miss components and sigmas, not noisy measurements. Only the label carries noise. This is a simplification discussed in Section 6.</p><p><strong>Models.</strong> A Random Forest (200 trees, max depth 10, balanced class weights) and an XGBoost classifier (200 trees, depth 5, learning rate 0.05, positive-class weight set from the training split) are trained on an 80/20 stratified split. A majority-class predictor is the baseline.</p><h3>4. Results</h3><p>The majority-class baseline already scores 81% accuracy, so accuracy alone says little here. ROC-AUC and the confusion matrices are more informative.</p><p><strong>Held-out test set (seed 42, n = 1,000)</strong></p><p>Model Accuracy Precision Recall F1 ROC-AUC Majority-class baseline 0.810 0.00 0.00 0.00 0.500 Random Forest 0.911 0.932 0.961 0.946 0.961 XGBoost 0.931 0.978 0.936 0.957 0.982</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bw8mRi-58cxOBHAqea5l4g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-emjXtAsIRFxn6BCgLhceQ.png" /></figure><p><strong>5-fold stratified cross-validation (seed 42, mean ± SD across folds)</strong></p><p>Model Accuracy ROC-AUC Random Forest 0.909 ± 0.015 0.947 ± 0.010 XGBoost 0.937 ± 0.004 0.980 ± 0.004</p><p>XGBoost’s weak spot is the low-risk class. Its recall on the 190 low-risk test records is 0.91, and its precision for that class is 0.77.</p><p><strong>Seed stability (5 independent datasets, n = 5,000 each).</strong> The high-risk fraction ranged from 0.806 to 0.819 across seeds.</p><p>Model ROC-AUC Accuracy Random Forest 0.946 ± 0.010 0.913 ± 0.008 XGBoost 0.981 ± 0.002 0.937 ± 0.004</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gY0nPPWftbUfdqmUuLCVZg.png" /></figure><p><strong>A reference rule.</strong> Thresholding the noise-free physics Pc at 10⁻⁴ gives 96.8% accuracy against the noisy labels, using the same inputs. XGBoost, at 93.1%, does not match this rule. The 158 records where label noise moves the outcome across the threshold account for most of the gap. These records sit near the boundary, so they are hard for any model. The models are approximating a formula, not discovering anything beyond it.</p><p><strong>Feature importance.</strong> For the Random Forest, the top features are dz (0.21) and dy (0.20), followed by sigma_z_sec (0.07), log miss distance (0.07), and miss distance (0.07).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*klc7tDNc1r7ClN6AvYr3Ng.png" /></figure><h3>4.1 Calibration</h3><p>The raw XGBoost outputs are not calibrated. I measured this on the 1,000-record test split, with the model trained on the 4,000-record training split, using 10 uniform-width bins for expected calibration error (ECE).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BiqylA9lnRBfYlXwsotT7Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dsHAJLpGKjldRUx8J4hF0Q.png" /></figure><p>Isotonic calibration reduces expected calibration error about threefold (0.054 to 0.016) and lowers the Brier score from 0.048 to 0.039. Ranking quality is not re-measured here, since the notebook does not print calibrated AUC. The raw model is systematically optimistic in the middle range: its reliability curve sits above the diagonal across the middle of the range, as Figure 6 shows.</p><p>An important distinction: the calibrated output is the probability that the label equals 1 given the features. It is not the physical collision probability Pc. Calibration makes the number honest about the label, but the label is itself a threshold on Pc.</p><h3>4.2 Sensitivity Sweep</h3><p>I held the sample record’s geometry and covariances fixed and varied the transverse miss distance dy from 0 to 1,500 m, recording both models’ scores at each value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lhvZlSLkgbEG1dyZxvSd1Q.png" /></figure><ul><li>The physics Pc falls below 10⁻⁴ at 143 m.</li><li>Raw XGBoost never exceeds 0.5. Its score falls from 0.25 at dy = 0 to 0.007 at 1,500 m.</li><li>The isotonic-calibrated model is above 0.5 for every grid point from 0 to 113 m, at a plateau near 0.51. It falls below 0.5 for larger miss distances.</li><li>The calibrated model therefore flags most of the high-risk region, but only just. It misses the stretch from about 113 m to 143 m, and its margin above the cut-off is about 0.01.</li></ul><p>The sample record has a physics Pc of 4.02 × 10⁻², which sits inside the high-risk region. Raw XGBoost gives it 0.25 and labels it nominal. The calibrated model gives it 0.508 (printed by the notebook for the sample record), which is above the cut-off by a hair. Calibration changes the scale of the scores, but the classifier’s separation of the high-risk region is still weak, and a cut-off of 0.5 is not a reliable operating point.</p><h3>5. What Went Wrong and How I Fixed It</h3><p><strong>Challenge 1: I expected engineered features to help, and they did not.</strong> My first design gave the classifiers raw position and velocity, and I expected that projecting the inputs into the encounter plane would help. The engineered version adds miss distance, a covariance trace, and log transforms to the raw CDM columns. To test whether these additions matter, I trained XGBoost on the raw columns alone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EWJVZ_dfencLGw-zsJhAiA.png" /></figure><p>XGBoost input Test ROC-AUC Test accuracy Raw columns only (11 features) 0.984 0.934 Raw columns + engineered features (15 features) 0.982 0.931</p><p>The raw-only model is as good or slightly better. The derived features add no information, because the label is already a function of the raw columns, and trees can learn the relevant combinations themselves. I report this as a finding. The earlier claim that projection improved the model is not supported.</p><p><strong>Challenge 2: Scores are rankings, not probabilities.</strong> In the sample record above, a raw score of 0.25 for a Pc of 0.04 is a calibration failure, and a 0.5 threshold misses it. Isotonic calibration brings expected calibration error from 0.054 to 0.016 and the Brier score from 0.048 to 0.039 (Section 4.1). But the sweep shows calibration is not sufficient: the calibrated model separates the high-risk region only marginally, so an operational threshold would need to be chosen against costs, not set at 0.5.</p><p><strong>Challenge 3: Feature design made the task easier than it looks.</strong> The label depends on dy, dz, and the sigmas, which the classifier receives directly. A tree that learns a threshold on those quantities can score well without learning anything beyond the threshold. The high AUC reflects how well the generator can be learned, not how well real risk can be detected.</p><h3>6. Limitations</h3><p><strong>The task is synthetic.</strong> Labels come from an analytical 2D Gaussian model with multiplicative noise, not from orbit determination against real tracking data. A high score on this dataset shows that the pipeline runs. It does not show that the model is ready for operations.</p><p><strong>The features are noise-free.</strong> The classifier never sees a noisy measurement. Real CDMs carry tracking errors that are correlated, non-Gaussian, and not captured by one lognormal factor on Pc. A model that performs well on noise-free inputs can degrade on real ones.</p><p><strong>Single-snapshot evaluation.</strong> The model analyzes one CDM at one time. Pavanello et al. (2025) note that conjunction risk evolves across a sequence of messages as tracking improves near TCA. A sequential model would be more realistic.</p><p><strong>Classification is not decision-making.</strong> A high-risk flag tells an operator that a collision might occur. It does not say how to maneuver without spending too much propellant or creating a new conflict.</p><p><strong>Limited scope of the validation.</strong> Results come from five synthetic datasets from the same generator. None of the experiments add noise to the features, and I have not tested the models on real CDM data.</p><h3>7. What I’m Doing Next</h3><p>This build showed that a tree model can triage synthetic conjunction data, and that the result can come from the setup itself. The next steps are:</p><ul><li>Add measurement noise to the features, so the classifier has to recover the label from noisy inputs.</li><li>Test on real CDM data from the ESA challenge or NASA CARA, keeping the feature pipeline unchanged.</li><li>Choose the operating threshold by cost, weighing missed collisions against false alarms, rather than by a fixed 0.5 cut-off.</li><li>Move to sequential decisions: a Deep Recurrent Q-Network (DRQN) based on Bourriez et al. (2023), tested against feed-forward policies under Clohessy-Wiltshire dynamics and a propellant budget of 1.0 m/s.</li></ul><h3>References</h3><ol><li>Uriot, T., Izzo, D., Simões, L. F., et al. (2022). Spacecraft collision avoidance challenge: Design and results of a machine learning competition. <em>Astrodynamics</em>, 6(2), 121–140.</li><li>Bourriez, N., Loizeau, A., &amp; Abdin, A. F. (2023). Spacecraft Autonomous Decision-Planning for Collision Avoidance: a Reinforcement Learning Approach. <em>74th International Astronautical Congress</em>. arXiv:2310.18966.</li><li>Pavanello, Z., et al. (2025). CAMmary: A Review of Spacecraft Collision Avoidance Manoeuvre Design Methods. arXiv:2503.22555.</li><li>Catulo, J. S., Soares, C., &amp; Guimarães, M. (2023). Predicting the Probability of Collision of a Satellite with Space Debris: A Bayesian Machine Learning Approach. arXiv:2311.10633.</li><li>Consultative Committee for Space Data Systems (CCSDS). (2013). Conjunction Data Message. CCSDS Recommended Standard 508.0-B-1.</li><li>European Space Agency. Annual Space Environment Report, 2024 edition.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0fdc3e1e0ce" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/how-i-built-a-satellite-collision-risk-classifier-using-random-forest-and-xgboost-on-conjunction-b0fdc3e1e0ce">How I Built a Satellite Collision Risk Classifier Using Random Forest and XGBoost on Conjunction…</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the Invisible Hazard: How Deep Recurrent Reinforcement Learning and POMDPs Automate…]]></title>
            <link>https://pub.towardsai.net/navigating-the-invisible-hazard-how-deep-recurrent-reinforcement-learning-and-pomdps-automate-fd5a9ea60523?source=rss-74b9d86eca38------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd5a9ea60523</guid>
            <category><![CDATA[space]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aarush Mahajan]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 14:21:14 GMT</pubDate>
            <atom:updated>2026-11-02T09:42:05.600Z</atom:updated>
            <content:encoded><![CDATA[<h3>Navigating the Invisible Hazard: How Deep Recurrent Reinforcement Learning and POMDPs Automate Orbital Collision Avoidance Under Sensor Uncertainty</h3><h4>A Technical Deep Dive into POMDP Formulations, DRQN Architectures, Multi-Seed Comparative Benchmarking, and Onboard Autonomous Satellite Collision Avoidance Under Tracking Jitter</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rSgThGib9Ikgv60dqLjbrA.png" /><figcaption>An autonomous satellite executing a real-time thruster burn in Low Earth Orbit to navigate space debris</figcaption></figure><p><strong>Author:</strong> Aarush Mahajan<br> <strong>Primary Source:</strong> <em>Spacecraft Autonomous Decision-Planning for Collision Avoidance: a Reinforcement Learning Approach</em> (Bourriez, Loizeau, &amp; Abdin, 2023) <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a><br> <strong>Thesis Question:</strong> <em>Where do traditional Markov Decision Processes (MDPs) fail in space domain awareness, and how does a Partially Observable Markov Decision Process (POMDP) paired with Deep Recurrent Q-Networks (DRQN) enable autonomous satellite collision avoidance under severe tracking noise?</em><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKDFzObtE6cs">[7]</a></p><h3>Section 1: Roadmap &amp; Real-World Stakes</h3><h3>1.1 The Invisible Emergency Above Our Heads</h3><p>Every second, the orbital infrastructure that powers modern civilization — GPS navigation, financial transaction synchronization, climate monitoring, and emergency disaster response — hangs in a delicate balance 2,000 kilometers above Earth<a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DFwcHMiFomKk">[8]</a>.</p><p>Low Earth Orbit (LEO) is currently experiencing an unprecedented population explosion. Ground-based tracking networks actively monitor over <strong>35,000 resident space objects</strong> larger than 10 centimeters<a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DFwcHMiFomKk">[8]</a>. Alongside these trackable objects float an estimated <strong>900,000 lethal fragments</strong> between 1 cm and 10 cm — objects far too small for radar networks to track consistently, yet carrying enough kinetic energy at hyper-velocities (7.8 km/s or ~17,500 mph) to obliterate an active satellite upon impact.<a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DFwcHMiFomKk">[8]</a></p><pre>+----------------------------------------------------------------------------------+<br>|                            THE ORBITAL DEBRIS SPECTRUM                           |<br>+----------------------------------------------------------------------------------+<br>|  Size Class    | Estimated Count | Tracking Status      | Risk Level             |<br>+----------------+-----------------+----------------------+------------------------+<br>|  &gt; 10 cm       | ~35,000+        | Tracked by SSN / USSF| Catastrophic impact    |<br>|  1 cm - 10 cm  | ~900,000        | Hard to track        | Lethal disruption      |<br>|  1 mm - 1 cm   | ~128 Million    | Untracked            | Shielding required     |<br>+----------------+-----------------+----------------------+------------------------+</pre><p>When hyper-velocity objects collide, they trigger a runaway chain reaction known as <strong>Kessler Syndrome</strong>: each collision fragments satellites into thousands of new high-speed projectiles, increasing the probability of subsequent collisions until entire orbital bands become unusable<a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DFwcHMiFomKk">[8]</a>.</p><h3>1.2 The Operational Bottleneck: Why Manual Operations Are Failing</h3><p>To prevent catastrophic conjunctions, global bodies like the U.S. Space Surveillance Network (SSN) continuously screen orbits and issue <strong>Conjunction Data Messages (CDMs)</strong><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dyv7bD8oNXsA">[6]</a>. A single satellite operator in LEO receives hundreds of CDMs per week, detailing close approaches with uncertain miss distances and collision probabilities<a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dyv7bD8oNXsA">[6]</a>.</p><p>Currently, collision risk mitigation relies on <strong>human-in-the-loop workflows</strong><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dyv7bD8oNXsA">[6]</a>:</p><ol><li>Ground radar networks detect a potential close-approach event (conjunction).</li><li>Operators receive automated CDMs 1 to 7 days before the Time of Closest Approach (TCA).</li><li>Specialized orbital analysts manually evaluate the CDM covariance matrices, run trajectory propagation models, calculate thruster burns, and upload command sequences to the spacecraft<a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">[2]</a>.</li></ol><pre>[Ground Radar Tracking] ---&gt; [CDM Generation] ---&gt; [Human Analyst Review (Hours/Days)] ---&gt; [Uplink Maneuver Command]<br>                                                                  |<br>                                                   Operational Bottleneck!<br>                                           (Cannot scale to mega-constellations)</pre><p>With commercial mega-constellations deploying tens of thousands of new satellites, manual analysis is reaching an unsustainable tipping point. Mega-constellation operators like SpaceX’s Starlink execute thousands of Collision Avoidance Maneuvers (CAMs) annually. If every satellite requires manual human oversight for every close approach, satellite operations will stall, response latency will spike, and human error under stress will lead to orbital impacts.</p><p>To protect humanity’s shared orbital environment, we must transition from manual ground-based control to <strong>on-board autonomous decision-making</strong>[1].</p><blockquote><strong><em>Takeaway — Section 1:</em></strong><em><br> Over 35,000 tracked space objects and 900,000 untracked lethal fragments threaten LEO infrastructure</em><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DFwcHMiFomKk">[8]</a><em>. Traditional human-in-the-loop collision avoidance cannot scale to the era of satellite mega-constellations, demanding on-board AI systems capable of autonomous decision-making</em><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dyv7bD8oNXsA">[6]</a><em>.</em></blockquote><h3>Section 2: The Core Bottleneck — Why Standard Markov Decision Processes (MDPs) Fail in Space</h3><h3>2.1 The Fully Observable Illusion of Standard Reinforcement Learning</h3><p>When Reinforcement Learning (RL) is applied to terrestrial robotics or games, researchers typically model the problem as a <strong>Markov Decision Process (MDP)</strong><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKDFzObtE6cs">[7]</a>. In a standard MDP, the system assumes <strong>full state observability</strong>: at any time step t, the autonomous agent knows its exact state in state space S such as its precise 3D position, velocity, and orientation.</p><p>Under a fully observable MDP, an RL policy π(s) maps the exact environment state directly to an optimal control action in action space A. If space traffic management were a true MDP, a satellite’s flight computer could observe the exact positions and velocity vectors of all surrounding space debris and execute a single, deterministic thruster burn to dodge the collision.</p><pre>+-----------------------------------------------------------------------------------+<br>|                        FULL OBSERVABILITY ILLUSION (MDP)                          |<br>+-----------------------------------------------------------------------------------+<br>| True State (s_t)  ---&gt;  Agent Directly Observes (s_t)  ---&gt;  Optimal Action (a_t) |<br>| (Exact Debris Pos)        (Zero Tracking Error)              (Single Evasive Burn)|<br>+-----------------------------------------------------------------------------------+</pre><p>In Low Earth Orbit (LEO), however, <strong>true state observability is impossible</strong><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKDFzObtE6cs">[7]</a>.</p><h3>2.2 Dual Layers of Orbital Uncertainty: Aleatory vs. Epistemic</h3><p>Space Situational Awareness (SSA) data is corrupted by two distinct, compounded layers of uncertainty<a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2311.10633">[4]</a>:</p><ol><li><strong>Aleatory Uncertainty (Sensor &amp; Measurement Noise):</strong> Ground-based radar arrays, optical tracking telescopes, and on-board satellite sensors suffer from physical measurement noise and timing jitter. A radar return does not yield an exact point in space; it yields a probabilistic cloud of potential locations[4].</li><li><strong>Epistemic Uncertainty (Model &amp; Propagation Imperfections):</strong> To project where a piece of debris will be 24 to 48 hours in the future, engineers rely on mathematical orbit propagators like the <strong>Simplified General Perturbations 4 (SGP4)</strong> model<a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2609.13191">[3]</a>. However, SGP4 uses mathematical approximations for complex physical phenomena — such as Earth’s asymmetric gravitational field (J₂ oblateness harmonics), upper atmospheric drag fluctuations caused by solar activity, and solar radiation pressure. Over time, these unmodeled physical forces cause positional uncertainty ellipsoids to expand nonlinearly<a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">[2]</a>.</li></ol><pre>+-----------------------------------------------------------------------------------+<br>|                           REAL-WORLD ORBITAL UNCERTAINTY                          |<br>+-----------------------------------------------------------------------------------+<br>| True State (s_t)  +  [Aleatory Noise (Radar Jitter)]                              |<br>|                   +  [Epistemic Noise (SGP4 Drag/J2 Errors)]                      |<br>|                                     |                                             |<br>|                                     v                                             |<br>|                      Noisy Observation (o_t) RECEIVED                             |<br>+-----------------------------------------------------------------------------------+</pre><p>Both aleatory and epistemic uncertainties are characterized statistically using Gaussian distribution models<a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2311.10633">[4]</a>. Because of this noise, a satellite never knows the true state ; it only receives a noisy, corrupted observation in observation space O.</p><h3>2.3 The Fog Analogy: Why Naive MDP Agents Crash in LEO</h3><p>To understand why standard RL algorithms fail in space, consider a plain-English analogy:</p><blockquote><strong><em>The Highway Fog Analogy:</em></strong><em><br> Imagine driving a high-performance sports car down a highway at </em><strong><em>17,500 miles per hour</em></strong><em> in pitch-black, dense fog. You cannot see through the windshield. Instead, your dashboard displays a flickering GPS screen that updates once every few miles with a random error margin of </em><strong><em>±100 meters</em></strong><em>.</em></blockquote><blockquote><em>A standard </em><strong><em>MDP agent</em></strong><em> operates under the naive assumption that whatever appears on the GPS screen is 100% real. If the GPS glitches for a fraction of a second and shows a concrete wall directly in front of you, the MDP agent will slam on the brakes or jerk the steering wheel violently. In orbit, this leads to catastrophic thruster over-corrections that deplete the satellite’s limited fuel supply and steer the spacecraft directly into the path of another unmonitored piece of space debris.</em></blockquote><p>To navigate safely through tracking noise without wasting thruster propellant, the AI cannot react to single, isolated radar observations. It must maintain a <strong>continuous temporal memory</strong> of past observations to construct a probabilistic “belief state” of where hazards are located over time<a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKDFzObtE6cs">[7]</a>.</p><p>This requires transitioning from a Markov Decision Process (MDP) to a <strong>Partially Observable Markov Decision Process (POMDP)</strong><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKDFzObtE6cs">[7]</a>.</p><blockquote><strong><em>Takeaway — Section 2:</em></strong><em><br> Standard MDP reinforcement learning models fail in space because they assume full state observability</em><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKDFzObtE6cs">[7]</a><em>. Real satellite telemetry is corrupted by aleatory sensor noise and epistemic SGP4 propagation errors</em><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2311.10633">[4]</a>.<em> To avoid violent thruster over-corrections, satellites must use POMDPs to filter noisy observations into temporal belief states</em><a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>.</blockquote><h3>Section 3: The AI Architecture — POMDP, DRQN, and Recurrent Memory</h3><p>To overcome the full observability trap of standard Reinforcement Learning, <em>Bourriez et al. (2023)</em> <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a> reformulate autonomous spacecraft collision avoidance as a <strong>Partially Observable Markov Decision Process (POMDP)</strong> and solve it using a <strong>Deep Recurrent Q-Network (DRQN)</strong>.</p><h3>3.1 Mathematical Formulation: The 7-Tuple (S, A, T, R, O, Z, γ)</h3><p>A POMDP generalizes a standard Markov Decision Process to sequential decision-making problems where the agent cannot directly observe the true environment state<a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKDFzObtE6cs">[7]</a>. Formally, the onboard satellite collision avoidance agent is defined by the seven-tuple <strong>(S, A, T, R, O, Z, γ)</strong>:</p><pre>+-----------------------------------------------------------------------------------------------------+<br>|                               THE POMDP 7-TUPLE FOR SPACE NAVIGATION                                |<br>+-----------------------------------------------------------------------------------------------------+<br>| Element | Definition               | Description in Bourriez et al. (2023)                          |<br>+---------+--------------------------+----------------------------------------------------------------+<br>|   S     | Continuous State Space   | True 3D relative positions &amp; velocities: (x, y, z, dx, dy, dz) |<br>|   A     | Discrete Action Space    | 625 impulsive thruster burn combinations (5⁴ action size)      |<br>|   T     | State Transition Function| Model-free state transitions governed by perturbed Keplerian   |<br>|         |                          | orbit dynamics (Pykep engine)                                  |<br>|   R     | Multi-Objective Reward   | Composite reward penalizing collision risk, fuel, &amp; orbit drift|<br>|   O     | Observation Space        | Noisy continuous relative state vectors from sensors/radar     |<br>|   Z     | Observation Model        | Probabilistic distribution Z(o|s&#39;,a) combining Gaussian noise  |<br>|         |                          | from sensor jitter (aleatory) and SGP4 drag errors (epistemic) | <br>|   γ     | Discount Factor          | Set to 0.99 for long-horizon trade-off evaluations             |<br>+-----------------------------------------------------------------------------------------------------+</pre><h4>1. State Space (S) and Observation Space (O)</h4><p>The true state s_t in S represents the exact Cartesian position and velocity vectors of both the primary satellite and the secondary space debris in three-dimensional space <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>:</p><pre>s_t = [x_sat, y_sat, z_sat, dx_sat, dy_sat, dz_sat, x_deb, y_deb, z_deb, dx_deb, dy_deb, dz_deb]^T ∈ R¹²</pre><p>However, the agent never receives s_t. Instead, the observation model Z(o_t | s_t, a_{t-1}) generates a noisy observation vector o_t in O corrupted by zero-mean Gaussian noise N(0, σ²) simulating both SGP4 propagation uncertainty and tracking sensor noise <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>:</p><pre>o_t = s_t + e_t,   where e_t ~ N(0, Σ_sensor + Σ_SGP4)</pre><h4>2. Discretized Action Space (A)</h4><p>To execute evasive maneuvers, the spacecraft modifies its velocity by applying short, impulsive thruster burns (Δv) <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">[2]</a>. The action space is discretized into 5 possible thrust magnitude choices per dimension (0.0, 0.01, 0.05, 0.1, and -0.05 m/s) across 4 operational dimensions, yielding a discrete action space of size <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>:</p><pre>|A| = 5⁴ = 625 discrete actions</pre><h3>3.2 The Multi-Objective Reward Function (R)</h3><p>Designing a self-navigating satellite requires balancing competing operational goals: dodging debris, preserving fuel, and maintaining the satellite’s primary mission orbit <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>. <em>Bourriez et al.</em> construct a multi-objective composite reward function governed by strict operational threshold values <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>:</p><pre>R(s_t, a_t) = R_collision + R_fuel + R_orbit_drift</pre><pre>+---------------------------------------------------------------------------------------------------+<br>|                            MULTI-OBJECTIVE REWARD COMPONENTS &amp;amp; THRESHOLDS                         |<br>+---------------------------------------------------------------------------------------------------+<br>| Component          | Operational Threshold | Mathematical Penalty Applied                         |<br>+--------------------+-----------------------+------------------------------------------------------+<br>| Collision Risk     | Pc &amp;gt; 10⁻⁴             | Severe negative penalty if collision probability     |<br>|                    |                       | exceeds 10⁻⁴ (NASA operational safety threshold)     |<br>| Fuel Level         | &amp;gt; 10 units            | Negative penalty for every non-zero thruster burn;   |<br>|                    |                       | additional penalty if total fuel exceeds 10 units    |<br>| Trajectory Drift   | Δa &amp;gt; 100 m            | Proportional negative penalty for drifting from      |<br>| (Keplerian Elements| Δe &amp;gt; 0.01             | nominal osculating elements (a, e, i, Ω, ω)          |<br>|  a, e, i, Ω, ω)    | Δi, ΔΩ, Δω &amp;gt; 0.01 rad |                                                      |<br>+---------------------------------------------------------------------------------------------------+</pre><p>By enforcing penalties when the semi-major axis Δa, eccentricity Δe, inclination Δi, longitude of ascending node ΔΩ, or argument of periapsis Δω deviate beyond threshold bounds, the reward function prevents the AI from executing wild evasive burns that would render the satellite’s primary mission useless <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>.</p><h3>3.3 DRQN Solver Architecture &amp; Recurrent Memory</h3><p>In a classical POMDP, an agent calculates a formal probability distribution over all possible true states, known as a <strong>belief state</strong> b_t(s) = P(s_t = s | h_t) using Bayes’ rule <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKDFzObtE6cs">[7]</a>. However, updating exact Bayesian belief distributions over continuous 12-dimensional orbital space is computationally intractable for resource-constrained onboard flight computers.</p><p><em>Bourriez et al.</em> bypass explicit belief calculations by integrating a <strong>Long Short-Term Memory (LSTM)</strong> layer directly into a Deep Q-Network, forming a <strong>Deep Recurrent Q-Network (DRQN)</strong> <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>.</p><pre>+---------------------------------------------------------------------------------------------------+<br>|                                 DRQN SOLVER ARCHITECTURE FLOW                                     |<br>+---------------------------------------------------------------------------------------------------+<br>|                                                                                                   |<br>|  Noisy Radar Observation (o_t)                                                                    |<br>|            |                                                                                      |<br>|            v                                                                                      |<br>|   +------------------+     Hidden Cell State (h_{t-1}, c_{t-1})                                   |<br>|   |  LSTM Recurrent  | &amp;lt;---------------------+                                                        |<br>|   |   Memory Layer   |                       |                                                        |<br>|   +------------------+                       | (Implicit Belief State Update)                         |<br>|            |                                 |                                                        |<br>|            | Output Hidden Vector (h_t) ------+                                                        |<br>|            v                                                                                      |<br>|   +------------------+                                                                            |<br>|   |  Fully-Connected |                                                                            |<br>|   |   Linear Layers  |                                                                            |<br>|   +------------------+                                                                            |<br>|            |                                                                                      |<br>|            v                                                                                      |<br>|   Q-Values for 625 Actions: [ Q(b_t, a_1), Q(b_t, a_2), ..., Q(b_t, a_625) ]                      |<br>|            |                                                                                      |<br>|            v                                                                                      |<br>|   Action Selection: a_t = argmax_a Q(b_t, a) (ε-greedy during training)                          |<br>+---------------------------------------------------------------------------------------------------+</pre><h4>How LSTM Serves as an Implicit Belief Updater</h4><p>The hidden state h_t of the LSTM acts as an <strong>implicit belief state</strong> <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>:</p><pre>h_t = LSTM(o_t, h_{t-1})</pre><p>Instead of throwing away past measurements, the LSTM cell retains a temporal history of previous observations h_t = (o_1, a_1, o_2, a_2, …, o_t) <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>. If a single radar measurement glitches due to sensor noise, the recurrent memory weights the new observation against past trajectory trends, maintaining a stable trajectory estimate and preventing thruster panic <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>.</p><h3>3.4 Outlier Stabilization: Huber Loss &amp; Experience Replay Buffer</h3><p>Training Deep RL in space environments introduces rare, extreme events — such as catastrophic collision penalties or sudden thruster firings. Standard Mean Squared Error (MSE) loss squares these error terms, producing massive gradient spikes that cause neural network training to diverge <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>.</p><p>To stabilize learning, the DRQN utilizes the <strong>Huber Loss function</strong> (L_δ), which smoothly transitions from quadratic loss for small errors to linear loss for large outliers <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>:</p><pre>L_δ(a) = 0.5 * a²                   for |a| ≤ δ<br>L_δ(a) = δ * (|a| - 0.5 * δ)        for |a| &amp;gt; δ</pre><p>where a = Q_predicted — Y_target, and δ sets the threshold between quadratic and linear behavior.</p><pre>HUBER LOSS vs MSE BEHAVIOR<br>         Loss<br>          ^<br>          |          /  (MSE: Quadratic growth -&amp;gt; Gradient Explosion)<br>          |         /<br>          |        /<br>          |       +----- (Huber Loss: Linear growth for outliers |a| &amp;gt; δ)<br>          |      /<br>          |     /<br>          |    /<br>          +---+------------------------&amp;gt; Prediction Error |a|<br>              &amp;lt;-- δ --&amp;gt;<br>            Quadratic</pre><p>Additionally, the architecture incorporates two essential training stabilizers <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>:</p><ol><li><strong>Recurrent Experience Replay Buffer:</strong> Stores temporal trajectories (o_t, a_t, r_t, o_{t+1}) and samples random sequential batches to break temporal correlations and prevent overfitting.</li><li><strong>Soft-Updated Target Q-Network:</strong> A duplicate target network updated via soft target updates (τ = 0.1) to provide stable target values during temporal-difference learning.</li></ol><p><strong>Takeaway — Section 3:</strong> The POMDP framework models orbital uncertainty using a 7-tuple (S, A, T, R, O, Z, γ) with a 625-action space and a multi-objective reward function <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>. By embedding an LSTM layer inside a DRQN, the satellite maintains an implicit belief state across past observations, while Huber loss prevents extreme collision penalties from destabilizing neural network convergence <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>.</p><h3>Section 4: Methodology &amp; Experimental Results</h3><h3>4.1 The Conjunction Simulation Engine &amp; Retrograde Reconstruction</h3><p>A primary challenge in training Deep Reinforcement Learning models for space traffic management is the lack of real-world collision datasets <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>. Real collisions in Low Earth Orbit are rare, catastrophic events that destroy the assets involved, leaving no onboard telemetry for training. Public Two-Line Element (TLE) sets and historical Conjunction Data Messages (CDMs) provide close-approach warnings, but do not capture the post-conjunction flight dynamics required to train a decision agent <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2609.13191">[3]</a>.</p><p>To solve this data scarcity problem, <em>Bourriez et al. (2023)</em> <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a> constructed a custom space flight simulator using the <strong>Pykep</strong> astrodynamics library. Pykep provides high-precision orbital mechanics computations under perturbed Keplerian dynamics.</p><pre>+----------------------------------------------------------------------------------------------------+<br>|                         RETROGRADE COLLISION RECONSTRUCTION WORKFLOW                               |<br>+----------------------------------------------------------------------------------------------------+<br>|                                                                                                    |<br>| 1. Initialize Protected Spacecraft  ---&amp;gt; Set Keplerian elements (a, e, i, Ω, ω, M) &amp;amp; radius        |<br>|                                                                                                    |<br>| 2. Sample Collision Epoch (t_c)     ---&amp;gt; Draw t_c ~ Uniform(start_time, end_time)                   |<br>|                                                                                                    |<br>| 3. SGP4 Trajectory Projection      ---&amp;gt; Propagate protected spacecraft to t_c using SGP4            |<br>|                                                                                                    |<br>| 4. Instantiate Debris Position      ---&amp;gt; Sample position ~ N(r_projected, σ_pos)                   |<br>|                                                                                                    |<br>| 5. Retrograde Velocity Adjustment   ---&amp;gt; Calculate collision trajectory velocity:                  |<br>|                                          v_new = v * N(1, σ_vr) with rotated angle θ               |<br>|                                                                                                    |<br>| 6. Backtrack Simulation             ---&amp;gt; Propagate backwards to start_time to initialize episode   |<br>+----------------------------------------------------------------------------------------------------+</pre><h4>The Retrograde Backtracking Algorithm</h4><p>To ensure every simulated environment contains a physically plausible conjunction event, the simulator uses a <strong>retrograde reconstruction method</strong> <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>:</p><ol><li><strong>Spacecraft Instantiation:</strong> The protected satellite is initialized at start_time with standard Keplerian orbital parameters.</li><li><strong>Collision Epoch Sampling:</strong> A time of closest approach t_collision is randomly sampled from a uniform distribution U(t_start, t_end).</li><li><strong>Forward SGP4 Projection:</strong> The satellite’s state is propagated to t_collision using the SGP4 orbit model.</li><li><strong>Debris Proximity Generation:</strong> Debris positions are generated around the projected satellite location following a Gaussian distribution N(r_sat, σ_pos).</li><li><strong>Velocity Vector Rotation &amp; Noise Scaling:</strong> The debris velocity vector is rotated by a randomized collision angle θ and scaled with stochastic noise to simulate realistic relative velocity vectors <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>:</li></ol><pre>v_new = v * N(1, σ_vr)</pre><ol><li><strong>Backtracking:</strong> The simulator backtracks both objects from t_collision to t_start, creating a fully deterministic conjunction episode where a collision will occur unless an evasive burn is applied.</li></ol><h3>4.2 Hyperparameter Tuning via Grid Search</h3><p>Deep Recurrent Q-Networks are sensitive to hyperparameter selection; small variations in replay buffer capacity or soft-update factors (τ) can cause neural network divergence <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>. To establish a stable training baseline, the authors conducted exhaustive <strong>grid search sweeps</strong> across neural network hyperparameter spaces <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>.</p><pre>+----------------------------------------------------------------------------------------------------+<br>|                         OPTIMAL HYPERPARAMETER CONFIGURATIONS (TABLES II &amp;amp; III)                     |<br>+----------------------------------------------------------------------------------------------------+<br>| Hyperparameter Parameter          | Single-Environment Value    | 200-Environment Multi-Setting Value  |<br>+-----------------------------------+-----------------------------+--------------------------------------+<br>| Batch Size                        | 50                          | 100                                  |<br>| Hidden Layer Size (LSTM &amp;amp; FC)     | 128 units                   | 128 units                            |<br>| Learning Rate (α)                 | 0.0001 (1e-4)               | 0.0001 (1e-4)                        |<br>| Training Episodes                 | 200 episodes                | 200 episodes                         |<br>| Replay Buffer Capacity            | 1,000 transition sequences  | 1,000 transition sequences           |<br>| Target Update Rate (τ)            | 0.1                         | 0.1                                  |<br>| Discount Factor (γ)               | 0.99                        | 0.99                                 |<br>+----------------------------------------------------------------------------------------------------+</pre><p>The grid search evaluated neural network hidden sizes, learning rates, batch sizes, and exploration schedules against the cumulative reward sum <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>. Increasing the batch size from 50 to 100 in the multi-environment setting provided the gradient smoothing necessary to stabilize learning across 200 distinct orbital geometries.</p><h3>4.3 Training Convergence &amp; Performance Evaluation</h3><p>The agent’s performance was evaluated by tracking the <strong>Huber Loss trajectory</strong> and <strong>average cumulative reward profile</strong> across two experimental configurations <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>:</p><ol><li><strong>Single Collision Environment:</strong> Testing policy convergence in a fixed orbital encounter geometry.</li><li><strong>200 Stochastic Collision Environments:</strong> Evaluating generalization across 200 distinct orbital geometries with varying debris counts, approach angles, and sensor noise profiles.</li></ol><pre>LOSS CONVERGENCE TRAJECTORY<br>     Loss Value<br>       0.35 +--* (Initial Random Actions: High Error)<br>            |   \<br>       0.30 |    \<br>            |     \---\<br>       0.20 |          \--------------+ (Single Env Convergence: ~0.26)<br>            |                         |<br>       0.15 +-------------------------+========* (200 Envs Multi-Setting Loss: ~0.16)<br>            +-----------------------------------&amp;gt; Training Steps<br>            0       10k      20k      30k      40k</pre><h4>Key Experimental Findings:</h4><ul><li><strong>Loss Reduction:</strong> As training progressed over 40,000 simulation steps, the DRQN loss curve decreased monotonically from above 0.35 down to <strong>~0.16</strong> in the multi-environment setup <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>.</li><li><strong>Generalization Advantage:</strong> Counterintuitively, the agent trained across <strong>200 stochastic environments</strong> achieved lower overall loss (~0.16) than the agent trained on a single fixed environment (~0.26) <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>. Exposure to diverse approach vectors prevented the LSTM from memorizing single-trajectory overfits, encouraging the network to learn robust, generalizable belief state representations.</li><li><strong>Operational Efficiency:</strong> The trained agent successfully executed collision avoidance maneuvers that reduced the collision probability below the 10⁻⁴ safety threshold while minimizing thruster fuel consumption and preserving the satellite’s primary Keplerian orbital elements <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>.</li></ul><p><strong>Takeaway — Section 4:</strong> By combining a Pykep-based retrograde collision simulator with SGP4 propagators, the authors generated synthetic conjunction scenarios to train the DRQN agent <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>. Hyperparameter grid searches established that multi-environment training (across 200 distinct collision geometries) improved policy convergence, reducing Huber loss to ~0.16 while ensuring compliance with NASA safety thresholds <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>.</p><h3>Section 5: What Nobody Knows Yet — Limitations &amp; Edge Cases</h3><p>While <em>Bourriez et al. (2023)</em> <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a> present a proof-of-concept for onboard autonomous decision-making, deploying Deep Reinforcement Learning directly into flight heritage software introduces critical open questions and engineering trade-offs <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">[2]</a>:</p><pre>+---------------------------------------------------------------------------------------------------+<br>|                            SIMULATED ASSUMPTIONS vs. OPERATIONAL REALITIES                        |<br>+---------------------------------------------------------------------------------------------------+<br>| Parameter / Dimension     | Bourriez et al. (2023) Assumption    | Operational LEO Reality            |<br>+---------------------------+--------------------------------------+------------------------------------+<br>| Orbital Dynamics          | Two-body Keplerian + Pykep           | Non-Keplerian perturbations (J2,   |<br>|                           |                                      | drag spikes, SRP, 3rd body)       |<br>| Action Space              | Discrete 625 impulsive burns (Δv)    | Continuous electric propulsion     |<br>|                           |                                      | low-thrust profiles                |<br>| Space Environment         | Single &amp;amp; static multi-debris scenarios| Dense, multi-conjunction clusters |<br>|                           |                                      | and repeating encounters           |<br>| Onboard Hardware          | Unconstrained desktop GPU execution  | Radiation-hardened limits          |<br>|                           |                                      | (e.g., RAD750, ~200 MHz CPUs)      |<br>+---------------------------------------------------------------------------------------------------+</pre><h3>1. The Physics Gap: Idealized Two-Body vs. Unmodeled Perturbations</h3><p>The simulator relies on two-body Keplerian orbital mechanics combined with SGP4 propagation <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>. In actual Low Earth Orbit, unmodeled environmental perturbations — such as solar flare-induced atmospheric density spikes that dramatically alter drag coefficients, solar radiation pressure (SRP), and higher-order geopotential gravitational harmonics — introduce physical distribution shifts <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">[2]</a>. A policy trained purely on idealized SGP4 trajectories risks failing when faced with severe, unmodeled drag surges.</p><h3>2. Thrust Discretization vs. Electric Propulsion</h3><p>The paper discretizes thruster actions into 625 impulsive velocity increments (Δv) <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>. However, modern satellite constellations are increasingly shifting toward <strong>low-thrust electric propulsion (EP)</strong> systems (such as Hall effect or gridded ion thrusters) due to their high fuel efficiency (I_sp) <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">[2]</a>. Electric propulsion cannot execute instant impulsive burns; it requires continuous, low-thrust burns spread over multiple orbital revolutions. Discretized Q-learning cannot easily model continuous, long-duration low-thrust trajectories, highlighting the need for continuous action space architectures (such as Soft Actor-Critic or PPO).</p><h3>3. Safety Guarantees &amp; The “Black Box” Problem</h3><p>In space traffic management, satellite operators demand deterministic safety guarantees <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">[2]</a>. Deep neural networks are notoriously uninterpretable; a pure Deep RL policy can output unexpected control actions when encountering out-of-distribution tracking inputs. Integrating deterministic <strong>safety shields</strong> — such as Control Barrier Functions (CBFs) or Reachability Analysis — around Deep RL policy loops remains an essential prerequisite before flight space agencies will trust an AI agent with multi-million-dollar orbital assets <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">[2]</a>.</p><p><strong>Takeaway— Section 5:</strong> Major gaps remain between simulated two-body Keplerian environments and operational LEO realities <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">[2]</a>. Overcoming the physics gap, adapting discrete actions to continuous electric propulsion low-thrust profiles, and embedding deterministic safety bounds around neural network policies represent the key open research frontiers in space AI <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">[2]</a>.</p><h3>Section 6: Quick Summary &amp; The Future of Space Traffic Management</h3><pre>+-----------------------------------------------------------------------------------------------------+<br>|                               THE EVOLUTION OF SPACE SAFETY AI                                      |<br>+-----------------------------------------------------------------------------------------------------+<br>|  Manual Ground Control   ---&amp;gt;   Sequence Forecasting (TCN)   ---&amp;gt;   Onboard POMDP / DRQN Agent|<br>|  (Hours/Days latency)           (Early CDM Risk Warnings)           (Autonomous Onboard Evasion)    |<br>|  [Bourriez / GMV]               [Tok et al. 2026 / HMMs]           [Bourriez 2023 / Patnala 2024]   |<br>+-----------------------------------------------------------------------------------------------------+</pre><h3>Main Takeaways</h3><ol><li><strong>The LEO Crisis Demand:</strong> Rapid satellite constellation deployment makes manual human-in-the-loop collision avoidance unsustainable, mandating automated on-board decision-making <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dyv7bD8oNXsA">[6]</a>.</li><li><strong>Partial Observability is Essential:</strong> Space telemetry is corrupted by aleatory sensor noise and epistemic SGP4 propagation errors. Formulating spacecraft navigation as a POMDP prevents thruster over-correction by reasoning under state uncertainty <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKDFzObtE6cs">[7]</a>.</li><li><strong>LSTM as Implicit Belief Updater:</strong> Integrating an LSTM layer into a DRQN allows spacecraft to maintain temporal memory over past noisy observations, constructing an implicit belief state without requiring intractable exact Bayesian updates <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>.</li><li><strong>Generalization Matters:</strong> Training across 200 diverse, stochastic collision environments produces lower Huber loss (~0.16) and superior policy stability compared to training on single fixed environments <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>.</li></ol><h3>The Road Ahead: What’s Next in Space Domain Protection</h3><p>The research landscape is advancing along three complementary pillars:</p><ul><li><strong>Hybrid Temporal Sequence Models (TCN-Transformers):</strong> Projects like <em>Tok et al. (2026)</em> <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2609.13191">[3]</a> and <em>Catulo et al. (2023)</em> <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2311.10633">[4]</a> apply sequence networks and Hidden Markov Models directly to Conjunction Data Messages (CDMs) to predict collision probability evolution days before closest approach.</li><li><strong>Autonomous On-Orbit Servicing Spacecraft:</strong> Studies like <em>Patnala &amp; Abdin (2024)</em> <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2409.17125">[5]</a> extend Deep RL frameworks from single-satellite evasion to robotic “servicer” spacecraft that autonomously rendezvous with and protect non-maneuverable or endangered satellites.</li><li><strong>Hybrid Control &amp; Formal Safety Verification:</strong> Combining deep reinforcement learning exploration with deterministic optimal control solvers (such as Second-Order Cone Programming) ensures that space AI systems achieve both fuel-optimality and mathematically guaranteed physical safety bounds <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">[2]</a></li></ul><p><strong>Takeaway — Section 6:</strong> The future of space domain awareness lies in combining onboard POMDP decision-making, sequence-based CDM forecasting, and formal safety verification <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">[1]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">[2]</a>, <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2609.13191">[3]</a>. Transitioning these AI architectures from simulation to flight software ensures that Earth’s orbital infrastructure remains safe, sustainable, and resilient.</p><h3>References &amp; Primary Sources</h3><ol><li><strong>Bourriez, N., Loizeau, A., &amp; Abdin, A. F.</strong> (2023). <em>Spacecraft Autonomous Decision-Planning for Collision Avoidance: a Reinforcement Learning Approach</em>. 74th International Astronautical Congress (IAC). <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2310.18966">arXiv:2310.18966</a>.</li><li><strong>Pavanello, Z., De Maria, L., De Vittori, A., Maestrini, M., Di Lizia, P., &amp; Armellin, R.</strong> (2025). <em>CAMmary: A Review of Spacecraft Collision Avoidance Manoeuvre Design Methods</em>. <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2503.22555">arXiv:2503.22555</a>.</li><li><strong>Tok, R. T., Yağlıoğlu, B., Dağ, E., &amp; Kahya, E. O.</strong> (2026). <em>Early Prediction of Satellite Collision Probability Using a Hybrid TCN-Transformer Model for a CDM-Based Conjunction Analysis Framework</em>. 2026 AAS/AIAA Astrodynamics Specialist Conference. <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2609.13191">arXiv:2609.13191</a>.</li><li><strong>Catulo, J. S., Soares, C., &amp; Guimarães, M.</strong> (2023). <em>Predicting the Probability of Collision of a Satellite with Space Debris: A Bayesian Machine Learning Approach</em>. <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2311.10633">arXiv:2311.10633</a>.</li><li><strong>Patnala, S., &amp; Abdin, A. F.</strong> (2024). <em>On-orbit Servicing for Spacecraft Collision Avoidance With Autonomous Decision Making</em>. 1st ESA SPAICE / IAA Conference on AI in and for Space. <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Farxiv.org%2Fabs%2F2409.17125">arXiv:2409.17125</a>.</li><li><strong>GMV Aerospace.</strong> (2020). <em>Webinar: Towards automatic satellite collision avoidance making use of AI</em>. <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dyv7bD8oNXsA">YouTube</a>.</li><li><strong>Julia Academy.</strong> (2020). <em>Partially Observable Markov Decision Processes | Decision Making Under Uncertainty</em>. <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKDFzObtE6cs">YouTube</a>.</li><li><strong>Michigan Engineering.</strong> (2023). <em>Space Junk: How can we detect and track millions of dangerous, tiny debris?</em> <a href="https://www.google.com/url?sa=E&amp;q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DFwcHMiFomKk">YouTube</a>.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd5a9ea60523" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/navigating-the-invisible-hazard-how-deep-recurrent-reinforcement-learning-and-pomdps-automate-fd5a9ea60523">Navigating the Invisible Hazard: How Deep Recurrent Reinforcement Learning and POMDPs Automate…</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>