Your Backtest Doesn’t Care If Your Broker Disconnects at 3 AM
The layer of automated trading that no tutorial covers — and the five silent failures that taught me to build it.
There’s no shortage of X posts about automated trading. Pine Script strategies. Webhook setups. Monte Carlo simulations showing median returns north of 100% and drawdowns under 2%. First live orders completing end-to-end in just over a second. The screenshots look great. Statistically, the systems look excellent.
What you almost never see is what happens after TradingView fires that JSON webhook.
Not the strategy logic. Not the entry rules. The infrastructure underneath — the part that determines whether your perfectly-backtested system actually trades the way your backtest said it would, when nobody is watching at 3 AM and something silently breaks.
This article is about that part. Specifically, it’s about five silent failures I have personally hit running ES and MES futures through Interactive Brokers TWS over the past two years. Every one of them taught me a piece of the infrastructure my system runs on today. And every one of them is the kind of thing that won’t show up in any backtest, any Monte Carlo, any walk-forward analysis — because none of those tools model the failures of the system that runs them.
If you’re new to live automated trading, the assumption almost everyone makes when they first go live is that once the webhook fires, the hard part is done.
The hard part is just beginning.
The Illusion of “It Works”
There is a moment, very early in your live trading career, when everything appears to be working. The backtest is profitable. The first live order executes cleanly. You watch a fill come through in real time and feel a kind of competence that you’ve never felt before. You scale up. You add capital. You sleep better.
This is the most dangerous moment in algorithmic trading.
Backtests measure your strategy. They do not measure your infrastructure. The backtest assumes the broker is connected. It assumes the order arrives at the exchange. It assumes positions reconcile correctly after every fill. It assumes the data feed is fresh. It assumes nothing breaks.
Production assumes nothing.
Every assumption your backtest makes is something production can violate, and most violations are silent. The system doesn’t crash. It doesn’t throw an error. It just stops doing the thing you thought it was doing — and unless you’ve built explicit machinery to detect that, you won’t know until your account statement disagrees with your dashboard.
Here are the five.
Failure 1: The Overnight Disconnect
Interactive Brokers performs a nightly server-side reset that drops TWS’s session. The window is brief — usually under a minute — and it happens around the same time every day. Connection drops also occur for less predictable reasons: network blip on your end, IB infrastructure work, a flaky Wi-Fi router.
Here is what the failure looks like from inside your code:
Your trading process is still running. The Python interpreter is happy. Your webhook receiver is listening on its port. TradingView fires an alert, sends it to your webhook, and your handler picks it up correctly. You call ib.placeOrder(). No exception is raised. The system continues to the next alert.
But nothing executes. Because the socket to TWS is gone.
ib_insync (the Python wrapper for the IB API) will queue the order locally and pretend everything is fine until your event loop tries to actually talk to TWS. By the time you notice, you’ve missed a fill — or worse, missed a stop.
The fix is auto-reconnect, but auto-reconnect is harder than it sounds. You have to:
- Detect that the socket has actually died, not just that no message has arrived recently
- Increment your Client ID, because TWS may still hold the previous one in a stale socket (Error 326)
- Coordinate the reconnect across all the threads that might trigger it (your health-check thread, your webhook handler, your dashboard), or two reconnect attempts will fight each other and produce a worse failure than the original disconnect
- Reconcile your positions after reconnecting, in case anything happened during the gap
And Error 326 doesn’t arrive as a Python exception — it comes through the ib.errorEvent callback, while connectAsync itself raises a TimeoutError. If you're only catching exceptions, you'll miss it.
There’s one more wrinkle worth knowing about: even with TWS configured to auto-restart instead of auto-logoff (so it cycles cleanly without prompting for credentials), IBKR invalidates the platform’s authentication token once a week, every Sunday at 1:00 AM ET. The next restart after that needs a fresh manual login. This is documented IBKR behavior, not a quirk.
The most popular open-source mitigation is IBC (IbcAlpha/IBC on GitHub), which automates TWS by detecting login dialogs and filling in your credentials programmatically. With IBC running, the daily restart is fully hands-off, and after the Sunday 1:00 AM ET token reset, IBC will refill your username and password automatically the next time TWS starts. What IBC cannot do is acknowledge the second-factor authentication on your behalf. IBKR's IB Key 2FA sends a push notification to the IBKR Mobile app on your phone, and a human has to tap "Approve" — there is no API or programmatic way around this. IBC will keep re-issuing login attempts and prompting the alert until you acknowledge it, but the tap itself is unavoidable.
The practical consequence is that even with IBC, a fully unattended setup still requires one human touchpoint per week, on Sunday after 1:00 AM ET. I’m not done trying to eliminate that step, but as of today I haven’t found a way that doesn’t compromise security. Plan for it — schedule a Sunday-evening or Monday-morning check, and make sure your monitoring tells you when the system is sitting at a 2FA prompt instead of trading.
I have a threading.Lock with a two-second acquire timeout protecting every reconnect path in my system. It exists because I learned the hard way that concurrent reconnects on the same ib object cause both attempts to fail.
Failure 2: The Subscription That Lies
This one took me five separate incidents to fully understand.
The IB API has a subscription model for streaming data. You call reqPnL() once, and IB pushes you P&L updates whenever your account state changes. It's elegant. It's efficient. It is also fragile in ways that aren't documented anywhere.
Here is what kept happening on my system:
ib.isConnected() would return True. The TCP socket was fine. The event loop was processing messages. My P&L callback was registered. And yet my dashboard showed yesterday's P&L all morning, while TWS itself showed different numbers.
The subscription had silently died. The callback wasn’t being called anymore. Or — even more confusing — it was being called, but always with the same stale value, while the underlying account state had changed.
The fix is what I now think of as multi-signal staleness detection. You can’t trust any single signal:
- Recency check — when did the callback last fire? If it’s been more than 60 seconds during market hours and your account has open positions, that’s suspicious.
- Value-change check — is the value actually moving? Callbacks that fire but never deliver new data are a real failure mode, not a theoretical one.
- Forced refresh — every 10 minutes, cancel the subscription and resubscribe from scratch. Treat this as a nuclear safety net for failure modes you haven’t catalogued yet.
And underneath all of that, you need to know that ib.cancelPnL() takes the account ID string, not the subscription object returned by reqPnL(). The IB API accepts the wrong parameter type silently. No exception. No error log. The cancel just doesn't happen. Stale entries accumulate in ib.wrapper.pnlKey2ReqId, and the next reqPnL() call eventually hits an AssertionError because the key already exists.
This single API quirk was the hidden root cause of weeks of incidents that looked like they had different causes. Every “fix” I deployed appeared to work. None of them did, because the cancellation step was failing silently the entire time.
Failure 3: The Half-Open Connection
Get Willow the Trader’s stories in your inbox
Join Medium for free to get updates from this writer.
This is the most insidious failure on the list, because every health check you’ve built will tell you the system is fine.
TCP sockets can enter a state where one side has effectively gone away — no traffic flowing, no responses coming back — but the socket itself is still open from the other side’s perspective. The kernel sees no FIN packet. Your application sees a connected socket. Heartbeats report alive. Logs say healthy.
P&L callbacks would stop, account updates would continue on their normal schedule, and the connection looked healthy to every check I had. The only way to detect it was to actually ask TWS a question that required a fresh response: reqCurrentTimeAsync(), with a 15-second timeout.
If the event loop can’t get a current-time response back in 15 seconds, the connection is dead even though it looks alive. Force a disconnect, reconnect cleanly, verify the subscription path is healthy again.
The lesson generalizes beyond IB: any system depending on a long-lived socket needs an active health check, not just a passive one. Counting incoming messages is insufficient. You need to send something and verify you got an answer.
Failure 4: The Orphaned Position After a Contract Roll
Futures roll quarterly. If you trade ES or MES, you’ve experienced this: the front-month contract approaches expiry, liquidity migrates to the next contract, and at some point you switch over.
If your system uses a continuous symbol like MES1! or ESH26, you also have to handle the moment of switching. Roll early or roll late, but you cannot afford to be ambiguous.
The failure mode looks like this:
You hold a long position on the current contract. Your roll logic switches you to the next contract — maybe four days before expiry, maybe at a specific date. A close signal arrives. Your code resolves “MES” to the new contract, asks IBKR “what’s my position on this contract?” and gets back zero. Your handler returns “no order needed.”
Meanwhile, your actual position on the old contract is still sitting there, fully open, with no risk management attached.
The fix is to scan all positions of the same base symbol when a close signal arrives, not just the position on the contract you currently consider active. Auto-flatten the orphans before processing the main order. Treat break statements in position-scanning loops as bugs by default.
There’s also a quirk specific to IB worth flagging: the contract objects returned by ib.positions() do not include the exchange field. If you try to use one of those contract objects directly to close a position, you'll get IBKR Error 321 (missing order exchange). You have to explicitly set contract.exchange = "CME" before calling placeOrder().
Failure 5: The External Monitor That Wasn’t Monitoring Anything Useful
This was the failure that made me rebuild my entire monitoring strategy.
I had Uptime Robot pinging my heartbeat endpoint every five minutes. The endpoint returned HTTP 200 if the process was running. For weeks, every check returned 200. My monitor showed 100% uptime. I was sleeping well.
Underneath, my system was disconnecting from TWS every five minutes and reconnecting. Twenty-four hours of cycling. Zero alerts.
The endpoint was returning 200 because the web process was healthy. The web process was fine. The dependency it relied on — the TWS connection — was not. And my monitor had no way to know, because the endpoint had been designed to “always return 200, put dependency status in the response body” — which is correct for general-purpose web apps and exactly wrong for a trading system.
For a 24/7 automated trader, the most dangerous silent failure is “the app is alive but can’t trade.” Your external monitor has to reflect trade execution capability, not process health.
The fix is to separate the two signals. Web-process health and trade-execution capability are different questions, and they deserve different answers. The web-health signal stays 200-always — that’s correct for general monitoring. The trade-capability signal returns a non-200 status when IBKR is disconnected, and that’s the one Uptime Robot watches. The always-200 pattern isn’t wrong; it’s just wrong for the question that actually matters here.
The Pattern Behind All Five
Every silent failure in this list has the same shape.
The happy path looks healthy. Process running. Socket open. Logs flowing. Dashboard responding. No exceptions in your error tracker.
There is a hidden dependency that can fail without raising an error. A subscription that stops delivering. An API that silently accepts the wrong type. A continuous symbol that resolves to a different contract than yesterday.
The system has no way to know it’s broken. Because you didn’t build the layer that knows.
The fix has the same shape every time:
- Detection. Track the actual thing you care about — a fresh callback, a changing value, a successful fill — and alarm when it stops.
- Escalation. Don’t just retry. Retry → resubscribe → reconnect → alert a human. Each level stronger than the last, with cooldowns to prevent loops.
- Awareness of legitimate quiet. Markets close. CME has a daily halt window from 5 to 6 PM ET. Weekends exist. Don’t trigger recovery on an empty schedule.
This isn’t glamorous. It’s not the part of automated trading anyone writes about. It is also the part that determines whether the system you built actually trades the way your backtest said it would.
Why This Matters More Than Your Win Rate
Imagine a strategy with a 60% win rate.
Now imagine it silently misses one trade in five because the broker connection was dead at the moment a signal arrived. You don’t know which trade you missed. The system reports its normal trades correctly. Your dashboard looks fine.
Your effective win rate is now 48%. Your expectancy has collapsed. Your equity curve is degraded in a way that no backtest, no Monte Carlo, no walk-forward analysis would ever predict — because none of those tools model the failures of the system that runs them.
Most of the time, a missed trade is just a missed trade. But sometimes the missed trade is a stop that should have closed a losing position, and the system holds that position until you wake up. Then it isn’t just expectancy — it’s an account-level event.
The strategy is maybe 20% of the work of running an automated trading system. The other 80% is the infrastructure that determines whether your strategy gets to run at all.
Where to Start
If you’re running an algorithmic system live right now and any of this is new, here is a short list, in order of impact:
- Add an external monitor that pings your trade-execution capability, not just your web process. If your broker is disconnected, your monitor should know.
- Track the freshness of every callback you depend on. If a feed has been silent for longer than expected during market hours, alarm.
- Verify positions after every reconnect, before processing the next signal. A reconnect that doesn’t include a reconciliation step is a reconnect that can leak risk.
- Test failure modes deliberately. Pull the network cable. Kill TWS. Restart the broker connection mid-trade. Watch what your system does. If you’ve never done this, the first time it happens in production will be the first time you find out.
Monte Carlo doesn’t model your Internet dropping. The strategy is the part you can see — the infrastructure is the part that decides whether anything you built ever runs.
If you’re at the start of running a real algorithmic system, you’re going to discover all of this over the next year. You’ll be running real money against real markets, and at some point your managed signal-relay platform will have an outage, or your exchange API will return an unexpected response, or your system will hold a position through an event it didn’t predict. You’ll learn the same things every other automated trader has learned, just by your own path.
If this article saves anyone a week of incident debugging at 3 AM, it has paid for itself.
Willow the Trader runs an automated futures system on ES and MES through Interactive Brokers TWS. He writes about systematic trading and AI tooling at @TechTradingUSA.
