The Three-Day Bug Hunt: How a Simple String Replace Broke Everything
The Three-Day Bug Hunt: How a Simple String Replace Broke Everything
December 15, 2025
Sometimes the most frustrating bugs are the ones that seem obvious in hindsight. Today I finally cracked a bug that had been haunting my automated trading system for three days.
The Symptom
Trade #63, a SHORT position on SOL-PERP, was opened at 23:21. Fifteen minutes later, at the first quick monitoring check, the system decided the position didn't exist and closed it with $0 P&L.
The log told the story:
ā ļø Trade 63 has no position on exchange. Closing in database.
š Closing trade 63 (reason: manual)
š° P&L: $0.00
A perfectly valid trade, killed by the system itself.
The Hunt
I'd been circling this bug for days. The symptoms kept shifting:
- Day 1: "Orphan warnings" - positions detected on exchange but not in DB
- Day 2: "Cache desync" - the paper position cache wasn't updating
- Day 3: Positions detected but immediately closed as "not found"
Each day I'd fix something, deploy, and think it was solved. Each morning I'd wake up to find another variation of the same fundamental problem.
The Root Cause
After deep-diving into the code flow today, I found it. One line. In trade_opener.py:
symbol=proposal['symbol'].replace('/USD', '').replace('-PERP', '')
The code was "normalizing" symbols by stripping the -PERP suffix. So when a trade proposal came in for SOL-PERP, it got saved to the database as just SOL.
But the paper position cache? That stored it as SOL-PERP.
When PositionTracker later read the trade from the database, it saw symbol = SOL. No -PERP suffix. So it routed to the spot adapter instead of the perpetual adapter. The spot adapter had no position for SOL. Result: "No position on exchange. Closing in database."
The comment said # Normalize to base symbol. It sounded so reasonable.
The Fix
One line change:
symbol=proposal['symbol'].replace('/USD', '') # Only strip /USD, keep -PERP
Now the trade stores SOL-PERP, matches the cache key, and routes correctly.
The Lesson
Data consistency across components is non-negotiable. When you have:
- A database storing trade records
- A memory cache storing position state
- A routing function that decides which adapter to use
They all need to agree on the key format. SOL vs SOL-PERP might look minor, but it was the difference between a working system and one that killed its own trades.
The "normalization" comment was the trap. It made the code look intentional and correct. But the author (past me, probably) didn't trace through what happened downstream when PositionTracker tried to look up that "normalized" symbol.
Three Days of Debugging
Was this a waste of time? Partly. But I also:
- Fixed the TradeCloser to properly receive the perpetual adapter (Sprint 33)
- Fixed order ID collisions in paper trading (Sprint 32)
- Added better error handling for "not found" order cancellations
The system is more robust now. But the core bug - the one that actually killed Trade #63 - was a single .replace('-PERP', '') that should never have been there.
Tomorrow I'll watch the logs carefully. But I think this one is finally dead.
Building Trader-7 in public. Follow along at jamiewatters.work