116 lines
6.1 KiB
Markdown
116 lines
6.1 KiB
Markdown
# Hourly Backtest Catastrophic Failure Analysis (2026-02-11)
|
|
|
|
## Results Summary
|
|
- Period: 3 months hourly (Nov 12, 2025 - Feb 11, 2026)
|
|
- Total Return: -11.44% ($88,563 from $100k)
|
|
- CAGR: -38.48%, Sharpe: -2.53, Sortino: -0.38
|
|
- Win Rate: 10% (2/20), Profit Factor: 0.08
|
|
- All 20 trades occurred in first 8 trading days, then system went to 100% cash permanently
|
|
|
|
## Trade-by-Trade Analysis
|
|
All trades between Nov 12-21 (8 trading days, ~56 hourly bars):
|
|
|
|
### Buys
|
|
1. Nov 12 17:00 - MU @$245.29 (89 shares = $21.8k)
|
|
2. Nov 12 17:00 - AMD @$255.39 (86 shares = $21.9k)
|
|
3. Nov 12 20:00 - GOOGL @$287.08 (76 shares = $21.8k)
|
|
4. Nov 12 20:00 - CAT @$573.69 (38 shares = $21.8k)
|
|
5. Nov 12 20:00 - SNOW @$269.80 (28 shares = $7.6k) -- total 5 positions, 95% invested
|
|
|
|
### Sell Cascade (Nov 13-21)
|
|
Nov 13: GOOGL -$636 (-2.9%), MU -$694 (-3.2%), SNOW -$226 (-3.0%), CAT -$744 (-3.4%)
|
|
Nov 13: Rebuy ASML, AAPL
|
|
Nov 14: ASML -$705 (-3.4%), AMD -$1,352 (-6.2%!!) -- exceeded 4% max loss cap
|
|
Nov 14: Rebuy MU, GOOGL, CAT, SNOW
|
|
Nov 17: AAPL -$440 (-2.1%), MU -$699 (-3.3%)
|
|
Nov 17: Rebuy AMD, GOOGL, ASML
|
|
Nov 18: AMD -$1,582 (-7.6%!!), CAT -$493 (-2.4%), GOOGL -$426 (-2.0%)
|
|
Nov 18: Rebuy AAPL, MU
|
|
Nov 19: MU -$941 (-4.6%), SNOW -$39 (-0.6%)
|
|
Nov 19: Rebuy AMD, ISRG, SNOW
|
|
Nov 20: AMD -$1,170 (-5.8%!), ISRG -$385 (-1.9%), SNOW -$164 (-2.6%), ASML -$442 (-2.2%)
|
|
Nov 20: Rebuy MU
|
|
Nov 21: MU -$1,274 (-6.6%!!), AAPL +$275 (+1.4%) -- ONLY WIN
|
|
|
|
### Key Observations
|
|
1. AMD losses: -$1,352, -$1,582, -$1,170 = -$4,104 total (36% of all losses). ALL exceeded the 4% max loss cap.
|
|
2. MU losses: -$694, -$699, -$941, -$1,274 = -$3,608 (32% of all losses).
|
|
3. After Nov 21: drawdown hit 11.65%, exceeding 10% halt. System went to 100% cash PERMANENTLY.
|
|
4. Equity curve shows 0 positions from Nov 21 through Feb 11 (461 bars of nothing).
|
|
5. Only 2 wins out of 20: GOOGL +$701, AAPL +$275. Total wins = $976.
|
|
|
|
## Root Cause #1: Absurd Indicator Period Scaling (FUNDAMENTAL)
|
|
|
|
The 7x multiplier creates these hourly indicator periods:
|
|
- RSI: 14 * 7 = 98 bars (14 trading days)
|
|
- MACD fast: 12 * 7 = 84 bars
|
|
- MACD slow: 26 * 7 = 182 bars (26 trading days)
|
|
- MACD signal: 9 * 7 = 63 bars
|
|
- EMA short: 9 * 7 = 63 bars
|
|
- EMA long: 21 * 7 = 147 bars
|
|
- EMA trend: 50 * 7 = 350 bars (50 trading days)
|
|
- ADX: 14 * 7 = 98 bars
|
|
- Bollinger: 20 * 7 = 140 bars
|
|
- Volume MA: 20 * 7 = 140 bars
|
|
- Momentum (ROC): 63 * 7 = 441 bars (63 trading days)
|
|
|
|
min_bars() = max(182+63, 98+1, 350, 98*2, 140, 441) + 5 = 446 bars
|
|
|
|
This means the system needs ~64 trading days (446/7) of WARMUP before it can even produce a valid signal. For a 3-month backtest, that eats most of the data. The indicators that DO produce values are extremely slow-moving and unresponsive to hourly price action.
|
|
|
|
### Why 7x is Wrong
|
|
The daily parameters (RSI-14, MACD 12/26/9) are designed for daily price action noise. On hourly bars, there are 6.5-7 bars per day, so naively you'd think 7x preserves the "look-back in days." But this ignores that:
|
|
1. Hourly bars have fundamentally different noise characteristics (mean-reverting intraday patterns)
|
|
2. A 98-period RSI on hourly bars is insanely slow -- it would take a massive multi-week move to push RSI to oversold/overbought
|
|
3. MACD with 182-period slow EMA cannot detect hourly momentum shifts
|
|
4. The strategy's edge is supposed to be momentum + pullback detection. A 441-bar momentum period on hourly data measures ~3-month trends, not tactical momentum.
|
|
|
|
## Root Cause #2: Drawdown Halt is Terminal
|
|
|
|
MAX_DRAWDOWN_HALT = 10%, MAX_DRAWDOWN_RESUME = 5%.
|
|
|
|
Once drawdown exceeds 10%, the system stops buying. It only resumes when drawdown recovers to 5%. But with 0 positions, there's no way to recover! The portfolio sits in cash forever.
|
|
|
|
This is a logical impossibility: you can't recover from drawdown without taking new positions, but the system won't take new positions until drawdown recovers.
|
|
|
|
## Root Cause #3: Max Loss Cap Not Working
|
|
|
|
AMD lost -6.2%, -7.6%, -5.8% in three trades despite MAX_LOSS_PCT = 4%.
|
|
MU lost -6.6% in one trade.
|
|
|
|
The check_stop_loss_take_profit function checks `pnl_pct <= -MAX_LOSS_PCT` but this only triggers on the NEXT bar after the loss occurs. If a stock gaps down or moves 6% in one hourly bar, the 4% cap is breached before the check runs. On hourly timeframe, overnight gaps can easily exceed 4%.
|
|
|
|
## Root Cause #4: All Indicators Fire Simultaneously at Warmup Edge
|
|
|
|
When indicators first become valid (right after the warmup period), all the EMA crossovers, MACD crossovers, etc. fire at once. This creates a burst of buy signals, putting 95% of capital at risk immediately. The system went from 0 to 5 positions in 3 hours on Nov 12.
|
|
|
|
## Proposed Fixes
|
|
|
|
### Fix 1: Proper Hourly Indicator Periods (NOT 7x scaling)
|
|
Use empirically appropriate periods for hourly timeframe:
|
|
- RSI: 14 (same as daily -- 14 hourly bars captures 2 trading days of momentum)
|
|
- MACD: 12/26/9 (same as daily -- these already work on any timeframe)
|
|
- EMA short: 20, EMA long: 50, EMA trend: 100 (~14 trading days)
|
|
- ADX: 14
|
|
- Bollinger: 20
|
|
- Volume MA: 20
|
|
- ATR: 14
|
|
- Momentum ROC: 63 (same as daily -- 63 hourly bars = ~9 trading days)
|
|
|
|
Rationale: Most technical indicators were designed for BARS, not calendar time. RSI-14 means "14 bars of price action" regardless of timeframe. The 7x scaling was conceptually wrong -- it assumed indicators need calendar-day equivalence, but they need bar-count equivalence.
|
|
|
|
### Fix 2: Remove or Redesign Drawdown Halt for Backtesting
|
|
Options:
|
|
a) Remove drawdown halt entirely in backtesting (it's more appropriate for live trading where you want manual review)
|
|
b) Make drawdown halt time-based: halt for N bars, then auto-resume
|
|
c) Change to a gradual reduction: reduce position size by 50% instead of going to 0
|
|
|
|
### Fix 3: Limit Initial Deployment Speed
|
|
Don't go from 0 to 5 positions in one bar. Add a "ramp-up" period where max positions increases gradually (e.g., 1 per day for first week).
|
|
|
|
### Fix 4: Tighter Max Loss Cap with Intra-Bar Checks
|
|
For hourly mode, use tighter max loss (3%) or check at open price vs entry to catch gap losses earlier.
|
|
|
|
### Fix 5: Use SIP Feed Instead of IEX
|
|
Change `feed=iex` to `feed=sip` for consolidated market data. IEX-only volume is unreliable for volume-based signals.
|