Robust Portfolio Optimisation Engine
Mean-variance optimisation is an error maximiser. It takes noisy estimates, trusts them completely, and bets heavily on whatever looked best last year. I built a walk-forward backtester to measure how much that actually costs, then tested two of the textbook fixes against each other on identical inputs. It grew out of my second-year essay on Michaud's resampled efficiency as an answer to Markowitz.
- Universe
- 7 ETFs · equity, bonds, gold, property
- Period
- 2005–2026 · 5,430 days
- Lookback
- 252 days
- Rebalance
- monthly
- Costs
- 10 bps turnover
- Evaluation
- walk-forward, out of sample
- Price datadata/
- Estimate inputsestimation/
- Choose weightsoptimisation/
- Backtest honestlybacktest/
- Compare to benchmarkbenchmarks/
- AI market viewsphase 3
Markowitz loses to equal weight, and not narrowly. It earns less while taking more risk: 18.4% volatility against 13.1%. That's the opposite of what a method built to optimise risk is supposed to do. It also traded 349% a year to get there.
| strategy | ann. return | ann. vol | Sharpe | max DD | turnover |
|---|---|---|---|---|---|
| Equal weight (1/N) | 7.86% | 13.10% | 0.643 | −36.7% | 2.4% |
| Markowitz (λ=3) | 7.44% | 18.36% | 0.483 | −31.7% | 349% |
| Markowitz + Ledoit-Wolf | 7.44% | 18.36% | 0.483 | −31.6% | 348% |
| Markowitz + resampling | 8.28% | 14.07% | 0.636 | −29.5% | 220% |
| Minimum variance | 3.67% | 5.35% | 0.700 | −19.7% | 30.4% |
| Minimum variance + LW | 3.59% | 5.32% | 0.689 | −19.8% | 29.9% |
Why it loses
- It isn't diversifying.
It holds 1.6 of 7 assets on average, with 85% in the biggest one. Every month it picks a favourite, piles in, and changes its mind the next month. That's not a portfolio, it's a sequence of bets.
- Costs aren't the cause.
With costs switched off it still loses (0.502 vs 0.643). The decisions are bad, not just expensive.
- The expected returns carry no signal.
In a typical window, not even the best-looking asset has a mean return distinguishable from zero (median largest |t| = 1.79, against 1.96). The covariance matrix is fine. The problem is μ.
Shrinkage is a correct implementation of a tool aimed at a problem this project doesn't have. It chose a median intensity of 0.055 and changed nothing. Resampling goes after μ, and the optimiser goes from making one bet to holding a real portfolio.
Not reporting the best row
λ is the one number in the strategy I choose rather than estimate, so it's the easiest place to fit the model to its own test set. I fixed λ=3 from the literature before running anything, then published the whole curve instead of the best result.
Performance improves the more the optimiser ignores its own return forecasts, steadily from λ=1 to λ=50. One lucky value would be suspicious. A trend across a 50-fold range is a property of the data, and it points to the same conclusion as the diagnostics.
What I got wrong along the way
- The off-by-oneOne character,
:tagainst:t+1, separates a valid backtest from a fantasy. Nothing in the output would look wrong. - Forward-fill vs backfill
ffill()andbfill()are one character apart, and one of them is lookahead bias with no visible symptom. There's now a test for it. - A test that couldn't failMy first engine test passed because the rule meant to churn the portfolio never churned. Next time the test gets written before the engine.
The dangerous errors here aren't the ones that crash. They're the ones that produce a plausible number. findings.md, §12
Caveats, stated up front
Resampling still doesn't beat 1/N. 0.636 against 0.643 is a tie at best, and 1/N gets there trading 2.4% a year instead of 220%. Minimum variance tops the Sharpe column by hiding 88% in bonds during an exceptional run for bonds. Next up: regime awareness (Phase 2), then LLM-generated views into Black-Litterman (Phase 3), evaluated out of sample like everything else. I've written my prediction for Phase 3 down in advance: it helps, but by less than resampling did.
