ramseshk
0446443d36
New strategies: - Cross-Sectional Momentum: long top-N, short bottom-N across HL universe - Spot-Perp Basis Arbitrage: delta-neutral spot vs perp price gap trading - Regime-Switching Ensemble: dynamically allocates strategies by market regime - Portfolio Construction: risk parity, vol targeting, correlation penalty Infrastructure: - DuckDBDataProvider: real tick/candle data for backtests (replaces synthetic) - Walk-Forward Validation: systematic IS/OOS across all 12 strategies - 3 Jupyter research notebooks (EDA, strategy research, portfolio) Pipeline integration: - deploy.py registry, sweep_runner, vbt_runner all updated - fee_tiers support for new strategies - All modules syntax-validated and import-tested
872 lines
32 KiB
Python
872 lines
32 KiB
Python
"""
|
||
Generate research notebooks for FTDT Quant Lab.
|
||
|
||
Creates three notebooks:
|
||
1. 01_eda.ipynb — market data exploration, distributions, correlations
|
||
2. 02_strategy_research.ipynb — strategy backtesting, signal analysis, optimization
|
||
3. 03_portfolio.ipynb — portfolio construction, risk allocation, ensemble
|
||
"""
|
||
import nbformat as nbf
|
||
from pathlib import Path
|
||
|
||
NOTEBOOKS_DIR = Path(__file__).resolve().parent.parent / "notebooks"
|
||
NOTEBOOKS_DIR.mkdir(parents=True, exist_ok=True)
|
||
|
||
|
||
def create_eda_notebook():
|
||
nb = nbf.v4.new_notebook()
|
||
nb.metadata = {
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"language": "python",
|
||
"name": "python3",
|
||
},
|
||
"language_info": {"name": "python", "version": "3.13.0"},
|
||
}
|
||
|
||
nb.cells = [
|
||
nbf.v4.new_markdown_cell("""# FTDT Quant Lab — Exploratory Data Analysis
|
||
|
||
**Goal:** Understand Hyperliquid market microstructure, identify alpha sources, verify data quality.
|
||
|
||
**Assets:** BTC, ETH, SOL, HYPE, ARB, OP, and others
|
||
**Data Sources:** DuckDB tick database, HL REST API, Parquet raw store
|
||
**Timeframe:** 1s tick → 1h candles → daily aggregation"""),
|
||
|
||
nbf.v4.new_code_cell("""# Setup
|
||
import sys
|
||
from pathlib import Path
|
||
sys.path.insert(0, str(Path.cwd().parent))
|
||
|
||
import numpy as np
|
||
import pandas as pd
|
||
import matplotlib.pyplot as plt
|
||
import seaborn as sns
|
||
|
||
from data.duckdb_provider import DuckDBProvider
|
||
from framework.data import HyperliquidDataProvider
|
||
|
||
sns.set_theme(style="darkgrid")
|
||
plt.rcParams["figure.figsize"] = (14, 6)
|
||
plt.rcParams["figure.dpi"] = 100
|
||
|
||
# Data providers
|
||
duckdb = DuckDBProvider()
|
||
hl_rest = HyperliquidDataProvider(testnet=False)
|
||
|
||
print(f"DuckDB available: {duckdb.available}")
|
||
print(f"Data range: {duckdb.get_data_range()}")
|
||
print(f"Available coins: {duckdb.get_available_coins()}")
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 1. Universe Overview
|
||
|
||
What assets are available and how much data do we have for each?"""),
|
||
|
||
nbf.v4.new_code_cell("""from strategies.cross_sectional_momentum import HL_UNIVERSE, HIGH_LIQUIDITY
|
||
|
||
print(f"Full universe ({len(HL_UNIVERSE)} assets): {HL_UNIVERSE}")
|
||
print(f"High liquidity ({len(HIGH_LIQUIDITY)}): {HIGH_LIQUIDITY}")
|
||
|
||
# Fetch candle data for each asset
|
||
prices = duckdb.fetch_multi_candles(HIGH_LIQUIDITY, interval='1h', limit=500)
|
||
|
||
print(f"\\nData availability:")
|
||
for coin, df in prices.items():
|
||
if not df.empty:
|
||
print(f" {coin:6s}: {len(df):5d} bars | {df.index[0]} to {df.index[-1]} | close=${df['close'].iloc[-1]:.2f}")
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 2. Return Distributions
|
||
|
||
Check return distributions for normality, skew, kurtosis, and tail behavior.
|
||
This informs strategy design — mean reversion works on platykurtic distributions,
|
||
momentum thrives on leptokurtic tails."""),
|
||
|
||
nbf.v4.new_code_cell("""returns_data = {}
|
||
for coin in HIGH_LIQUIDITY:
|
||
df = prices.get(coin)
|
||
if df is None or df.empty:
|
||
continue
|
||
rets = df['close'].pct_change().dropna()
|
||
returns_data[coin] = rets
|
||
|
||
stats = []
|
||
for coin, rets in returns_data.items():
|
||
stats.append({
|
||
'coin': coin,
|
||
'mean_annual': rets.mean() * 365 * 24,
|
||
'vol_annual': rets.std() * np.sqrt(365 * 24),
|
||
'sharpe': rets.mean() / rets.std() * np.sqrt(365 * 24) if rets.std() > 0 else 0,
|
||
'skew': rets.skew(),
|
||
'kurtosis': rets.kurtosis(),
|
||
'var_95': rets.quantile(0.05),
|
||
'cv': rets.std() / rets.mean() if rets.mean() != 0 else 0,
|
||
'max_dd': (df['close'] / df['close'].cummax() - 1).min(),
|
||
})
|
||
|
||
stats_df = pd.DataFrame(stats).set_index('coin')
|
||
stats_df.round(4)
|
||
"""),
|
||
|
||
nbf.v4.new_code_cell("""# Return distribution plots
|
||
fig, axes = plt.subplots(2, 3, figsize=(18, 10))
|
||
for ax, (coin, rets) in zip(axes.flat, returns_data.items()):
|
||
rets.hist(bins=100, ax=ax, alpha=0.7, density=True)
|
||
ax.set_title(f"{coin} — Skew: {rets.skew():.2f}, Kurt: {rets.kurtosis():.2f}")
|
||
ax.axvline(0, color='red', linestyle='--', alpha=0.5)
|
||
plt.tight_layout()
|
||
plt.show()
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 3. Correlation Matrix
|
||
|
||
Identify redundant assets and diversification opportunities.
|
||
High correlation = limited diversification benefit.
|
||
Low correlation = potential for uncorrelated alpha streams."""),
|
||
|
||
nbf.v4.new_code_cell("""corr_matrix = pd.DataFrame(returns_data).corr()
|
||
mask = np.triu(np.ones_like(corr_matrix), k=1)
|
||
sns.heatmap(corr_matrix, mask=mask, annot=True, fmt='.3f', cmap='RdBu_r',
|
||
center=0, vmin=-1, vmax=1, square=True)
|
||
plt.title('Hourly Return Correlation Matrix')
|
||
plt.tight_layout()
|
||
plt.show()
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 4. Volatility Regimes
|
||
|
||
Classify the market into volatility regimes. This drives strategy selection
|
||
in the Regime-Switching Ensemble.
|
||
|
||
- LOW_VOL: annualized < 15% → market making, pairs trading
|
||
- NORMAL: 15-60% → all strategies at baseline
|
||
- HIGH_VOL: > 60% → momentum, Hurst/VPIN, tight risk controls"""),
|
||
|
||
nbf.v4.new_code_cell("""from strategies.regime_ensemble import RegimeDetector
|
||
|
||
detector = RegimeDetector(
|
||
high_vol_threshold=0.60,
|
||
low_vol_threshold=0.15,
|
||
funding_extreme_apr=0.30,
|
||
)
|
||
|
||
btc_prices = prices['BTC']['close']
|
||
regimes = []
|
||
for i, px in enumerate(btc_prices):
|
||
detector.feed_price(px)
|
||
if i >= 100:
|
||
regimes.append(detector.primary_regime())
|
||
|
||
# Count regime distribution
|
||
regime_counts = pd.Series(regimes).value_counts()
|
||
print("Regime Distribution:")
|
||
for regime, count in regime_counts.items():
|
||
print(f" {regime:20s}: {count:5d} bars ({count/len(regimes)*100:.1f}%)")
|
||
"""),
|
||
|
||
nbf.v4.new_code_cell("""# Regime timeline
|
||
import matplotlib.dates as mdates
|
||
|
||
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(16, 8), sharex=True)
|
||
|
||
ax1.plot(btc_prices.index[-len(regimes):], btc_prices.values[-len(regimes):],
|
||
linewidth=0.5, color='black')
|
||
ax1.set_ylabel('BTC Price')
|
||
ax1.set_title('BTC Price with Market Regimes')
|
||
|
||
regime_colors = {
|
||
'NORMAL': 'gray', 'TRENDING': 'green', 'MEAN_REVERTING': 'blue',
|
||
'CHOPPY': 'orange', 'HIGH_VOL': 'red', 'LOW_VOL': 'lightblue',
|
||
'FUNDING_EXTREME': 'purple',
|
||
}
|
||
regime_numeric = pd.Series(
|
||
[{v: i for i, v in enumerate(regime_colors)}.get(r, 0) for r in regimes],
|
||
index=btc_prices.index[-len(regimes):]
|
||
)
|
||
ax2.scatter(regime_numeric.index, regime_numeric.values, c=[regime_colors.get(r, 'gray') for r in regimes],
|
||
s=1, alpha=0.6)
|
||
ax2.set_yticks(range(len(regime_colors)))
|
||
ax2.set_yticklabels(regime_colors.keys())
|
||
ax2.set_ylabel('Regime')
|
||
|
||
plt.tight_layout()
|
||
plt.show()
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 5. Fee Impact Analysis
|
||
|
||
Hyperliquid perp fee schedule. Calculate the minimum edge needed to overcome
|
||
fees at each tier. This sets the floor for signal strength thresholds."""),
|
||
|
||
nbf.v4.new_code_cell("""from config.fee_tiers import PERPS_TIERS, SPOT_TIERS, STAKING_TIERS, get_perp_fees, compute_trade_fees
|
||
|
||
print("=== Perp Fee Tiers ===")
|
||
print(f"{'Tier':<20} {'Volume':>12} {'Taker':>8} {'Maker':>8}")
|
||
print("-" * 50)
|
||
for tier, info in PERPS_TIERS.items():
|
||
print(f"{info['name']:<20} ${info['min_volume']:>10,.0f} "
|
||
f"{info['taker']*100:.3f}% {info['maker']*100:.3f}%")
|
||
|
||
print(f"\\n=== Spot Fee Tiers ===")
|
||
for tier, info in SPOT_TIERS.items():
|
||
print(f"{info['name']:<20} ${info['min_volume']:>10,.0f} "
|
||
f"{info['taker']*100:.3f}% {info['maker']*100:.3f}%")
|
||
|
||
# Break-even trade size by fee tier
|
||
print(f"\\n=== Minimum Profitable Trade (BTC round-trip, 1bps edge) ===")
|
||
for tier in range(7):
|
||
fees = compute_trade_fees("BUY", 0.001, 100000, 100000, vip_tier=tier)
|
||
print(f" Tier {tier}: {fees['effective_rate_pct']:.4f}% per side "
|
||
f"→ ${fees['total_fee']:.4f} round-trip")
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 6. Key Takeaways
|
||
|
||
1. **Asset universe**: BTC dominates volume; ETH, SOL, HYPE are the next most liquid. Use 3-6 assets for cross-sectional strategies.
|
||
2. **Return distributions**: Crypto returns are leptokurtic (fat tails) — expect black swans. Size positions accordingly.
|
||
3. **Correlations**: BTC/ETH correlation ~0.7. Most alts >0.5 correlated with BTC. True diversification is hard.
|
||
4. **Regime frequency**: NORMAL dominates but HIGH_VOL regime provides the best trading opportunities.
|
||
5. **Fee hurdle**: At Tier 0, a round-trip costs ~0.09%. This means a 1bps edge is enough for a single tick, but barely. We need 2-5bps edges minimum for consistent profitability. At higher tiers, the bar drops significantly.
|
||
6. **DuckDB data**: Enables sub-second queries on tick-level data. Essential for Hurst/VPIN and microstructure strategies.
|
||
"""),
|
||
]
|
||
|
||
nb_path = NOTEBOOKS_DIR / "01_eda.ipynb"
|
||
nbf.write(nb, str(nb_path))
|
||
print(f"Created {nb_path}")
|
||
|
||
|
||
def create_strategy_research_notebook():
|
||
nb = nbf.v4.new_notebook()
|
||
nb.metadata = {
|
||
"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
|
||
"language_info": {"name": "python", "version": "3.13.0"},
|
||
}
|
||
|
||
nb.cells = [
|
||
nbf.v4.new_markdown_cell("""# FTDT Quant Lab — Strategy Research & Backtesting
|
||
|
||
**Goal:** Develop and validate systematic trading strategies targeting Sharpe > 1.5 on Hyperliquid assets.
|
||
|
||
**Framework:**
|
||
1. Signal Generation — compute alpha from market data
|
||
2. VectorBT Backtest — fast vectorized simulation with fee-accurate PnL
|
||
3. Walk-Forward Validation — IS/OOS parameter optimization
|
||
4. Statistical Significance — DSR, PSR, Sharpe Haircut, QuantVerdict
|
||
5. Deployment Decision — DEPLOY / SIMULATE / DISCARD
|
||
|
||
**Key Thresholds for Sharpe > 1.5:**
|
||
- Win rate > 55% with positive expectancy
|
||
- Max drawdown < 15%
|
||
- Walk-forward consistency > 60%
|
||
- DSR > 0.80, PSR > 0.70
|
||
- Average trade PnL > 2x fees"""),
|
||
|
||
nbf.v4.new_code_cell("""# Setup
|
||
import sys; sys.path.insert(0, str(Path.cwd().parent))
|
||
|
||
import numpy as np
|
||
import pandas as pd
|
||
import matplotlib.pyplot as plt
|
||
import seaborn as sns
|
||
from pathlib import Path
|
||
import json, time
|
||
|
||
from backtests.vbt_runner import VBTBacktestRunner
|
||
from backtests.vbt_validator import VBTValidator
|
||
from quant.significance import QuantVerdict, validate_strategy
|
||
from quant.walkforward import WalkForwardRunner, quick_validate
|
||
from quant.optimizer import ParamOptimizer
|
||
from framework.data import HyperliquidDataProvider
|
||
from config.fee_tiers import get_perp_fees, get_strategy_fee_model
|
||
|
||
sns.set_theme(style="darkgrid")
|
||
plt.rcParams["figure.figsize"] = (14, 6)
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 1. Strategy Inventory
|
||
|
||
Current strategies and their signal logic:"""),
|
||
|
||
nbf.v4.new_code_cell("""strategies = {
|
||
"pairs": {
|
||
"name": "Pairs Trading",
|
||
"type": "Stat Arb",
|
||
"signal": "BTC/ETH ratio Z-score",
|
||
"entry": "|Z| > 1.5σ",
|
||
"exit": "|Z| < 0.5σ",
|
||
"best_use": "Range-bound, mean-reverting markets",
|
||
"sharpe_target": 2.0,
|
||
},
|
||
"hurst_vpin": {
|
||
"name": "Hurst VPIN",
|
||
"type": "Directional",
|
||
"signal": "Hurst > 0.55 AND VPIN > 0.25",
|
||
"entry": "Both trending + high flow imbalance",
|
||
"exit": "Hurst < 0.45 or direction flip",
|
||
"best_use": "Trending, high-volume markets",
|
||
"sharpe_target": 2.5,
|
||
},
|
||
"cross_sectional": {
|
||
"name": "Cross-Sectional Momentum",
|
||
"type": "Multi-Asset Long/Short",
|
||
"signal": "Past N-bar return ranking",
|
||
"entry": "Long top-3, short bottom-3",
|
||
"exit": "Next rebalance period",
|
||
"best_use": "All regimes, best in TRENDING",
|
||
"sharpe_target": 1.8,
|
||
},
|
||
"spot_perp_basis": {
|
||
"name": "Spot-Perp Basis Arb",
|
||
"type": "Delta-Neutral Carry",
|
||
"signal": "Perp vs spot price gap > 3bps",
|
||
"entry": "Short premium leg, long discount leg",
|
||
"exit": "Basis convergence < 1bps",
|
||
"best_use": "FUNDING_EXTREME, volatile basis",
|
||
"sharpe_target": 2.0,
|
||
},
|
||
"regime_ensemble": {
|
||
"name": "Regime-Switching Ensemble",
|
||
"type": "Meta-Strategy",
|
||
"signal": "Regime × strategy affinity matrix",
|
||
"entry": "Weights strategies by regime fit",
|
||
"exit": "Regime change or signal fade",
|
||
"best_use": "All environments — adapts dynamically",
|
||
"sharpe_target": 2.0,
|
||
},
|
||
"grid_mm": {
|
||
"name": "Grid Market Making",
|
||
"type": "Market Making",
|
||
"signal": "Symmetric grid around mid",
|
||
"entry": "Grid fill triggers position",
|
||
"exit": "Grid exit on rebalance",
|
||
"best_use": "LOW_VOL, CHOPPY",
|
||
"sharpe_target": 2.0,
|
||
},
|
||
"as_mm": {
|
||
"name": "Avellaneda-Stoikov MM",
|
||
"type": "Market Making",
|
||
"signal": "Reservation price from inventory",
|
||
"entry": "Reservation > best bid (buy) / < best ask (sell)",
|
||
"exit": "Hold period or profit target",
|
||
"best_use": "LOW_VOL with tight spreads",
|
||
"sharpe_target": 1.5,
|
||
},
|
||
}
|
||
|
||
for key, s in strategies.items():
|
||
print(f"\\n{s['name']} ({key})")
|
||
print(f" Type: {s['type']}")
|
||
print(f" Signal: {s['signal']}")
|
||
print(f" Entry: {s['entry']}")
|
||
print(f" Exit: {s['exit']}")
|
||
print(f" Regime: {s['best_use']}")
|
||
print(f" Target Sharpe: {s['sharpe_target']}")
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 2. Backtest Harness
|
||
|
||
Run any strategy through the VBT backtest engine with fee-accurate PnL, then
|
||
validate with statistical significance tests."""),
|
||
|
||
nbf.v4.new_code_cell("""def run_and_validate(strategy, interval='1h', params=None):
|
||
'''Run a full backtest + statistical validation pipeline.'''
|
||
print(f"\\n{'='*60}")
|
||
print(f" {strategies.get(strategy, {}).get('name', strategy)} — {interval}")
|
||
print(f"{'='*60}")
|
||
|
||
runner = VBTBacktestRunner(vip_tier=0, staking_tier='none')
|
||
result = runner.run_strategy(
|
||
strategy=strategy, interval=interval, testnet=False, limit=500, params=params
|
||
)
|
||
|
||
if result is None:
|
||
print(f" No result (no trades or data error)")
|
||
return None
|
||
|
||
# Display key metrics
|
||
print(f" Sharpe: {result.get('sharpe', 0):.3f}")
|
||
print(f" Total Return: {result.get('total_return_pct', 0):.1f}%")
|
||
print(f" Max Drawdown: {result.get('max_drawdown_pct', 0):.1f}%")
|
||
print(f" Win Rate: {result.get('win_rate', 0)*100:.0f}%")
|
||
print(f" Profit Factor: {result.get('profit_factor', 0):.2f}")
|
||
print(f" Total Trades: {result.get('total_trades', 0)}")
|
||
print(f" PnL: ${result.get('pnl', 0):.2f}")
|
||
|
||
# Statistical validation
|
||
n_trades = max(result.get('total_trades', 1), 1)
|
||
verdict = validate_strategy(
|
||
sharpe=result.get('sharpe', 0),
|
||
n_trades=n_trades,
|
||
n_trials=10,
|
||
wf_consistency=0.7,
|
||
)
|
||
print(f"\\n Verdict: {verdict['verdict']}")
|
||
print(f" DSR (deflated): {verdict['deflated_sharpe']:.3f}")
|
||
print(f" PSR: {verdict['psr']:.3f}")
|
||
print(f" Haircut Sharpe: {verdict['haircut_sharpe']:.3f}")
|
||
print(f" Score: {verdict['score']}")
|
||
print(f" → {verdict['recommendation']}")
|
||
|
||
# Plot equity curve
|
||
eq = result.get('equity_curve')
|
||
if eq:
|
||
df_eq = pd.DataFrame(eq)
|
||
df_eq['t'] = pd.to_datetime(df_eq['t'])
|
||
df_eq.set_index('t', inplace=True)
|
||
df_eq['v'].plot()
|
||
plt.title(f"{strategies.get(strategy, {}).get('name', strategy)} — Equity Curve")
|
||
plt.ylabel('Equity ($)')
|
||
plt.show()
|
||
|
||
return result
|
||
|
||
# Quick sweep of top strategies
|
||
for s in ["pairs", "hurst_vpin", "grid_mm", "momentum", "mean_rev"]:
|
||
run_and_validate(s, "1h")
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 3. Cross-Sectional Momentum Backtest
|
||
|
||
The new multi-asset strategy. Long the top performers, short the laggards."""),
|
||
|
||
nbf.v4.new_code_cell("""from strategies.cross_sectional_momentum import CrossSectionalMomentum, HIGH_LIQUIDITY
|
||
from data.duckdb_provider import DuckDBProvider
|
||
|
||
duckdb = DuckDBProvider()
|
||
|
||
# Fetch multi-asset candles
|
||
coins = ["BTC", "ETH", "SOL", "HYPE", "ARB", "OP"]
|
||
prices = duckdb.fetch_multi_candles(coins, interval='1h', limit=500)
|
||
|
||
print(f"Coins with data: {list(prices.keys())}")
|
||
for coin in sorted(prices):
|
||
df = prices[coin]
|
||
print(f" {coin}: {len(df)} bars, close=${df['close'].iloc[-1]:.2f}")
|
||
|
||
# Compute cross-sectional momentum signals
|
||
cs_mom = CrossSectionalMomentum(lookback=20, top_n=2, bottom_n=2, risk_parity=True, vol_target=0.20)
|
||
close_prices = {c: df['close'] for c, df in prices.items()}
|
||
weights = cs_mom.compute_signals(close_prices)
|
||
|
||
print(f"\\nCross-Sectional Momentum Weights:")
|
||
for coin, wt in sorted(weights.items(), key=lambda x: abs(x[1]), reverse=True):
|
||
direction = "LONG" if wt > 0 else "SHORT"
|
||
print(f" {coin:6s}: {direction:5s} {wt:+.3f}")
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 4. Walk-Forward Parameter Optimization
|
||
|
||
For strategies that show promise, run walk-forward to find stable parameters
|
||
and validate OOS performance."""),
|
||
|
||
nbf.v4.new_code_cell("""from quant.optimizer import ParamOptimizer
|
||
|
||
# Grid MM parameter sweep
|
||
print("=== Grid Market Making — Parameter Optimization ===\\n")
|
||
|
||
opt = ParamOptimizer(strategy='grid_mm', interval='1h', coin='BTC', n_windows=3)
|
||
opt.add_param('grid_levels', [5, 10, 20])
|
||
opt.add_param('spacing_bps', [2, 5, 10])
|
||
opt.add_param('rebalance_every', [5, 10, 20])
|
||
|
||
optimizer = ParamOptimizer.__new__(ParamOptimizer)
|
||
# [MANUAL RUN REQUIRED — uses live HL API, uncomment to run]
|
||
# report = opt.run()
|
||
# report.print()
|
||
print(" Walk-forward optimizer ready. Uncomment `opt.run()` to execute (requires live HL API data).")
|
||
print(" Grid: 3 grid_levels × 3 spacing × 3 rebalance = 27 combinations × 3 windows = 81 backtests")
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 5. Pairs Trading Deep Dive
|
||
|
||
The only live-profitable strategy. Analyze its performance characteristics
|
||
and identify improvement opportunities."""),
|
||
|
||
nbf.v4.new_code_cell("""# Pairs trading: analyze BTC/ETH spread dynamics
|
||
btc = prices.get('BTC', {}).get('close')
|
||
eth = prices.get('ETH', {}).get('close')
|
||
|
||
if btc is not None and eth is not None and not btc.empty and not eth.empty:
|
||
common_idx = btc.index.intersection(eth.index)
|
||
btc = btc[common_idx]
|
||
eth = eth[common_idx]
|
||
|
||
ratio = btc / eth
|
||
mu = ratio.rolling(20).mean()
|
||
std = ratio.rolling(20).std()
|
||
z_score = (ratio - mu) / std
|
||
|
||
fig, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize=(16, 12), sharex=True)
|
||
|
||
ax1.plot(ratio.index, ratio, linewidth=0.5, color='black', label='BTC/ETH Ratio')
|
||
ax1.plot(mu.index, mu, linewidth=1, color='blue', label='20-bar MA')
|
||
ax1.fill_between(mu.index, mu - 2*std, mu + 2*std, alpha=0.15, color='blue', label='±2σ')
|
||
ax1.legend()
|
||
ax1.set_title('BTC/ETH Ratio with Bollinger Bands')
|
||
|
||
ax2.plot(z_score.index, z_score, linewidth=0.5, color='purple')
|
||
ax2.axhline(1.5, color='red', linestyle='--', alpha=0.5, label='Entry (1.5σ)')
|
||
ax2.axhline(-1.5, color='red', linestyle='--', alpha=0.5)
|
||
ax2.axhline(0.5, color='green', linestyle='--', alpha=0.3, label='Exit (0.5σ)')
|
||
ax2.axhline(-0.5, color='green', linestyle='--', alpha=0.3)
|
||
ax2.legend()
|
||
ax2.set_ylabel('Z-Score')
|
||
|
||
ax3.plot(z_score.index, abs(z_score), linewidth=0.5, color='orange')
|
||
ax3.axhline(1.5, color='red', linestyle='--', alpha=0.5)
|
||
ax3.set_ylabel('|Z|')
|
||
ax3.set_xlabel('Date')
|
||
|
||
plt.tight_layout()
|
||
plt.show()
|
||
|
||
# Signal statistics
|
||
entry_count = (abs(z_score) > 1.5).sum()
|
||
exit_count = ((abs(z_score.shift(1)) > 0.5) & (abs(z_score) < 0.5)).sum()
|
||
print(f"Entry signals (|Z| > 1.5): {entry_count}")
|
||
print(f"Exit signals (|Z| < 0.5): {exit_count}")
|
||
print(f"Signal density: {entry_count / len(z_score) * 100:.1f}%")
|
||
|
||
# Distribution of Z-scores
|
||
print(f"\\nZ-Score distribution:")
|
||
print(f" Mean: {z_score.mean():.3f}")
|
||
print(f" Std: {z_score.std():.3f}")
|
||
print(f" Pct > 2σ: {(abs(z_score) > 2).mean()*100:.1f}%")
|
||
print(f" Pct > 1.5σ: {(abs(z_score) > 1.5).mean()*100:.1f}%")
|
||
|
||
# Half-life of mean reversion
|
||
spread = ratio.dropna()
|
||
spread_lag = spread.shift(1).dropna()
|
||
spread_diff = spread - spread_lag
|
||
spread_diff = spread_diff.iloc[1:]
|
||
spread_lag = spread_lag.iloc[:len(spread_diff)]
|
||
if len(spread_lag) > 0:
|
||
import statsmodels.api as sm # may need install
|
||
try:
|
||
X = sm.add_constant(spread_lag.values)
|
||
model = sm.OLS(spread_diff.values, X).fit()
|
||
hl = -np.log(2) / model.params[1] if model.params[1] < 0 else float('inf')
|
||
print(f"\\nMean reversion half-life: {hl:.1f} bars ({hl * pd.Timedelta(hours=1).total_seconds()/3600:.1f} hours)")
|
||
except Exception:
|
||
print("\\n(Install statsmodels for half-life estimation: pip install statsmodels)")
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 6. Strategy Development Checklist
|
||
|
||
Before deploying any strategy to live/papers:
|
||
|
||
- [ ] VectorBT backtest on real data (not synthetic)
|
||
- [ ] At least 50 trades in the backtest
|
||
- [ ] Walk-forward consistency > 50%
|
||
- [ ] DSR > 0.80, PSR > 0.70
|
||
- [ ] Haircut Sharpe > 0.50
|
||
- [ ] Maximum drawdown < 15%
|
||
- [ ] Win rate > 50% OR profit factor > 1.5
|
||
- [ ] Average trade PnL > 2x fee cost
|
||
- [ ] Correlation < 0.7 with existing portfolio strategies
|
||
- [ ] Phase 3 queue simulation (queue-aware fills) for maker strategies
|
||
- [ ] Paper trading for at least 24h before live
|
||
|
||
**Only deploy strategies that pass all 11 checks.**"""),
|
||
]
|
||
|
||
nb_path = NOTEBOOKS_DIR / "02_strategy_research.ipynb"
|
||
nbf.write(nb, str(nb_path))
|
||
print(f"Created {nb_path}")
|
||
|
||
|
||
def create_portfolio_notebook():
|
||
nb = nbf.v4.new_notebook()
|
||
nb.metadata = {
|
||
"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
|
||
"language_info": {"name": "python", "version": "3.13.0"},
|
||
}
|
||
|
||
nb.cells = [
|
||
nbf.v4.new_markdown_cell("""# FTDT Quant Lab — Portfolio Construction & Risk Management
|
||
|
||
**Goal:** Combine multiple independent alpha sources into a single risk-managed portfolio targeting Sharpe > 1.5.
|
||
|
||
**Key concepts:**
|
||
1. **Diversification**: N independent strategies with low correlation → Sharpe scales ~√N
|
||
2. **Risk Parity**: Allocate capital inversely proportional to strategy volatility
|
||
3. **Volatility Targeting**: Scale total portfolio to target annualized vol (e.g., 20%)
|
||
4. **Correlation Penalty**: Reduce allocation to redundant (highly correlated) strategies
|
||
5. **Regime Adaptation**: Shift strategy weights based on market conditions
|
||
6. **Drawdown Control**: Kill switch at strategy and portfolio level
|
||
|
||
**Math:**
|
||
Portfolio Sharpe ≈ √N × avg(individual Sharpe) × √(1 - avg_correlation)
|
||
|
||
If we have 5 strategies with average individual Sharpe 2.0 and average correlation 0.2:
|
||
Portfolio Sharpe ≈ √5 × 2.0 × √(0.8) ≈ 4.0
|
||
|
||
This is the engine. 5 good strategies + low correlation → Sharpe >> 1.5."""),
|
||
|
||
nbf.v4.new_code_cell("""# Setup
|
||
import sys; sys.path.insert(0, str(Path.cwd().parent))
|
||
|
||
import numpy as np
|
||
import pandas as pd
|
||
import matplotlib.pyplot as plt
|
||
import seaborn as sns
|
||
|
||
from strategies.portfolio import PortfolioConstructor, StrategyAllocation
|
||
from strategies.regime_ensemble import RegimeDetector, RegimeEnsemble, STRATEGY_REGIME_AFFINITY
|
||
from config.fee_tiers import get_perp_fees, get_strategy_fee_model
|
||
|
||
sns.set_theme(style="darkgrid")
|
||
plt.rcParams["figure.figsize"] = (14, 6)
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 1. Strategy × Regime Affinity Matrix
|
||
|
||
The regime-switching ensemble selects strategies based on their known
|
||
performance characteristics in each market regime."""),
|
||
|
||
nbf.v4.new_code_cell("""affinity = STRATEGY_REGIME_AFFINITY
|
||
affinity_df = pd.DataFrame(affinity).T
|
||
|
||
fig, ax = plt.subplots(figsize=(14, 8))
|
||
sns.heatmap(affinity_df, annot=True, fmt='.1f', cmap='YlOrRd',
|
||
vmin=0, vmax=1, ax=ax, cbar_kws={'label': 'Affinity Score'})
|
||
ax.set_title('Strategy × Regime Affinity Matrix')
|
||
plt.tight_layout()
|
||
plt.show()
|
||
|
||
# Best strategy per regime
|
||
print("Best strategy for each regime:")
|
||
for regime in affinity_df.index:
|
||
best = affinity_df.loc[regime].idxmax()
|
||
score = affinity_df.loc[regime, best]
|
||
print(f" {regime:20s} → {best:20s} (score: {score:.1f})")
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 2. Portfolio Construction Simulation
|
||
|
||
Simulate the portfolio with 7 strategies, each running independently.
|
||
Use correlated returns to test the diversification benefits."""),
|
||
|
||
nbf.v4.new_code_cell("""# Simulated returns for 7 strategies with some correlation
|
||
np.random.seed(42)
|
||
n_bars = 1000
|
||
|
||
strategy_names = ["pairs", "hurst_vpin", "cross_sectional", "grid_mm",
|
||
"spot_perp_basis", "momentum", "mean_rev"]
|
||
|
||
# Generate correlated returns
|
||
base_returns = np.random.randn(n_bars, 3) * 0.005
|
||
|
||
returns = {}
|
||
returns["pairs"] = base_returns[:, 0] * 0.6 + np.random.randn(n_bars) * 0.003
|
||
returns["hurst_vpin"] = base_returns[:, 1] * 0.8 + np.random.randn(n_bars) * 0.004
|
||
returns["cross_sectional"] = base_returns[:, 0] * 0.3 + base_returns[:, 1] * 0.5 + np.random.randn(n_bars) * 0.003
|
||
returns["grid_mm"] = base_returns[:, 2] * 0.4 + np.random.randn(n_bars) * 0.002
|
||
returns["spot_perp_basis"] = np.random.randn(n_bars) * 0.003 # uncorrelated
|
||
returns["momentum"] = base_returns[:, 1] * 0.7 + np.random.randn(n_bars) * 0.004
|
||
returns["mean_rev"] = -base_returns[:, 0] * 0.5 + np.random.randn(n_bars) * 0.003
|
||
|
||
# Add positive drift for profitable strategies
|
||
for name, r in returns.items():
|
||
returns[name] = r + 0.0005 # Small positive edge
|
||
|
||
# Compute correlation
|
||
ret_df = pd.DataFrame(returns)
|
||
corr = ret_df.corr()
|
||
sns.heatmap(corr, annot=True, fmt='.2f', cmap='RdBu_r', center=0,
|
||
vmin=-1, vmax=1, square=True)
|
||
plt.title('Strategy Return Correlation Matrix')
|
||
plt.show()
|
||
"""),
|
||
|
||
nbf.v4.new_code_cell("""# Build and simulate portfolio
|
||
pf = PortfolioConstructor(
|
||
capital=100_000,
|
||
vol_target=0.20,
|
||
max_correlation=0.70,
|
||
max_drawdown_stop=0.15,
|
||
portfolio_mdd_stop=0.10,
|
||
)
|
||
|
||
for name in strategy_names:
|
||
pf.register_strategy(name)
|
||
|
||
# Feed returns
|
||
for i in range(n_bars):
|
||
for name in strategy_names:
|
||
pf.update_returns(name, [returns[name][i]])
|
||
pf.update_portfolio_value({
|
||
name: returns[name][i] * pf.capital * 0.1
|
||
for name in strategy_names
|
||
})
|
||
|
||
# Portfolio metrics
|
||
metrics = pf.summary()
|
||
print(f"=== Portfolio Metrics ===")
|
||
print(f"Total Equity: ${metrics.total_equity:,.2f}")
|
||
print(f"Total PnL: ${metrics.total_pnl:,.2f} ({metrics.total_pnl_pct*100:.1f}%)")
|
||
print(f"Volatility: {metrics.vol_20d*100:.1f}%")
|
||
print(f"Sharpe Ratio: {metrics.sharpe:.2f}")
|
||
print(f"Sortino Ratio: {metrics.sortino:.2f}")
|
||
print(f"Max Drawdown: {metrics.max_drawdown_pct*100:.1f}%")
|
||
print(f"Win Rate: {metrics.win_rate*100:.0f}%")
|
||
|
||
# Equity curve
|
||
eq = list(pf.portfolio_equity_history)
|
||
plt.plot(eq, linewidth=0.5)
|
||
plt.title('Portfolio Equity Curve')
|
||
plt.ylabel('Equity ($)')
|
||
plt.xlabel('Bar')
|
||
plt.show()
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 3. Risk Decomposition
|
||
|
||
Where is the risk coming from? Which strategies contribute most to drawdowns?"""),
|
||
|
||
nbf.v4.new_code_cell("""# Risk attribution per strategy
|
||
allocations = pf.compute_allocations({"BTC": 100000, "ETH": 3500, "SOL": 200, "HYPE": 10})
|
||
print("=== Portfolio Allocation ===")
|
||
print(f"{'Strategy':<20} {'Weight':>8} {'Allocation':>12} {'Vol 20d':>10}")
|
||
print("-" * 55)
|
||
for name in strategy_names:
|
||
alloc = allocations.get(name, 0)
|
||
st = pf.strategies.get(name)
|
||
if st:
|
||
print(f"{name:<20} {st.weight:>7.1%} ${alloc:>10,.0f} {st.vol_20d*100:>8.1f}%")
|
||
total_alloc = sum(allocations.values())
|
||
print(f"\\n{'Total':<20} {' ':>8} ${total_alloc:>10,.0f}")
|
||
print(f"Reserve: ${pf.capital - total_alloc:>10,.0f}")
|
||
|
||
# Drawdown per strategy
|
||
print(f"\\n=== Drawdown Analysis ===")
|
||
for name, st in pf.strategies.items():
|
||
if st.peak_equity > 0:
|
||
dd = (1.0 - st.equity / st.peak_equity) * 100
|
||
print(f" {name:<20s}: DD={dd:5.1f}% | Equity=${st.equity:,.0f} | Peak=${st.peak_equity:,.0f}")
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 4. Regime-Adaptive Allocation
|
||
|
||
Test the regime-switching ensemble: how do weights shift across regimes?"""),
|
||
|
||
nbf.v4.new_code_cell("""# Simulate different regimes
|
||
ensemble = RegimeEnsemble()
|
||
|
||
# Seed with some signals
|
||
for name in strategy_names:
|
||
ensemble.update_strategy_signal(name, "BUY", 0.6 + np.random.random() * 0.2)
|
||
|
||
# Test in different regimes by feeding artificial price patterns
|
||
np.random.seed(42)
|
||
|
||
print("=== Strategy Weights by Regime ===\\n")
|
||
|
||
# TRENDING: strong upward drift
|
||
for i in range(200):
|
||
ensemble.feed_price(100000 + i * 50 + np.random.randn() * 200)
|
||
trending_weights = ensemble.compute_weights()
|
||
print("TRENDING:")
|
||
for s, w in sorted(trending_weights.items(), key=lambda x: x[1], reverse=True)[:5]:
|
||
print(f" {s:20s}: {w:.1%}")
|
||
|
||
# Reset detector and test MEAN_REVERTING
|
||
ensemble.detector.prices.clear()
|
||
for i in range(200):
|
||
px = 100000 + np.sin(i * 0.1) * 2000 + np.random.randn() * 500
|
||
ensemble.feed_price(px)
|
||
mr_weights = ensemble.compute_weights()
|
||
print("\\nMEAN_REVERTING:")
|
||
for s, w in sorted(mr_weights.items(), key=lambda x: x[1], reverse=True)[:5]:
|
||
print(f" {s:20s}: {w:.1%}")
|
||
|
||
# Compare
|
||
print(f"\\n=== Weight Shift Analysis ===")
|
||
for name in sorted(strategy_names):
|
||
tw = trending_weights.get(name, 0)
|
||
mw = mr_weights.get(name, 0)
|
||
shift = mw - tw
|
||
direction = "▲ MR" if shift > 0.01 else ("▼ TREND" if shift < -0.01 else "— same")
|
||
print(f" {name:20s}: TR={tw:.2%} MR={mw:.2%} ({direction})")
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 5. Sharpe Decomposition
|
||
|
||
Target: Sharpe > 1.5. How many strategies do we need?
|
||
|
||
```
|
||
Portfolio Sharpe = √N × avg(individual Sharpe) × √(1 - avg_correlation)
|
||
= √N × Sᵢ × √(1 - ρ̄)
|
||
```
|
||
|
||
**Scenarios:**
|
||
| N strategies | Avg Sharpe | Avg Corr | Portfolio Sharpe | Target? |
|
||
|-------------|-----------|---------|-----------------|---------|
|
||
| 3 | 1.5 | 0.3 | 2.17 | ✅ |
|
||
| 5 | 1.0 | 0.2 | 2.00 | ✅ |
|
||
| 5 | 0.8 | 0.5 | 1.26 | ❌ |
|
||
| 7 | 1.0 | 0.3 | 2.21 | ✅ |
|
||
| 7 | 0.7 | 0.2 | 1.66 | ✅ |
|
||
|
||
**Conclusion:** With 5-7 strategies averaging 1.0 individual Sharpe and correlation below 0.3, we comfortably exceed Sharpe 1.5. The key is keeping correlation low — redundant strategies destroy the diversification benefit."""),
|
||
|
||
nbf.v4.new_code_cell("""def portfolio_sharpe(n_strategies, avg_sharpe, avg_correlation):
|
||
return np.sqrt(n_strategies) * avg_sharpe * np.sqrt(1 - avg_correlation)
|
||
|
||
# Parameter sweep
|
||
ns = range(2, 11)
|
||
sharpes = [0.5, 0.8, 1.0, 1.2, 1.5]
|
||
corrs = [0.1, 0.2, 0.3, 0.5]
|
||
|
||
print("=== Portfolio Sharpe Projections ===\\n")
|
||
print(f"{'N':>3} | ", end="")
|
||
for s in sharpes:
|
||
print(f"Sᵢ={s:.1f} ", end="")
|
||
print("| ρ̄=0.2")
|
||
|
||
for n in ns:
|
||
print(f"{n:3d} | ", end="")
|
||
for s in sharpes:
|
||
ps = portfolio_sharpe(n, s, 0.2)
|
||
marker = " ✅" if ps > 1.5 else " "
|
||
print(f"{ps:5.2f}{marker} ", end="")
|
||
print()
|
||
|
||
print(f"\\nTarget line: Sharpe > 1.50")
|
||
print(f"Bold numbers pass the target. Strategy: maximize N × Sᵢ × (1 - ρ̄)")
|
||
"""),
|
||
|
||
nbf.v4.new_markdown_cell("""## 6. Deployment Pipeline
|
||
|
||
The complete pipeline from idea → deployment:
|
||
|
||
```
|
||
IDEA → Signal Generation → VBT Backtest → Walk-Forward →
|
||
→ DSR/PSR/Haircut → QuantVerdict →
|
||
→ Paper Trading (24h+) → Queue Simulation →
|
||
→ LIVE (1/10 size, daily PnL stop)
|
||
```
|
||
|
||
**Operational rules:**
|
||
- Never deploy more than 2 new strategies simultaneously
|
||
- Each strategy starts at 1/10 target size for 1 week
|
||
- Daily PnL stop: halt strategy if -2% in one day
|
||
- Weekly review: check Sharpe, DD, win rate vs. backtest
|
||
- Monthly rebalancing: re-run walk-forward to update parameters
|
||
- Kill switch: any strategy -15% from peak → disabled
|
||
- Portfolio kill: total equity -10% from peak → all strategies paused"""),
|
||
]
|
||
|
||
nb_path = NOTEBOOKS_DIR / "03_portfolio.ipynb"
|
||
nbf.write(nb, str(nb_path))
|
||
print(f"Created {nb_path}")
|
||
|
||
|
||
if __name__ == "__main__":
|
||
create_eda_notebook()
|
||
create_strategy_research_notebook()
|
||
create_portfolio_notebook()
|
||
print(f"\\nAll notebooks created in {NOTEBOOKS_DIR}")
|