rams/ftdt-quant-lab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
145 Commits 1 Branch 0 Tags
d96730183473ba01c2ea11ecdaeee86535b79df2
Commit Graph

10 Commits

Author SHA1 Message Date
ramseshk 20ee340cef feat: VBT visualization + validation pipeline, HFT tick viz, DuckDB loader 
Track 1 — VBT Candle-Frequency Pipeline:
- backtests/vbt_validator.py: VBTValidator with 11 checks — timestamp monotonicity,
  duplicates, NaN, data gaps, lookahead bias, signal alignment, density,
  coincident entry/exit, min trade count, fee application, benchmark comparison.
  ValidationReport dataclass with errors/warnings/stats. Validates VBT results
  or raw signal arrays.
- backtests/vbt_viz.py: VBTVisualizer with 10+ Plotly chart methods — equity
  curve with benchmark, drawdown, rolling Sharpe/Sortino/vol, trade markers,
  returns distribution with normal fit, monthly PnL heatmap, gross vs net,
  holding periods, parameter sensitivity heatmaps, dashboard compositor,
  HTML save (self-contained, CDN Plotly). All methods handle empty/null inputs.
- backtests/vbt_report.py: Markdown + HTML report generator — structured
  sections for implementation summary, performance metrics, cost analysis,
  validation results, signal analysis, known limitations, next steps.
  batch_report() for mass report generation from results directory.
- backtests/vbt_runner.py: Added run_benchmark() (buy-and-hold VBT portfolio),
  validate() (integrated VBTValidator), run_with_report() (fetch→validate→
  backtest→visualize→save in one call).

Track 2 — HFT Tick Pipeline:
- backtests/tick_viz.py: 9-panel HFT dashboard — price+trade markers,
  spread dynamics, top-of-book depth, microprice vs mid, OBI/OFI panel,
  VPIN toxicity with thresholds, event timeline (PnL from tick_runner),
  markout curves at 6 horizons. Parquet→pandas→Plotly pipeline.
  Dark-themed HTML output for microstructure review.
- data/duckdb_load.py: Parquet→DuckDB loader — creates l2_snapshots,
  trades, funding tables with schema. Pre-computed 1s rollup views for
  microprice, OFI, trade imbalance. Markout queries directly in SQL.
  Incremental loading with load_state tracking.

CLI Integration:
- cli.py: Added 'report' (full VBT report), 'validate' (check existing
  results), 'hft' (tick dashboard generation) commands. Fixed argparse
  help string escaping.

355 tests passing (34 new).
 2026-08-11 12:22:11 +08:00
ramseshk 7517163142 feat: configurable grid params + auto walk-forward optimizer 
Track 1 — Grid param sweep in vbt_runner:
  - _generate_signals accepts params dict: grid_levels, spacing_bps, rebalance_every
  - run_strategy passes params through to signals
  - _strategy_params reflects actual runtime params
  - Grid param sweep results: spacing is critical, levels don't matter
    Tight spacing (1-2bps) = 1 trade, positive EV
    Wide spacing (20bps+) = many trades, negative EV
    Candle simulation can't model grid MM fills accurately

Track 6 — quant/optimizer.py:
  - ParanOptimizer: automated IS/OOS parameter walk-forward
  - add_param() to define parameter grid
  - Composite score: Sharpe × sqrt(trades) for robustness
  - IS optimization per window, OOS testing per window
  - WFParamWindow + OptimizerReport with consistency + stable params

Grid MM walk-forward results (3 windows):
  W0: IS S=-3.75 → OOS S=+2.23 (+12.7%, 1t)
  W1: IS S=+2.52 → OOS S=-3.49 (-19.0%, 11t)
  W2: IS S=-3.30 → OOS S=0.00 (0t)
  Consistency: 33.3%, Stable params: {levels=5, spacing=1bps, rebalance=5}

  Verdict: Candle-based grid MM is fundamentally unreliable.
  Real fills require queue simulation with L2 data.
 2026-08-10 16:47:09 +08:00
ramseshk 543537e33f feat: quant validation framework — DSR, PSR, Haircut, regimes, walk-forward 
Three-module quant framework replacing 'sort by Sharpe' with proper
statistical validation:

quant/significance.py (15 tests):
  - deflated_sharpe_ratio(): adjusts for N trials (Harvey & Liu 2015)
  - probabilistic_sharpe_ratio(): P(True SR > benchmark) given T, skew, kurt
  - sharpe_haircut(): expected OOS Sharpe after selection bias deflation
  - QuantVerdict: DEPLOY / SIMULATE / DISCARD with 5-point scoring
  - validate_strategy(): one-shot validation function

quant/regimes.py (8 tests):
  - classify_regime(): trending_up/down, ranging, volatile
  - RegimeClassifier: stateful rolling-window classifier
  - conditional_performance(): per-regime trade statistics

quant/walkforward.py (5 tests):
  - WalkForwardRunner: sequential IS/OOS window optimization
  - WFWindow/WFReport: structured walk-forward results
  - consistency score, performance decay, concatenated OOS equity
  - significance_report() integration

Walk-forward results (real HL data with date-sliced windows):
  grid_mm 1h:    2/4 pos, OOS S=-0.45,  74t, haircut=-22.66 → DISCARD
  momentum 4h:   2/4 pos, OOS S=-1.47, 116t, haircut=-45.35 → DISCARD
  composite_mm 1h: 2/4 pos, OOS S=+2.97, 6t, haircut=+43.25 → SIMULATE

28 tests total
 2026-08-10 16:20:56 +08:00
ramseshk 745174f0e6 fix: iceberg strategy — bool dtype + consecutive spike detection 
Root cause: iceberg signal generator produced object-dtype entries
that crashed VectorBT's numba JIT compiler with 'non-precise type
array(pyobject, 1d, C)'. All 28 sweep combos returned 0 trades.

Fixes:
- iceberg: lower vol spike threshold (1.3→1.15), require 2/3
  consecutive same-direction spikes (not just single bar)
- exit when spike subsides (not arbitrary 5-bar hold)
- .astype(bool) on all entries/exits before returning from
  _generate_signals, preventing numba JIT errors

Results (36/36 succeeded):
  iceberg 1d 2000b BTC  S=0.18  85t  ret=9.38%  (best)
  iceberg 15m  100b BTC  S=-36.34 4t  ret=-0.18% (worst)
  Consistently negative Sharpe except 1d interval —
  volume-spike following loses on sub-daily timescales
 2026-08-07 16:27:06 +08:00
ramseshk 0162e83138 feat: Hyperliquid fee schedule — all tiers, staking, maker rebates 
config/fee_tiers.py — complete rewrite:
  - 7 perps fee tiers (T0-T6) matching HL docs:
    T0: 0.045/0.015% → T6: 0.024/0.000%
  - 7 spot fee tiers (T0-T6):
    T0: 0.070/0.040% → T6: 0.025/0.000%
  - 7 staking tiers (none → diamond):
    multiplier 1.00 → 0.60 (40% discount)
  - 3 maker rebate tiers (>0.5%, >1.5%, >3% maker ratio)
    extra -0.001% to -0.003% on positive maker rates
  - compute_trade_fees() — per-trade fee breakdown
  - fee_tier_from_volume(), staking_tier_from_hype()
  - STRATEGY_FEE_MODELS: maker/taker classification per strategy

backtests/vbt_runner.py:
  - Accept vip_tier, staking_tier, maker_rebate_tier at init
  - Auto-detect fee model per strategy (maker vs taker)
  - compute_trade_fees() for per-trade fee calculation
  - Include fee_info in result JSON

dashboard/server.py:
  - /api/vbt/run accepts fee_tier/aking_tier/maker_rebate params
  - Trade normalization uses proper HL fee schedule per strategy
  - /api/vbt/result/{filename}/recalc — recalc trades with new tiers
  - /api/vbt/fee_tiers — get full fee schedule as JSON

dashboard/static/vbt.html:
  - Fee tier selector (T0-T6) + staking tier selector
  - Auto-recalculate on tier change when a result is selected
  - Fee rate shown in trade log header (e.g. 0.045%)
 2026-08-07 15:44:43 +08:00
ramseshk 9d817ac2fa feat: VBT trade log — show asset, entry/exit prices, Hyperliquid fees 
Backend (vbt_runner.py):
  - Add asset (BTC/ETH) to each trade record
  - Compute per-trade fee using HL taker rate (0.05%)
    entry_fee = size * entry_px * fee_rate
    exit_fee = size * exit_px * fee_rate
  - Add pnl_gross (before fees) and pnl_net (after fees)
  - Add fee_rate field for transparency

Server (server.py):
  - Normalize old backtest trades: add missing asset, fee,
    pnl_net, pnl_gross fields
  - Holyliquid default fee rate: 0.05% taker

Frontend (vbt.html):
  - Trade log table now shows:
    Time | Side + Asset | Size | Entry | Exit | Fee | PnL (net) | Duration
  - Asset shown as inline badge in Side column
  - Fee column with explicit USD amount
  - PnL now explicitly labeled 'net' (after fees)
  - Fallback to old 'pnl' field for legacy backtest files
 2026-08-07 15:35:33 +08:00
ramseshk 887a33f278 feat: trade log table, strategy params panel, B+W color scheme 
Dashboard:
- Trade log table: all trades with time, side, size, entry/exit price, PnL, duration
  in scrollable panel below charts
- Strategy params panel: displays all coefficients (z_entry, gamma, obi_entry,
  grid_levels, etc.) for the selected strategy
- Color scheme: professional black/white
  • positive: #03A9F4 (light blue)
  • negative: #FF5252 (red)
  • neutral: #777 (gray)
  • backgrounds: #0a0a0a / #111 / #181818
  • borders: #222 / #333

VBT runner:
- _extract_metrics now captures trades from pf.trades.records_readable
  (Avg Entry Price, Avg Exit Price, PnL, Return, Duration, Direction)
- _strategy_params() returns key coefficients per strategy type
- _empty_result includes empty trades/params

New vbt_server.py: minimal standalone dashboard (no live trading machinery,
no memory guard, no broadcast loop) — avoids crashing issues
 2026-08-07 12:53:52 +08:00
ramseshk 3606e7f92e feat: proper Grid MM, Composite MM, Hurst/VPIN, Iceberg, A-S strategies 
New strategies (strategies/nt/):
- GridMMNT: symmetric limit order grid around mid-price, captures spread from
  oscillation. Simulates fills from candle high/low. Rebuilds grid every 20 bars.
- CompositeMMNT: weighted ensemble of OBI (30%) + A-S inventory skew (40%) +
  Hurst/VPIN (30%). Votes: +1 long, -1 short, 0 neutral. Entry |score| > 0.5.
- IcebergNT: volume spike detection for whale accumulation. Dual-mode:
  candle proxy (volume > avg*2.5, >= 3 consecutive same-direction) and
  L2 wall detection (single level > avg*3). Exit on stop-loss/time/spike-fade.

Fixed strategies:
- Hurst/VPIN VBT: added proper VPIN proxy from candle volume (buy_vol when close
  > open, sell_vol when close < open). 50-bar rolling VPIN window. Signal:
  H>0.55 AND VPIN>0.25 AND |direction|>0.05. Exit: H<0.45 or direction flips.
- Hurst/VPIN paper trader: added HurstVPINLive integration (was missing entirely)
- A-S VBT: replaced placeholder spread filter with proper A-S simulation using
  reservation price formula (mid - q*gamma*sigma^2*tau), inventory tracking
- A-S NT formula: fixed to standard: mid - q*gamma*sigma^2*tau (was scaled by
  notional and gamma_scale improperly)
- Iceberg VBT: new volume spike detection replacing the old trend proxy

Registry: all 7 strategies now ✅ (pairs, hurst_vpin, as_mm, obi, grid_mm,
         composite_mm, iceberg)

VBT backtest results (500 BTC 1h bars):
  pairs:       -2.81%  13 trades   38% win
  hurst_vpin:  -0.77%   1 trade    (VPIN now active, very selective)
  as_mm:       -16.38%  73 trades  29% win
  obi:         -7.19%  15 trades    7% win
  grid_mm:     -4.79%  22 trades  33% win
  iceberg:     0 trades (threshold strict for 1h BTC data)
 2026-08-07 11:42:32 +08:00
ramseshk 37da46a016 feat: proper Order Book Imbalance strategy for BTC-USD on HL 
strategies/nt/obi_nt.py:
- Dual-mode OBI: candle proxy (backtest) + real L2 orderbook (live)
- Volume-based imbalance: buy_vol / (buy_vol + sell_vol) over rolling window
- Entry when |imbalance| > 0.35, exit on reversion < 0.10
- Stop-loss 2%, take-profit 0.5%, cooldown 3 bars
- compute_signal(price, orderbook=None) for paper trader integration

backtests/vbt_runner.py:
- Replaced placeholder z-score with proper volume-based OBI
- Buy vol = volume where close > open, sell vol = volume where close < open
- Rolling window imbalance computation
- Parameter sweep support with 12 combos tested

Registered across: deploy.py, nt_runner.py, dashboard, strategies/nt/__init__

Verified:
- VectorBT OBI backtest: 15 trades, -7.2% on default (window=20)
- Param sweep best: w=30 t=0.35 → sharpe -0.82, 49% win, 23% DD
- Real L2 orderbook signal: BUY obi=0.880 (bids 88% of depth)
- NT backtest engine: 201 bars, 8 days, 236ms
 2026-08-07 10:50:43 +08:00
ramseshk f5ffe4baee feat: NautilusTrader + VectorBT unified framework for Hyperliquid 
Add complete framework for testing and deploying quant strategies:

Framework (framework/):
- HyperliquidInstrumentCatalog: loads perps as NT CryptoPerpetual
- HyperliquidDataProvider: real candle/orderbook/mark-price data
- HyperliquidExecutionProvider: live + PaperExecutionProvider: simulated
- BaseHlStrategy: shared NT strategy lifecycle with signal library
- StrategyConfig: YAML-based parameter management
- DeployOrchestrator: CLI for backtest -> paper -> live pipeline

Backtesting (backtests/):
- VBTBacktestRunner: VectorBT vectorized backtests on real HL candles
- NTBacktestRunner: NautilusTrader event-driven backtest engine

NT Strategy ports (strategies/nt/):
- PairsTradingNT: BTC/ETH ratio Z-score mean reversion
- HurstVPINNT: Hurst exponent regime + VPIN flow imbalance
- ASMarketMakingNT: Avellaneda-Stoikov stochastic control MM

E2E verified: real HL candles fetch, VectorBT backtest (Sharpe 5.2
on Hurst/VPIN), instrument catalog, deploy CLI --list, strategy signals.
Existing live/node.py and paper_trader.py unchanged.
 2026-08-06 17:23:49 +08:00