00bb06434ec3f2c352a50dfc17398f408b18caf7
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ramseshk
|
20ee340cef |
feat: VBT visualization + validation pipeline, HFT tick viz, DuckDB loader
Track 1 — VBT Candle-Frequency Pipeline: - backtests/vbt_validator.py: VBTValidator with 11 checks — timestamp monotonicity, duplicates, NaN, data gaps, lookahead bias, signal alignment, density, coincident entry/exit, min trade count, fee application, benchmark comparison. ValidationReport dataclass with errors/warnings/stats. Validates VBT results or raw signal arrays. - backtests/vbt_viz.py: VBTVisualizer with 10+ Plotly chart methods — equity curve with benchmark, drawdown, rolling Sharpe/Sortino/vol, trade markers, returns distribution with normal fit, monthly PnL heatmap, gross vs net, holding periods, parameter sensitivity heatmaps, dashboard compositor, HTML save (self-contained, CDN Plotly). All methods handle empty/null inputs. - backtests/vbt_report.py: Markdown + HTML report generator — structured sections for implementation summary, performance metrics, cost analysis, validation results, signal analysis, known limitations, next steps. batch_report() for mass report generation from results directory. - backtests/vbt_runner.py: Added run_benchmark() (buy-and-hold VBT portfolio), validate() (integrated VBTValidator), run_with_report() (fetch→validate→ backtest→visualize→save in one call). Track 2 — HFT Tick Pipeline: - backtests/tick_viz.py: 9-panel HFT dashboard — price+trade markers, spread dynamics, top-of-book depth, microprice vs mid, OBI/OFI panel, VPIN toxicity with thresholds, event timeline (PnL from tick_runner), markout curves at 6 horizons. Parquet→pandas→Plotly pipeline. Dark-themed HTML output for microstructure review. - data/duckdb_load.py: Parquet→DuckDB loader — creates l2_snapshots, trades, funding tables with schema. Pre-computed 1s rollup views for microprice, OFI, trade imbalance. Markout queries directly in SQL. Incremental loading with load_state tracking. CLI Integration: - cli.py: Added 'report' (full VBT report), 'validate' (check existing results), 'hft' (tick dashboard generation) commands. Fixed argparse help string escaping. 355 tests passing (34 new). |
||
|
ramseshk
|
3073415d33 |
feat: funding arb strategy, queue-aware paper fills, WQI live integration
- strategies/funding_arb_strategy.py: full backtestable funding rate carry module with entry/exit thresholds, position tracking, funding payment accounting, basis stop-loss, max-hold timeout. Includes backtest_funding_arb() and run_funding_discovery() for threshold optimization - live/node_v2.py: replaced naive random fills with QueueAwareFillModel (sim/fills.py) with queue-priority simulation; integrated WQI predictor and funding arb strategies; per-coin WQI signal generation every 3 ticks; funding arb metrics in dashboard - cli.py: added 'funding' command for funding rate distribution analysis and threshold backtesting - tests/test_funding_arb.py: 20 tests covering entry/exit logic, fee accounting, signal generation, backtesting, and node integration 321 tests passing (20 new). |
||
|
ramseshk
|
50d63e1ecc |
feat: HFT infrastructure — tick backtest runner, VPIN-gated A-S maker, WQI predictor, queue-aware fills
- backtests/tick_runner.py: TickBacktestRunner replays stored Parquet L2/trade events through sim/engine.py with queue position modeling, producing PnL breakdowns, equity curves, VPIN curves, and QuantVerdict significance reports - VPINGatedASMaker: VPIN-toxicity-gated A-S market maker with inventory skew and dynamic spread widening; blocks quoting when VPIN >= alarm threshold - sim/engine.py: Added SimConfig.from_fee_tier() factory — constructs sim config from Hyperliquid fee tier (VIP + staking) - sim/fills.py: Added QueueAwareFillModel — realistic queue-priority fill simulation replacing random fills in paper trading - strategies/wqi_predictor.py: WQI z-score directional strategy with adverse selection gating, timeout exit, stop-loss, and take-profit - cli.py: Added 'tick', 'markout' analysis, and 'discover' signal-discovery commands for end-to-end tick-level HFT research pipeline 301 tests passing (23 new). |
||
|
ramseshk
|
543537e33f |
feat: quant validation framework — DSR, PSR, Haircut, regimes, walk-forward
Three-module quant framework replacing 'sort by Sharpe' with proper statistical validation: quant/significance.py (15 tests): - deflated_sharpe_ratio(): adjusts for N trials (Harvey & Liu 2015) - probabilistic_sharpe_ratio(): P(True SR > benchmark) given T, skew, kurt - sharpe_haircut(): expected OOS Sharpe after selection bias deflation - QuantVerdict: DEPLOY / SIMULATE / DISCARD with 5-point scoring - validate_strategy(): one-shot validation function quant/regimes.py (8 tests): - classify_regime(): trending_up/down, ranging, volatile - RegimeClassifier: stateful rolling-window classifier - conditional_performance(): per-regime trade statistics quant/walkforward.py (5 tests): - WalkForwardRunner: sequential IS/OOS window optimization - WFWindow/WFReport: structured walk-forward results - consistency score, performance decay, concatenated OOS equity - significance_report() integration Walk-forward results (real HL data with date-sliced windows): grid_mm 1h: 2/4 pos, OOS S=-0.45, 74t, haircut=-22.66 → DISCARD momentum 4h: 2/4 pos, OOS S=-1.47, 116t, haircut=-45.35 → DISCARD composite_mm 1h: 2/4 pos, OOS S=+2.97, 6t, haircut=+43.25 → SIMULATE 28 tests total |
||
|
ramseshk
|
e0be9f4d40 |
feat: advanced microstructure — sequencer latency, dealer GEX, tick regime, triangular arb
4 new modules with 20 tests: #22 Sequencer Latency Detector (live/monitors/sequencer_latency.py): Detects stale-state windows between WebSocket and REST API. - WebSocket vs REST timestamp delta tracking - Transport latency percentiles (p50, p99) - Liquidation-triggered stale-state detection - Stale asset identification for cross-margin arbitrage #25 Dealer GEX (microstructure/dealer_gex.py): Dealer Gamma Exposure estimation via Black-Scholes. - Per-strike gamma × OI × spot² GEX computation - Pin level detection (strikes where dealers are long gamma) - Net GEX aggregation - Signal: fade_breakout (long gamma pinning) vs ride_momentum (short gamma amplification) - nearest_pin() for distance-to-magnet calculation #30 Tick-Size Regime Exploitation (microstructure/tick_regime.py): Detects when asset price approaches tick-size boundaries. - Hyperliquid tick schedule (BTC 0.1/0.5, ETH 0.01/0.05, SOL 0.001/0.005) - Boundary approach detection with configurable threshold - Linear trend estimation for expected bars-to-cross - Signal: widen_quotes or tighten_quotes with urgency classification - Per-coin state tracking #33 Triangular Latency Arb (live/monitors/triangular_arb.py): Cross-venue A→B→C triangular arbitrage detection. - Internal triangular: BTC-USDT → ETH-BTC → ETH-USDT - Cross-venue: price discrepancy across slow/fast venues - Latency gap detection between venue pairs - Implied cross-rate computation vs direct quote - Minimum spread threshold gating |
||
|
ramseshk
|
5304534e38 |
feat: advanced microstructure modules — HLP, Hawkes, Whipsaw, Term Structure, Liq Waterfall, Spoof Detector
6 new modules with 46 new tests (230 total): #21 HLP Vault Monitor (live/monitors/hlp_vault.py): Tracks Hyperliquid's native protocol market maker at address 0xfefefe... Queries clearinghouseState + metaAndAssetCtxs. - Delta exposure per asset (notional + PnL) - Overextension detection (notional exceeds M threshold) - Rebalancing signals: fade_short when HLP too short, fade_long when HLP too long (front-run forced rebalancing) - Toxicity score: HLP losing money = absorbing informed flow - Historical delta tracking #29 Hawkes Processes (microstructure/hawkes.py): Multivariate Hawkes calibrator for limit order book dynamics. - MLE calibration via SGD gradient descent on log-likelihood - Branching ratio enforcement (alpha/beta < 0.99 for stationarity) - Intensity computation λ_i(t) with cross-excitation - Activity forecasting (expected event count in horizon) - Synthetic event generator (Ogata thinning) - Pure functions: hawkes_intensity, hawkes_log_likelihood, generate_hawkes_events #23 Funding Whipsaw Trader (live/strategies/funding_whipsaw.py): Premium index decay trading in final 60s of funding epoch. - Detects deterministic convergence of premium→0 at settlement - Time-scaled position sizing (larger closer to settlement) - Auto-close after funding epoch completes - Confidence scoring based on premium magnitude #32 Term Structure Monitor (live/monitors/term_structure.py): Perp/quarterly/bi-quarterly futures basis curve trading. - Quarterly-perp basis with z-score anomaly detection - BiQ-quarterly curve steepness monitoring - Fair quarterly price via interest rate parity + funding carry - Calendar spread signals: buy_basis, sell_basis, curve_steepener, curve_flattener #24 Liquidation Waterfall (live/monitors/liq_waterfall.py): Cross-margin liquidation order prediction. - Margin ratio tracking (equity / maintenance margin) - Danger/critical level classification - Asset liquidation priority: maintenance / book_liquidity ratio (least liquid asset relative to margin = dumped first) - Strategy output: widen_spreads on target, tighten on rest #31 Spoof Detector (microstructure/spoof_detector.py): Adversarial ML-style spoofing pattern recognition. - Rule 1: Large order far from mid, cancelled immediately - Rule 2: Cancel right before trade approaches price level - Rule 3: Oversized order with no fill within short lifetime - Spoof probability (rolling window ratio) - Cancel-to-fill ratio monitoring |
||
|
ramseshk
|
e58c5951b7 |
feat: Phase 5 — integration layer (analytics pipeline, production node v2, CLI) + 12 tests
live/integrator.py (AnalyticsPipeline):
Real-time pipeline: data → microstructure → signals.
Accumulates book snapshots + trades, computes OBI, VPIN, microprice,
spread, depth, trade imbalance, HFT regime, and emits composite
signal with confidence and breakdown. Per-coin isolation.
live/node_v2.py (ProductionNode):
Rebuilt production node integrating ALL Phase 1-4 modules:
- REST data fetching (order book, mark prices, funding rates)
- AnalyticsPipeline per coin for real-time microstructure signals
- Treasury for position/capital/PnL/breaker management
- ToxicityFilter integration via HlMakerPool makers
- HlMakerPool for per-coin A-S quoting
- CrossVenueMonitor, FundingBasisMonitor, LiquidationRiskOverlay
- Paper trading with probabilistic fill simulation
- Dashboard metrics JSON output (equity, treasury, analytics, maker)
- Periodic status logging
cli.py (unified CLI):
Subcommands integrating all modules:
collect — Run Hyperliquid data collector to Parquet
analyze — Run microstructure analytics on stored data
simulate — Run market-making simulator on stored data
run — Start production trading node (paper or live)
backtest — Run VectorBT backtest
12 integration tests (all pass):
- AnalyticsPipeline: empty, book, trade, VPIN, emit, regime, isolation
- ProductionNode: creation, tick cycle (3 ticks), metrics JSON output
- CLI: import verification
Total test suite: 184 tests, all passing.
|
||
|
ramseshk
|
4f66ef36a9 |
feat: Phase 4 — controlled strategy deployment module + 38 tests
New live/ sub-modules for production-ready market making:
live/filters/toxicity.py (ToxicityFilter):
VPIN-based pre-trade filter. Accumulates buy/sell volume, computes
VPIN via microstructure module, produces quoting decision:
- allow_quoting: bool
- size_multiplier: 0.0–1.0 (graduated reduction approaching alarm)
- granular thresholds (threshold vs alarm) with smooth reduction
live/treasury.py (Treasury):
Central capital/risk management — single source of truth:
- Position tracking per coin (opening, closing, average entry)
- Realized + unrealized PnL computation
- Pre-trade constraint checks (inventory limits, fee estimates)
- Circuit breaker (drawdown, trade count, toxic fill rate, API errors)
- Liquidation distance monitoring
- Automatic cooldown reset after trip expiry
live/makers/hl_btc_eth.py:
HlMaker — per-coin market maker integrating:
- AvellanedaStoikovMaker (Phase 3) for optimal quotes
- ToxicityFilter for pre-trade gating
- Treasury for position/risk checks
HlMakerPool — manages multiple HlMaker instances with shared treasury
and coordinated observe_all()/quote_all()
live/monitors/cross_venue.py (CrossVenueMonitor):
Cross-exchange lead-lag detection via cross-correlation at multiple
lags. Spot premium (basis proxy) computation. Multi-venue summary.
live/monitors/funding_basis.py (FundingBasisMonitor):
Funding regime classification, momentum detection, carry PnL
estimation, basis spread analysis. Uses microstructure/funding.py.
live/monitors/liq_risk.py (LiquidationRiskOverlay):
Per-position liquidation distance monitoring with tiered warnings
(safe/warning/danger/critical). Recommended position reduction.
38 tests across 4 files (all pass):
test_live_filters.py (5)
test_live_maker.py (9)
test_live_monitors.py (11)
test_live_treasury.py (13)
Total test suite: 172 tests, all passing.
|
||
|
ramseshk
|
639dd4fb6d |
feat: Phase 3 — event-driven market-making simulator + 53 tests
New sim/ module — 7 files + init, replays stored L2/trade data
through a realistic market-making simulation:
sim/engine.py (SimulationEngine):
Event-driven core — processes L2 updates, trades, mark prices
sequentially. Orchestrates queue model, maker quotes, fill sim,
constraints, scenarios. Supports periodic re-quoting and
stale order cancellation.
sim/queue.py (QueueModel):
Price-time FIFO queue per price level. Tracks where maker orders
sit in queue. Simulates order eating by aggressor trades.
fill_probability() — Poisson thinning model for fill odds.
sim/maker.py:
AvellanedaStoikovMaker — stochastic control quoting with
aeta, k, tau parameters. Reservation price based on inventory.
quote() and quote_with_skew() with configurable inventory tilt.
GridMaker — evenly-spaced grid quoting at N levels.
sim/fills.py:
FillSimulator — partial fills, adverse selection probability,
cancel latency (gaussian RTT). FillEvent/CancelEvent tracking.
adverse_selection_intensity() — measures post-fill price moves.
sim/constraints.py:
InventoryConstraint — long/short/net/gross position limits.
FundingConstraint — hourly funding cost estimation.
FeeSchedule — maker/taker fee calculation.
LiquidationRisk — liquidation price and safety distance.
CircuitBreaker — PnL, trade count, toxic rate, slippage trips.
ConstraintManager — unified pre-trade constraint check.
sim/scenario.py:
ScenarioEngine — randomized exchange downtimes, latency spikes,
volatility bursts. State query per sim_time for spread/trade-rate.
sim/reporter.py:
PnLReporter — component-level PnL breakdown:
spread_capture, inventory_pnl, fees, funding, adverse_selection.
SimulationStats — trade counts, fill rates, drawdown, sharpe.
Equity curve tracking and max drawdown computation.
53 new tests across 4 files (all pass):
test_sim_queue.py (12) — order placement, FIFO, fills, cancels
test_sim_maker.py (9) — A-S quotes, inventory skew, grid maker
test_sim_constraints.py (14) — limits, funding, fees, liquidation, breakers
test_sim_reporter.py (12) — PnL components, equity curve, stats
test_sim_engine.py (6) — full engine integration
Total test suite: 134 tests, all passing.
|
||
|
ramseshk
|
fcfc136384 |
feat: Phase 2 — microstructure analytics + 81 tests
New microstructure/ module with pure-function analytics: microstructure/book.py: microprice() — depth-weighted mid price mid_price() — simple bid/ask midpoint order_book_imbalance() — ranged [-1, 1] volume skew depth_imbalance() — imbalance at fixed price distance spread_stats() — spread, spread_bps, mid, bid, ask depth_resiliency() — bid/ask volume within impact radius queue_depletion_prob() — Poisson fill probability at level batch_book_stats() — aggregate stats across snapshots microstructure/trades.py: classify_lee_ready() — Lee-Ready aggressor classification classify_bulk_lee_ready() — batch classification with mids/bids/asks compute_markouts() — forward mid-price change at configurable horizons markout_summary() — mean/std/t-stat per side per horizon trade_volume_profile() — size bucket distribution trade_arrival_rate() — rolling trades/sec with burst detection microstructure/toxicity.py: compute_vpin() — volume-synchronized informed trading probability compute_vpin_time_series() — rolling VPIN with alarm threshold fill_toxicity() — adverse price movement post-trade adverse_selection_ratio() — per-side adverse selection liquidation_clustering() — cluster detection in liquidation events microstructure/funding.py: funding_regime() — classify regime (neutral/positive/negative/high) funding_predictability() — AR(1) autocorrelation analysis funding_carry_pnl() — cumulative carry PnL estimation basis_spread() — perp premium over spot (bps) basis_convergence_speed() — mean-reversion half-life via AR(1) microstructure/signals.py: composite_signal() — weighted OBI + trade + VPIN + funding signal SignalPipeline — stateful pipeline accumulating book/trade updates detect_hft_regime() — regime classifier for HFT strategy selection Bug fixes in Phase 1: - data/latency.py: proper linear-interpolation percentiles - data/normalizer.py: UTC timezone for naive datetimes - data/normalizer.py: detect_sequence_gap returns gap-1 (missing count) - microstructure/toxicity.py: consistent vpin_value key in compute_vpin 81 tests across 4 test files (store, normalizer, latency, microstructure) |
||
|
ramseshk
|
50f8f4f970 |
Refactor: review, fix, and test entire codebase
Live node: - Fix null-handling for open_ords and get_fills requests - Cap equity_history, strategy_equity at 600-1000 entries (memory leak fix) - Dynamic strategy count in startup log - Loop error recovery: catch exceptions, backoff 5s, continue Dashboard server: - Fix backtest detail API: check HISTORICAL_DIR first - This was causing all historical detail views to show zeros Tests (5 suites, all passing): 1. Signal generation: Mean Reversion VWAP + Momentum + Pairs + OBI 2. Backtest: SPX mean reversion on 500-point series 3. Hurst/VPIN: 15 signals from 280 dollar bars 4. Memory guard: RSS monitoring, GC thresholds 5. Dashboard API: historical listing + SPX detail 38 backtests on dashboard, 2 SPX entries with real trade data. |