00bb06434e
fix: collector WebSocket trade parsing + DuckDB 1.5 API compatibility
...
data/collectors/hyperliquid.py:
- Fixed _handle_trades to handle HL WebSocket trade data as list[dict]
(each trade dict has its own 'coin' field) instead of assuming single dict
- Fixed _poll_funding and _poll_open_interest to guard against
data[0] being list vs dict (mainnet vs testnet response format difference)
data/duckdb_load.py:
- Replaced deprecated duckdb.from_sequence() with executemany() for
l2_snapshots, trades, and funding table inserts (DuckDB 1.5.x API)
Tick runner verified on real mainnet BTC data:
- 324 events (45 L2 + 279 trades), VPIN-gated A-S MM, 1 trade, 98 cancels
- HFT dashboard generated with 8 panels from real tick data
2026-08-11 14:45:48 +08:00
20ee340cef
feat: VBT visualization + validation pipeline, HFT tick viz, DuckDB loader
...
Track 1 — VBT Candle-Frequency Pipeline:
- backtests/vbt_validator.py: VBTValidator with 11 checks — timestamp monotonicity,
duplicates, NaN, data gaps, lookahead bias, signal alignment, density,
coincident entry/exit, min trade count, fee application, benchmark comparison.
ValidationReport dataclass with errors/warnings/stats. Validates VBT results
or raw signal arrays.
- backtests/vbt_viz.py: VBTVisualizer with 10+ Plotly chart methods — equity
curve with benchmark, drawdown, rolling Sharpe/Sortino/vol, trade markers,
returns distribution with normal fit, monthly PnL heatmap, gross vs net,
holding periods, parameter sensitivity heatmaps, dashboard compositor,
HTML save (self-contained, CDN Plotly). All methods handle empty/null inputs.
- backtests/vbt_report.py: Markdown + HTML report generator — structured
sections for implementation summary, performance metrics, cost analysis,
validation results, signal analysis, known limitations, next steps.
batch_report() for mass report generation from results directory.
- backtests/vbt_runner.py: Added run_benchmark() (buy-and-hold VBT portfolio),
validate() (integrated VBTValidator), run_with_report() (fetch→validate→
backtest→visualize→save in one call).
Track 2 — HFT Tick Pipeline:
- backtests/tick_viz.py: 9-panel HFT dashboard — price+trade markers,
spread dynamics, top-of-book depth, microprice vs mid, OBI/OFI panel,
VPIN toxicity with thresholds, event timeline (PnL from tick_runner),
markout curves at 6 horizons. Parquet→pandas→Plotly pipeline.
Dark-themed HTML output for microstructure review.
- data/duckdb_load.py: Parquet→DuckDB loader — creates l2_snapshots,
trades, funding tables with schema. Pre-computed 1s rollup views for
microprice, OFI, trade imbalance. Markout queries directly in SQL.
Incremental loading with load_state tracking.
CLI Integration:
- cli.py: Added 'report' (full VBT report), 'validate' (check existing
results), 'hft' (tick dashboard generation) commands. Fixed argparse
help string escaping.
355 tests passing (34 new).
2026-08-11 12:22:11 +08:00
fcfc136384
feat: Phase 2 — microstructure analytics + 81 tests
...
New microstructure/ module with pure-function analytics:
microstructure/book.py:
microprice() — depth-weighted mid price
mid_price() — simple bid/ask midpoint
order_book_imbalance() — ranged [-1, 1] volume skew
depth_imbalance() — imbalance at fixed price distance
spread_stats() — spread, spread_bps, mid, bid, ask
depth_resiliency() — bid/ask volume within impact radius
queue_depletion_prob() — Poisson fill probability at level
batch_book_stats() — aggregate stats across snapshots
microstructure/trades.py:
classify_lee_ready() — Lee-Ready aggressor classification
classify_bulk_lee_ready() — batch classification with mids/bids/asks
compute_markouts() — forward mid-price change at configurable horizons
markout_summary() — mean/std/t-stat per side per horizon
trade_volume_profile() — size bucket distribution
trade_arrival_rate() — rolling trades/sec with burst detection
microstructure/toxicity.py:
compute_vpin() — volume-synchronized informed trading probability
compute_vpin_time_series() — rolling VPIN with alarm threshold
fill_toxicity() — adverse price movement post-trade
adverse_selection_ratio() — per-side adverse selection
liquidation_clustering() — cluster detection in liquidation events
microstructure/funding.py:
funding_regime() — classify regime (neutral/positive/negative/high)
funding_predictability() — AR(1) autocorrelation analysis
funding_carry_pnl() — cumulative carry PnL estimation
basis_spread() — perp premium over spot (bps)
basis_convergence_speed() — mean-reversion half-life via AR(1)
microstructure/signals.py:
composite_signal() — weighted OBI + trade + VPIN + funding signal
SignalPipeline — stateful pipeline accumulating book/trade updates
detect_hft_regime() — regime classifier for HFT strategy selection
Bug fixes in Phase 1:
- data/latency.py: proper linear-interpolation percentiles
- data/normalizer.py: UTC timezone for naive datetimes
- data/normalizer.py: detect_sequence_gap returns gap-1 (missing count)
- microstructure/toxicity.py: consistent vpin_value key in compute_vpin
81 tests across 4 test files (store, normalizer, latency, microstructure)
2026-08-07 14:34:18 +08:00
a7f811eb81
feat: Phase 1 — real-time & historical data system
...
New data/ module with:
- data/store.py: Parquet-based raw message storage with background writer
thread. Messages partitioned by channel/coin/date. Thread-safe queue.
Supports pyarrow Parquet with zstd compression. Includes read_range()
helper for replay.
- data/collectors/hyperliquid.py: HL WebSocket + REST collector
- WebSocket: l2Book (full book reconstruction), trades, allMids (mark prices)
- REST pollers: funding rates, predicted funding, open interest, liquidations
- Per-coin OrderBook class with snapshot/update reconstruction
- Sequence gap detection with per-coin re-snapshot on gap
- Latency tracking (exchange transport, signal, order, roundtrip)
- Periodic stats reporter (book stats + latency summary every 60s)
- CLI entrypoint: python -m data.collectors.hyperliquid --coins BTC ETH
- data/normalizer.py: Timestamp normalization (ms, s, ISO strings from
HL/Binance/Bybit/OKX/Coinbase/Deribit) + SequenceTracker with gap detection
- data/latency.py: Rolling-window latency metrics (p50/p90/p95/p99) for
transport, signal computation, order submission, and roundtrip
- Added pyarrow + aiohttp to requirements.txt
2026-08-07 14:28:21 +08:00
ramseshk