 ramseshk
|
a7f811eb81
|
feat: Phase 1 — real-time & historical data system
New data/ module with:
- data/store.py: Parquet-based raw message storage with background writer
thread. Messages partitioned by channel/coin/date. Thread-safe queue.
Supports pyarrow Parquet with zstd compression. Includes read_range()
helper for replay.
- data/collectors/hyperliquid.py: HL WebSocket + REST collector
- WebSocket: l2Book (full book reconstruction), trades, allMids (mark prices)
- REST pollers: funding rates, predicted funding, open interest, liquidations
- Per-coin OrderBook class with snapshot/update reconstruction
- Sequence gap detection with per-coin re-snapshot on gap
- Latency tracking (exchange transport, signal, order, roundtrip)
- Periodic stats reporter (book stats + latency summary every 60s)
- CLI entrypoint: python -m data.collectors.hyperliquid --coins BTC ETH
- data/normalizer.py: Timestamp normalization (ms, s, ISO strings from
HL/Binance/Bybit/OKX/Coinbase/Deribit) + SequenceTracker with gap detection
- data/latency.py: Rolling-window latency metrics (p50/p90/p95/p99) for
transport, signal computation, order submission, and roundtrip
- Added pyarrow + aiohttp to requirements.txt
|
2026-08-07 14:28:21 +08:00 |
|