ramseshk
a7f811eb81
New data/ module with: - data/store.py: Parquet-based raw message storage with background writer thread. Messages partitioned by channel/coin/date. Thread-safe queue. Supports pyarrow Parquet with zstd compression. Includes read_range() helper for replay. - data/collectors/hyperliquid.py: HL WebSocket + REST collector - WebSocket: l2Book (full book reconstruction), trades, allMids (mark prices) - REST pollers: funding rates, predicted funding, open interest, liquidations - Per-coin OrderBook class with snapshot/update reconstruction - Sequence gap detection with per-coin re-snapshot on gap - Latency tracking (exchange transport, signal, order, roundtrip) - Periodic stats reporter (book stats + latency summary every 60s) - CLI entrypoint: python -m data.collectors.hyperliquid --coins BTC ETH - data/normalizer.py: Timestamp normalization (ms, s, ISO strings from HL/Binance/Bybit/OKX/Coinbase/Deribit) + SequenceTracker with gap detection - data/latency.py: Rolling-window latency metrics (p50/p90/p95/p99) for transport, signal computation, order submission, and roundtrip - Added pyarrow + aiohttp to requirements.txt
17 lines
635 B
Python
17 lines
635 B
Python
"""
|
|
Real-time and historical data system for Hyperliquid and cross-venue data.
|
|
|
|
Collectors: WebSocket streaming + REST polling for L2 books, trades,
|
|
funding rates, mark/index prices, open interest, liquidation events.
|
|
|
|
Store: Parquet-based raw message storage with background writer.
|
|
Normalizer: timestamp alignment, sequence gap detection.
|
|
Latency: exchange vs signal latency tracking.
|
|
"""
|
|
|
|
from data.store import RawMessageStore
|
|
from data.normalizer import normalize_timestamp, detect_sequence_gap
|
|
from data.latency import LatencyTracker
|
|
|
|
__all__ = ["RawMessageStore", "normalize_timestamp", "detect_sequence_gap", "LatencyTracker"]
|