{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2094c7c7",
   "metadata": {},
   "source": [
    "# FTDT Quant Lab — Exploratory Data Analysis\n",
    "\n",
    "**Goal:** Understand Hyperliquid market microstructure, identify alpha sources, verify data quality.\n",
    "\n",
    "**Assets:** BTC, ETH, SOL, HYPE, ARB, OP, and others  \n",
    "**Data Sources:** DuckDB tick database, HL REST API, Parquet raw store  \n",
    "**Timeframe:** 1s tick → 1h candles → daily aggregation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eabc713b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup\n",
    "import sys\n",
    "from pathlib import Path\n",
    "sys.path.insert(0, str(Path.cwd().parent))\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from data.duckdb_provider import DuckDBProvider\n",
    "from framework.data import HyperliquidDataProvider\n",
    "\n",
    "sns.set_theme(style=\"darkgrid\")\n",
    "plt.rcParams[\"figure.figsize\"] = (14, 6)\n",
    "plt.rcParams[\"figure.dpi\"] = 100\n",
    "\n",
    "# Data providers\n",
    "duckdb = DuckDBProvider()\n",
    "hl_rest = HyperliquidDataProvider(testnet=False)\n",
    "\n",
    "print(f\"DuckDB available: {duckdb.available}\")\n",
    "print(f\"Data range: {duckdb.get_data_range()}\")\n",
    "print(f\"Available coins: {duckdb.get_available_coins()}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bed20cce",
   "metadata": {},
   "source": [
    "## 1. Universe Overview\n",
    "\n",
    "What assets are available and how much data do we have for each?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad451959",
   "metadata": {},
   "outputs": [],
   "source": [
    "from strategies.cross_sectional_momentum import HL_UNIVERSE, HIGH_LIQUIDITY\n",
    "\n",
    "print(f\"Full universe ({len(HL_UNIVERSE)} assets): {HL_UNIVERSE}\")\n",
    "print(f\"High liquidity ({len(HIGH_LIQUIDITY)}): {HIGH_LIQUIDITY}\")\n",
    "\n",
    "# Fetch candle data for each asset\n",
    "prices = duckdb.fetch_multi_candles(HIGH_LIQUIDITY, interval='1h', limit=500)\n",
    "\n",
    "print(f\"\\nData availability:\")\n",
    "for coin, df in prices.items():\n",
    "    if not df.empty:\n",
    "        print(f\"  {coin:6s}: {len(df):5d} bars | {df.index[0]} to {df.index[-1]} | close=${df['close'].iloc[-1]:.2f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2eef87d",
   "metadata": {},
   "source": [
    "## 2. Return Distributions\n",
    "\n",
    "Check return distributions for normality, skew, kurtosis, and tail behavior.\n",
    "This informs strategy design — mean reversion works on platykurtic distributions,\n",
    "momentum thrives on leptokurtic tails."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0c7e04e",
   "metadata": {},
   "outputs": [],
   "source": [
    "returns_data = {}\n",
    "for coin in HIGH_LIQUIDITY:\n",
    "    df = prices.get(coin)\n",
    "    if df is None or df.empty:\n",
    "        continue\n",
    "    rets = df['close'].pct_change().dropna()\n",
    "    returns_data[coin] = rets\n",
    "\n",
    "stats = []\n",
    "for coin, rets in returns_data.items():\n",
    "    stats.append({\n",
    "        'coin': coin,\n",
    "        'mean_annual': rets.mean() * 365 * 24,\n",
    "        'vol_annual': rets.std() * np.sqrt(365 * 24),\n",
    "        'sharpe': rets.mean() / rets.std() * np.sqrt(365 * 24) if rets.std() > 0 else 0,\n",
    "        'skew': rets.skew(),\n",
    "        'kurtosis': rets.kurtosis(),\n",
    "        'var_95': rets.quantile(0.05),\n",
    "        'cv': rets.std() / rets.mean() if rets.mean() != 0 else 0,\n",
    "        'max_dd': (df['close'] / df['close'].cummax() - 1).min(),\n",
    "    })\n",
    "\n",
    "stats_df = pd.DataFrame(stats).set_index('coin')\n",
    "stats_df.round(4)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79e3e1a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return distribution plots\n",
    "fig, axes = plt.subplots(2, 3, figsize=(18, 10))\n",
    "for ax, (coin, rets) in zip(axes.flat, returns_data.items()):\n",
    "    rets.hist(bins=100, ax=ax, alpha=0.7, density=True)\n",
    "    ax.set_title(f\"{coin} — Skew: {rets.skew():.2f}, Kurt: {rets.kurtosis():.2f}\")\n",
    "    ax.axvline(0, color='red', linestyle='--', alpha=0.5)\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a68d3cb8",
   "metadata": {},
   "source": [
    "## 3. Correlation Matrix\n",
    "\n",
    "Identify redundant assets and diversification opportunities.\n",
    "High correlation = limited diversification benefit.\n",
    "Low correlation = potential for uncorrelated alpha streams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed22c624",
   "metadata": {},
   "outputs": [],
   "source": [
    "corr_matrix = pd.DataFrame(returns_data).corr()\n",
    "mask = np.triu(np.ones_like(corr_matrix), k=1)\n",
    "sns.heatmap(corr_matrix, mask=mask, annot=True, fmt='.3f', cmap='RdBu_r',\n",
    "            center=0, vmin=-1, vmax=1, square=True)\n",
    "plt.title('Hourly Return Correlation Matrix')\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cb1d28b",
   "metadata": {},
   "source": [
    "## 4. Volatility Regimes\n",
    "\n",
    "Classify the market into volatility regimes. This drives strategy selection\n",
    "in the Regime-Switching Ensemble.\n",
    "\n",
    "- LOW_VOL: annualized < 15% → market making, pairs trading\n",
    "- NORMAL: 15-60% → all strategies at baseline\n",
    "- HIGH_VOL: > 60% → momentum, Hurst/VPIN, tight risk controls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71535eb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from strategies.regime_ensemble import RegimeDetector\n",
    "\n",
    "detector = RegimeDetector(\n",
    "    high_vol_threshold=0.60,\n",
    "    low_vol_threshold=0.15,\n",
    "    funding_extreme_apr=0.30,\n",
    ")\n",
    "\n",
    "btc_prices = prices['BTC']['close']\n",
    "regimes = []\n",
    "for i, px in enumerate(btc_prices):\n",
    "    detector.feed_price(px)\n",
    "    if i >= 100:\n",
    "        regimes.append(detector.primary_regime())\n",
    "\n",
    "# Count regime distribution\n",
    "regime_counts = pd.Series(regimes).value_counts()\n",
    "print(\"Regime Distribution:\")\n",
    "for regime, count in regime_counts.items():\n",
    "    print(f\"  {regime:20s}: {count:5d} bars ({count/len(regimes)*100:.1f}%)\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d5645fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Regime timeline\n",
    "import matplotlib.dates as mdates\n",
    "\n",
    "fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(16, 8), sharex=True)\n",
    "\n",
    "ax1.plot(btc_prices.index[-len(regimes):], btc_prices.values[-len(regimes):],\n",
    "         linewidth=0.5, color='black')\n",
    "ax1.set_ylabel('BTC Price')\n",
    "ax1.set_title('BTC Price with Market Regimes')\n",
    "\n",
    "regime_colors = {\n",
    "    'NORMAL': 'gray', 'TRENDING': 'green', 'MEAN_REVERTING': 'blue',\n",
    "    'CHOPPY': 'orange', 'HIGH_VOL': 'red', 'LOW_VOL': 'lightblue',\n",
    "    'FUNDING_EXTREME': 'purple',\n",
    "}\n",
    "regime_numeric = pd.Series(\n",
    "    [{v: i for i, v in enumerate(regime_colors)}.get(r, 0) for r in regimes],\n",
    "    index=btc_prices.index[-len(regimes):]\n",
    ")\n",
    "ax2.scatter(regime_numeric.index, regime_numeric.values, c=[regime_colors.get(r, 'gray') for r in regimes],\n",
    "            s=1, alpha=0.6)\n",
    "ax2.set_yticks(range(len(regime_colors)))\n",
    "ax2.set_yticklabels(regime_colors.keys())\n",
    "ax2.set_ylabel('Regime')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa092524",
   "metadata": {},
   "source": [
    "## 5. Fee Impact Analysis\n",
    "\n",
    "Hyperliquid perp fee schedule. Calculate the minimum edge needed to overcome\n",
    "fees at each tier. This sets the floor for signal strength thresholds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "565628e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from config.fee_tiers import PERPS_TIERS, SPOT_TIERS, STAKING_TIERS, get_perp_fees, compute_trade_fees\n",
    "\n",
    "print(\"=== Perp Fee Tiers ===\")\n",
    "print(f\"{'Tier':<20} {'Volume':>12} {'Taker':>8} {'Maker':>8}\")\n",
    "print(\"-\" * 50)\n",
    "for tier, info in PERPS_TIERS.items():\n",
    "    print(f\"{info['name']:<20} ${info['min_volume']:>10,.0f} \"\n",
    "          f\"{info['taker']*100:.3f}% {info['maker']*100:.3f}%\")\n",
    "\n",
    "print(f\"\\n=== Spot Fee Tiers ===\")\n",
    "for tier, info in SPOT_TIERS.items():\n",
    "    print(f\"{info['name']:<20} ${info['min_volume']:>10,.0f} \"\n",
    "          f\"{info['taker']*100:.3f}% {info['maker']*100:.3f}%\")\n",
    "\n",
    "# Break-even trade size by fee tier\n",
    "print(f\"\\n=== Minimum Profitable Trade (BTC round-trip, 1bps edge) ===\")\n",
    "for tier in range(7):\n",
    "    fees = compute_trade_fees(\"BUY\", 0.001, 100000, 100000, vip_tier=tier)\n",
    "    print(f\"  Tier {tier}: {fees['effective_rate_pct']:.4f}% per side \"\n",
    "          f\"→ ${fees['total_fee']:.4f} round-trip\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49c93517",
   "metadata": {},
   "source": [
    "## 6. Key Takeaways\n",
    "\n",
    "1. **Asset universe**: BTC dominates volume; ETH, SOL, HYPE are the next most liquid. Use 3-6 assets for cross-sectional strategies.\n",
    "2. **Return distributions**: Crypto returns are leptokurtic (fat tails) — expect black swans. Size positions accordingly.\n",
    "3. **Correlations**: BTC/ETH correlation ~0.7. Most alts >0.5 correlated with BTC. True diversification is hard.\n",
    "4. **Regime frequency**: NORMAL dominates but HIGH_VOL regime provides the best trading opportunities.\n",
    "5. **Fee hurdle**: At Tier 0, a round-trip costs ~0.09%. This means a 1bps edge is enough for a single tick, but barely. We need 2-5bps edges minimum for consistent profitability. At higher tiers, the bar drops significantly.\n",
    "6. **DuckDB data**: Enables sub-second queries on tick-level data. Essential for Hurst/VPIN and microstructure strategies.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.13.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}