3 Commits

Author	SHA1	Message	Date
ramseshk	11182b47f8	Fix ML calibration: logistic regression + Platt/isotonic + realistic NWP errors ... Calibration overhaul: - Logistic regression mode for synthetic/bootstrap data (prevents LightGBM overfit) - 3-layer calibration stack: raw LR → Platt scaling → isotonic regression - Extreme probability smoothing: blend toward 0.5 when raw>0.95 or raw<0.05 - Platt preferred over isotonic (isotonic produces step functions with few points) - Continuous precipitation probability in bootstrap (beta distribution, not just 0/100) - Realistic NWP forecast errors: temp σ=2.0°C, rain calibration bias, diurnal-aware noise - Outlier injection: 10% of days have 2-3x larger errors (typhoon/low-pressure days) - LR model + StandardScaler saved as _lr.pkl alongside .lgb marker Results: - temp_gt_30c: AUC=0.987, Brier=0.049, predictions vary 20-85% per day - rain_gt_0mm: AUC=0.979, Brier=0.042, predictions vary 15-85% per day - temp_gt_35c: AUC=0.713 (realistic — extreme heat is hard to predict)	2026-08-11 10:50:05 +08:00
ramseshk	03f9ea2129	Add spatial features, typhoon model, ERA5 pipeline, portfolio Kelly ... Tier 2 enhancements: - SpatialWeatherClient: multi-station Open-Meteo fetcher for all HK locations Extracts urban heat island delta, coastal-inland gradients, wind convergence, precipitation spatial heterogeneity, composite instability index - TyphoonModel: data-driven signal probability for T1/T3/T8/T10 Climatological base rates + conditional transition probabilities Currently active T1 signal → 25% T3/24h, 10% T8/72h, 22% T8/120h ENSO modulation, active storm proximity boost, month-specific seasonality - ERA5 download/process pipeline via CDS API Downloads hourly reanalysis for HK region, processes to daily training format Output schema matches Open-Meteo for seamless feature compatibility - PortfolioKelly: correlation-aware simultaneous Kelly sizing Covariance matrix from historical outcome correlations Prevents over-betting on correlated rain/temp/wind markets Σ⁻¹ μ vector formulation, regularized inversion, independent fallback - MLPredictor updated: integrates spatial + typhoon + portfolio Kelly record_outcome feeds both calibration AND portfolio correlation matrix	2026-08-10 17:56:02 +08:00
ramseshk	7d7a67bd20	Add ML prediction pipeline — LightGBM, calibration fix, ensemble disagreement ... Tier 1 ML enhancements: - Feature engineering (37 features across 5 groups: thermal, dynamic, moisture, temporal, interaction) from NWP model output - 7 LightGBM probability models for rain/temp/wind thresholds - Temperature-scaled probabilities to prevent overconfidence on bootstrap data - MLPredictor: unified inference pipeline replacing heuristic sigmoids - Ensemble disagreement signals (composite spread → edge amplification) - Fixed calibration loop: update_calibration() now functional (EMA of errors) - record_outcome() wired for post-resolution feedback - Nautilus strategy updated: ML predictions take priority, heuristics as fallback - Historical backtest engine with Sharpe/ROI/max-DD simulation - Bootstrap training data generator from HK climate normals Run: python ml/train.py && python ml/backtest.py --edge 50	2026-08-10 17:50:07 +08:00