ramseshk
3b065775f5
Phase 1: Data Engineering
- Refactored notebooks into src/{scraper,features,models,optimization,bot}
- FBref scraper with proxy rotation + Playwright Cloudflare bypass
- Fantacalcio.it integrated scraper (authenticated API + HTML fallback)
- api-football RapidAPI client for supplementary xG/xA/injuries
- RAG news pipeline: Gazzetta, Sky Sport, Di Marzio → injury/suspension/tactical extraction
- 26/27 season config: teams, scoring rules, name mappings, news sources
Phase 2: SOTA ML Architecture
- GBM Ensemble (LightGBM + CatBoost + XGBoost) with stacked blending
- Bootstrap ensemble for uncertainty quantification
- SinhArcsinh distribution head (ported from original TF Probability)
- Card classifiers (yellow/red), penalty model, goal probability (Poisson)
- Temporal GNN for player interaction modeling (crosses→goals, passes→assists)
- Optuna hyperparameter tuning with time-series CV
Phase 3: Operations Research
- Auction solver: MILP knapsack with PuLP (budget + role constraints)
- Grid Auction (Asta a Griglia): Minimax game theory bidding strategy
- Weekly lineup optimizer: MCTS maximizing win probability vs opponent
- Modificatore Difesa integration + captain selection
- Transfer market analyzer: buy-low/sell-high via xG regression to mean
- Opponent behavior modeling from historical lineage patterns
Phase 4: Agentic Workflow
- Telegram bot: auto-briefing (Friday + Sunday morning)
- Tactical briefing generator with start/sit recommendations
- GitHub Actions CI/CD: scheduled pipeline (scrape → predict → notify)
Infrastructure:
- 31 pytest unit tests (features, models, scraper, optimization)
- requirements.txt (lightgbm, catboost, xgboost, optuna, pulp, playwright, langchain)
- Makefile with install/test/lint/scrape/train/bot targets
- Jupyter notebook: 26_27_strategy.ipynb demonstrating auction + matchday 1 mockup
- Completely rewritten README.md with architecture diagram
49 lines
1.6 KiB
Python
49 lines
1.6 KiB
Python
"""Tests for the scraping modules."""
|
|
|
|
import pytest
|
|
|
|
|
|
class TestProxyManager:
|
|
def test_initialization(self):
|
|
from src.scraper.proxy_manager import ProxyManager
|
|
|
|
pm = ProxyManager()
|
|
assert pm.pool_size == 0
|
|
assert not pm.has_proxies
|
|
|
|
def test_add_proxy(self):
|
|
from src.scraper.proxy_manager import ProxyManager
|
|
|
|
pm = ProxyManager()
|
|
pm.add_proxy("http://proxy1:8080")
|
|
pm.add_proxy("http://proxy2:8080")
|
|
assert pm.pool_size == 2
|
|
|
|
|
|
class TestBrowserFallback:
|
|
def test_cloudflare_detection(self):
|
|
from src.scraper.browser_fallback import is_cloudflare_blocked
|
|
|
|
assert is_cloudflare_blocked("Checking your browser... Cloudflare")
|
|
assert is_cloudflare_blocked("Just a moment...")
|
|
assert not is_cloudflare_blocked("<html><body>Normal page</body></html>")
|
|
|
|
|
|
class TestFantacalcioScraper:
|
|
def test_normalize_name(self):
|
|
from src.features.player_features import PlayerFeatureBuilder
|
|
|
|
assert PlayerFeatureBuilder.normalize_name("Victor Osimhen") == "osimhen"
|
|
assert PlayerFeatureBuilder.normalize_name("Khvicha Kvaratskhelia") == "kvaratskhelia"
|
|
assert PlayerFeatureBuilder.normalize_name("Çalhanoğlu") == "calhanoglu"
|
|
|
|
|
|
class TestFBrefScraper:
|
|
def test_constants(self):
|
|
from src.scraper.fbref_scraper import FBREF_BASE, STAT_CATEGORIES, KEEPER_CATEGORIES
|
|
|
|
assert "11" in FBREF_BASE # Serie A competition ID
|
|
assert "stats" in STAT_CATEGORIES
|
|
assert "keepers" in KEEPER_CATEGORIES
|
|
assert "keepersadv" in KEEPER_CATEGORIES
|