ramseshk
3b065775f5
Phase 1: Data Engineering
- Refactored notebooks into src/{scraper,features,models,optimization,bot}
- FBref scraper with proxy rotation + Playwright Cloudflare bypass
- Fantacalcio.it integrated scraper (authenticated API + HTML fallback)
- api-football RapidAPI client for supplementary xG/xA/injuries
- RAG news pipeline: Gazzetta, Sky Sport, Di Marzio → injury/suspension/tactical extraction
- 26/27 season config: teams, scoring rules, name mappings, news sources
Phase 2: SOTA ML Architecture
- GBM Ensemble (LightGBM + CatBoost + XGBoost) with stacked blending
- Bootstrap ensemble for uncertainty quantification
- SinhArcsinh distribution head (ported from original TF Probability)
- Card classifiers (yellow/red), penalty model, goal probability (Poisson)
- Temporal GNN for player interaction modeling (crosses→goals, passes→assists)
- Optuna hyperparameter tuning with time-series CV
Phase 3: Operations Research
- Auction solver: MILP knapsack with PuLP (budget + role constraints)
- Grid Auction (Asta a Griglia): Minimax game theory bidding strategy
- Weekly lineup optimizer: MCTS maximizing win probability vs opponent
- Modificatore Difesa integration + captain selection
- Transfer market analyzer: buy-low/sell-high via xG regression to mean
- Opponent behavior modeling from historical lineage patterns
Phase 4: Agentic Workflow
- Telegram bot: auto-briefing (Friday + Sunday morning)
- Tactical briefing generator with start/sit recommendations
- GitHub Actions CI/CD: scheduled pipeline (scrape → predict → notify)
Infrastructure:
- 31 pytest unit tests (features, models, scraper, optimization)
- requirements.txt (lightgbm, catboost, xgboost, optuna, pulp, playwright, langchain)
- Makefile with install/test/lint/scrape/train/bot targets
- Jupyter notebook: 26_27_strategy.ipynb demonstrating auction + matchday 1 mockup
- Completely rewritten README.md with architecture diagram
213 lines
9.0 KiB
Markdown
213 lines
9.0 KiB
Markdown
# Fantabeto 26/27
|
|
|
|
<div align="center">
|
|
|
|
**Fantacalcio Bayesian Estimated Team's Outcome**
|
|
|
|
*SOTA Machine Learning for Serie A Fantasy Football Dominance*
|
|
|
|
[](https://python.org)
|
|
[](LICENSE)
|
|
|
|
</div>
|
|
|
|
---
|
|
|
|
## Architecture Overview
|
|
|
|
Fantabeto 26/27 is a complete rewrite of the original 2023/24 codebase, upgraded with production-grade data engineering, state-of-the-art ML models, and advanced optimization algorithms. It predicts **Fantacalcio Modified Scores** (voto + bonus/malus), probabilistic card distributions, goal probabilities, and penalty chances — then generates optimal lineups via Monte Carlo Tree Search.
|
|
|
|
```
|
|
┌──────────────────────────────┐
|
|
│ Data Ingestion │
|
|
│ FBref │ Fantacalcio │ API │
|
|
└──────────────┬───────────────┘
|
|
│
|
|
┌──────────────▼───────────────┐
|
|
│ Feature Engineering │
|
|
│ Fatigue │ Pitch Tilt │ RAG │
|
|
│ Weather │ Name Matching │
|
|
└──────────────┬───────────────┘
|
|
│
|
|
┌─────────────────────────┼─────────────────────────┐
|
|
│ │ │
|
|
┌─────────▼──────────┐ ┌───────────▼───────────┐ ┌─────────▼──────────┐
|
|
│ GBM Ensemble │ │ Card Classifiers │ │ T-GNN │
|
|
│ LightGBM + CatBoost │ │ Yellow/Red/Penalty │ │ Player Interactions │
|
|
│ + XGBoost │ │ + Goal Probability │ │ │
|
|
└─────────┬──────────┘ └───────────┬───────────┘ └─────────┬──────────┘
|
|
│ │ │
|
|
└─────────────────────────┼─────────────────────────┘
|
|
│
|
|
┌──────────────▼───────────────┐
|
|
│ Optimization Engine │
|
|
│ Auction (MILP) │ Lineup (MCTS)│
|
|
│ Transfers │ Opponent Model │
|
|
└──────────────┬───────────────┘
|
|
│
|
|
┌──────────────▼───────────────┐
|
|
│ Telegram Bot + CI/CD │
|
|
│ Fri/Sun briefings │ Actions │
|
|
└──────────────────────────────┘
|
|
```
|
|
|
|
## Key Features
|
|
|
|
### Data Pipeline
|
|
- **FBref Scraper**: Rotating proxies + Playwright Cloudflare bypass
|
|
- **Fantacalcio.it Integration**: API → Excel votes/stats with HTML fallback
|
|
- **api-football**: Real-time xG, injuries, fixtures (via RapidAPI)
|
|
- **News RAG Pipeline**: Italian sports news (Gazzetta, Sky Sport, Di Marzio) → injuries, suspensions, tactical shifts via LLM extraction
|
|
|
|
### SOTA ML Architecture
|
|
- **GBM Ensemble** (LightGBM + CatBoost + XGBoost): Stacked blending with Ridge meta-learner, bootstrap uncertainty
|
|
- **Temporal GNN**: Player-to-player interaction modeling (winger crosses → striker goals)
|
|
- **Card Classifiers**: Yellow/red card probability with SMOTE class imbalance handling
|
|
- **Penalty Model**: Team-specific penalty taker heuristics
|
|
- **Goal Probability**: Poisson regression for goal count prediction
|
|
- **Optuna**: Hyperparameter tuning with time-series cross-validation
|
|
|
|
### Optimization Engine
|
|
- **Auction (MILP)**: Multi-period stochastic knapsack via PuLP/Gurobi
|
|
- Budget allocation per role (GK, DEF, MID, FWD)
|
|
- Grid Auction (Asta a Griglia) game theory
|
|
- **Weekly Lineup (MCTS)**: Win-probability maximization vs opponent projection
|
|
- Modificatore Difesa (defense modifier)
|
|
- Captain selection optimization
|
|
- Opposition weakness exploitation
|
|
- **Transfer Market (Svincolati)**: Buy-low/sell-high via xG divergence + regression to the mean
|
|
|
|
### Agentic Workflow
|
|
- **Sunday Morning Bot**: Telegram/Discord → auto briefing with start/sit recommendations
|
|
- **GitHub Actions**: Automated Friday + Sunday pipeline (scrape → predict → notify)
|
|
- **Tactical Briefing**: Narrated decision rationale (e.g., "Start X over Y — opponent left-back injured")
|
|
|
|
## Quick Start
|
|
|
|
### Prerequisites
|
|
- Python 3.11+
|
|
- Playwright: `playwright install chromium`
|
|
- [Optional] RapidAPI key for api-football
|
|
- [Optional] Fantacalcio.it account for vote API access
|
|
- [Optional] Telegram Bot Token for notifications
|
|
|
|
### Installation
|
|
|
|
```bash
|
|
git clone https://github.com/uPeppe/fantabeto.git
|
|
cd fantabeto
|
|
make install
|
|
```
|
|
|
|
### Environment Variables
|
|
|
|
Copy `.env.example` and fill in:
|
|
```bash
|
|
FANTACALCIO_TOKEN=your_fc_access_token
|
|
RAPIDAPI_KEY=your_rapidapi_key
|
|
TELEGRAM_BOT_TOKEN=your_telegram_bot_token
|
|
TELEGRAM_CHAT_ID=your_chat_id
|
|
OPENAI_API_KEY=your_openai_key # for LLM-enhanced news extraction
|
|
```
|
|
|
|
### Usage
|
|
|
|
```bash
|
|
# Full pipeline: scrape → features → train
|
|
make pipeline
|
|
|
|
# Generate matchday predictions
|
|
make predict
|
|
|
|
# Run tests
|
|
make test
|
|
|
|
# Send Telegram briefing
|
|
make bot
|
|
```
|
|
|
|
### Python API
|
|
|
|
```python
|
|
from src.pipeline import Pipeline
|
|
|
|
pipeline = Pipeline()
|
|
|
|
# Scrape historical seasons
|
|
pipeline.scrape_fbref(["2024-2025", "2025-2026"], current=True)
|
|
|
|
# Build feature dataset
|
|
dataset = pipeline.build_features()
|
|
|
|
# Train models
|
|
model = pipeline.train_models(
|
|
X=dataset.drop(columns=["fantavote"]),
|
|
y=dataset["fantavote"],
|
|
)
|
|
|
|
# Run lineup optimization
|
|
result = pipeline.optimize_lineup("data/predictions.xlsx", "my_squad.xlsx")
|
|
```
|
|
|
|
## Package Structure
|
|
|
|
```
|
|
src/
|
|
├── scraper/
|
|
│ ├── fbref_scraper.py # FBref.com Serie A scraper
|
|
│ ├── fantacalcio_scraper.py # Fantacalcio.it votes/rosters
|
|
│ ├── api_football.py # api-football RapidAPI client
|
|
│ ├── proxy_manager.py # Rotating proxy pool
|
|
│ └── browser_fallback.py # Playwright Cloudflare bypass
|
|
├── features/
|
|
│ ├── vote_processor.py # Vote → unified database
|
|
│ ├── player_features.py # Player-level feature builder
|
|
│ ├── match_features.py # Per-match feature matrix
|
|
│ ├── advanced_metrics.py # Fatigue, Tilt, Weather
|
|
│ └── news_rag.py # News ingestion + entity extraction
|
|
├── models/
|
|
│ ├── gbm_model.py # LightGBM/CatBoost/XGBoost ensemble
|
|
│ ├── tgcn_model.py # Temporal GNN interactions
|
|
│ ├── distribution_head.py # SinhArcsinh + Bernoulli
|
|
│ ├── card_model.py # Cards + Penalties + Goals
|
|
│ └── train.py # Optuna tuning + pipeline
|
|
├── optimization/
|
|
│ ├── auction_solver.py # MILP auction strategy
|
|
│ ├── lineup_solver.py # MCTS lineup selection
|
|
│ ├── transfer_analyzer.py # Buy-low/Sell-high analysis
|
|
│ └── opponent_model.py # Opponent behavior modeling
|
|
├── bot/
|
|
│ ├── telegram_bot.py # Telegram bot client
|
|
│ └── briefing.py # Tactical briefing generator
|
|
└── pipeline.py # Full pipeline orchestrator
|
|
```
|
|
|
|
## 2026/27 Season Configuration
|
|
|
|
Key season data in `config/`:
|
|
- `26_27_teams.yaml` — Teams, promoted/relegated, API season IDs
|
|
- `fantasy_scoring.yaml` — FVM scoring rules, Modificatore, role quotas
|
|
- `news_sources.yaml` — RSS feeds for Italian sports news
|
|
- `name_fix.yaml` — FBref ↔ Fantacalcio name mappings
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
# Full test suite
|
|
make test
|
|
|
|
# With coverage
|
|
pytest tests/ --cov=src --cov-report=term
|
|
```
|
|
|
|
## Credits
|
|
|
|
- [Fantacalcio.it](https://www.fantacalcio.it) — The game and vote data
|
|
- [FBref.com](https://fbref.com) — Comprehensive football statistics
|
|
- [parth1902/Scrape-FBref-data](https://github.com/parth1902/Scrape-FBref-data) — Original scraping inspiration
|
|
- [amiles2233/ff_prob](https://github.com/amiles2233/ff_prob) — Bayesian NN inspiration
|
|
|
|
## License
|
|
|
|
MIT — See [LICENSE](LICENSE)
|