Major refactor: Fantabeto 26/27 — modular package, GBM ensemble, MILP/MCTS optimization
Phase 1: Data Engineering
- Refactored notebooks into src/{scraper,features,models,optimization,bot}
- FBref scraper with proxy rotation + Playwright Cloudflare bypass
- Fantacalcio.it integrated scraper (authenticated API + HTML fallback)
- api-football RapidAPI client for supplementary xG/xA/injuries
- RAG news pipeline: Gazzetta, Sky Sport, Di Marzio → injury/suspension/tactical extraction
- 26/27 season config: teams, scoring rules, name mappings, news sources
Phase 2: SOTA ML Architecture
- GBM Ensemble (LightGBM + CatBoost + XGBoost) with stacked blending
- Bootstrap ensemble for uncertainty quantification
- SinhArcsinh distribution head (ported from original TF Probability)
- Card classifiers (yellow/red), penalty model, goal probability (Poisson)
- Temporal GNN for player interaction modeling (crosses→goals, passes→assists)
- Optuna hyperparameter tuning with time-series CV
Phase 3: Operations Research
- Auction solver: MILP knapsack with PuLP (budget + role constraints)
- Grid Auction (Asta a Griglia): Minimax game theory bidding strategy
- Weekly lineup optimizer: MCTS maximizing win probability vs opponent
- Modificatore Difesa integration + captain selection
- Transfer market analyzer: buy-low/sell-high via xG regression to mean
- Opponent behavior modeling from historical lineage patterns
Phase 4: Agentic Workflow
- Telegram bot: auto-briefing (Friday + Sunday morning)
- Tactical briefing generator with start/sit recommendations
- GitHub Actions CI/CD: scheduled pipeline (scrape → predict → notify)
Infrastructure:
- 31 pytest unit tests (features, models, scraper, optimization)
- requirements.txt (lightgbm, catboost, xgboost, optuna, pulp, playwright, langchain)
- Makefile with install/test/lint/scrape/train/bot targets
- Jupyter notebook: 26_27_strategy.ipynb demonstrating auction + matchday 1 mockup
- Completely rewritten README.md with architecture diagram
This commit is contained in:
ramseshk
@@ -1,80 +1,212 @@
|
||||
<font size = "+3"><b>fantabeto</b></size>
|
||||
# Fantabeto 26/27
|
||||
|
||||
Fantacalcio Bayesian Estimated Team's Outcome
|
||||
<div align="center">
|
||||
|
||||
<i>Machine learning model for predicting Serie A players performance in a match, in terms of Fantacalcio (italian fantasy football) scores.</i>
|
||||
**Fantacalcio Bayesian Estimated Team's Outcome**
|
||||
|
||||
https://pub.towardsai.net/how-i-won-at-italian-fantasy-football-fantacalcio-using-machine-learning-ce8fc3fdcaef
|
||||
*SOTA Machine Learning for Serie A Fantasy Football Dominance*
|
||||
|
||||
The aim of this project is to predict Fantacalcio (Serie A fantasy football) player performances (vote and fantavote, respectively their match rating and that summed to the bonus/malus given by goals, assists and cards), using players and teams data from http://fantacalcio.it and http://fbref.com.
|
||||
[](https://python.org)
|
||||
[](LICENSE)
|
||||
|
||||
"outputs" folder contains predictions for the next Serie A matchdays, and an excel file for analysis. A list of players can be inserted to help selecting an optimal line-up.
|
||||
</div>
|
||||
|
||||
The code has been adapted for <b>Serie-A season 2023/2024</b>, updating the statistics database and taking into account the stats from other leagues, for the players who are at their first season in Serie A (see rookies stats folder).
|
||||
---
|
||||
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
Two neural network models are trained for predicting vote and fantavote for outfield players and goalkeepers, for which the clean sheet probability is also an output.
|
||||
The outputs of these models are not raw predictions, but probability distributions, in the form of SinhArcsinh, which is a skewed distribution, meaning that the probability density is asymmetric.
|
||||
For predicting clean sheet probability, a Bernoulli distribution is instead used (a sample of which would be clean sheet = 1, with a given probability p, or clean sheet = 0 with probability 1-p)
|
||||
|
||||
See the following code and plot to show an example of vote and fantavote probability distributions.
|
||||
In this case, the player would be an attacking one, whose fantavote distribution is very skewed to the right (it is very more probable to score a goal and receive a 10 = 7+3 fantavote, than having an awful performance with a 4 fantavote!).
|
||||
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
def sinh_archsinh_pdf(x, mu, sigma, eps, delta):
|
||||
mul = 2 / np.sinh( np.arcsinh(2) * delta)
|
||||
z = (x - mu) / (sigma*mul)
|
||||
S = np.sinh( -eps + (1/delta) * np.arcsinh(z))
|
||||
return np.exp(-0.5 * S * S) * np.sqrt(1 + S * S) / ( sigma * mul * delta ) / np.sqrt(1 + z * z) / np.sqrt(2 * np.pi)
|
||||
|
||||
x = np.arange(start = 0, stop = 30, step = 0.001)
|
||||
pxv = sinh_archsinh_pdf(x, 6.06, 0.62, 0.33, 1.06)
|
||||
pxf = sinh_archsinh_pdf(x, 5.6657, 1.224146, 0.795868, 1.9983)
|
||||
plt.plot(x, pxv, label = 'vote', color = 'b')
|
||||
plt.plot(x, pxf, label = 'fantavote', color = 'g')
|
||||
plt.fill_between(x, pxv, color = 'lightblue')
|
||||
plt.fill_between(x, pxf, color = 'lightgreen')
|
||||
mv = np.average(x, weights = pxv)
|
||||
mf = np.average(x, weights = pxf)
|
||||
plt.vlines(x = mv, color = 'b', ymin = 0, ymax = 3, linestyle = 'dashed', label = 'mean vote = ' + '{:.2f}'.format(mv))
|
||||
plt.vlines(x = mf, color = 'g', ymin = 0, ymax = 3, linestyle = 'dashed', label = 'mean fantavote = ' + '{:.2f}'.format(mf))
|
||||
plt.legend()
|
||||
plt.xlim([0, 20])
|
||||
plt.ylim([0, 1])
|
||||
plt.ylabel('Probability Density')
|
||||
plt.xlabel('Vote')
|
||||
plt.title('Beto (A) estimated performance')
|
||||
plt.show()
|
||||
Fantabeto 26/27 is a complete rewrite of the original 2023/24 codebase, upgraded with production-grade data engineering, state-of-the-art ML models, and advanced optimization algorithms. It predicts **Fantacalcio Modified Scores** (voto + bonus/malus), probabilistic card distributions, goal probabilities, and penalty chances — then generates optimal lineups via Monte Carlo Tree Search.
|
||||
|
||||
```
|
||||
┌──────────────────────────────┐
|
||||
│ Data Ingestion │
|
||||
│ FBref │ Fantacalcio │ API │
|
||||
└──────────────┬───────────────┘
|
||||
│
|
||||
┌──────────────▼───────────────┐
|
||||
│ Feature Engineering │
|
||||
│ Fatigue │ Pitch Tilt │ RAG │
|
||||
│ Weather │ Name Matching │
|
||||
└──────────────┬───────────────┘
|
||||
│
|
||||
┌─────────────────────────┼─────────────────────────┐
|
||||
│ │ │
|
||||
┌─────────▼──────────┐ ┌───────────▼───────────┐ ┌─────────▼──────────┐
|
||||
│ GBM Ensemble │ │ Card Classifiers │ │ T-GNN │
|
||||
│ LightGBM + CatBoost │ │ Yellow/Red/Penalty │ │ Player Interactions │
|
||||
│ + XGBoost │ │ + Goal Probability │ │ │
|
||||
└─────────┬──────────┘ └───────────┬───────────┘ └─────────┬──────────┘
|
||||
│ │ │
|
||||
└─────────────────────────┼─────────────────────────┘
|
||||
│
|
||||
┌──────────────▼───────────────┐
|
||||
│ Optimization Engine │
|
||||
│ Auction (MILP) │ Lineup (MCTS)│
|
||||
│ Transfers │ Opponent Model │
|
||||
└──────────────┬───────────────┘
|
||||
│
|
||||
┌──────────────▼───────────────┐
|
||||
│ Telegram Bot + CI/CD │
|
||||
│ Fri/Sun briefings │ Actions │
|
||||
└──────────────────────────────┘
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
|
||||
|
||||
|
||||
### Data Pipeline
|
||||
- **FBref Scraper**: Rotating proxies + Playwright Cloudflare bypass
|
||||
- **Fantacalcio.it Integration**: API → Excel votes/stats with HTML fallback
|
||||
- **api-football**: Real-time xG, injuries, fixtures (via RapidAPI)
|
||||
- **News RAG Pipeline**: Italian sports news (Gazzetta, Sky Sport, Di Marzio) → injuries, suspensions, tactical shifts via LLM extraction
|
||||
|
||||
### SOTA ML Architecture
|
||||
- **GBM Ensemble** (LightGBM + CatBoost + XGBoost): Stacked blending with Ridge meta-learner, bootstrap uncertainty
|
||||
- **Temporal GNN**: Player-to-player interaction modeling (winger crosses → striker goals)
|
||||
- **Card Classifiers**: Yellow/red card probability with SMOTE class imbalance handling
|
||||
- **Penalty Model**: Team-specific penalty taker heuristics
|
||||
- **Goal Probability**: Poisson regression for goal count prediction
|
||||
- **Optuna**: Hyperparameter tuning with time-series cross-validation
|
||||
|
||||
There is code also for computing the expected points outcome of a particular line-up, by sampling multiple times from the players' points distribution and considering bonus for Defense Modifier ("Modificatore") and Clean Sheet. This leads to estimating the team's points probability distribution.
|
||||
### Optimization Engine
|
||||
- **Auction (MILP)**: Multi-period stochastic knapsack via PuLP/Gurobi
|
||||
- Budget allocation per role (GK, DEF, MID, FWD)
|
||||
- Grid Auction (Asta a Griglia) game theory
|
||||
- **Weekly Lineup (MCTS)**: Win-probability maximization vs opponent projection
|
||||
- Modificatore Difesa (defense modifier)
|
||||
- Captain selection optimization
|
||||
- Opposition weakness exploitation
|
||||
- **Transfer Market (Svincolati)**: Buy-low/sell-high via xG divergence + regression to the mean
|
||||
|
||||
|
||||
### Agentic Workflow
|
||||
- **Sunday Morning Bot**: Telegram/Discord → auto briefing with start/sit recommendations
|
||||
- **GitHub Actions**: Automated Friday + Sunday pipeline (scrape → predict → notify)
|
||||
- **Tactical Briefing**: Narrated decision rationale (e.g., "Start X over Y — opponent left-back injured")
|
||||
|
||||
Credits:
|
||||
## Quick Start
|
||||
|
||||
http://Fantacalcio.it - The game! And of course, a lot of data, including votes, players list and probable line-ups.
|
||||
### Prerequisites
|
||||
- Python 3.11+
|
||||
- Playwright: `playwright install chromium`
|
||||
- [Optional] RapidAPI key for api-football
|
||||
- [Optional] Fantacalcio.it account for vote API access
|
||||
- [Optional] Telegram Bot Token for notifications
|
||||
|
||||
http://FBRef.com - Plenty of stats for football players and teams.
|
||||
### Installation
|
||||
|
||||
https://github.com/amiles2233/ff_prob - Inspiration, for using Tensorflow Probability and Bayesian Neural Networks for this task.
|
||||
```bash
|
||||
git clone https://github.com/uPeppe/fantabeto.git
|
||||
cd fantabeto
|
||||
make install
|
||||
```
|
||||
|
||||
https://github.com/parth1902/Scrape-FBref-data - FBref data scraping code.
|
||||
### Environment Variables
|
||||
|
||||
#fantacalcio #fantasy-football #serie-a
|
||||
Copy `.env.example` and fill in:
|
||||
```bash
|
||||
FANTACALCIO_TOKEN=your_fc_access_token
|
||||
RAPIDAPI_KEY=your_rapidapi_key
|
||||
TELEGRAM_BOT_TOKEN=your_telegram_bot_token
|
||||
TELEGRAM_CHAT_ID=your_chat_id
|
||||
OPENAI_API_KEY=your_openai_key # for LLM-enhanced news extraction
|
||||
```
|
||||
|
||||
#machine-learning #ai #neural-networks
|
||||
### Usage
|
||||
|
||||
#python #tensorflow #tensorflow-probability
|
||||
```bash
|
||||
# Full pipeline: scrape → features → train
|
||||
make pipeline
|
||||
|
||||
# Generate matchday predictions
|
||||
make predict
|
||||
|
||||
# Run tests
|
||||
make test
|
||||
|
||||
# Send Telegram briefing
|
||||
make bot
|
||||
```
|
||||
|
||||
### Python API
|
||||
|
||||
```python
|
||||
from src.pipeline import Pipeline
|
||||
|
||||
pipeline = Pipeline()
|
||||
|
||||
# Scrape historical seasons
|
||||
pipeline.scrape_fbref(["2024-2025", "2025-2026"], current=True)
|
||||
|
||||
# Build feature dataset
|
||||
dataset = pipeline.build_features()
|
||||
|
||||
# Train models
|
||||
model = pipeline.train_models(
|
||||
X=dataset.drop(columns=["fantavote"]),
|
||||
y=dataset["fantavote"],
|
||||
)
|
||||
|
||||
# Run lineup optimization
|
||||
result = pipeline.optimize_lineup("data/predictions.xlsx", "my_squad.xlsx")
|
||||
```
|
||||
|
||||
## Package Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── scraper/
|
||||
│ ├── fbref_scraper.py # FBref.com Serie A scraper
|
||||
│ ├── fantacalcio_scraper.py # Fantacalcio.it votes/rosters
|
||||
│ ├── api_football.py # api-football RapidAPI client
|
||||
│ ├── proxy_manager.py # Rotating proxy pool
|
||||
│ └── browser_fallback.py # Playwright Cloudflare bypass
|
||||
├── features/
|
||||
│ ├── vote_processor.py # Vote → unified database
|
||||
│ ├── player_features.py # Player-level feature builder
|
||||
│ ├── match_features.py # Per-match feature matrix
|
||||
│ ├── advanced_metrics.py # Fatigue, Tilt, Weather
|
||||
│ └── news_rag.py # News ingestion + entity extraction
|
||||
├── models/
|
||||
│ ├── gbm_model.py # LightGBM/CatBoost/XGBoost ensemble
|
||||
│ ├── tgcn_model.py # Temporal GNN interactions
|
||||
│ ├── distribution_head.py # SinhArcsinh + Bernoulli
|
||||
│ ├── card_model.py # Cards + Penalties + Goals
|
||||
│ └── train.py # Optuna tuning + pipeline
|
||||
├── optimization/
|
||||
│ ├── auction_solver.py # MILP auction strategy
|
||||
│ ├── lineup_solver.py # MCTS lineup selection
|
||||
│ ├── transfer_analyzer.py # Buy-low/Sell-high analysis
|
||||
│ └── opponent_model.py # Opponent behavior modeling
|
||||
├── bot/
|
||||
│ ├── telegram_bot.py # Telegram bot client
|
||||
│ └── briefing.py # Tactical briefing generator
|
||||
└── pipeline.py # Full pipeline orchestrator
|
||||
```
|
||||
|
||||
## 2026/27 Season Configuration
|
||||
|
||||
Key season data in `config/`:
|
||||
- `26_27_teams.yaml` — Teams, promoted/relegated, API season IDs
|
||||
- `fantasy_scoring.yaml` — FVM scoring rules, Modificatore, role quotas
|
||||
- `news_sources.yaml` — RSS feeds for Italian sports news
|
||||
- `name_fix.yaml` — FBref ↔ Fantacalcio name mappings
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Full test suite
|
||||
make test
|
||||
|
||||
# With coverage
|
||||
pytest tests/ --cov=src --cov-report=term
|
||||
```
|
||||
|
||||
## Credits
|
||||
|
||||
- [Fantacalcio.it](https://www.fantacalcio.it) — The game and vote data
|
||||
- [FBref.com](https://fbref.com) — Comprehensive football statistics
|
||||
- [parth1902/Scrape-FBref-data](https://github.com/parth1902/Scrape-FBref-data) — Original scraping inspiration
|
||||
- [amiles2233/ff_prob](https://github.com/amiles2233/ff_prob) — Bayesian NN inspiration
|
||||
|
||||
## License
|
||||
|
||||
MIT — See [LICENSE](LICENSE)
|
||||
|
||||
Reference in New Issue
Block a user