Runbook — What To Do
Quick reference for fixing common issues. Also available via Telegram: /help
CRITICALNo OddsAPI odds for a match
Symptoms: Telegram: "No OddsAPI odds for X vs Y — skipped". Match shows no_data in simulation.
Fix: Check match coverage, then restart to force re-discovery and fresh odds fetch.
Telegram:
/check then /restartWebsite: /monitoring → Recheck
CRITICALNo Betfair odds / session stale
Symptoms: All evaluations show no_data. Betfair session file older than 12h.
Fix: Restart forces fresh certlogin through EC2 proxy.
Telegram:
/restartWARNINGNo upcoming matches
Symptoms: Monitoring shows "0 upcoming matches". No evaluations scheduled.
Fix: Restart forces populate_bf_matches on first heartbeat. If international break — this is normal.
Telegram:
/restartCRITICALWrong bet placed
Symptoms: Bet on wrong match or wrong side due to translation error.
Fix: Cancel the bet on the emergency page. Refunds stake to bankroll.
Website: /emergency → Cancel Bet
CRITICALOddsAPI credits draining fast
Symptoms: System alert about credit burn rate. Credits dropping >5K/hour.
Fix: Kill rogue OddsAPI processes (fetch_historical, fetch_multisport etc.)
Telegram:
/killrogueWARNINGSimulation not evaluating matches
Symptoms: No bets for several days. No "SIM BET" notifications on Telegram.
Fix: Check coverage first, then restart. May be international break (normal).
Telegram:
/check then /restartWebsite: /monitoring → check Bet Activity
CRITICALsimulation-runner crashed
Symptoms: Telegram commands stop responding. System alert: "simulation-runner waiting restart".
Fix: Full restart of all services.
Telegram:
/fixallWARNINGMatch result not settling
Symptoms: Bet stuck as "pending" for 24h+. OddsAPI scores returns 0 for that league.
Fix: Manually settle with the correct score.
INFOMatch postponed
Symptoms: Bet stuck as "pending" but match was postponed/cancelled.
Fix: Void the bet to refund stake.
CRITICALProxy down (BrightData or EC2)
Symptoms: Monitoring shows "BF Proxy" red. Betfair API calls failing.
Fix: For BrightData: check brightdata.com dashboard (balance, zone status). For EC2: reboot via AWS Console or Telegram.
Telegram:
/reboot-proxy (EC2 only)Website: /monitoring → What to do section
INFOBetting should be paused
Symptoms: Model issues, wrong predictions, want to stop temporarily.
Fix: Pause betting. Signals still evaluated — you get notifications showing what would have been bet.
Telegram:
/pause (then /resume when ready)Website: /emergency → Pause button
CRITICALEverything broken
Symptoms: Multiple alerts, nothing working, can't figure out what's wrong.
Fix: Nuclear option — restarts Docker + all PM2 services.
Telegram:
/fixallWebsite: /emergency → Fix All button
CRITICALPM2 processes completely gone
Symptoms: Telegram doesn't respond. Dashboard down. SSH shows empty pm2 list.
Fix: Requires SSH access: pm2 delete all && pm2 start ecosystem.config.js && pm2 save
INFOOA→BF translation needed
Symptoms: Match shows as uncovered on /translate/oddsapi. "Awaiting reconcile" tag. Live cron auto-matches >99% of events; manual review only for the residue.
Fix: Open dashboard and click "Run reconcile now" — it auto-links if a BF candidate scores ≥0.85. Otherwise review and approve manually. Or send /coverage_check on Telegram to see whether anything is actually drifting.
Telegram:
/coverage_checkWebsite: /translate/oddsapi
INFOCoverage health check
Symptoms: You want to confirm OA→BF translation is at 100% (matches kicking off in next 7 days all have a BF mapping) and the daily reconcile cron is fresh.
Fix: Three equivalent ways: send /coverage_check on Telegram (priad_sos_bot), click "🩺 Health check" on the dashboard, or run packages/odds-binning/coverage_health_check.py on rocket. All three return the same metrics + issue list.
Telegram:
/coverage_checkWebsite: /translate/oddsapi → 🩺 Health check