← Back to projects

Egoist

A 1v1 Rocket League bot trained from scratch with reinforcement learning.

Actively training
Python 3.11PyTorch (CUDA)RLGym v2RocketSimPPORLBot

Egoist is a 1v1 Rocket League bot trained from scratch with reinforcement learning: PPO over the RocketSim simulator through RLGym v2, running on a single desktop machine. This devlog documents the training, evaluation, and real-game deployment pipeline, along with every experiment and what went wrong along the way, unpolished.

There’s no public playable demo yet. The bot runs locally via RLBot.

champion_1099200556

1.1 billion steps: entropy comes down and the opponent pool is ready

Current status

Training steps1.1 billion and counting (run heading toward 1.5B in progress)
Current championchampion_1099200556
Latest result179 wins / 21 losses across 200 games against the previous champion
Policy entropy3.94 (theoretical max 4.50); started at 4.42
Training speed~12,000 steps/s with 20 workers, i7-12700H + 8 GB GPU
Level of playchases, shoots, and defends with judgment; still far from a good human player

Next in the queue, already built and tested: an opponent pool (25% of episodes against frozen past champions) and varied start states (40% kickoff, 45% real states from pro replays, 15% random physics) so aerials, saves, and recoveries show up without hand-coding them.

How it works

  • Environment: RLGym 2.0 + RocketSim 2.2, 1v1, 92-value DefaultObs observation, 90 discrete actions (LookupTableAction), one decision every 8 ticks (15 per second).
  • Rewards: goal +10, concede −10, touch 0.2, velocity to ball 0.05, ball velocity to goal 0.18. Nothing specific for jumps or mechanics: mechanics are meant to emerge, not be programmed.
  • Learning: PPO (rlgym-ppo, a vendored fork with a fix to the advantage computation, GAE), 512×512 network, entropy coefficient 0.003, self-play.
  • Evaluation: matches between frozen checkpoints identified by SHA-256, promotion to champion only with a statistically significant win in 100-200 games; reproducible jump audits; comparison against pro replays.
  • Live game: the bot runs on RLBot with the same observation reconstructed from the live game packet, stochastic sampling, and physics-tick synchronization.

The GAE fix mentioned above is about this. The advantage of a transition is computed by accumulating temporal-difference residuals:

δt=rt+γV(st+1)−V(st),A^t=∑l=0∞(γλ)l δt+l\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t), \qquad \hat A_t = \sum_{l=0}^{\infty} (\gamma\lambda)^l\, \delta_{t+l}

The bug was in how V(st+1)V(s_{t+1}) was obtained: instead of the value of the state that actually follows each transition, the original backend took the value of the next transition in the buffer, which in a buffer packed with many episodes is almost never the same state.

What was learned, in three lines

  1. Cutting the touch reward doesn’t remove jumps: it cost 100M steps for no gain, and the bot touched the ball 25% less. Reverted.
  2. The entropy coefficient was the brake: lowering it from 0.01 to 0.003 got the policy to commit within 100M steps, with half the jumps, triple the ground time, and a 179-21 result against the previous champion.
  3. Windows 11 pushes training onto the E-cores when an app is in the foreground: pinning the processes to the P-cores multiplied throughput by 2.5. It’s now automatic.

Compared against 19 professional RLCS replays (~368M bot steps at the time of that comparison), Egoist is at 10-15% of a human’s touches and goals per second, but 25-33% of their car/ball speed. It moves with real energy, but converts that into contact and goals far less often. This motivated raising the weight of ball_velocity_to_goal at the next 500M-step milestone.