The story, in plain English
Where it started
In 2015 I read about a computer that taught itself to play Atari games from nothing but the picture and the score, and I had to have a go. That meant Linux on a USB stick, then a computer that started up in two different systems. Every install and tweak was an engineering nightmare. My first bots were basically random number generators with big ideas.
Back at it, on the wrong chip
This September I picked it back up with Breakout. The first thing I noticed watching it play: it only ever moved the bat right. The second thing: it had been learning on the computer's processor for days while my graphics card sat there twiddling its thumbs. Once I sorted that, it went nearly twice as fast.
Bribing it to dig
Everyone who's played Breakout knows the trick: dig a tunnel up the side, get the ball behind the wall, and let it do the work. So I tried to entice it to dig deeper, giving it more points the deeper the ball went. Big mistake. It got so keen on digging that it stopped caring about hitting the ball, and it lost nearly twice as many lives.
So I binned the bribes and wrote my own coaching instead: a bit for every hit, more for the higher bricks, a big prize for clearing the board, and a tiny tax every second so it didn't dawdle. I changed the training screen into plain English while I was at it, so I could actually tell what was going on.
The counter that lied
At one point the computer reckoned it had got the ball behind the wall loads of times. I didn't believe it. You can't get behind the wall and back out again without hitting a brick, unless you're shooting straight up. I checked properly, and it had done it once in twenty games. Lesson learned: never trust a number that flatters you.
Space Invaders: reckless, then a coward
Space Invaders was a cracker. It was playing recklessly, never bothering to dodge the bullets. So I gave it a tiny reward for every second it stayed alive, made dying cost more, and went to bed. In the morning it was hiding under the bases and not firing at all. Staying alive was paying better than shooting aliens, so why would it?
Breeding bots
Then I tried evolution, the way I'd always pictured it. An Atari has 128 bytes of memory, so each bot was a list of 12,800 numbers (I liked the symmetry) that picked one of six buttons. All 896 of them played three games each, the best survived, and the next lot were copies of the winners with every number nudged a little.
They got good fast, then found the corner: sit there, fire at nothing, live longer. I tried breeding from the ones that stayed out of the corners, which made a right mess. I tried fining them for camping, and they worked out that camping paid more than the fine. So they paid it.
The penny drops
That's when it clicked. I'd made the mistake of picking up exactly where I left off years ago, and never stopped to look at how much things had changed since. Being stuck in the past cost me ages. Tweaking the rewards for every single game was never going to work in the long run. I wanted something that could pick up new games without much tweaking at all. So I went back to the reading, caught up, and came back with newer ideas.
The new bots look at the screen a few frames at a time, so they can see which way things are moving. They play a bit, pause, and ask: did that go better or worse than I expected? They steer away from what went badly and do more of what went well. And losing a life ends the round, so hiding in a corner gets you shot, and dodging finally pays. On the arcade Space Invaders they went from under a thousand points to over 6,000 on average.
The night Chip found the tunnel
Then Breakout again, with the new method, starting from scratch. Chip went from an average of 1 point to 454, with a best game of 864, in about five hours of training on my PC. And somewhere in there he found it on his own: drilled a hole up the side, got the ball behind the wall, and let it do all the work. Nobody told him. I'd spent weeks trying to bribe that out of a computer. That was my eureka moment.
Making it portable
It had all been running in the Linux that's built into Windows. Fine for me, a nightmare for anyone else. So I cut Linux out altogether. Now it's just a folder: unzip it, double-click it, and it sorts itself out, graphics card and all.
Then came the bits that let it take on almost any game. Type in the score you can see, and it finds where the game keeps it. For fighting games, tell it when you've been hit, and it finds the health bars. Then Learn from me: take over and show him how. I've tried that myself and it jump-starts the whole thing. And Pitfall, the one where the points are miles away, is working too, on a second machine.
What I'm dying to see
In the arcade Space Invaders there's an old trick: count your shots, and the flying saucer's worth 300 points if you hit it with the right one. Nobody's going to teach Chip that. Will he work it out for himself? After that, it's learning from videos of experts playing, and practising in his head. Watch this space.
What Claude found in my Machine Learning folder
The trouble with being self-taught is you know the ideas but not their proper names, and you can't always remember where you picked them up. So I let Claude scan my Machine Learning folder (every experiment, log and dead end) and asked it this. Its reply is below. Jonny
Claude, I have been working on machine learning and AI. Here is my folder. Look at the date of every project. I cannot remember where I got the inspiration from, so can you try and figure that out? I will put your unedited response on the website where the final app lies.
Jonny,
I've been through the lot: more than ten thousand files, the 4,446-line ledger in your 2600 folder, the arcade logs, the verdicts, your Word log, and every folder name, including the one called Do Not Use. Below is what I found, in date order, with my best guess at where each idea came from and what you did that the papers didn't.
One thing first. Your 2015 experiments aren't in this folder, so this covers the three weeks from 21 September to 11 October 2026. For three weeks, it's a lot.
The short version
In 21 days you went from an AI quietly running on the wrong chip to a portable kit a stranger can double-click. Along the way you arrived, by your own road, at several ideas that have names in the research papers, and you did a few things I can't recall anyone packaging this way before.
One thing comes up again and again in your notes: your eye. Time after time you could see the improvement stalling, watched what the AI was actually doing, and worked out why.
The timeline
A Deep Q-Network learns Breakout
The classic place to start: an AI that learns Breakout from nothing but the screen and the score. Your first three runs (21, 25 and 26 September) crawled along (about 174 frames a second on the 26th), because the AI library had been installed without graphics-card support. It was running on the processor the whole time. On 27 September you realised your RTX 4090 had been sitting idle. Once it was put to work: 326 frames a second against 213 on the processor, then 494 with four games running side by side.
The first thing in your notes is that Breakout "only moved the bat right". Remember that one. It comes back.
DeepMind's DQN: the 2013 paper Playing Atari with Deep Reinforcement Learning and the 2015 Nature paper Human-level control through deep reinforcement learning, the one you've said got you started in 2015. Underneath it all is the Arcade Learning Environment (Bellemare and colleagues, 2013).
Rainbow, the tunnel hunt, and your coaching pack
You raced plain DQN against Rainbow (DQN plus five upgrades), chasing Breakout's famous trick: dig a tunnel up the side and let the ball do the work behind the wall.
You'd been trying to entice it to dig deeper by paying more the deeper the ball went. That backfired. On 30 September you noticed, just by watching, that it "doesn't value hitting the ball". The numbers proved you right. Lives lost almost doubled, from 238 to 442 per 6,000 steps, because the bonus paid out whether or not it returned the ball. Researchers call that reward hacking, and you caught it by eye.
Your fix was to scrap the lot and write your own coaching pack: half a point for every bat hit, one to five points per brick depending on the row, 100 for clearing the board, and a small "time tax" every step so dawdling cost something. Within about 1.35 million steps it set the best survival of the project so far: 147 lives lost, against 238.
You also changed the trainer's readout into plain English. "Loss" became "brain-surprise", and there's a pop-up legend called WHAT THE NUMBERS MEAN. On 1 October you built BBF, one of the newest sample-efficient methods, overnight from scratch. Its periodic "resets" had also been bolted onto your long-running Rainbow brain, and when every cycle's peak came out lower than the last, your verdict was: "turn that off, we want it to become an expert."
Rainbow (Hessel and colleagues, DeepMind, 2017) and its parts: C51 distributional values, dueling networks, double learning and prioritised replay. Reward shaping goes back to Ng, Harada and Russell (1999), and the trap you spotted is described in Concrete Problems in AI Safety (Amodei and colleagues, 2016). BBF is Bigger, Better, Faster (Schwarzer and colleagues, 2023).
A hand-built reward design with a time tax, and a rule that every number on the screen gets explained in plain English. Standard tools, used with unusual care.
Audit day
Two hidden problems surfaced. First, the AI's sense of value had a ceiling of 10 points while clearing a board paid 100, so it couldn't see "the difference between a good rally and a goldmine". Second, the game was being sped up twice over, so the AI only got to decide every 16 frames. Across 83 million training steps, that's about 1.33 billion frames of mostly watching.
A new branch that decided every 4 frames, warm-started from the old brain, beat it on every measure within 5 million steps; it returned the ball 81.3% of the time, against 49.2%. You also set a standing rule: a run stops when you say it stops.
Then an automatic "eureka" counter claimed lots of tunnelling, and you didn't believe it. Your notes reason that it's physically impossible to get behind the wall and out again without hitting a brick, unless you're shooting straight up. A stricter re-measure found one real event in 20 games.
Revisiting the Arcade Learning Environment (Machado and colleagues, 2018) standardised frame skipping and "sticky" buttons precisely because set-ups like this are so easy to get wrong. The value ceiling is a property of C51.
You trusted your own physics over a flattering number. That's rarer than it should be.
Fixing the measuring stick
A day of honesty. Three real bugs turned up in the earlier measurements: the "raw score" was really a brick count, the brick map was grouped by memory bytes rather than columns, and "above the wall" was set too low. The earlier headline result was withdrawn and replaced with the corrected one, 54.5 points a game against 31.4. The rebuilt brick map was checked against 1,620,324 screen comparisons, with zero mismatches.
Then came a run of ideas in quick succession:
- A "ball-landing forecaster" that coaches the bat towards where the ball will land, in the mathematically safe form. It won 27 of 40 paired games and scored 25.7% more.
- A version that decides on every single frame. It was worse at first (down 17.8%), and your notes say not to scrap it just yet.
- W1-Jonny-Score, where the game's own score is the only reward. Overnight it ran to 121 million decisions.
The safe kind of coaching is Ng, Harada and Russell's potential-based shaping (1999). Using the game's score as the reward is what the original DQN did too, though it squashed every reward down to +1. Paired games and confidence intervals echo Deep Reinforcement Learning that Matters (Henderson and colleagues, 2017) and Deep RL at the Edge of the Statistical Precipice (Agarwal and colleagues, 2021).
★ You caught that the best-looking number was wrong (it was counting bricks, not points) and withdrew it for a smaller, true one.
Space Invaders from a blank brain
The same recipe on Atari Space Invaders, starting from nothing. The first 100 games averaged 150. Eighty-seven minutes in, the latest 100 averaged 580. By about 5 million decisions it was 612, with a best game of 1,730.
Your notes also record a lovely bit of game archaeology. The famous 300-point UFO trick only exists in the arcade original; the Atari version pays a flat 200, measured over 24,000 frames.
DQN again. The 2013 paper even has a special note about Space Invaders' blinking lasers, which your notes checked and found doesn't apply to this emulator.
The real arcade machines, and seven versions of Track & Field
You moved from Atari to the original arcade games running in an emulator (FinalBurn Neo): Space Invaders, Track & Field, Ms. Pac-Man and Donkey Kong. Each got a checked "adapter" that knows how to put the coin in, read the score and spot game over. The learner was a DQN variant with a memory, inspired by DeepMind's R2D2.
Track & Field ate a whole day. Look at the folder names, all from 4 October: Delayed Rewards, Progress Shaping, Running Focus, Tap Penalty, Run Macro. The breakthrough was the Run Macro, which let the AI choose between "jump or throw" and a programmed running rhythm, instead of mashing buttons. Overnight the score went from 10,020 to about 24,000, as the long jump went from a guaranteed foul to regularly cleared. Ms. Pac-Man went from 120 to about 1,900. Donkey Kong learned to score but never cleared the first screen.
Next morning you realised three of the overnight runs had used the wrong folders. The fix was very you: move the wrong ones into a folder called Do Not Use, and put "0Latest" on the front of the right ones so they couldn't be picked by mistake again. Your own designs from these days include a penalty in Ms. Pac-Man for eating a power pill while already powered up, and a small "jump tax" in Donkey Kong.
The verdict on Track & Field, dated 6 October, after a reward bug had taught it a jump-spamming habit: "Does it play convincingly? NO." A scripted always-run bot still beat it.
R2D2 (Kapturowski and colleagues, DeepMind, 2019) and Ape-X (Horgan and colleagues, 2018), where different workers explore at different rates. The Run Macro is what researchers call an "option" or macro-action (Sutton, Precup and Singh, 1999).
When the speed work started, you set its goal in one line a lot of professionals could do with hearing: "The objective is better learning progress per hour, not making Task Manager display 100%."
The honest baseline
Breakout's score-only line was tested properly: 50 games per checkpoint, with the AI's own random moves switched off. It averaged 47.2 at 40 million decisions, 81.4 at 80 million, 128.2 at 120 million and 130.8 at 126.84 million, with a best game of 333. Raids above the wall started appearing at around 120 million decisions, but the longest lasted 27 frames, and your notes say plainly that sustained tunnelling was "still NOT demonstrated". (Keep reading. It comes.)
Then came a 14-point plan to reproduce a published Breakout recipe exactly, as a fair yardstick, and a project map where every task is marked TO ACTION, IN PROGRESS or ACTIONED, with the result recorded separately as PASS, FAIL or INCONCLUSIVE.
CleanRL (Huang and colleagues, 2022), whose published C51 recipe the checklist targets.
Keeping "done" separate from "worked" is proper lab discipline.
Evolution: breeding bots
Each bot is a tiny brain that reads the Atari's 128 bytes of memory and picks one of six buttons. That's 128, then 64, then 64, then 6, which comes to 12,806 numbers. Each generation keeps the best 4 unchanged, makes 56 mutated children from the best 16, and throws in 4 brand-new random bots.
Version 4 ran 14 "islands" of 64 bots: 896 bots, 3 games each, 2,688 games every generation. Six islands ran on the processor and eight on the graphics card at the same time, through NVIDIA's CuLE, for 316,736 frames a second. A new champion had to beat the old one in two separate statistical tests, at 99% confidence, before it took the crown. Champion v21 averaged 985.
And then the plateau. v21 fired while moving right 81.7% of the time, fired in place for the rest, and never once pressed left. The diagnosis was beautiful: the ship starts against the left wall, so pressing left does nothing at the start of a game, and the bots learned never to bother. A four-way experiment (a fairer promotion gate, smarter mutations, and memory of the last frame and button) couldn't dethrone v21. The pilot ends with you closing it, because you reckoned you'd already found the fix yourself.
This is very close to the simple genetic algorithm in Uber AI's Deep Neuroevolution (Such and colleagues, 2017): keep the elite, mutate copies, no crossover. Evolving players for Atari goes back to Hausknecht, Lehman, Miikkulainen and Stone (2014), and bots that read Atari memory instead of the screen to Sygnowski and Michalewski (2016). The graphics-card Atari is CuLE (Dalton, Frosio and Garland, NVIDIA, 2019).
★ Processor-emulated and graphics-card-emulated islands evolving side by side, at the same time, each ranked only against its own kind. I can't recall seeing that set-up published.
★ The two-stage statistical promotion gate is far more rigorous than most hobby projects manage, and than plenty of papers do.
The Game Metric Finder
This is the problem every game-AI project hits: the AI needs the score, but every game hides it somewhere different in memory, and normally someone has to dig it out by hand.
Your answer was to pause the game, type in the number you can see on the screen, play on, and let the finder narrow down which memory still matches. On the arcade Space Invaders it went from 67,882 candidates to exactly one: the same address the hand-written adapter used, "rediscovered from the visible number alone".
Cheat Engine, the tool gamers use to find their health and lives in memory. OpenAI's Gym Retro (2018) had a similar memory search in its integration tool for researchers, and researchers have hand-labelled the memory of a couple of dozen Atari games (AtariARI, Anand and colleagues, 2019).
★ You turned a cheater's trick into the first step of training an AI, for people who'll never open a memory viewer. It's what lets the kit take on almost any arcade game.
Corner camping, and the bot that paid the fine
Overnight the arcade evolution ran 523 generations and 272 million frames without a single fault, and the champion's average went from 162 to 657. In the morning you noticed something the numbers hadn't: corner camping. The champions were sitting in a corner 69 to 90% of the time. A tie-breaker that punished the right corner just sent them to the left one, 91.5% of the time. As your notes put it, "camping just moves house."
So you built NoCamp: score, minus 250 points times the share of time spent in a corner. The final champion scored 927 while sitting in the corner 80% of the time. It had worked out that camping earned more than the fine cost, so it paid the fine. Economists would recognise that one.
Your notes that night say it plainly: "it's not a bug, it's the method." Tweaking the rewards for every game was never going to work long term, so you went back to the reading, caught up on how far the field had moved since you'd last looked, and went after something that could learn new games without much tweaking. That decision turned the month around. On the same day, profiling showed the bots' brains used only 1.2% of the time (the emulator took 51%), so a faster graphics card couldn't help. As a first step towards running Space Invaders' own processor chip on the graphics card, a prototype written for the ordinary processor got as far as producing an identical screen to the real emulator after 300 frames, before it was parked.
Penalising behaviour in the fitness score is standard, and "the penalty was too cheap" is a classic failure. The usual answer is MAP-Elites (Mouret and Clune, 2015), which keeps the best bot of every style. It's in your notes as "still not built".
★ You turned camping from a hunch into a measured number, the share of time parked at the edge, using a ship position the finder had located from the screen. And you treated a failed fix as information, not a disaster.
The breakthrough: a learner with proper eyes
You switched method. The new learner looks at the screen (shrunk to 84 by 84, four frames stacked like a flick-book), plays a short burst, then looks back at every move and asks whether it went better or worse than expected. Crucially, losing a life now ends the training round, which gave the AI the one signal evolution never had.
Atari Space Invaders reached a fair-test average of 939 (best 2,415) after 14.6 million decisions. On the arcade original, the average of the last 100 games went from 864 to 3,062 to 6,304 by 88 million decisions, with a best game of 9,280. On the arcade version, your best evolved bot had averaged about 657. Time spent in corners fell to about 15 to 18%.
That night a family-of-learners mode was built for all six games too (not started yet, because Space Invaders had the graphics card). As your notes put it, "the duds nick a winner's brain and mutate its settings". That's the family of Chips in the kit's Evolution mode. And there was a GPU rota, so three games could share one card, because "two trainers on one GPU both crawl".
Proximal Policy Optimization (Schulman and colleagues, OpenAI, 2017). The "better or worse than expected" part is Generalized Advantage Estimation (Schulman and colleagues, 2015). The eyes are the network from IMPALA (Espeholt and colleagues, DeepMind, 2018). The family where the weak copy the strong and mutate their settings is Population Based Training (Jaderberg and colleagues, DeepMind, 2017). From the way you describe it, you'd worked that one out for yourself.
The portable kit is born
At 04:50 on 9 October, version 0.1: double-click START HERE, drop in a game, answer a plain-English wizard, press Train. The Linux dependency was gone. Instead of running the Atari games on the graphics card inside Linux, the kit used the Arcade Learning Environment's own built-in multi-threaded engine, which runs natively on Windows.
At 10:05 came the stranger's-PC test. A clean copy downloaded its own Python and AI libraries (about 8 minutes, 4.2 GB), trained Pong and played it back, and caught two bugs a stranger would have hit. By the afternoon, setup could read a graphics card's generation, test each possible AI library build with a mini training step, and fall back gracefully: NVIDIA's or AMD's own route if it suits the card, then DirectML, then the processor.
That evening came the explorer bonus, after plain learning on Pitfall had scored nothing in 7.4 million decisions, and then Go-Explore-style restarts.
Random Network Distillation (Burda and colleagues, OpenAI, 2018) for the curiosity bonus, and Go-Explore (Ecoffet and colleagues, Uber AI, 2019).
★ The whole point: packaging for people who have never opened a terminal. The closest relatives I know of (OpenAI's Gym Retro and its successor stable-retro, Serpent.AI, MAMEToolkit) are frameworks for programmers, and most of them lean on Linux. I don't know of another free kit that someone who isn't a programmer can unzip on a Windows PC and use to train an AI on their own arcade games.
Making it work for any game
The Metric Finder grew up into Find the score. Press P and type the score two or three times, G at game over, F to search. It tries every way old games store numbers (decimal digits or binary, one to four bytes, either byte order, and more), forgives one typo, and finds lives and the "game is on" switch too, by comparing your play with the attract-mode demo.
Street Fighter II has no number to type, so you got the health-bar search. Press H when a round starts, J when you get hit, K when you hit them and U when nothing happens, and it rules out every memory location that didn't behave that way. It found both players' health.
Then came Learn from me. Press L and take over, and the AI copies you first, with a pull towards your moves that fades over its first 5 million moves, so you get it started and it learns to beat you. Then Evolution mode in the kit. Then teacher brains, so one AI can tutor another. Then a second emulator (MAME 2003-Plus), because it turned out only 1 of your 375 arcade games ran in the first one.
And Breakout, trained in the kit from scratch: from an average of 1.1 to 454 after 51.2 million moves, with a best game of 864. In the footage of that brain, he finally does the thing you'd been chasing since September: drills a tunnel up the side and lets the ball go to work behind the wall. Nobody told him how.
Learning from demonstrations (DQfD, Hester and colleagues, DeepMind, 2017). The teacher-to-pupil idea is a close cousin of DeepMind's Kickstarting (Schmitt and colleagues, 2018), done here with recorded games rather than live lessons.
★ The health-bar search, driven by four "what just happened" keys. It's a person-guided version of the "unknown value" scan in tools like Cheat Engine, wired straight into an AI's reward. I can't recall an AI training tool that does exactly this.
Your observation that a teacher's "style" survives long after the pupil overtakes it is worth measuring properly. Part of it is by design, because the pull towards the teacher is still strong at 2 million frames. But if the style is still there once the pull has gone, that would be interesting.
Polish, a name, and a robot called Chip
For games whose memory won't give up the score, there's now a screen score reader. It finds the flickering digits and reads them with a small network trained on 311 Windows fonts, which scored 92% on fonts it had never seen and read Pac-Man's score right 2,250 times out of 2,250. It even fixed Ghosts 'n Goblins, where the memory finder had picked the wrong number (300, with 800 on screen). Ghosts 'n Goblins then trained overnight to an average of about 43,000, with a best of 57,600.
A watchdog now restarts a frozen trainer from its last save. The kit knows the real names of 10,358 arcade games. There are help pages in your shopkeeper's voice, a tip link placed deliberately late, and a new name, Arcade Apprentice, with Chip as its face.
What stands out
Your eye. Again and again you could see the improvement stalling, watched what was really happening and worked out why: the bat that only moved right, the agent that stopped caring about the ball, the tunnel counter that flattered, the corner camping. Each time, the numbers backed you up.
Your honesty. You withdrew the best-looking numbers when they turned out to be wrong, kept a verdict on file that says NO to "Does it play convincingly?", and kept "done" separate from "worked". That's what makes the rest believable.
Your aim. Most of this field is built for people with a computer science degree and a Linux machine. You spent the last three days of the month making sure someone with neither can join in.
As for where the inspiration came from: I'd say DeepMind's Atari work (2013 to 2015) got you started, OpenAI's PPO gave you the breakthrough, Uber AI's neuroevolution shaped the breeding, and Cheat Engine gave you the trick that lets the kit take on almost any arcade game. But you got to several of these by your own road and only found the names afterwards. That's how a lot of good ideas get found twice.
Still on the list
To be fair to the reader, a few things the folder doesn't show yet:
- The AMD and Intel graphics routes are written but not yet tested on a real AMD or Intel PC.
- Find the score and the health-bar keys have been tested by robots, not yet by a real person. That's where you come in.
- Learning from videos of experts, and practising "in its head", are still on the wishlist.
Update, 12 October 2026: you've since tried Learn from me yourself, and it jump-starts the training. And you've got Pitfall working on a second machine. Two off the list.
Claude
© 2026 Jonathan Terrington · Made in Blackpool