A visual explanation of reinforcement learning

How does your brain help you learn a new skill?

The Details

To try to answer this question, we’ll build a digital being with a simple brain, and watch it learn a new behavior.

In our previous article, we created a creature called a spark that could steer towards light. And we saw how small changes to its neural circuitry gave rise to very different behaviors.

• Fear
• Aggression
• Attraction
• Love
• Neural circuit
• Show one spark

Back then, we’d wired these circuits into the spark’s brain by hand. But now, we want it to learn the same behaviors by itself.

After all, you didn’t come into this world knowing how to walk, talk or read. While some of your abilities, such as breathing or swallowing, are wired by evolution, you gained others through an arduous process of trial and error.

We want our spark to do the same. To give you an intuition for this learning process, we’re going to hand over the controls to you.

We’d like you to wire the spark’s brain.

You wire its brain

But first, we need to tell you what you’ll be wiring. The spark’s brain has just four neurons — two neurons serving as its eyes, and two neurons that wiggle its flippers.

There are straight connections between neurons on the same side of its body, crossed connections between neurons on opposite sides, and drive, which controls the spark’s speed when there’s no light. straight crossed drive So its brain wiring boils down to just three numbers — the strength of each kind of connection, and the drive.

Now let’s set this spark loose. Use the sliders below to wire its brain as it moves. We’ll keep score.

Your goal: Make the spark love the light by getting a score above 0.5.

Can you make the spark spend time inside the light?

• Neural circuit
• Score
• Target
• 0.11
• Tries straight crossed drive
• Hint
• Reset

Finding the right configuration takes a lot of trial and error. (If you get stuck, click the hint button above for a nudge in the right direction.)

Let’s visualize what you just did. To guide the spark to the light, you tweaked three numbers. We can represent these numbers as the three axes of a cube.

The payoff of this picture is that we can visualize your spark’s learning trajectory as a path within this cube.

Your learning path... — To guide the spark, you probably nudged a slider one way, saw whether this made things better or worse, and then used this feedback to decide your next move.

Let’s distill this into a learning algorithm — a step-by-step procedure by which the spark can rewire its brain through practice.

Here’s how it’ll work: The spark gets many tries to steer towards the light. Before every trial, its brain wiring will change a little at random, just like you wiggled the sliders to and fro.

If the trial goes better than previous ones, then the spark gets a reward and its wiring shifts in the direction of this random change. And if the trial goes worse, the spark gets a punishment, and its wiring shifts the other way.

This isn’t that different from how you learned how to walk or ride a bike. Falling down was your punishment, and the thrill of balancing was your reward.

Can this simple algorithm help the spark find the light?

A spark learns by itself

In the beginning, the spark is a novice, with a brain that’s wired together at random. Unsurprisingly, it doesn’t get very far.

Before learning: Stuck

The baby spark can spin in place, but it can’t yet move around. But it’s about to get better…

After 50 tries: Meandering

After 50 tries of this learning algorithm, our infant spark has taken its first steps. And while it won’t break any speed records, it’s now on the move…

After 250 tries: Avoiding

After 250 tries, our juvenile spark has picked up speed. But it veers away from the light at the last moment.

After 500 tries: Charging

After 500 tries, our adolescent spark boldly charges towards the light. But it hasn’t yet figured out how to slow down or stop.

After 2,000 tries, our spark is now an adult — an expert at seeking and following the light. It clings to the light and can turn on a dime. (Try moving the light to see the spark follow it.)

Like before, we can visualize this learning process as a path.

The spark’s learning path after

2,000 tries...

0 tries...

This isn’t a path through physical space. Instead, it’s a path through the space of possible brains. It shows how the spark’s brain changed as it learned.

Other novice sparks would start from different points, and they would each trace out their own learning trajectory.

Many sparks’ learning paths after

6,000 tries...

0 tries...

This tangled web illustrates the learning paths of hundreds of novice sparks, each starting with a differently wired brain. As if by magic, all of these threads converge to the point marked ♥︎.

The paths aren’t straight lines: Learning isn’t linear. There are setbacks, wrong turns and mistakes along the way. And it isn’t necessarily fast. Some of the learners took thousands of tries to reach their goal.

You’ll notice that the paths are all jittery. That’s because every step involves randomness. We can smooth out this jitter by averaging over many trajectories at every point, revealing a map of the flow of learning.

• The sparks learn to love
• 3D view
• Side view
• Top view

It’s worth taking a moment to appreciate how surprising this is — given enough tries, the same learning process turned every novice into an expert light seeker.

Of the myriad possible arrangements of this spark’s brain, only a tiny fraction generate the behavior that we want. There are far more ways to fail than to succeed.

And yet, you’ve just witnessed a small miracle. The spark navigated this labyrinth of dead ends and rewired its brain to find the light, guided by nothing but reward and punishment.

Learning new tricks

Learning doesn’t stop once you become an adult. We learn throughout our lives and adapt to changing circumstances.

Our sparks can do this too. For example, what if the light represents a harmful chemical or a dangerous predator instead of a food source?

The spark is now rewarded for avoiding the light rather than for seeking it, and this leads to new behaviors:

• The sparks learn to flee or freeze
• 3D view
• Side view
• Top view
• If the light is harmful, then

LOVE becomes the worst strategy, and the sparks learn to FLEE (escape as fast as possible) or to FREEZE (play dead).

If instead the spark is rewarded for approaching the light with high speed, then it learns to charge towards the light, i.e. to FIGHT.

• The sparks learn to fight
• 3D view
• Side view
• Top view

Much like a child learns from its environment, the spark can learn very different behaviors — fleeing, freezing, fighting or following the light — depending on how it’s rewarded.

• A spark is shaped by its environment
• Flee
• Freeze
• Fight
• Love
• Neural circuit
• Pause light

· Spark is shown moving at two-thirds speed.

You’ve now seen how a novice spark can rewire its brain and learn new behaviors through a process known as reinforcement learning.

Biological brains also learn through reinforcement. For example, baby songbirds learn to sing by babbling. They try out random variations of their song. If a variation gets closer to their memory of an adult’s song, they get a burst of dopamine (and if it gets further away, the dopamine dips). Guided by these rewards and punishments, they can recreate the song from memory, even with no teacher present !

Now, we do have to come clean about something. This spark’s brain had just four neurons and three numbers that determined how these neurons were connected. This simple structure allowed us to watch its brain learn in three dimensions.

Your brain has around 86 billion neurons and over a hundred trillion connections. The space of possible ways in which it could be wired is therefore unimagi