Inverse Reinforcement Learning

RL vs Inverse RL

Inverse Reinforcement Learning (IRL) can reduce some forms of reward hacking, but it does not automatically eliminate reward hacking.

Why IRL helps

In ordinary RL, you explicitly specify a reward:

“Get to the destination as quickly as possible.”

The agent may discover an unintended loophole:

Intended:
reach destination → high reward

Hacked solution:
exploit simulator bug → high reward without accomplishing the real objective

With IRL, instead of giving the agent the reward function, you give it expert demonstrations:

“Watch how a competent human accomplishes this task.”

IRL tries to infer the underlying reward/preference:

demonstrations→infer R(s,a)

Then the agent optimizes the inferred reward.

This is attractive because the designer doesn’t have to specify every detail of the objective.

But IRL has its own problem

The inferred reward can be wrong.

Get Farshad Noravesh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Suppose a human drives:

Road → slows down → checks intersection → turns → reaches destination

IRL might infer:

“The human values safety.”

But perhaps the human was actually following a traffic rule, or avoiding a particular road.

This is the inverse problem: many different reward functions can explain the same behavior.