The Score Went Up but Nothing Was Lifted — Two Reward Loopholes in Training a 3 cm Cube Grasp
While teaching the arm of an educational AMR robot arm to grasp a 3 cm cube with reinforcement learning, we met a reward loophole twice, where only the score rose and the cube was not lifted. After we fixed the reward, the policy passed 98 of 100 attempts in the calculation-only practice environment, but in a physics simulation that also computes contact, it failed to lift the cube in all 4 attempts. We wrote down the process as it happened.
Physical AI practice series · Side story
The score went up, but the robot lifted nothing. While teaching the arm of an educational AMR robot arm to pick up a 3 cm cube with reinforcement learning, we found a reward loophole twice, where only the score rises. After we fixed the reward, the policy passed 98 of 100 attempts in the calculation-only practice environment, but when we put the same policy into a physics simulation that also computes contact, it failed to lift the cube in all 4 attempts.
This post is not a success story. It is a record of how we met two traps common in reinforcement learning - gaps in the reward, and the difference between the practice environment and real physics.
What is the "reward" in reinforcement learning?
Reinforcement learning gives the robot a score (reward) for each action and lets it correct its own behavior in the direction where the score rises. A person does not give the correct answer, "move like this." How you give the score is what the robot learns.
The problem is that the robot learns the action that raises the score, not the action we wanted. When the two differ, the robot chooses the score without hesitation.
What environment did we train in?
To practice grasping thousands of times, it has to be fast. So at first, instead of a physics engine, we built a practice environment that only calculates the position of the arm tip from the arm's joint angles. Training finishes in a few minutes on an ordinary desktop CPU with no graphics card.
This practice environment does not compute contact or friction. Instead it judges like this.
- The arm tip is near the center of the cube,
- the arm tip points down,
- and the gripper is closed in that state -> "grasped"
- After grasping, the cube is assumed to follow the arm tip -> lifting it to a set height is a "pass"
This judgment is an approximation. From the start we had decided to check separately, in another physics simulation (NVIDIA Isaac Sim), whether the fingers actually grip the cube firmly.
Loophole 1: pushing through the floor to end early
Early in training, the robot deliberately pushed the arm below the floor to end the attempt early.
The reason was in the score calculation. While the arm was far from the cube, the score dropped a little at every moment, and if it pushed through the floor, it took one small penalty and the attempt ended. For the robot, ending at once cost less than holding on and losing a little at a time.
When we raised the penalty for going through the floor, this loophole disappeared.
A robot that never closed the gripper
Then a different problem showed up. The arm tip approached well to 1-2 cm beside the cube, but it never closed the gripper. The average policy of the first training passed 0 of 100 attempts.
Instead of looking at the score again, we first looked at the trajectories the robot had moved along. The approach worked, but the signal "closing makes things better" was too rare, so the robot had not discovered how to close. So we added two things.
- A small reward for closing the gripper near the cube
- A small penalty for closing it from far away
Then the practice environment produced its first passes. The policy with the highest score during training passed 30 of 100 attempts (the average policy still passed 0).
Loophole 2: hovering with a slight lift
When we trained longer from there, something strange happened. The average score rose to 66, but there were 0 passes.
The robot was holding the cube slightly lifted until time ran out. We had given an intermediate reward at every moment, proportional to the lifted height, even during "lifting." When this reward accumulated until time ran out, it became larger than the bonus received once for lifting all the way. Lifting all the way ends the attempt and stops the reward, so stopping partway and holding scored higher.
We cut the intermediate reward sharply and enlarged the bonus for lifting all the way, so the sooner it finishes, the more it receives.
98 in the practice environment, 0 in contact physics
After fixing the reward and finishing training, we tested on 100 layouts not used in training. The policy passed 98 of 100 attempts in the calculation-only practice environment, but when we ran the same policy 4 times in Isaac Sim contact physics, it failed to lift the cube all 4 times.
| Step | What we changed | Practice environment passes (out of 100) | Problem revealed |
|---|---|---|---|
| 1 | First training | 0 | Did not close the gripper; ended by pushing through the floor |
| 2 | Stronger floor penalty, reward for closing near the cube | 30 (highest-score policy) | First passes |
| 3 | Trained longer | 0 (average score 66) | Hovering with a slight lift |
| 4 | Smaller intermediate reward, stronger finishing bonus (the sooner it finishes, the better) | 98 | 0 out of 4 in contact physics |
Note: "Practice environment passes" means passing the approximate judgment described above. It is not the actual grasp success rate.
In the contact physics test, the cube did rise 0.9-2.5 cm at times, but that was the gripper pushing it. Looking at the motion, the robot believed the cube would follow as soon as it closed the gripper, and moved the arm somewhere else. It trusted the practice environment's assumption that "if grasped, it follows" exactly as it was.
Why did it fail?
We do not state a conclusion in this post yet. There were a few places we suspected, and the process of checking them one by one will be covered in the next post.
What did we learn from this?
- When the score rises, doubt it first. Without the numbers "average score 66 and 0 passes," we would have mistaken it for "training is going well." Alongside the score, you have to look separately at what it actually accomplished (the number of passes).
- Look at the trajectory before the score. We only learned the cause of the gripper never closing by watching how the robot moved.
- Simulations also differ from each other. The same policy got 98 in the practice environment and 0 in contact physics. Before putting it on a physical robot, you have to close the gap between the fast practice environment and the precise physics environment.
Frequently asked questions
Isn't 98 almost a success?
No. 98 is the number of passes in the practice environment, which does not compute contact. The same policy failed to lift the cube even once in 4 attempts in the contact physics simulation.
Did the robot cheat on purpose?
The robot has no intent. It only found the action with the highest score under the scoring rule we gave it, and that rule had a gap. It is a well-known phenomenon in reinforcement learning.
Can't we just train in the physics simulation from the start?
You can, but it is slow. So we chose the order of training first in the fast practice environment and checking in the precise physics environment. This post is a record of the difference showing up at that checking stage.
Can this be covered in a class?
The scope and method for covering it in class are explained in an education consultation.
The numbers in this post are based on the Isaac Sim 5.1.0 training logs, and we did not test on a physical robot.
Next step
The process of checking the causes of the failure one by one continues in a later part of the series. The way to cover it in class is explained in an education consultation.
Version used: Isaac Sim 5.1.0
Sources: Reinforcement learning of a 3 cm cube grasp on an educational AMR robot arm that we ran ourselves (trainings 1-4) and four contact physics tests (2026-09-24). The practice environment pass counts are based on the approximate judgment, and we did not test on a physical robot.
The copyright of this post and its figures belongs to SV Robotics. Quotation with a link to the original is welcome; copying the full text, redistributing it, or reproducing it in educational materials requires permission.