We Moved an Approach Learned in Simulation to the Physical Robot — Why We Wrote Nine Limits Next to the Numbers
We put an approach policy trained with reinforcement learning in a simulated environment onto a physical AMR robot arm and compared it with a hand-written rule, three runs each from the same position. Approach time was 13.2 s versus 19.4 s, and the "lower the camera two steps at once" behavior learned in simulation appeared on the physical robot at about the same distance. The result comes with conditions attached, though - one position, three runs each, and a low battery.
Compared three runs each from the same position, approach time was 13.2 s versus 19.4 s. This is the result of putting a policy trained by reinforcement learning in a simulated environment onto a physical educational AMR robot arm. The behavior learned in simulation, "lower the camera two steps at once," also appeared on the physical robot at almost the same distance.
But these numbers come from one position, three runs each, and a low battery. So this post spends more space on the conditions attached to the numbers than on the numbers themselves.
In Part 6, we trained the approach to the block eleven times with thousands of simulated robots. This post is about moving that policy onto the physical robot.
How did we compare them?
To keep the comparison fair, we made the code of the two apps identical and changed only the decision method in step 2, the approach to the block.
- Reinforcement learning app: a policy trained for the approach makes the decisions
- Rule app: a hand-written rule makes the approach decisions
- Finding the block, aligning the angle, grasping and putting it down are the same in both apps.
We placed the block about 1 m from the robot, to the front left, and ran each app three times from the same position. All six runs grasped the block. The times were recalculated from the raw logs the robot left behind.
What were the results?
| Six physical runs (same position) | Reinforcement learning policy | Hand-written rule |
|---|---|---|
| Step 2: approach | 13.2 s (13.1-13.3) | 19.4 s (18.4-19.9) |
| Whole task (find to put down) | 56.3 s | 63.3 s |
| Camera lowerings | 4 | 7 (one step each) |
The approach step was about 6 seconds faster. Under these conditions we can say the reinforcement learning side was faster. We cannot widen that to "reinforcement learning is faster than rules." The reasons are in the limits below.
Did the behavior learned in simulation appear on the physical robot?
What we weighed most in this comparison was the behavior, more than the time. We placed the physical logs next to the simulated-environment logs of the same policy.
| Behavior | Simulated environment (final policy, 8 episodes) | Physical robot (3 runs) | Judgment |
|---|---|---|---|
| First lowering | All 8 right after the start, two steps at once | Two steps at once at about 1 m | Learned behavior |
| Second lowering | 0.26-0.29 m | About 0.28 m | Learned behavior |
| Number of lowerings | 4-5 | 4 | Learned behavior |
Lowering two steps at once is a behavior the hand-written rule does not have. The rule always lowers one step at a time. A method the policy found by itself in the simulated environment appeared on the physical robot at the same distance and in the same order.
Two more things look like reasons for the speedup. We did not test either behavior on its own, however.
- It keeps making decisions without stopping even when the block briefly disappears from view - we have never tested this behavior separately.
- It aligns sideways while approaching - there is an auxiliary rule that both apps share, so it cannot be separated from the policy's share.
Is this the result of the policy alone?
No. On the physical robot we did not use the trained policy as it was. We added an auxiliary rule that both apps use identically. Three of its parts are conditions the policy did not meet during training.
- Lowering the camera while moving - during training the robot stopped while lowering, but on the physical robot it keeps moving forward while lowering when it is far away.
- Previous-command input - when a lowering starts, the speed actually sent goes into the policy in place of "the speed issued just before."
- A motion around the block - the rotation that aligns the angle is added to the policy's command. The training environment had no such rotation.
Also, on the physical robot, once the robot is close enough to the block it passes to the next step without waiting for the policy to stop. So 13.2 s versus 19.4 s is the result of the bundle "policy + the auxiliary rule common to both apps." Because both apps use the same auxiliary rule, we take the difference to come from the approach decision method.
What are the nine limits of this comparison?
- It is one position (about 1 m, front left), one color (blue) and three runs each, and we did not alternate the two methods (rule 1 run, then reinforcement learning 3 runs, then rule 2 runs).
- The battery voltage was below the manufacturer's recommended range.
- The comparison target is the current rule. The lowering size and speed steps are values set by a person, so refining the rule could shrink the difference.
- The arrival criteria differ between the two apps. The reinforcement learning side stopped closer and more squarely in front, and was still faster (the side unfavorable to reinforcement learning).
- We did not align the block's tilt.
- The lowering and stop times are approximations from short-interval logs, and we write the start distance only as "about 1 m."
- The angle of one entry in the camera pose table differs from the physical robot, so the absolute distance values may have errors. Both apps use the same table, so the comparison itself remains valid.
- The rule app is a version whose code was aligned for this comparison, not the version we originally used.
- During the comparison, the app's input stopped and the battery ran down once. This did not affect the result numbers.
We do not compare simulated-environment times and physical times directly. The 21.7 s of the simulated environment in Part 6 and the 13.2 s on the physical robot here measure different spans, because the start distance, the point of handing over to the next step and the auxiliary rule are all different.
What did we learn from this?
- Change only one thing in a comparison. We aligned the code of the two apps and changed only the approach decision, so we could say where the difference came from.
- Look at behavior more than time. That "lower two steps at once" appeared on the physical robot was firmer evidence than the number 13.2 s.
- Do not write it as if it were the policy alone. An honest result also lists the auxiliary rule we added when moving to the physical robot.
- Attach the limits to the numbers. Moving the numbers without one position, three runs and a low battery is exaggeration.
Frequently asked questions
Is reinforcement learning faster than a hand-written rule?
Under these conditions (one position, three runs each, low battery), the approach step was faster, 13.2 s versus 19.4 s. We do not yet have results from comparing the two methods alternately over several positions at normal voltage, so we cannot say it is faster in general.
Did what was learned in simulation work on the physical robot?
The behavior of lowering the camera two steps at once, the distance of the second lowering and the number of lowerings came out the same in the simulated environment and on the physical robot. On the physical robot, however, we also used an auxiliary rule that the policy had not trained with.
Is it wrong to compare simulated-environment time with physical time?
The values do not measure the same span, because the start distance, the point of handing over to the next step and the auxiliary rule used together all differ. So we do not place the two numbers side by side.
Can this process be covered in a class?
The scope and method for covering it in class are explained in an education consultation.
This comparison is a physical result from one position, one color and three runs each, and the conditions are listed in "What are the nine limits of this comparison?" above. The simulated robot does not turn wheels but gives velocity to the body directly, so the sideways slip of mecanum wheels is not reproduced.
Next step
We will publish results from alternating the two methods over several positions at normal voltage in this series as soon as we have them. The way to cover the process from simulation to the physical robot in class is explained in an education consultation.
Sources: A physical comparison on an educational AMR robot arm that we ran ourselves (September 2026). The times were recalculated from the raw logs the robot left behind. We did not include the detailed settings of the training environment and the physical robot app in this post.
The copyright of this post and its figures belongs to SV Robotics. Quotation with a link to the original is welcome; copying the full text, redistributing it, or reproducing it in educational materials requires permission.