Eleven Trainings of Approaching with Thousands of Simulated Robots — Four Collapsed on Gaps in the Reward Table
We used reinforcement learning to teach only one step, approaching the block, to an educational AMR robot arm. Of 11 trainings, four collapsed on reward loopholes such as "standing still," and one that looked good by training score never actually arrived. The final policy arrived in 199 of 200 simulated runs but fell short of the target time.
We trained 11 times, and four of them collapsed. This is a record of teaching an educational AMR robot arm one step, approaching the block, with reinforcement learning. We caught the collapsed trainings not by the training score but by a separate evaluation of 200 runs each. The final policy arrived at the target in 199 of 200 simulated runs, but it did not reach the target time we first set, even in simulation.
In part 5 we trained a small problem with two numbers on the CPU in a short time. This part is about a much larger problem, trained by moving thousands of simulated robots at the same time.
Why did we train only the "approach" step?
Picking up a block takes four steps. ① Find the block → ② Approach the block → ③ Circle around the block to adjust the angle → ④ Reach out and pick it up. When we ran the rules written by a person, the step that took the longest was ② approaching.
In this step, three goals collide with each other.
- You have to go fast, but if you go fast the block drops below the bottom of the camera image and you lose it.
- To see a nearby block you have to tilt the camera down, and the robot stops while it tilts.
- You have to stand at the exact spot to pick it up, but if you go a little too far you push the block away.
The other three steps are solved exactly by calculation, and some movements, such as moving the arm, are dangerous if wrong, so we used the rules written by a person as they were.
Principle — Solve by calculation what can be solved by calculation, and use training only where the goals collide and the right answer is hard to fix.
How did we train it?
In Isaac Lab we trained by moving thousands of simulated robots at the same time. At short intervals, the robot looks at where the block appears, how far the camera is tilted, what it did just before and so on, and decides forward speed, sideways speed, and whether to tilt the camera. The rule that makes these decisions is called the policy.
Rendering the camera images of thousands of robots would be too slow, so instead of rendering images we calculated "whether the block is visible from this camera pose, and where on the image it appears" and fed that in. In exchange, we added values that arrive late and jitter, like a physical camera.
Before starting training, we reproduced the rules written by a person in the same way inside the simulation. At first we carried the rules over by guesswork, and 57 of 64 runs passed the block; only after we ported the physical robot's program as it was did we get a tendency that matched the physical robot's records. If the comparison baseline is wrong, we cannot say whether training improved anything.
How did we decide the reward table?
Reinforcement learning looks for the behavior that earns the most points. So the reward table is "what we want." We set the order of importance as not losing it, not overshooting > standing exactly > going fast.
| Item | Add/subtract | Meaning |
|---|---|---|
| Approach | Add | A little at a time as the robot gets closer to the block |
| Arrival | Add | A lot when it arrives |
| Time | Subtract | The longer it takes, the bigger the loss |
| Losing the block | Subtract | It has to find it again |
| Overshoot | Subtract | If it goes past the target, the episode ends there |
| Camera tilt | Subtract | Reduce needless tilting |
Arrival was judged not by values measured through the camera but by the real distance inside the simulation. This is to block the loophole where a jittery measurement happens to fall inside the range and scores points.
What happened over the 11 trainings?
| Run | What we changed | Result (200 simulated runs) |
|---|---|---|
| Run 1 | First training | Hardly moved — "standing still" |
| Run 2 | Some robots started near the block | Still standing still |
| Rule imitation | Made it imitate the person's rules first | 0 arrivals (76 overshoots) |
| Run 3 | Started from the imitation and trained | 181 arrivals, median 22.4 s |
| Run 4 | Trained longer | Faster, but with the camera angle off by ±3°, 33 overshoots |
| Run 5 | Added the camera error all at once, large | Back to standing still |
| Run 6 | Added the error gradually, from small | 185 arrivals |
| Run 7 | Trained longer with a larger error | 0 arrivals — the training score looked good |
| Run 8 | Added a smooth acceleration/deceleration reward | 200 arrivals, median 19.9 s |
| Run 9 | Added an assumption not confirmed on the physical robot | Not faster, so discarded |
| Run 10 | Randomized the starting conditions | 199 arrivals — a habit of sidestepping to one side remained |
| Run 11 | Left-right symmetric training (last) | The lean disappeared but 77 arrivals → not used |
Run 1: a student too afraid of penalties to take the exam
The reward table cut a lot of points when the robot lost the block while approaching, and cut fewer when it stood still. The robot chose not to move at all. When there is a gap in the reward table, reinforcement learning finds exactly that gap.
Rule imitation: rare movements are not copied
When we first had it imitate the rules written by a person, it never tilted the camera. The moments of tilting are only a tiny part of all the examples, so learning "do not tilt" is right most of the time. Only after we gave a large weight to the rare tilting examples did it imitate tilting. Imitation alone did not reach the block, but training began to work from Run 3, which started here.
Run 5: make it hard all at once and it gives up
The Run 4 policy passed the block whenever the camera angle was even slightly off. So we added the error all at once, large, and the robot went back to standing still. It appears to have given up on approaching itself once overshoots became frequent. Only in Run 6, where we raised the error gradually from small values, did overshoots decrease.
Run 7: you cannot trust the training score alone
In Run 7, the average score earned during training looked normal. But when we tested it separately 200 times, it never arrived once. We estimate that easy attempts that start close to the block were mixed in and inflated the score. So for every run we also ran an evaluation of 200 runs under the same conditions, separate from the training score.
We asked for smooth movement, and it got faster
We added the condition "do not be abrupt, either when starting or when stopping." We did not put a limit on acceleration; we only added a reward that subtracts more as acceleration gets larger.
| 200 simulated runs | Run 6 | Run 8 (smooth acceleration/deceleration) |
|---|---|---|
| Arrivals | 185 | 200 |
| Median arrival time | 21.0 s | 19.9 s |
| Maximum acceleration | 1.69 m/s² | 0.61 m/s² |
We added one more condition and it got faster instead. The speed graph changed from the "square staircase" of sudden starts and stops to a "rounded hill." We did not test separately why it got faster; it appears to be because the time spent speeding up again after a sudden stop became shorter.
How did we choose the final policy?
Run 8 arrived in all 200 runs, but in a test that fed in the physical robot's driving records and looked only at the policy's answers, there were cases of sidestepping away from the block. We checked again with a "poke test," which changes only one input and looks at the answer. Run 10 did not remove the lean completely either, and Run 11, trained left-right symmetric, removed the lean but arrivals fell to 77.
So we stopped training at Run 11 and added one rule to the Run 10 policy. "If the block is to the side and the policy tries to sidestep the opposite way, change the sideways velocity to 0."
That one line created a new problem. In 200 mock runs that chain commands and results together without a robot, the robot kept standing still in 5 of them. The policy gave only a tiny sidestep in the opposite direction, and the rule turned it to 0, so the two blocked each other. After we added an escape rule, "if the command stays near 0 for a while, briefly use the command from the person-written rules," the robot moved in all 200 runs.
| Final policy (200 simulated runs) | Result |
|---|---|
| Arrived / overshot / lost | 199 / 0 / 0 (1 timeout) |
| Median arrival time | 21.7 s |
| Maximum acceleration | 0.59 m/s² |
| Camera angle off by ±3° | 198 arrivals, 0 overshoots |
In the same simulation, the maximum acceleration of the person-written rules was 1.71 m/s². However, the rule figure is the median over all 200 runs while the policy figure is the median over the runs that arrived, so the ranges measured differ slightly.
What have we not achieved yet?
- We did not reach the target time of 18 seconds or less. Even in simulation it is 21.7 seconds.
- We could not build a fully left-right symmetric policy. For now it is blocked with one rule.
- The evidence for the simulated robot's movement is weak. We tuned the simulated robot with values from a test on the physical robot, but the robot's movement record in that test was not its actual movement but a value calculated from the commands we sent (the same problem as in part 5). We re-evaluated assuming that at very low speed the robot actually moves less than commanded (80%, 60%). Arrivals did not waver, but how much faster the policy is than the person-written rules shrank from 3.8 seconds to 0.9 seconds at 60%. Only after re-measuring the actual movement can we state the gain from training accurately.
What did we learn from this?
- A gap in the reward table will always be found. If "standing still" costs less, the robot stands still.
- Look at the training score and the actual task evaluation separately. Run 7 had a good score but 0 arrivals.
- Rare movements need extra weight to be copied. A movement that appears only occasionally, like tilting, disappears if left alone.
- Raise the difficulty gradually. Raise it a lot at once and the training gives up.
- Test the safety rule too. A line that blocks one problem can create another.
Frequently asked questions
Is a reinforcement learning policy better than rules written by a person?
All the comparisons in this post are values inside simulation. In simulation it moved more smoothly but did not reach the target time, and because the evidence for the simulated robot's movement is weak, how much faster it is must be measured again. The comparison on the physical robot will be described, with its conditions, in the next part.
How do you judge whether training is going well?
We did not look only at the score earned during training. Separately from training, we ran 200 tests under the same conditions and counted arrivals, overshoots and losses. Run 7 looked good by score but had 0 arrivals in this evaluation.
What computer do you need to run thousands of simulated robots?
We trained on one PC with a graphics card. We did not render camera images but substituted calculation, which let us run thousands at the same time.
Can this process be covered in a class?
The scope and method covered in a class are explained in an education consultation.
All the numbers in this post are values from the Isaac Lab 2.3.2 simulation. The simulated robot is driven by giving velocity directly to the body without turning the wheels, so the sideways slip of mecanum wheels is not reproduced.
Next step
In the next part, we plan to put this policy on the physical robot and describe the result of comparing it with the person-written rules in the same place, together with the conditions and limits. The method covered in a class is explained in an education consultation.
Version used: Isaac Lab 2.3.2
Sources: The educational AMR robot arm block-approach reinforcement learning that we carried out ourselves (2026-09-28). The detailed settings of the training environment (reward values, arrival criteria, neural network structure, training settings) are not included in this post.
The copyright of this post and its figures belongs to SV Robotics. Quotations that credit the original link are welcome; full reproduction, redistribution and reprinting as teaching material require permission.