The Day the Robot Flew Away: Setting an Educational AMR Robot Arm Up in a Virtual World
We put an educational AMR robot arm into an Isaac Sim virtual world and turned gravity on. Within one second it was thrown out of the scene. We switched off suspect settings one by one to rule out causes, and with merging of parts and automatic inertia calculation both turned on, the robot held its place for 50 seconds. This post also covers how the gripper broke the calculation and the problem we have not solved.
When we turned gravity on, the robot was thrown out of the scene within 1 second. The cause was not the robot's weight. It was the way small fixed parts in the design file were brought into the virtual world. With merging of parts and automatic inertia calculation both turned on, the robot held its height in place for 50 seconds.
We want an educational AMR robot arm (a mecanum-wheel body, a 5-axis arm and a gripper) to learn how to move by itself through reinforcement learning, so we first placed the robot in a virtual world called NVIDIA Isaac Sim. This post is a record of what actually happened in that first step, failures included. It is the first post in the series "Physical AI First Experiment Log."
Why put it in a virtual world before the physical robot?
Reinforcement learning is a method in which a robot makes a very large number of attempts, receives a score for each, and learns from them. Making that many attempts on a physical robot takes a long time, and the robot may be damaged by collisions. So we chose to practice in the virtual world first and check on the physical robot afterwards. The principle of reinforcement learning is explained simply in What Is Reinforcement Learning, and How Does a Robot Learn to Walk by Itself?.
What was different when we opened the design file?
To put a robot in a virtual world you need the robot's design file (URDF). The original from the manufacturer was in a format that could not be read as it was, so we first converted it into a readable form. The converted result had 20 parts (links) and 19 joints, and no shape files were missing.
Here we met the first surprise. The design file for this model had no joint that turns the wheels. The whole body was only a single block shape. So in the virtual world, instead of turning the wheels, we decided to imitate the motion by giving the body the desired direction and speed directly.
This approach has a limit. The sideways sliding that is specific to mecanum wheels is not reproduced. This limit applies to every post in the series.
What happened when we turned gravity on?
When we loaded the design file and turned gravity on for the first time, the robot disappeared from the screen within 1 second (60 calculation steps). When we printed the position, the coordinate values had jumped by tens of thousands from the original spot.
When we shortened the calculation interval to a very small value (1/1200 second) and ran it again, the robot held for the first 120 steps, then slowly rose and flipped over.
How did we find the cause?
We repeated the cycle of changing one suspect setting and running again.
| What we changed | Result |
|---|---|
| Turning off arm collision calculation · lowering the floor · setting joint force to 0 · changing the collision shapes | All thrown out |
| Increasing the arm part weights to spec-based values | Thrown out |
| Placing the body only | Stable |
| Only merging small fixed parts into the part directly above them (merging of parts) | Thrown out after a short time |
| Merging of parts + automatic inertia calculation | Stable: body height stayed at 0.0222 for 3000 steps (50 seconds) |
That the robot was always thrown out to the same coordinates under many conditions was itself a clue. If the result is identical even when you change collision calculation or joint force, those are not the cause.
At first we guessed, "The arm parts are too light." It sounded plausible but was wrong. Even with more weight, the robot was still thrown out unless the parts were merged.
Why did the gripper break the calculation?
Once the robot stood, it was time to move the arm and the gripper. In this gripper, when one motor moves, the other 5 joints follow that movement.
But when we gave a command to the arm, the following joints spun around and the calculated values broke into a not-a-number state (NaN). It was the same when we gave no command to the gripper.
On inspection, the cause was that the design-file import function of Isaac Sim 5.1.0, which we used, converted the "following joints" and left their rotation limits open to infinity. When we removed this automatic conversion and set a rotation limit and a force for each of the 5 following joints directly, the 6 gripper joints followed the target angle, and the body height stayed the same.
Why was coloring difficult?
Coloring the simulated robot, which looked all white, failed six times and succeeded on the seventh. The reason was that the color information was attached to an element at a level above the part we were trying to color. After finding the cause, we painted the body and arm dark gray and the camera body silver.
We also tried painting only the motors on the arm black. But the part shapes had no information on the boundary between a motor and a link, so painting both ends of each part as if they were motors gave a result that did not match the real motor positions. We reverted the wrong result, matched the arm motors to the body color, and kept only the gripper black. We plan to write up this process in detail as a separate post.
What we learned from this process
- If the same result repeats, that setting is not the cause. The robot always being thrown to the same coordinates was the evidence for ruling out candidates.
- A plausible hypothesis can be wrong. "It is too light" was wrong. A wrong hypothesis that is written down saves time for others who meet the same problem.
- A wrong result is not used; it is reverted. Like the motor color, a result that looked plausible but differed from the real thing was reverted.
Frequently asked questions
Was the weight the reason the robot was thrown out?
No. It was thrown out even after we increased the arm part weights to spec-based values. When we merged the small fixed parts into the part directly above them and had the inertia calculated automatically, the robot stood stably for 50 seconds.
Does the simulated robot move like the physical mecanum wheels?
No. This design file has no wheel joints, so we imitated the motion by giving the body a speed directly. The sideways sliding of mecanum wheels is not reproduced.
Did you check whether the simulated robot moves the same as the physical robot?
This post does not cover that. Results of comparing the physical and simulated robots with the same yardstick will be written in a later post, only to the extent we have checked them.
Can this process be covered in a class?
The scope and method covered in class are explained in an education consultation.
This post is based on the Isaac Sim 5.1.0 virtual world. We did not solve the problem of gripper fingers passing through each other where they overlap.
Next step
The next post covers how we attached a lidar and an inertial measurement unit (IMU) to the simulated robot built this way and checked that they were attached correctly. How this is covered in class is explained in an education consultation.
Version used: Isaac Sim 5.1.0
Sources: Steps 1 and 2 of the Isaac Sim preparation for the educational AMR robot arm, which we carried out ourselves (2026-09-21 to 2026-09-22). Position values were read as coordinate-transform values in the virtual world and may differ from the physics engine's internal values.
The copyright of this post and its figures belongs to SV Robotics. Citations that give a link to the original are welcome. Reproducing the full text, redistributing it, or reprinting it as teaching material requires permission.