auto-animated origami caterpillars

tl;dr here’s some animated origami caterpillars

backstory

I recently got access to a friend’s server running a video generation deep learning model, MiniMax H3 (by the eponymous MiniMax), and on a whim

I gave it the picture of origami caterpillars (mentioned in a previous post) and wrote “Generate a video where the origami caterpillars in the attached image move.”

(I had just read a make magazine article on creating a stop motion video game)

Try 1

The result was honestly pretty impressive but not quite what I wanted.

Try 2

So I added in “The caterpillars should move across the screen, not just wriggle”. I liked the results a lot more (this is the one at the top of the post).

Wow!! They’re so wriggly πŸ˜€

the spiral

This was followed by several rounds of attempting to fix the video further in various ways, but getting worse results each time. The main issue is that the caterpillar all the way in the back is scrunching backwards (obviously the eyes are the side with a few extra folds, can’t you tell? XD) instead of scrunching forwards. So I asked it to fix that, but then the tape started moving with the caterpillar, and the caterpillars moved slower and only did half a scrunch instead of a full wriggle, and kinda skidded around the book, etc. Alas.

subtleties

there are little subtleties that show this is not really real (i.e. I didn’t go forth and make stop motion by moving my little caterpillars). The table actually has gray-ish stripes running through it that the model “interprets” as shadows that disappear halfway through the image, if you look closely.

technical details

As mentioned earlier, this was running a Minimax H3 model with a “reference-to-video” checkpoint. The reference is a text prompt and optionally 1-2 videos. The “prompt” per the minimax docs is actually quite detailed and written by an LLM that then calls the video generation model.

Write a structured prompt file (markdown + YAML frontmatter) with shot descriptions, dialog, soundscape, music
Load it into a ComfyUI workflow (MiniMaxH3ImageToVideo node)
Submit via ComfyUI API β†’ poll for completion β†’ download the video
~5-6 min per 10s clip on a 64GB GPU, run sequentially (no parallelization)
It's image-to-video with integrated audio (dialog, sound effects, music all in one pass).

Each call returns with a video and some details, here’s what it returned with the first video prompt:

What I did:
Used your attached image as the I2V reference (Picture 1) for MiniMax H3 on the MITERS-XXXXX ComfyUI (port XXXXX)
Wrote an I2VA prompt describing the caterpillars' segmented paper bodies undulating and inching forward with a rippling crawling motion, while the camera slowly pushes in
16:9 widescreen, 0.4 MP, turbo mode, 8 seconds, seed 42

The model preserved the original image as the first frame and animated the caterpillars moving from there. If you'd like a different style of motion (e.g. more dramatic movement, different camera angle, longer duration, or higher quality with --no-turbo), let me know and I can re-generate with adjusted parameters.

I2VA = image to video with audio. ComfyUI is a way to run generative AI models that well, generate images/videos/audio.

Sleep now, enjoy the caterpillars!

CANSEND? CANDO! Controlling gripper motor via handtyping CAN protocol messages (Linux / Steadywin)

I got a gripper from some very nice SteadyWin folks at the boston robotics expo (part of boston robotics week / boston tech week) and finally got it working last week!! going to attach it to the UR5 once I’m done with the mount design.

 

After chatting with the steadywin folks on wechat, I got a link to their wiki.

The relevant pages are

How motor talk = registers

The way these quasi direct-drive (QDD — has small planetary gearbox, not direct drive) brushless DC (BLDC) motors with integrated controllers (driver is built into the housing) work: you write to a register to control the motor and read from the register to get statuses.

On to the practical part:

Hardware setup

The XT30(2+2) consists of the fat motor power wires and two small signal wires. [6]

Plug in the cable, then connect to 1) the power supply and 2) the USBCAN device.

After wiring up the gripper, make sure to also:

** Connect USB CAN to laptop. **

** Turn on power to motor **

Don’t forget the above steps πŸ™‚

Software setupΒ 

I got this working in Linux with the Steadywin motors.

In a nutshell:

$ uv venv
$ uv pip install python-can

$ sudo slcand -o -c -s8 /dev/ttyACM2  # [1]
$ sudo ip link set can0 up txqueuelen 1000 # [2]

# In a new terminal:
$ candump can0 # Listen to all the CAN messages back and forth

CANSEND? CANDO!

Sending CAN commands by hand

The overall flow

Check the motor is talks back and where it thinks it is using #280204 (ask for current position, should get a response from the motor), #21000103FFFFFF (set to position control), #34000101FFFFFF (set position to zero so we don’t get unexpected big movement — this needs to happen while torque is off, and we should get 01 in response, not 00), #00020101FFFFFF (torque on), #14020400040000 (command a position), #00020100FFFFFF (torque off).

The commands I used


cansend can0 001#01                  # ping β€” confirm it answers
cansend can0 001#02                  # status
cansend can0 001#280204              # read position
cansend can0 001#21000103FFFFFF      # mode = position (0x03, FFFFFF is for padding)
cansend can0 001#280204              # read position
cansend can0 001#34000101FFFFFF      # [3] set position to zero -- should get 01 as response, not 00 ! 
cansend can0 001#280204              # read position
cansend can0 001#00020101FFFFFF      # torque on (0x01)
cansend can0 001#14020400040000      # goal +1024
cansend can0 001#280204              # read position
cansend can0 001#00020100FFFFFF      # torque off (0x00)
cansend can0 001#02                  # check motor still alive. I use this because it's easy to remember

 

SUCCESS!!!!!!!111111!!!

The first high was when the motor actually talked back, the second was when it finally moved !!!!!!!!!!!!111111111!!!!!!!!!!!!!1

Parenthetical Notes

[1] Use sudo dmesg | tail to find the port, you’ll see something like […] ttyACM2: USB ACM device. (See later section on udev for how to fix it so the address doesn’t change each time you plugin the device).

[2] The TX Queue length increases the buffer so that while the motor is still processing a command, incoming commands may not get dropped. AFAIK.

[3] Reading candump. The successful reply looks like the following, where (can0 = the interface, 001 = motor CAN id, [1] for length of message, 01 for message — aka success)

can0  001   [1]  01

[6]Β  Iif you’re fancy you can terminate the XT30(2+2) data wire in wire ferrules with a crimper, then they fit nicely into the usb can headers)

UR5 on wheels: 80-20 and 3d prints

ur5 mount

Dane helped source a ton of 80-20 to make a nice wheeled mount for the UR5 (a really nice one tbh) so it doesn’t take over the entire living room all the time. Then we needed to actually attach the arm to the base. I remember around miters we would sometimes hang up mistakes as a funny sort of hall-of-shame. So here’s my pic. I did a 20 hr print (dat 70% infill 10 wall loops yo) that turned out to have indexing issues. Then I was like “heck it we’ll do the simplest possible mount”, dropped the indexing pins, checked the convenient ur5 base had room for the socket cap heads, and printed a … flat block with 8 holes in it. Hey. I’m far from a shop and far from free aluminum blocks, don’t have a saw, and just needed the robot off the floor for a bit (couchsurfer woo). I spent an hour printing the middle blue one, and then realized I’d measured to the inside with calipers then spec’d to the centers with CAD.The next one I was a bit smarter and printed it with no top wall, extrude cuts, and it printed in ~17 minutes. Hold it up to the robot arm:

Yay it checks out! Here’s my 3d printed block attached to the 80-20 lol. I think as a fresh grad I would have felt bad about it. Now I’m years out of my PhD (?!), feel less like I need to prove anything to anyone (i can just point to other stuff I’ve done), and don’t care so long as it gets the job done and I can move on to what I want to work on.

And now mounted. when the robot arm cracks the 3d print, flings itself off of the base, and sends me to the ER, you can tell me “I told you so”

projects blog (nouyang)