Robotic Arms in Art

Exploring the creative potential of the Franka Research 3 arm through contemplative motion, interactive painting, and emergent choreography.

Context: NYU Shanghai · Interactive Media Arts × Franka Robotics
Role: Research Assistant · Prototyping · Interaction Design
Focus: Robotic Expression · Computer Vision · Reinforcement Learning
Timeline: Spring 2025

Mandate

Collaborate with Assistant Arts Professor Andy Garcia to investigate how industrial-grade robotics can support artistic practice. The team comprised Professor Garcia, research assistants Taojie Zhang and Evan Xiao, and the Franka Robotics support crew.

Approach

We began with off-the-shelf Desk tooling to understand motion planning fundamentals, then moved into franky-control Python workflows to gain full access to the robot’s seven-axis kinematics. Each prototype deepened our technical skillset while prompting questions about authorship, embodiment, and agency.

Project 1 · Contemplation

Our first exploration focused on meditative repetition. The Franka Research 3 arm drags combs, stones, and brushes through fine sand, erases the pattern, and begins again. The slow cadence invites viewers to breathe with the machine—no narrative, just an endless act of attention.

What we learned

Used Desk UI to choreograph precise Cartesian trajectories while monitoring joint torque limits.
Captured the relationship between sand texture, end-effector speed, and perceived calm.
Developed safe-reset routines so the arm could loop indefinitely for exhibition hours.

Contemplative sand patterns generated by the Franka arm.

Franka arm smoothing sand during the Contemplation piece

Zen garden patterns created by the robotic arm

Project 2 · The Mimic Brush

Next, we transformed the robot into a collaborative painter. Using TensorFlow.js hand pose detection, visitors could guide the brush by simply moving their hands. Forward, backward, up, and down gestures mapped to Franka’s Cartesian commands, turning the piece into a responsive, shared performance.

Exhibition insights

Balanced responsiveness and safety by smoothing gesture noise before sending velocity commands.
Designed UI prompts and pacing to help newcomers adapt to the robot’s deliberate tempo.
Observed families and children using the system, validating the accessible design goal.

Visitors guiding live watercolor strokes via hand pose detection.

Preparing the Franka arm with a watercolor brush attachment

Participant leading the robotic brush with hand gestures

Close-up of brush strokes created with human-robot collaboration

Visitors interacting with the Mimic Brush installation

Project 3 · Signals of the Void

In the final study we explored emergent behavior. Three speakers around the robot pulse audio cues. Reinforcement learning rewards the arm for discovering joint configurations that move toward louder signals, prompting an ambient search across space—a kinetic listening exercise rather than a scripted routine.

System architecture

SAC-inspired policy adjusts all seven joints, sampling microphone input at the gripper.
Reward function encourages both exploitation of strong signals and constant exploration.
Audio origin shifts every 30 minutes, prompting new “choreography” across the gallery.

Ambient reinforcement learning routine seeking the loudest signal.

Franka arm listening for the strongest audio signal in a darkened gallery

Long exposure showing the arm's exploratory motion path

Visitors observing the reinforcement learning choreography

“The Franka arm became a co-performer. Each iteration altered our sense of where authorship lives—sometimes in the code, sometimes in the machine, sometimes in the audience response.”
Andy Garcia · Assistant Arts Professor, NYU Shanghai

Outcomes

Across the semester we developed three working installations, an internal toolkit for choreographing responsive motion, and practical guidelines for staging robotic art in gallery settings. The work continues as we refine the RL pipelines and explore collaborative performances with dancers and musicians.

Thanks to Franka Robotics for equipment support and technical guidance throughout the residency.