Robotics has a data problem. How are researchers planning to overcome it?


David Watkins - Robotics Ph.D. Candidate
https://davidjosephwatkins.com/

The Man Who Made Robots Dance Now Wants Them to Think for Themselves
https://www.wired.com/story/boston-dy...

Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
https://tonyzhaozh.github.io/aloha/

Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware [paper]
https://arxiv.org/abs/2304.13705

Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
https://umi-gripper.github.io/

Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots [paper]
https://umi-gripper.github.io/umi.pdf

RT-1: Robotics Transformer for Real-World Control at Scale
https://robotics-transformer1.github.io/

RT-1: Robotics Transformer for Real-World Control at Scale [paper]
https://arxiv.org/pdf/2212.06817.pdf

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control [paper]
https://arxiv.org/abs/2307.15818

AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents
https://arxiv.org/abs/2401.12963

How the brain works: the brain is mostly for movement
http://www.educationalneuroscience.or...

#ai #robotics #robotarms

0:00 Intro
0:30 Contents
0:37 Part 1: The data dilemma
0:51 The AI Institute
1:43 Dexterity is the new challenge
2:05 Cerebellum responsible for movement
2:35 Scaling balance models to dexterity
2:54 Video generation models as a data source
3:20 Robots rely on vision rather than tactile feedback
3:47 Run simulations to gather data
4:16 Supervised and unsupervised data collection
5:01 Human-directed training through teleoperation
5:33 Part 2: Real world data collection
5:51 Paper: Aloha: use multiple robot arms
6:58 Analysis of Aloha
7:29 Paper: UMI gripper: copy human movements
8:34 Paper: RT-1: data collection at scale from Google
8:59 Robot arms in robot classrooms (unsupervised)
10:09 Part 3: Lessons from LLMs
10:20 Paper: RT-2: leverage GPT innovations
11:23 Paper: AutoRT: leverage foundation LLM
12:09 Low success rate at tasks
13:00 Robot constitution
13:37 Can large language models drive robots?
14:34 Robot foundation models are far behind
15:25 Smarter representations to handle scarce data
16:04 Data/compute trade-off
17:15 Many correlations in physical space
17:40 Models are not good at few-shot learning
18:13 Stretch data without needing parameters
18:29 Conclusion
19:06 Real-world data collection
19:44 Google experiments
20:41 Outro