Summary
Terminology note: Robot models are also AI. To avoid confusion, I try to use “frontier models” to refer to the most capable general-purpose language and multimodal models, and “robot models” to refer to models trained specifically for robotic control.
There’s been a lot of progress in robot models in the last few years. This work has made use of progress in language and multimodal models, including by using small vision-language models (VLMs) for language and image processing. But robotics progress is only weakly coupled to frontier model progress: robotics labs are mostly trying to scale their own models based on their own data. Because existing non-frontier VLMs are already sufficient at the semantic tasks they’re used for in robotics (like parsing language commands and identifying objects), the VLM is typically not the bottleneck to further progress. As a result, new frontier model releases don’t usually translate directly into better robot models.
However, it seems possible that future frontier models could generalize more strongly to robotics. Above a certain threshold of capability, they might be able to quickly enable a lot of progress in robot models, either by controlling robots themselves or by dramatically improving the way robot models are trained. Importantly, frontier models could do this in ways that bypass the current robotics scaling path (rather than just making the existing approach scale faster). This means that robotics scaling laws might underestimate how quickly robotics could improve.
How robotics currently draws on frontier model progress
Robotics training builds on progress in language and multimodal models in two ways:
Many advanced robot policies use a vision-language model (VLM) to process language commands, understand the task, and analyze the scene. Low-level action planning and control, by contrast, are typically handled by robotics-specific components. The VLMs used in robot policies are often relatively small, non-frontier open-weight models. For example, Physical Intelligence’s π0 used a 3B open-source VLM, and Figure’s Helix uses a 7B open-weight VLM. The VLM doesn’t currently seem to be a bottleneck to robotics progress: once a VLM is good enough to provide the necessary semantic context about the task (e.g., by being able to reliably identify objects on a table), further advances in language or image understanding don’t necessarily produce better action planning. Consistent with this, new model releases don’t seem to have a dramatic impact in the robotics world.
The robotics industry seems to be using the lessons learned from training LLMs – like scaling pre-training data – to inform the process of training robot models.
Beyond that, robotics progress is only weakly coupled to progress in frontier models.
There are only a few exceptions to this that I’ve seen. One is Waddle Labs, a YC-backed startup trying to enable LLM agents (like Claude Code) to control robots either by directly writing control code or by calling smaller VLA models. This seems like one of the few labs that is set up to directly benefit from advances in frontier models (though I don’t know how well their approach is working in practice).
Frontier-model capabilities could generalize to robotics
I could imagine a few ways this could happen:
Frontier models themselves get capable and fast enough to be able to control robots. LLMs can already operate simple robotic interfaces. However, they are a long way from being able to operate complex robots like humanoids. This is because humanoids require lots of very fast motion control adjustments to stay upright, walk, and generally not fall over. This is currently handled by having purpose-built motion control models that process sensory information, make decisions, and communicate those decisions to joints hundreds to thousands of times per second. In the limit, a sufficiently capable and fast frontier model running on the robot might be able to handle all of the motion control itself, either by directly producing motor commands or by building and coordinating smaller motion-control subagents. More likely, before that point, advanced frontier models might get adept at calling VLAs or basic action control policies for moment-to-moment motion control, while the frontier model makes all the higher-level decisions. This is similar to what Waddle Labs is trying to build.
Frontier models help researchers train really good robot models. There are a few ways frontier models might rapidly scale robotics progress:
Rapidly scale up high-fidelity simulations. Simulations are currently not realistic enough to be sufficient to train models that transfer reliably to many real-world contexts. For example, it turns out to be pretty hard to accurately simulate things like friction. As such, for contact-rich tasks like manipulating delicate objects with a hand, robotics labs usually need to couple any simulation training they’re doing with real-world data. But a sufficiently advanced frontier model might be able to, say, infer the relevant dynamics and construct much more realistic simulations. Once simulations are good enough, they might allow robotics companies to quickly scale up their model training. In the limit, these simulations might be good enough that policies trained inside them transfer without additional real-world robot data.
Figure out how to make use of existing data. There’s no equivalent of internet text at scale for robotics. Instead, robotics data collection is bespoke and laborious, often involving lots of teleoperation data. Various labs are trying to use video data, but extracting action sequences from videos is difficult because videos don’t directly specify robot actions. To compensate, labs generally still mix video data with teleoperation data and real-world fine-tuning.1 An advanced frontier model might be able to infer enough missing information from existing troves of video data to let robotics progress scale much faster.
Figure out much better training methods. Advanced frontier models could invent new model architectures, data-generation methods, or entirely new robotics paradigms. Maybe with sufficient intelligence they could even derive sufficiently accurate world models from first principles/existing knowledge, without the need for much additional real-world data.
Importantly, if these approaches became sufficiently capable, none would necessarily require lots of new real-world data collection or experimentation. Current simulation- and video-based methods still rely heavily on real data, but this may not be a fundamental constraint. For example, immediately after a frontier-model developer releases a sufficiently capable new model, a robotics lab could conceivably use it to spin up a hyperrealistic simulation environment and train a highly capable robot model.
These improvements could produce a discontinuity with the current path of robotics progress. Robotics progress might be chugging along within one paradigm (e.g., collecting tons of teleoperation data), only for frontier models to accelerate capabilities through an entirely different paradigm, like using advanced simulation. If that happened, frontier models could bypass the standard robotics data-scaling path. We don’t yet have standard “robotics scaling laws.” But if standard relationships based on the current paradigm did start to emerge, we should be wary that extrapolating from them might underestimate just how quickly robotics could improve.
Some caveats
I don’t think it’s a given that progress in frontier models will generalize to robotics. Training sufficiently advanced robots might simply require more data than a frontier model could access without further real-world data collection – including data that can only be collected through boring, time-intensive means like teleoperation. I think this is a very real possibility. The fact that coding agents – despite being very useful in many adjacent domains – haven’t yet dramatically accelerated robotics progress might be evidence that it’s difficult to make progress without more real-world data collection.
I also think it’s possible that robot models get really good through the current paradigm before frontier models radically transform the process. Scaling up the usual end-to-end VLA training might be sufficient to get human-level robotics capabilities in short order, making spillover from frontier models largely irrelevant to robotics capabilities.
Thanks to Konstantin Pilz for helpful feedback.
One sort-of counterexample is Figure, which reports training a navigation capability using only egocentric human video data, with no robot demonstrations. But this result applied only to navigation; the underlying motion control systems were built from teleoperation and simulation data. Additionally, it appears that the video data was custom-collected (i.e., not based on existing troves of data that could be immediately incorporated into training).