Robots at Generalist AI can master physical tasks after watching short videos, adapting on the fly without prior training. Their improvisation hints at a leap toward real-world use, though reliability remains a challenge.
Inside a Cambridge, Massachusetts startup, robot arms are picking up new skills with a speed and flexibility that’s turning heads in the AI world. At Generalist AI, these machines can watch a brief instructional video-such as stacking cups or placing blocks in bowls-and then perform the task themselves, often without any prior training for that specific job. In one demonstration, a robot was told to sweep a block into a bowl using a dustpan and brush. When the brush was removed, the robot improvised, using the dustpan alone to flick the block into place.
Another striking example involved a two-armed robot that observed a video of someone unzipping a purse and removing banknotes. The robot then unzipped a different style of purse and carefully extracted the bills. When it struggled to grab the money, it switched from its right to its left gripper for a better angle-an action that surprised even the engineers, who noted the robot had never done that before.
Generalist AI’s approach draws inspiration from the way young children intuitively learn about the physical world. The company’s cofounder and CEO, Pete Florence, compared the breakthrough to the impact of GPT-3, which could perform new tasks simply by being prompted. Generalist AI’s robots are trained to understand the physics of their environment, allowing them to transfer knowledge from one scenario to another and improvise when faced with unexpected changes. In one case, a robot even used a banana to sweep up items when it was placed in front of it.
Traditionally, training robots for new tasks required thousands of examples and was easily disrupted by minor changes, such as lighting. Generalist AI, however, is building a general-purpose robotic model using data collected from humans wearing special camera-equipped gloves that mimic robot pincers. These gloves are used to perform various chores, generating a large and diverse dataset for training. The company has shipped hundreds of these grippers to workers in Mexico and other locations to expand its data collection efforts.
While the company is secretive about its exact training methods, it has developed its AI models entirely in-house, rather than relying on open-source language models. Experts like Danfei Xu of Georgia Tech and Karen Liu of Stanford University say Generalist AI stands out for its scale of data collection and its focus on physical interaction data that isn’t tied to a single robot type. Xu believes the company is closer than others to deploying robots in commercial settings, while Liu notes that their results suggest the strategy is paying off.
Despite these advances, Generalist AI acknowledges that its robots’ learning skills are not yet fully reliable. On average, a robot completes a demonstrated task about 59 percent of the time, far from the ideal 99 percent success rate. It’s also uncertain how well these abilities will generalize to every possible task or environment. Still, the potential for rapid skill acquisition in manufacturing and other industries is significant. This echoes challenges seen in other AI domains, such as when AI agents encounter unexpected obstacles in real-world business settings.
One evening, an engineer stacked cups in front of a two-armed robot just to see what would happen. The robot joined in, stacking the remaining cups into a neat pile, prompting a delighted reaction from the team. These moments of improvisation and adaptation are fueling optimism that robots could soon handle a wider range of real-world tasks with minimal human intervention.
Generalist AI was founded by Pete Florence, Andrew Barry, and Andy Zeng, all of whom previously worked at Google DeepMind and Boston Dynamics. The company has rapidly expanded its team and data collection network since its founding, with hundreds of camera-equipped grippers deployed globally. While Generalist AI has not disclosed its funding or valuation, its leadership’s track record and the scale of its research efforts have attracted significant attention from both academic and commercial robotics communities.
