OpenAI is also apparently ramping up its own robotics efforts. Last week, Caitlin Kalinowski, who previously led the development of virtual and augmented reality headsets at Meta, announced on LinkedIn that she joined OpenAI to work on hardware, including robotics.
Lachy Groom, a friend of OpenAI CEO Sam Altman and an investor and co-founder of Physical Intelligence, joins the team in the conference room to discuss the business side of the plan. The groom is wearing an expensive-looking hoodie and appears extremely youthful. He emphasizes that physical intelligence has plenty of potential to make a breakthrough in robot learning. “I just talked to Kushner,” he says, referring to Joshua Kushner, founder and managing partner of Thrive Capital, which led the startup’s seed investment round. He is, of course, also the brother of Donald Trump’s son-in-law, Jared Kushner.
Several other companies are currently pursuing a similar breakthrough. One of them, called Skild, founded by roboticists from Carnegie Mellon University, raised $300 million in July. “Just as OpenAI built ChatGPT for language, we are building a general-purpose brain for robots,” he says Deepak PathakCEO of Skild and adjunct professor at CMU.
Not everyone is sure this can be achieved in the same way that OpenAI cracked the AI language code.
There is simply no online repository of robotic activities similar to the text and graphics data available for LLM training. Either way, achieving breakthroughs in physical intelligence may require exponentially more data.
“Words in a sequence are, dimensionally speaking, a small toy compared to all the movement and activity of objects in the physical world,” says Illah Nourbakhsh, a roboticist at CMU who is not affiliated with Skild. “The degrees of freedom we have in the physical world are much more than just the letters of the alphabet.”
Ken Goldberg, a research fellow at the University of California, Berkeley who works on applying artificial intelligence to robots, warns that excitement around the idea of a revolution of data-powered robots and humanoids is reaching hype proportions. “To achieve the expected level of performance, we will need ‘good old engineering’, modularity, algorithms and metrics,” he says.
Russ Tedrakecomputer scientist at the Massachusetts Institute of Technology and vice president of robotics research at the Toyota Research Institute, says the success of LLM programs has prompted many roboticists, including himself, to rethink research priorities and focus on finding ways to continue learning through robots in a more ambitious scale. However, he admits that we still face huge challenges.