Novel type The gigantic language model, developed by scientists from the Allen Institute for AI (AI2), allows you to control the way of using training data even after building the model.
The up-to-date model, called Flexolmo, can challenge the current industry paradigm of gigantic artificial intelligence companies that withdraw data from the network, books and other sources – often with little respect for property – and then completely having resulting resulting models. When the data is baked today in the AI model, separating it from this model is a bit like an attempt to recover eggs from ready dough.
“Conventionally your data is in or outside,” says Ali Farhadi, CEO AI2 based in Seattle, Washington. “When I train this data, you lose control. And you have no choice, unless you force me to go through the next round of training worth many millions of dollars.”
The AI2 AI2 approach divides the training so that data owners can exert control. Those who want to bring data to the Flexolmo model can do it, first copying a publicly shared model known as “anchor”. Then they train the second model using their own data, combine the result with the anchor model and contribute to who is building the third and last model.
In this way, contributing means that the data themselves never have to be transferred. And due to the method of connecting the data owner model with the final, it is later possible to separate the data. The publisher of the magazine may, for example, bring the text from the archive of articles to the model, but later remove the submodel trained on the basis of this data, if there is a legal dispute or if the company opposes the operate of the model.
“The training is completely asynchronous,” says Sewon Min, a scientist from AI2, who managed technical work. “Data owners do not have to coordinate and the training can be conducted completely independently.”
The architecture of the Flexolmo model is the so -called “mix of experts”, a popular project, which is usually used to simultaneously combine several submits into a larger, more talented one. The key innovation of AI2 is a way to combine the subathmal that has been trained independently. This is achieved using a up-to-date scheme for representing values in the model so that his skills can be combined with others when the final model is launched.
To test the approach, Flexolmo researchers have created a set of data, which they call Flexmix from reserved sources, including books and websites. They used the Flexolmo project to build a model with 37 billion parameters, about one tenth size of the largest open source model with meta. Then they compared their model to several others. They discovered that this exceeded each individual model in all tasks, and also obtained 10 percent better in common references than two other approaches to the connection of independently trained models.
The result is a way to have a dough – and recover eggs. “You can simply give up the system without much damage and time of inference,” says Farhadi. “This is a completely new way of thinking about training these models.”