Differential adaptive linking accelerates enterprise SLM

Model combining is a fundamental AI process that enables organizations to reuse and combine existing trained models to achieve specific goals.

Today, enterprises can employ model coupling in a variety of ways, but many of the approaches are convoluted. A modern approach, the so-called Differential adaptive coupling (DAM) may be the answer to current model fusion challenges. DAM offers an novel solution for connecting AI models while potentially reducing computational costs.

Arcee A.Ibusiness focusing on capable, specialized diminutive language models, leads DAM research. The company, which raised funding in May 2024, has evolved from providing model training tools to become a full-fledged model delivery platform offering both open source and and commercial offers.

How DAM creates a modern path for connecting models

Combining can aid companies combine models specialized in different areas to create a modern model capable of operating in both areas.

The basic concept of data linkage is very well understood when it comes to structured data and databases. However, model linking is more abstract than structured data linking because the models’ internal representations are not as uncomplicated to interpret.

Thomas Gauthier-Caron, research engineer at Arcee AI and one of the authors of the DAM study, explained to VentureBeat that classic model fusion often relied on evolutionary algorithms. This approach can potentially be sluggish and unpredictable. DAM takes a different approach by leveraging established machine learning (ML) optimization techniques.

Gauthier-Caron explained that the goal of DAM is to solve the complexity of the model linking process. The company’s existing library, MergeKit, is useful for combining different models, but is convoluted due to different methods and parameters.

“We wondered, can we make it easier, can we have the machine optimize it for us rather than sitting in the weeds and tweaking all these parameters?” Gauthier-Caron said.

Instead of simply mixing models directly, DAM adapts based on the contribution of each model. DAM uses scaling factors for each column of the model weight matrix. It automatically learns the best settings for these coefficients by testing the performance of the combined model, comparing the output with the original models, and then adjusting the coefficients to obtain better results.

According to research, DAM performs competitively or better than existing methods such as evolutionary pooling, DARE-TIES, and Model soups. According to Gauthier-Caron, the technology represents a significant departure from existing approaches. He described evolutionary merging as a sluggish process in which it is not entirely clear how good the result will be and how long the merging process should take.

Combining is not an approach based on combining experts

Data scientists combine models in many different ways. One increasingly popular approach is mix of experts (MoE).

Gauthier-Caron emphasized that connecting models to DAM is completely different from MoE. He explained that MoE is a specific architecture that can be used to train language models.

The basic concept of model linking is that it starts from a point where the organization already has trained models. Training these models usually costs a lot of money, so engineers try to reuse existing trained models.

Practical applications and benefits of DAM for enterprise AI

One of the key advantages of DAM is the ability to effectively combine specialized models.

One such example given by Gauthier-Caron is a situation where an organization wanted to combine the Japanese model with a mathematical model. The goal of this combination is to create a model that can handle Japanese math well without requiring retraining. This is one area where DAM could potentially excel.

This technology is particularly critical when implementing generative artificial intelligence in enterprises, where efficiency and cost considerations are paramount. Helping create more capable ways of doing things at reduced costs is a key goal of Arcee. Therefore, DAM research is critical both for the company and, ultimately, for its users.

“Enterprise AI adoption comes down to performance, availability, scalability and cost,” Mark McQuade, co-founder and CEO of Arcee AI, told VentureBeat.