Join our daily and weekly newsletters to receive the latest updates and exclusive content on our industry-leading AI coverage. Find out more
Meta AI announced the release of open source software MobilnyLLMa set of mobile-optimized language models, with checkpoints and model code now available in Hugging Face. However, currently it is only available under the Creative Commons 4.0 non-commercial licensewhich means companies cannot operate it in commercial products.
Originally described in a research article published in July 2024 and covered by VentureBeat, MobileLLM is now fully available with open scales, marking a significant milestone in high-performance on-device AI.
The introduction of these open scales makes MobileLLM a more direct, if roundabout, competitor to Apple Intelligence, Apple’s hybrid on-device/private cloud AI solution composed of multiple models and shared with users iOS 18 operating system in the US and outside the EU this week. However, since it is narrow to research uses and requires download and installation from Hugging Face, it will likely remain narrow to IT and academic audiences for now.
Greater performance on mobile devices
MobileLLM aims to address the challenges of deploying AI models on smartphones and other resource-constrained devices.
With parameter counts ranging from 125 million to 1 billion, these models are designed to operate within the narrow memory and power capacity typical of mobile hardware.
Meta’s research suggests that by emphasizing architecture over sheer size, well-designed compact models can deliver solid AI performance directly on devices.
Troubleshooting scaling issues
MobileLLM’s design philosophy deviates from classic AI scaling laws that emphasize breadth and a gigantic number of parameters.
Instead, Meta AI research focuses on deep, gaunt architectures to maximize performance and improve how abstract concepts are captured in the model.
Yann LeCun, Chief Artificial Intelligence Officer at Meta, emphasized the importance of these in-depth strategies in enabling advanced AI on everyday hardware.
MobileLLM includes several innovations designed to augment the efficiency of smaller models:
• Depth over width: The models operate deep architectures, which perform better against broader but shallower architectures in small-scale scenarios.
• Embedding sharing techniques: They maximize weight efficiency, key to maintaining the model’s compact architecture.
• Group inquiry Note: Inspired by the work of Ainslie et al. (2023), this method optimizes attention mechanisms.
• Instant weight sharing by block: A novel strategy to reduce latency by minimizing memory traffic, helping maintain execution performance on mobile devices.
Performance metrics and comparisons
Despite their diminutive size, MobileLLM models perfectly cope with benchmark tasks. The 125 million and 350 million parameter versions show accuracy improvements of 2.7% and 4.3% compared to the previous state-of-the-art models (SOTA) on zero tasks.
Interestingly, the 350M version matches API call performance to the much larger Meta Llama-2 7B model.
These benefits show that well-designed smaller models can effectively handle intricate tasks.
Designed for smartphones and edge devices
The release of MobileLLM is part of Meta AI’s broader efforts to democratize access to advanced artificial intelligence technology.
As demand for AI on devices grows due to cloud costs and privacy concerns, models like MobileLLM will play a key role.
The models are optimized for devices with memory limitations of 6-12 GB, making them practical for integration with popular smartphones such as iPhone and Google Pixel.
Open but non-commercial
Meta AI’s decision to open source MobileLLM reflects the company’s stated commitment to collaboration and transparency. Unfortunately, the license conditions prohibit commercial use for nowso only researchers can benefit from it.
By providing both model weights and pre-training code, they invite the research community to expand and refine their work.
This could accelerate innovation in diminutive language models (SLM), making high-quality AI available without having to rely on extensive cloud infrastructure.
Developers and researchers interested in testing MobileLLM can now access models on Hugging Face, fully integrated with the Transformers library. As these compact models evolve, they promise to redefine how advanced artificial intelligence works in everyday devices.