At its core, machine learning is an experimental science. To innovate in AI, you must accept the possibility that common knowledge or methods that have worked in the past may not be the best way to solve fresh problems. It’s vital to rethink how you approach training data and evaluate performance metrics.
This isn’t always what teams want to hear when developing a fresh product; however, breakthroughs may be worth the extra days on the timeline. It’s a reminder of why many of us became data scientists, engineers, and innovators in the first place: we’re curious, and we’ll do whatever we can to solve even seemingly impossible problems.
I have witnessed the success of applying this concept firsthand with my team at Ultraleap, developing a variety of machine learning models that meet the tough hand tracking needs of businesses and consumers, creating the future of virtual interactions.
How challenges can become opportunities with machine learning (ML) experiments.
Many companies and industries have unique challenges in implementing machine learning that are not solved by the generic, one-size-fits-all solutions currently available on the market. This may be due to the complexity of their application domains, lack of budget and available resources, or being in a more niche market that may not attract the attention of enormous technology players. One such area is developing machine learning models for defect control in vehicle manufacturing. To be able to detect tiny defects over a enormous surface of a car on a moving assembly line, you have to cope with the limitation of low frame rate but high resolution.
My team and I face the same limitation when applying machine learning to hand tracking software – the resolution can be low, but the frame rate must be high. Hand tracking uses ML technology to identify human gestures, creating a more natural and realistic user experience in a virtual environment. The AR/VR headsets we develop this software for typically operate at the edge and have restricted computational power, so we cannot deploy machine learning models at mass. They must also respond faster than the speed of human perception. Moreover, given that this is a relatively juvenile space, we don’t have a lot of industry data to train on.
These challenges force us to be as artistic and curious as possible when developing hand tracking models – rethinking our training methods, questioning data sources, and experimenting not only with different approaches to model quantization, but also with compilation and optimization. We don’t stop at checking the model’s performance on a given dataset, we iterate on the data itself and experiment with model implementation. While this means that in the immense majority of cases we learn to solve “x” equations, it also means that our discoveries are even more valuable. For example, creating a system that can run with 1/100,000 of the processing power of, say, ChatGPT, while maintaining the imperceptibly low latency that makes your virtual hands precisely track your real hands. Solving these complex problems, while challenging, also gives us a commercial advantage – our tracking operates at 120 Hz compared to the 30 Hz norm, providing a better experience at the same power budget. This isn’t unique to our problems – many companies face specific challenges due to niche application domains that offer the tantalizing prospect of turning ML experiments into market advantage.
By its nature, machine learning is constantly evolving. Just like pressure creates diamonds, with enough experimentation we can create breakthrough machine learning solutions. However, as with any ML implementation, the basis of this experiment is data.
Evaluating data learning machine learning models
Artificial intelligence innovation often revolves around the model architectures used and the annotation, labeling, and cleaning of data. However, when solving sophisticated problems – for which prior data may be irrelevant or unreliable – this method is not always sufficient. In these cases, data teams need to innovate based on the data used for training. When training data, it is critical to evaluate what makes the data “good” for a particular apply case. If you can’t answer a question correctly, you need to approach your data sets differently.
While proxy metrics for data quality, accuracy, dataset size, model loss, and metrics are useful, there is always an element of the unknown that needs to be explored experimentally when it comes to training a machine learning model. At Ultraleap, we combine simulated and real data in a variety of ways, iterating on our datasets and sources and evaluating them based on the quality of the models they produce in the real world – we literally test them by hand. This has expanded our understanding of hand modeling for precise tracking regardless of the type of image and device being received, which is particularly useful when creating software compatible with XR headsets. Many headsets work with different cameras and chips, which means ML models must work with fresh data sources. Therefore, it is useful to have a diverse dataset.
If you want to explore all parts of the problem and all possible solutions, you must be open to the fact that your data may also be incomplete, and test your models in the real world. Our newest hand tracking platform, Hyperion, builds on our data evaluation and experimentation approach to provide a variety of hand tracking models to suit specific needs and apply cases, rather than a one-size-fits-all approach. Without escaping any part of the problem space, by questioning data, models, metrics and execution, we have models that are not only responsive and productive, but provide fresh capabilities such as tracking despite held objects or very tiny micro-gestures. Again, the message is that experimentation broad and deep can deliver unique product offerings.
Experimentation (from every angle) is key
The best discoveries require a tough fight; When it comes to true AI innovation, there is no substitute for experimentation. Don’t rely on what you know: answer questions by experimenting with a real application domain and measuring the model’s performance relative to your task. This is the most vital way to ensure that ML tasks are translated into specific business needs, expanding the scope of innovation and giving your organization a competitive advantage.
about the author