2023 was the year of generative AI breakthrough year—where organizations have begun looking for ways to integrate AI into every aspect of their technology and operations.
But as companies start taking a closer look at their AI deployments in the second half of 2024, the large question won’t be what they can do with the technology, but how much will it all cost? Because there’s no single overarching AI strategy, there’s often confusion about the overall price tag.
By understanding what type of AI you want to train, what your latency requirements are, how much training data you need, and what external data you will need, you can ensure that your business can innovate without breaking the bank.
Understanding the type of AI you are training
Knowing how elaborate the problem you’re trying to solve has a huge impact on the compute resources you need and the costs, both in training and implementation. Given the breadth of AI projects, from training chatbots to autonomous cars, understanding the models you’re working with and the resources required will be crucial to aligning costs with expectations.
AI tasks are hungry in every sense: they require a lot of processing power, storage capacity, and specialized hardware. As you escalate or decrease the complexity of the task at hand, you can run up a huge bill for sourcing components, such as the most desirable hardware — for example Nvidia A100 operates at a speed of approximately $10,000 per chipAnother example is the need to determine whether your project requires a completely recent model or an improvement on existing open source versions; both types of projects will have radically different budgets.
Storing training data
AI training requires a lot of data, and while it’s demanding to estimate, we can estimate that a immense AI model will require at least tens of gigabytes of data, and at most petabytes. For example, OpenAI is estimated to apply between 17 GB to 570 GB to 45 TB text data (OpenAI considers the actual size of the database to be proprietary information.) How large a dataset you need is a heated area of research right now, as is the number of parameters and hyperparameters. A general rule of thumb is that you need to have 10 times as many examples as parametersAs with all things AI, the apply case has a major impact on the amount of data needed, the number of parameters and hyperparameters included, and how these two things interact over time.
Delay Requirements
When considering the overall cost of building AI, it is also significant to recognize the amount of both indefinite and short-lived storage that is needed. Throughout the training process, the underlying dataset is constantly being transformed and therefore broken into pieces. Each of these subsets will need to be stored separately. Even if inference based on an already trained modelwhich will be the primary apply of the model once deployed, the time it takes the model is affected by caching, processing, and latency.
The physical location of data storage affects how quickly tasks can be completed. One way to solve this problem is to create short-lived storage on the same chips on which the processor is executing the task. Another way to solve this problem is to keep the entire cluster of processing and storage in one place in the data center and closer to the end user, as they do in TritonGPT at UC San Diego.
Incorporating third-party assistance
Once you have determined the specific needs of any AI project, one question you need to ask yourself is whether you need outsourcing aid. Many companies have developed existing models or are providers who can deliver the results you need at a fraction of the cost you would incur if you went it alone.
A good place to start is the open source community. Face hugging to see if its wide range of models, datasets and no-code tools can aid you. On the hardware side, there are specialist services such as Core fabric that offer effortless access to advanced GPUs at a much lower cost than time-honored vendors, or allow you to build your own system from scratch.
Savings on AI spending can add up
Keeping up with the ever-changing and evolving AI innovation industry doesn’t have to be hard. But as with previous cloud and large data hype cycles, investing without a clear understanding or direction can lead to overspending.
While it’s stimulating to speculate about when the industry will reach artificial general intelligence (AGI) or how to access the most powerful chips, don’t forget that the costs associated with implementation will be just as significant in determining how the industry will evolve. Understanding the most cost-effective options for developing AI solutions now will aid you plan for future AI innovation resources in the long term.
about the author