Photo by the author ideogram.Ai# Entry
With the boost in enormous language models (LLM), many applications powered by LLM have appeared in recent years. The implementation of LLM introduced functions that did not exist before.
Over time, many LLM models and products have become available, each with advantages and disadvantages. Unfortunately, there is still no standard way to access all these models, because each company can develop its own framework. Therefore, having an open source tool, such as Liteellm It is useful when you need normalized access to LLM application without any additional costs.
In this article, we will examine why Liteellm is beneficial to build LLM application.
Let’s get it.
# Benefit 1: unified access
The biggest advantage of Liteellm is its compliance with various models suppliers. The tool supports over 100 different LLM services via normalized interfaces, which allows us to access them, regardless of the model supplier used. This is especially useful if your applications employ many different models that must work interchangeably.
Several examples of the main models suppliers supported by Liteellm include:
- OpenAI and Azure OpenAI, like GPT-4.
- Anthropic, like Claude.
- AWS Bedrock & Sagemaker, supporting models such as Amazon Titan and Claude.
- Google Vertex AI, like Gemini.
- Hugging Face Hub and Ollam for Open Source models such as Llama and Mistral.
The standardized format is compatible with the OPENAI Framework, using the chat/completion scheme. This means that we can easily switch models without having to understand the pattern of the original model supplier.
For example, this is a Python code that uses the Google Gemini model with Litellm.
from litellm import completion
prompt = "YOUR-PROMPT-FOR-LITELLM"
api_key = "YOUR-API-KEY-FOR-LLM"
response = completion(
model="gemini/gemini-1.5-flash-latest",
messages=[{"content": prompt, "role": "user"}],
api_key=api_key)
response['choices'][0]['message']['content']You just need to get the model name and appropriate API keys with model supplier To access them. This flexibility makes Liteellm ideal for applications that employ many models or for models comparisons.
# Benefit 2: costs and optimization
When working with LLM applications, it is essential to track the employ of tokens and expenses for each implemented model and in all integrated suppliers, especially in real -time scenarios.
Litellm allows users to maintain a detailed employ log on the API interface connections, providing all necessary information for effective cost control. For example, the above “ending” connection will contain information about the employ of a token, as shown below.
usage=Usage(completion_tokens=10, prompt_tokens=8, total_tokens=18, completion_tokens_details=None, prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=None, cached_tokens=None, text_tokens=8, image_tokens=None))Access to hidden response parameters will also provide more detailed information, including costs.
With a output similar to below:
{'custom_llm_provider': 'gemini',
'region_name': None,
'vertex_ai_grounding_metadata': [],
'vertex_ai_url_context_metadata': [],
'vertex_ai_safety_results': [],
'vertex_ai_citation_metadata': [],
'optional_params': {},
'litellm_call_id': '558e4b42-95c3-46de-beb7-9086d6a954c1',
'api_base': 'https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash-latest:generateContent',
'model_id': None,
'response_cost': 4.8e-06,
'additional_headers': {},
'litellm_model_name': 'gemini/gemini-1.5-flash-latest'}There is a lot of information, but the most essential element is “Response_Cost” because it estimates the actual fee that you will incur during this connection, although it can still be balanced if the model provider offers free access. Users can also define Non -standard prices for models (for token or for a second) to accurately calculate the costs.
More advanced Cost tracking The implementation will also allow users to set up issuing a budget and limitAt the same time, connecting information about Liteellm costs with an analytical desktop to easier to aggregate information. It is also possible to provide non -standard label markers that will assist to assign costs to some employ or departments.
By ensuring detailed data on the employ of costs, Liteellm helps users and organizations optimize their costs and the LLM application budget.
# Benefit 3: ease of implementation
Litellm is intended for uncomplicated implementation, regardless of whether you employ it for local development or the production environment. Thanks to the modest resources required for the installation of the Python library, we can start Liteellm on our local laptop or host it in the implementation in a container with docker without the need for additional configuration.
Speaking of configuration, we can more effectively configure Liteellm using the YAML configuration file to mention all the necessary information, such as the model name, APi keys and all necessary non -standard LLM application settings. You can also employ a facilities database such as SQLITE or Postgresql to store its condition.
In the case of data privacy, you are responsible for your own privacy as a user implementing Liteellm, but this approach is safer because the data never leave a controlled environment, except for sending to LLM suppliers. One of the functions that Liteellm provides users of the corporate is single login (SSO), access control based on roles and audit diaries, if the application needs a safer environment.
In general, Litellm provides adaptable implementation options and configuration, while ensuring data security.
# Benefit 4: immunity functions
Resistance is crucial when building LLM applications, because we want our application to remain working even in the face of unexpected problems. To promote immunity, Litellm provides many functions that are useful in creating the application.
One of the functions that Liteellm has is built -in bufferingWhere users can buffer LLM hints and answers so that identical demands do not incur repetitive costs or delays. This is a useful function if our application often receives the same queries. The buffering system is adaptable, supporting both memory buffering and remote, for example in the case of vector database.
The next feature of Litellm is Automatic again againenabling users to configure the mechanism when the demands fail due to errors, such as the time limit or errors to limit the rate in order to automatically re -re -re -re -re -re -re -re -re -re -reassemble again again again again, it is also possible to configure additional re -configuration Failure mechanismssuch as using another model if the request has already reached the re -payment limit.
Finally, we can set a speed limit for specific demands per minute (rpm) or tokens per minute (TPM) to limit the level of employ. This is a great way to limit the integration of specific models to prevent failure and respect the application infrastructure requirements.
# Application
In the era of product growth, LLM has become much easier to build LLM applications. However, with so many models suppliers, it is hard to determine the standard of implementing LLM, especially in the case of multi -model system architecture. That is why Liteellm can assist us build LLM applications effectively.
I hope it helped!
Cornellius Yudha Wijaya He is a data assistant and data writer. Working full -time at Allianz Indonesia, he loves to share Python and data tips through social media and media writing. Cornellius writes on various AI topics and machine learning.