5 tips for building optimized monsters transformer hugging

Photo via editor Chatgpt

# Entry

Hugging It has become a standard for many AI programmers and scientists from data, because it drastically lowers the barrier in working with advanced AI. Instead of working with AI models from scratch, developers can access a wide range of preferred models without trouble. Users can also adapt these models to custom data sets and quickly implement them.

However, working with Transformers streams can become disordered and may not give an optimal pipeline. Therefore, we will examine five different ways to optimize transformer streams.

Let’s get it.

# 1. Requesting the party to apply

Often, when using transformers pipelines, we do not fully operate the graphics processing (GPU) unit. The processing of lots of input data can significantly boost the operate of GPU and boost the efficiency of the application.

Instead of processing one sample at once, you can operate a pipeline batch_size Parameter or transfer the list of inputs, so the model processes several input data in one pass forward. Here is an example of the code:

from transformers import pipeline

pipe = pipeline(
    task="text-classification",
    model="distilbert-base-uncased-finetuned-sst-2-english",
    device_map="auto"
)

texts = [
    "Great product and fast delivery!",
    "The UI is confusing and slow.",
    "Support resolved my issue quickly.",
    "Not worth the price."
]

results = pipe(texts, batch_size=16, truncation=True, padding=True)
for r in results:
    print(r)

By party demands, you can achieve a higher bandwidth with a minimal effect on delay.

# 2. Exploit lower precision and quantization

Many preferred models fail in the application because the development and production environments do not have enough memory. Lower numerical precision helps reduce memory consumption and accelerates inference without devoting high accuracy.

For example, here, how to operate half the GPU precision in the Transformers pipeline:

import torch
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    torch_dtype=torch.float16
)

Similarly, quantization techniques can compress the mass masses without noticeably degrading performance:

# Requires bitsandbytes for 8-bit quantization
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_8bit=True,
    device_map="auto"
)

The operate of lower precision and quantization in production usually accelerates pipelines and reduces memory consumption without a significant impact on the accuracy of the model.

# 3. Choose productive model architecture

In many applications you do not need the largest model to solve the task. Choosing a transformer’s lighter architecture, such as a distilled model, often gives a better delay and bandwidth with an acceptable compromise of accuracy.

Compact models or distilled versions such as DistilbertKeep most of the accuracy of the original model, but with a much smaller number of parameters, which causes faster inference.

Choose a model whose architecture is optimized in terms of inference and corresponds to the requirements of the accuracy of the task.

# 4. Lever buffering

Many systems are wasting their face, repeating costly work. Buffing can significantly boost efficiency by reusing the results of costly calculations.

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=False,
        use_cache=True
    )

Effective buffering reduces the time of calculation and improves response time, reducing delay in production systems.

# 5. Exploit an accelerated executive environment through optimal

Many pipelines work in Pythorch Not very optimal mode that adds copies of Python and additional copies of memory. Using Optimum with an open environment of the neural network round (ONNX) – VIA Onnx Runtime – converts the model to the stationary chart and combines operations, thanks to which the performance time can operate faster testicles on the central processing unit (CPU) or GPU with a lower cost. The result is usually faster inference, especially on the processor or mixed equipment, without changing the way the pipeline is called.

Install the required packages with:

pip install -U transformers optimum[onnxruntime] onnxruntime

Then convert the model with this code:

from optimum.onnxruntime import ORTModelForSequenceClassification

ort_model = ORTModelForSequenceClassification.from_pretrained(
    model_id,
    from_transformers=True
)

By transforming the pipeline for the time of ONNX by optimal, you can keep the existing pipeline code, while obtaining a lower delay and more productive inference.

# Wrapping

Transformers Pipelines is a pack of API in the Hulging Face framework, which facilitates the development of the AI ​​application by condensing the sophisticated code into simpler interfaces. In this article, we examined five tips to optimize the streams of hugging the face, from the requests of the party to apply, to the selection of productive model architecture, to the operate of buffering and more.

I hope it helped!

Cornellius Yudha Wijaya He is a data assistant and data writer. Working full -time at Allianz Indonesia, he loves to share Python and data tips through social media and media writing. Cornellius writes on various AI topics and machine learning.