An introductory guide to practical decoding of constraints

# Entry

Practical decoding of constraintsalso known as structured generation or guided decoding, involves engineering strategies to force a enormous language model (LLM) to produce text output that strictly follows a specific data schema, grammar, or regular expression() at the level token selection scene.

With this article’s introductory guide to practical constraint decoding, you’ll no longer have to beg your model to “output valid JSON without including any reductions,” just provide an example. Constraint decoding makes it mathematically impossible for LLM to deliver anything beyond the defined constraints.

# How does practical constraint decoding work?

While the typical LLM generation process acts as a “leap of faith” in which you pass a hint to the model and it may get exactly what you’re looking for (or not), practical constraint decoding requires a subtly different approach. It treats prompting and text generation as a unique, interwoven program. This allows you to lock out certain characters that are crucial to maintaining the specific required syntax, allowing the model to “fill in the blanks” between them.

How about we go into more detail? When LLM sends another token of its response, it initially creates a vector of raw results, or logits — one for each possible tag in the available vocabulary. This usually means thousands of possible options to choose from.

But when using practical constraint decoding, something happens before the inference process starts: a finite-state machine is builtduring which the target constraint is compiled – for example using the Pydantic model in Python. At a given inference stage, the finite state machine evaluates the current state and provides: list of allowed subsequent tokens. This “white list” exists used as a mask on the raw LLM logit vectorso that for any token outside this list its logit is set to negative infinity, i.e -inf in Python.

After the masking process, the model normally performs a softmax normalization and sampling process (based on parameters such as temperature, top-p or top-k) on the “surviving tokens” to finally select the most likely one and generate it.

It may seem that applying this process to an entire vocabulary of many thousands of words would significantly snail-paced down model inference. Good news: this is not the case. Contemporary Python libraries take the unchanging LLM vocabulary and pre-compile it before the user starts typing the prompt. The state machine will not have to search through the entire vocabulary and the latency overhead will be drastically reduced.

So what is the current gold standard for implementing practical constraint decoding? Probably, guidelines the library deserved this distinction. It allows us to define and pass Pydantic models, JSON schemas, or directly to a wrapped version of a pre-trained model, thus limiting its freedom in generating results.

# Example

Let’s go through an example. First install the outlines:

pip install outlines[transformers]

Now let’s move on to the code:

from pydantic import BaseModel
import outlines
from transformers import AutoTokenizer, AutoModelForCausalLM

class UserProfile(BaseModel):
    name: str
    age: int
    is_active: bool

model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

llm = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = outlines.from_transformers(llm, tokenizer)
result = model("Extract the user: John is a 34 year old pilot.", UserProfile)

print(result)

Exit:

{"name": "John", "age": 34, "is_active": true}

This example showed how to exploit the outline library to wrap a pre-trained model along with its tokenizer and restrict it to output JSON objects defined by a custom class we defined – called UserProfile and Pydantic inheritance BaseModel.

# Summary

Strengths:

• If used correctly, it provides a 100% guarantee of correct syntax, eliminating the need to parse blocks in the code.
• This helps dramatically save tokens in tooltips, no longer requiring several token-consuming examples to, for example, show the model what a valid JSON object should look like.
• It helps democratize diminutive models by transforming a “small” 1B parameter model, which would otherwise compromise JSON generation exploit cases, into a reliable data builder.

Limitations:

• If LLM needs to say it can’t answer something, but the schema forces it to generate an integer, for example, it will do so and will no longer be sincere in edge cases.
• When first running the Pydantic vs. LLM schema, you may experience a freeze for a few seconds while building the finite state machine, making the first run much slower, although subsequent runs will be smoother.

This article presents practical constraints decoding, exploring why it is necessary in certain LLM-based situations, how it works, and what is the most commonly used solution in the current landscape: the contour library. An example of its exploit is also provided.

Ivan Palomares Carrascosa is a thought leader, writer, speaker and advisor in the fields of Artificial Intelligence, Machine Learning, Deep Learning and LLM. Trains and advises others on the exploit of artificial intelligence in the real world.