Generating a structured language model with outlines

# Entry

Usually, when asking for an LLM – an abbreviation of “The big language model“- too neat, structured output like JSON objects, for example, it requires a combination of careful, speedy creation with a “pinch” of luck. Otherwise, it may be challenging for the model to produce the perfectly ordered result you want. Or at least it was like that until the novel was written open source library came on stage: guidelines.

This library is designed to prevent common problems experienced by LLM in these specific results-oriented operate cases, such as hallucinations. More specifically, it introduces a degree of deterministic certainty into the process of generating results.

Let’s discover what outlines allows us to do this through this illustrative article where we will show you some practical examples in Python!

# Employ case 1: Multiple choice classification for sentiment analysis

Before you fully dive into your first operate case, you may be wondering. How does this happen? outlines does it work and how does it guarantee the correctness of the results of the structural model? At the inference level, it masks “syntactically illegal” tokens during generation, rather than trying to fix bad text once generated. This makes it virtually impossible to break the rules underlying the specific output format you are looking for.

Let’s see the first example where we are building an analytics pipeline for customer service tickets and we want Exactly one option from a constrained, approved list of possible options. This is a bit like the problem of classification i generate.choice() is a function that helps us mimic this by forcing the model to choose one of the predefined literals or classes.

But first let’s install it next to it transformers to load pre-trained LLM:

pip install outlines[transformers]

The code below uses outlines.from_transformers() to load a pre-trained model using Hugging Face auto classes for the model and its associated tokenizer. But the icing on the cake is that they are both wrapped in outlines an object that will later facilitate the model tell what exactly to get. At the inference stage, we not only provide the user with a prompt asking them to classify the review, but also: Literal an object containing output constraints that the model should limit to:

import outlines
from transformers import AutoTokenizer, AutoModelForCausalLM
from typing import Literal

# 1. Loading the backend using standard Transformer-based models
model_name = "microsoft/Phi-3-mini-4k-instruct"

# We operate outlines to load the model with its from_transformers() function
model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained(model_name),
    AutoTokenizer.from_pretrained(model_name)
)

# 2. Calling the model directly, passing our approved strings as type constraints
sentiment = model(
    "Classify the sentiment of this customer review: 'I've been waiting two weeks for my delivery and it's still missing.'",
    Literal["Positive", "Negative", "Neutral"]
)

print(sentiment)

Exit:

A word of warning here: although the literal we defined is part of Python’s built-in typing module, and not outlinesour ready-to-use library still takes control of the model here: both the model and the tokenizer are wrapped in an object that enforces standard Python types, building a finite state machine under the hood that limits the output to only the options provided.

# Employ case 2: Generating a JSON object

This example first defines a Pydantic object, which defines the desired structure of a JSON object describing a fictional character with name, description, and age. Then it uses our previously packed ones outlines model by passing a character object to ensure that the generated output strictly follows the following structure for the requested JSON object:

from pydantic import BaseModel

# 1. Define a Pydantic model for the desired JSON structure
class Character(BaseModel):
    name: str
    description: str
    age: int

# 2. Using the outlines-wrapped model to generate a JSON output conforming to the Pydantic model
json_output = model(
    "Generate a JSON object describing a fictional character named 'Anya'.",
    Character,
    max_new_tokens=200
)

print(json_output)

Exit:

{ "name": "Anya", "description": "Anya is a young, adventurous woman with a passion for exploring new places and meeting new people. She has long, curly hair and bright green eyes that sparkle with curiosity. Anya is always eager to learn and loves to share her knowledge with others. She is kind-hearted and always willing to lend a helping hand to those in need. Anya's favorite hobbies include hiking, reading, and playing the guitar. She is a free spirit who values freedom and independence above all else." ,"age": 25 }

# Employ Case 3: Generate pure JSON for REST APIs

This third example, also related to JSON, is similar to the previous one, but in a slightly different context. Imagine you are creating an API that requires a well-defined JSON payload to update the database. Asking a standard LLM to get this result most often results in annoying trailing characters such as commas that can crash the JSON parser.

Using outlines, we redefine our JSON payload schema with a custom Pydantic-based class object.

from pydantic import BaseModel
from typing import Literal
import json

class ServerHealth(BaseModel):
    service_name: str
    uptime_seconds: int
    status: Literal["OK", "DEGRADED", "DOWN"]

# 1. Outlines should produce a raw string guaranteed to be valid JSON
raw_json_string = model(
    "Report the current status of the main Auth database.",
    ServerHealth,
    max_new_tokens=50
)

print(type(raw_json_string))  # This will just print: 

# 2. Pretty-printing
parsed_json = json.loads(raw_json_string)
print(json.dumps(parsed_json, indent=2))

Exit:

{
  "service_name": "auth_db_status",
  "uptime_seconds": 1623456789,
  "status": "OK"
}

# Final remarks

Since LLMs are trained to be chat aficionados who can break syntax or hallucinate to “sound human” in their conversations with us, getting them to produce reliable, structured output like pure JSON objects can seem a bit of a chore. Introduces a modern open-source library that introduces deterministic assurance to the LLM output generation process for better, more reliable generation of structured output. This article shows three straightforward but useful operate cases for this compelling tool for beginners.

Ivan Palomares Carrascosa is a thought leader, writer, speaker and advisor in the fields of Artificial Intelligence, Machine Learning, Deep Learning and LLM. Trains and advises others on the operate of artificial intelligence in the real world.