Evaluating hallucinations in a language model with GraphEval

# Entry

Hallucinations this is one of the most celebrated problems immense language models (LLM) that you may experience when generating a response. They occur when the model generates an answer that is factually incorrect, nonsensical, or simply contrived, usually due to the model’s lack of internal knowledge on the subject.

Although many solutions to the problem of model hallucinations have emerged in recent years, the methodological assessment framework for internally diagnosing them has been relatively less explored. One recent research by Amazon researchers proposes the utilize of knowledge graphs as a means of analyzing and detecting hallucinations occurring in LLM. The framework presented in the study is called GraphEval.

In this article, we’ll take a gentle, hands-on approach to illustrating the conceptual building blocks of GraphEval with a simulation-based, lightweight code example that you can easily try out on your computer.

# GraphEval in a nutshell

GraphEval uses knowledge graphs to identify and flag hallucinations in the results generated by LLM. Unlike classic performance metrics that provide single scores to evaluate aspects such as accuracy, confidence, etc., GraphEval uses a two-step evaluation process that emphasizes explainability, namely providing insight into exactly where the hallucination occurred.

To do this, GraphEval takes into account two steps:

• Constructing a knowledge graph based on the generated model response. The graph consists of semantic triples of form (Topic, relationship, object)where entities and objects correspond to nodes and relations correspond to the edges connecting these nodes.
• Evaluating each triplet in the constructed knowledge graph in the source context (the body of factual knowledge) using a natural language inference (NLI) model. Any triple that, according to the NLI engine, cannot be inferred from the context – because it is contradictory or neutral – is marked as a hallucination.

# Illustrating GraphEval with a code example

Before running the code simulating the GraphEval framework application, let’s make sure that we have the necessary libraries installed:

!pip install -q transformers networkx matplotlib torch

The purpose of the code example we will discuss in a moment is to explain how the GraphEval methodology works, so we will replace steps that would be computationally intensive under real-world conditions with simulated, lightweight alternatives.

Therefore, we will simulate background knowledge (context), which is assumed to contain fact-based information. In a production environment, this basic knowledge would result, for example, from searching for relevant documents within the company vector database of the search assisted generation (RAG) system.. For simplicity, here we directly create the truth context and store it source_context.

# The ground-truth context provided to the LLM
source_context = (
    "GraphEval is a hallucination evaluation framework based on representing information "
    "in Knowledge Graph (KG) structures. It acts as a pre-processing step and utilizes "
    "out-of-the-box NLI models to detect factual inconsistencies."
)

Now let’s assume that below is LLM’s original response to a user prompt such as “explain concisely what GraphEval is”. To begin the first step of the assessment process, we would ask the adjunct LLM to build a knowledge graph based on this response. Both the response and the prompt used to obtain the knowledge graph are shown below:

# The generated response we want to evaluate (contains a hallucination)
llm_output = (
    "GraphEval is an evaluation framework that uses Knowledge Graphs. "
    "It requires a highly expensive, enterprise-level server farm to operate."
)

# Prompt template that would theoretically be passed to a local/free LLM (e.g. Mistral-7B)
KG_EXTRACTION_PROMPT = f"""
You are an expert information extractor. Extract the core information from the following text as a Knowledge Graph.
Return the output strictly as a Python list of tuples in the format: (Subject, Relationship, Object).

Text: {llm_output}
"""

Once again, for simplicity and to avoid the ponderous computational overhead of running a huge LLM locally, assume the following triple graphs are obtained:

# Simulated extraction to bypass the ponderous computational load of running a massive LLM locally
extracted_triples = [
("GraphEval", "is", "evaluation framework"),
("GraphEval", "uses", "Knowledge Graphs"),
("GraphEval", "requires", "expensive enterprise server farm")
]
print("Extracted Triples:")
for t in extracted_triples:
print

Extracted Triples:
('GraphEval', 'is', 'evaluation framework')
('GraphEval', 'uses', 'Knowledge Graphs')
('GraphEval', 'requires', 'pricey enterprise server farm')

from transformers import pipeline

# Loading the open-source NLI model
print("Loading DeBERTa NLI model...")
nli_evaluator = pipeline("text-classification", model="cross-encoder/nli-deberta-v3-small")

def evaluate_triple(context, triple):
    subject, relation, obj = triple
    hypothesis = f"{subject} {relation} {obj}"

    # Checking if the context entails the hypothesis
    result = nli_evaluator({"text": context, "text_pair": hypothesis})

    # NLI models normally output: 'entailment', 'neutral', or 'contradiction'
    label = result['label'].lower()

    # In GraphEval, anything other than 'entailment' is flagged as a hallucination
    is_hallucinated = label != 'entailment'

    return is_hallucinated, label, hypothesis

# Running the evaluation pipeline
evaluation_results = []

print("n--- GraphEval Results ---")
for t in extracted_triples:
    is_hallucinated, nli_label, hypothesis = evaluate_triple(source_context, t)
    evaluation_results.append((is_hallucinated, nli_label))

    status = "🚨 HALLUCINATION" if is_hallucinated else "✅ GROUNDED"
    print(f"{status} | Triple: {t} | NLI Output: {nli_label}")

--- GraphEval Results ---
✅ GROUNDED | Triple: ('GraphEval', 'is', 'evaluation framework') | NLI Output: entailment
✅ GROUNDED | Triple: ('GraphEval', 'uses', 'Knowledge Graphs') | NLI Output: entailment
🚨 HALLUCINATION | Triple: ('GraphEval', 'requires', 'pricey enterprise server farm') | NLI Output: neutral

import networkx as nx
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

def visualize_grapheval(triples, eval_results):
    G = nx.DiGraph()
    edge_colors = []

    for (triple, res) in zip(triples, eval_results):
        sub, rel, obj = triple
        is_hallucinated = res[0]

        G.add_node(sub)
        G.add_node(obj)
        G.add_edge(sub, obj, label=rel)

        # Color-code the edges based on the NLI evaluation
        edge_colors.append('red' if is_hallucinated else 'green')

    # Set up the plot
    plt.figure(figsize=(10, 6))
    pos = nx.spring_layout(G, seed=42)

    # Draw nodes
    nx.draw_networkx_nodes(G, pos, node_color="lightblue", node_size=2500)
    nx.draw_networkx_labels(G, pos, font_size=10, font_weight="bold")

    # Draw edges and labels
    nx.draw_networkx_edges(G, pos, edge_color=edge_colors, width=2.5, arrowsize=20)
    edge_labels = nx.get_edge_attributes(G, 'label')
    nx.draw_networkx_edge_labels(G, pos, edge_labels=edge_labels, font_color="black")

    # Add legend
    green_patch = mpatches.Patch(color="green", label="Grounded (Entailment)")
    red_patch = mpatches.Patch(color="red", label="Hallucination (Neutral/Contradiction)")
    plt.legend(handles=[green_patch, red_patch], loc="lower right")

    plt.title("GraphEval Hallucination Map", fontsize=14, fontweight="bold")
    plt.axis('off')
    plt.tight_layout()
    plt.show()

# Render the knowledge graph
visualize_grapheval(extracted_triples, evaluation_results)