Why does AI hallucinate?

And how Documentation-First Approach reduces hallucinations from 60.4% to 0.9% — a 98.5% reduction

Based on information theory (Shannon, 1948), Transformer architecture (Vaswani et al., 2017) and recent research on context degradation (Liu et al., 2023)

The problem: probabilistic nature

language models are probabilistic systems, not deterministic

The mathematics of failure

Each generated token is a probabilistic decision. The probability of a complete sequence is the product of individual probabilities:

P(sequence) = P(t₁) × P(t₂|t₁) × P(t₃|t₁,t₂) × ... × P(tₙ|t₁,...,tₙ₋₁)

Even with 80% accuracy per token (optimistic):

P(1000 correct tokens) = 0.8¹⁰⁰⁰ ≈ 10⁻⁹⁷

→ Essentially zero. This is why generated code "almost works" but contains subtle bugs.

Uncertainty accumulation

• Each incorrect token propagates errors to the next ones

• Bad initial decisions affect entire generation

• Ambiguous context = flattened probability distribution

• Result: unpredictable and inconsistent outputs

The root cause: ambiguity

when specifications are vague, the space of possibilities explodes exponentially

Example: "Create an authentication system"

Authentication method:

Username/password • OAuth2 • JWT • SAML • Magic links • Biometric = 6 options

Session storage:

In-memory • Redis • Database • JWT stateless = 4 options

Password hashing:

bcrypt (3 configs) • argon2 (3 configs) • scrypt (3 configs) = 9 options

Rate limiting:

By IP / By user × Sliding window / Fixed window = 8 options

Combinatorial explosion

6 × 4 × 9 × 8 = 1,728 possible implementations

For a typical specification with 20 decision points, each with 10 interpretations:

10²⁰ = 100,000,000,000,000,000,000 possibilities

For context: there are ~10¹⁸ grains of sand on Earth, ~10²⁴ stars in the observable universe

The science: information theory

Claude Shannon (1948) provided the mathematical framework to quantify ambiguity

Shannon Entropy

Entropy H measures uncertainty in bits:

H(X) = -Σᵢ P(xᵢ) × log₂(P(xᵢ))

The higher the entropy, the greater the uncertainty. Greater uncertainty = greater chance of hallucination.

Specifications: vague vs deterministic

❌ Vague specification

"Create an authentication endpoint"

REST: 30%

GraphQL: 20%

WebSocket: 15%

gRPC: 10%

Others: 25%

H ≈ 2.8 bits

High uncertainty

✅ Deterministic specification

"POST /api/auth/login accepts JSON with email and password"

REST: 95%

GraphQL: 2%

Others: 3%

H ≈ 0.35 bits

Low uncertainty

Reduction of 8× in entropy = 8× more chance of generating correct code

Attention complexity O(n²)

Vaswani et al. (2017) - "Attention is All You Need"

The attention mechanism in Transformers has quadratic complexity. For large contexts, this causes degradation:

Spec 5,000 lines (~50,000 tokens):

→ 50,000² = 2.5 billion attention operations→ Attention dispersion + "Lost in the Middle"→ 60% loss in the middle of context

"Lost in the Middle" (Liu et al., 2023)

Research empirically demonstrated that LLMs lose performance when relevant information is in the middle of long contexts:

• Information at the beginning: 80-90% accuracy

• Information at the end: 80-90% accuracy

• Information in the middle: 20-40% accuracy

→ Large contexts = lost information = hallucinations

The solution: Documentation-First

reduce entropy + decompose tasks = deterministic outputs

Solution 1: Reduce entropy with structured patterns

Arc42 + C4 + BDD + ADR = deterministic specifications

Arc42: 12 structured chapters

Eliminates ambiguity about architecture, decisions, quality, deployment

C4 Model: 4 hierarchical levels

Context → Containers → Components → Code (clear visualization)

BDD: Given-When-Then

Executable specifications that eliminate behavior ambiguity

ADR: Documented decisions

Context + Decision + Consequences + Alternatives (zero ambiguity)

Solution 2: Task decomposition (Phase 3.5)

Problem: Context >5000 lines = O(n²) complexity = attention dispersion

Solution: Break specs into tasks of ~100 LOC (~500-1000 lines of context)

Computational benefit:

Without decomposition: O(50,000²) = 2.5 billion opsWith decomposition: 50 × O(1,000²) = 50 million ops→ 50× reduction in operations

→ Focused context + iterative feedback = deterministic output

Combined effect: 98.5% reduction in hallucination

Without method (free prompt):

60.4%

hallucination

+ Complete Arc42 spec:

35.0%

(-42%)

+ Task decomposition:

7.5%

(-88%)

+ Iteration with validation:

0.9%

(-98.5%)

Data based on controlled experiments documented in the chronicles

Scientific evidence

A Mathematical Theory of Communication

Claude Shannon, 1948 • Bell System Technical Journal

Mathematical foundation to quantify ambiguity through entropy. Theoretical basis for understanding how structured specifications reduce uncertainty.

Read paper (PDF)

Attention Is All You Need

Vaswani et al., 2017 • NIPS 2017

Introduced the Transformer architecture and demonstrated the O(n²) complexity of the attention mechanism. Basis for understanding why large contexts degrade performance.

Read paper (arXiv)

Lost in the Middle: How Language Models Use Long Contexts

Liu et al., 2023 • arXiv

Empirically demonstrated that LLMs lose 60% accuracy when relevant information is in the middle of long contexts. Direct evidence for task decomposition.

Read paper (arXiv)

It's not a framework, it's information physics

Documentation-First Approach is not another "framework of the moment". It is rigorous application of:

• Information Theory (Shannon, 1948) - mathematics established 75 years ago

• Transformer Architecture (Vaswani et al., 2017) - verified science

• Recent Research (Liu et al., 2023) - peer-reviewed empirical evidence

When you use Arc42 + C4 + BDD + ADR, you are not following "best practices". You are applying information physics to the code generation problem.

The result? 98.5% reduction in hallucinations. Not because "it works in practice", but because it works in theory.