Why does AI hallucinate?
And how Documentation-First Approach reduces hallucinations from 60.4% to 0.9% — a 98.5% reduction
Based on information theory (Shannon, 1948), Transformer architecture (Vaswani et al., 2017) and recent research on context degradation (Liu et al., 2023)
The problem: probabilistic nature
language models are probabilistic systems, not deterministic
The mathematics of failure
Each generated token is a probabilistic decision. The probability of a complete sequence is the product of individual probabilities:
P(sequence) = P(t₁) × P(t₂|t₁) × P(t₃|t₁,t₂) × ... × P(tₙ|t₁,...,tₙ₋₁)Even with 80% accuracy per token (optimistic):
P(1000 correct tokens) = 0.8¹⁰⁰⁰ ≈ 10⁻⁹⁷→ Essentially zero. This is why generated code "almost works" but contains subtle bugs.
Uncertainty accumulation
• Each incorrect token propagates errors to the next ones
• Bad initial decisions affect entire generation
• Ambiguous context = flattened probability distribution
• Result: unpredictable and inconsistent outputs
The root cause: ambiguity
when specifications are vague, the space of possibilities explodes exponentially
Example: "Create an authentication system"
Authentication method:
Username/password • OAuth2 • JWT • SAML • Magic links • Biometric = 6 options
Session storage:
In-memory • Redis • Database • JWT stateless = 4 options
Password hashing:
bcrypt (3 configs) • argon2 (3 configs) • scrypt (3 configs) = 9 options
Rate limiting:
By IP / By user × Sliding window / Fixed window = 8 options
Combinatorial explosion
6 × 4 × 9 × 8 = 1,728 possible implementationsFor a typical specification with 20 decision points, each with 10 interpretations:
10²⁰ = 100,000,000,000,000,000,000 possibilitiesFor context: there are ~10¹⁸ grains of sand on Earth, ~10²⁴ stars in the observable universe
The science: information theory
Claude Shannon (1948) provided the mathematical framework to quantify ambiguity
Shannon Entropy
Entropy H measures uncertainty in bits:
H(X) = -Σᵢ P(xᵢ) × log₂(P(xᵢ))The higher the entropy, the greater the uncertainty. Greater uncertainty = greater chance of hallucination.
Specifications: vague vs deterministic
❌ Vague specification
"Create an authentication endpoint"
REST: 30%
GraphQL: 20%
WebSocket: 15%
gRPC: 10%
Others: 25%
H ≈ 2.8 bits
High uncertainty
✅ Deterministic specification
"POST /api/auth/login accepts JSON with email and password"
REST: 95%
GraphQL: 2%
Others: 3%
H ≈ 0.35 bits
Low uncertainty
Reduction of 8× in entropy = 8× more chance of generating correct code
Attention complexity O(n²)
Vaswani et al. (2017) - "Attention is All You Need"
The attention mechanism in Transformers has quadratic complexity. For large contexts, this causes degradation:
Spec 5,000 lines (~50,000 tokens):
→ 50,000² = 2.5 billion attention operations→ Attention dispersion + "Lost in the Middle"→ 60% loss in the middle of context"Lost in the Middle" (Liu et al., 2023)
Research empirically demonstrated that LLMs lose performance when relevant information is in the middle of long contexts:
• Information at the beginning: 80-90% accuracy
• Information at the end: 80-90% accuracy
• Information in the middle: 20-40% accuracy
→ Large contexts = lost information = hallucinations
The solution: Documentation-First
reduce entropy + decompose tasks = deterministic outputs
Solution 1: Reduce entropy with structured patterns
Arc42 + C4 + BDD + ADR = deterministic specifications
Arc42: 12 structured chapters
Eliminates ambiguity about architecture, decisions, quality, deployment
C4 Model: 4 hierarchical levels
Context → Containers → Components → Code (clear visualization)
BDD: Given-When-Then
Executable specifications that eliminate behavior ambiguity
ADR: Documented decisions
Context + Decision + Consequences + Alternatives (zero ambiguity)
Solution 2: Task decomposition (Phase 3.5)
Problem: Context >5000 lines = O(n²) complexity = attention dispersion
Solution: Break specs into tasks of ~100 LOC (~500-1000 lines of context)
Computational benefit:
Without decomposition: O(50,000²) = 2.5 billion opsWith decomposition: 50 × O(1,000²) = 50 million ops→ 50× reduction in operations→ Focused context + iterative feedback = deterministic output
Combined effect: 98.5% reduction in hallucination
Without method (free prompt):
60.4%
hallucination
+ Complete Arc42 spec:
35.0%
(-42%)
+ Task decomposition:
7.5%
(-88%)
+ Iteration with validation:
0.9%
(-98.5%)
Data based on controlled experiments documented in the chronicles
Scientific evidence
A Mathematical Theory of Communication
Claude Shannon, 1948 • Bell System Technical Journal
Mathematical foundation to quantify ambiguity through entropy. Theoretical basis for understanding how structured specifications reduce uncertainty.
Read paper (PDF)Attention Is All You Need
Vaswani et al., 2017 • NIPS 2017
Introduced the Transformer architecture and demonstrated the O(n²) complexity of the attention mechanism. Basis for understanding why large contexts degrade performance.
Read paper (arXiv)Lost in the Middle: How Language Models Use Long Contexts
Liu et al., 2023 • arXiv
Empirically demonstrated that LLMs lose 60% accuracy when relevant information is in the middle of long contexts. Direct evidence for task decomposition.
Read paper (arXiv)It's not a framework, it's information physics
Documentation-First Approach is not another "framework of the moment". It is rigorous application of:
• Information Theory (Shannon, 1948) - mathematics established 75 years ago
• Transformer Architecture (Vaswani et al., 2017) - verified science
• Recent Research (Liu et al., 2023) - peer-reviewed empirical evidence
When you use Arc42 + C4 + BDD + ADR, you are not following "best practices". You are applying information physics to the code generation problem.
The result? 98.5% reduction in hallucinations. Not because "it works in practice", but because it works in theory.