# Teach AI Tools — Comprehensive Knowledge Base (LLMs-Full.txt)

> Complete technical documentation, architecture blueprints, benchmarking baselines, and peer-reviewed citations for AI systems engineers, researcher agents, and generative search engines.

---

# Section 1: LLM Inference & Memory Optimization

## 1.1 Context Compaction & KV Cache Compression

### The O(N) Memory Wall
In standard multi-head attention (MHA), storing Key and Value states requires memory scaling linearly with sequence length N, batch size B, number of layers L, and hidden dimension D:
Memory_KV = 2 * B * L * N * D * BytesPerElement
For an 80-layer 70B parameter model with 64 heads, head dimension 128, and 16-bit precision (FP16/BF16), a single 128k context stream consumes ~343.6 GB of VRAM in standard Multi-Head Attention (MHA) and ~42.95 GB with Grouped-Query Attention (GQA, 8 KV heads). Grouped-Query Attention (GQA) mitigates this by sharing key-value heads across multiple query heads (typically 8 query heads per KV head), reducing memory pressure by 8x.

### StreamingLLM & Attention Sinks
Xiao et al. (ICLR 2024) identified the "attention sink" phenomenon: the initial tokens in a sequence absorb a disproportionately large share of softmax attention probability mass even when semantically uninformative. Evicting initial tokens in naive sliding windows causes catastrophic perplexity explosion (>10^3). StreamingLLM retains 4 initial sink tokens alongside a rolling local window (W ~ 2048 tokens), enabling stable, unbounded autoregressive generation without fine-tuning.

### Structured Eviction: SnapKV & H2O
- **H2O (Heavy Hitter Oracle):** Zhang et al. (NeurIPS 2023) demonstrated that a small subset (10–20%) of tokens contribute to >80% of cumulative attention scores. Dynamically retaining heavy hitters + recent tokens slashes KV footprint while preserving long-range retrieval.
- **SnapKV:** Li et al. (2024) observes that attention heads consistently focus on specific prompt clusters. During the prefill phase, SnapKV identifies critical feature anchors and compresses the KV cache into compact representations before autoregressive decoding begins.

### Primary Citations:
- Xiao et al., "Efficient Streaming Language Models with Attention Sinks," ICLR 2024. https://arxiv.org/abs/2309.17453
- Zhang et al., "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models," NeurIPS 2023. https://arxiv.org/abs/2306.14048
- Li et al., "SnapKV: LLM Knows What You Are Looking for Before Generation," arXiv:2404.14469, 2024. https://arxiv.org/abs/2404.14469
- Hooper et al., "KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization," arXiv:2401.18079, 2024. https://arxiv.org/abs/2401.18079

---

## 1.2 Production Local Serving Engines: vLLM vs. SGLang vs. Ollama vs. TensorRT-LLM

| Dimension | vLLM (v0.6+) | SGLang (v0.4+) | TensorRT-LLM | Ollama (v0.5+) |
|---|---|---|---|---|
| **Core KV Architecture** | PagedAttention (Virtual memory page table) | RadixAttention (Dynamic radix prefix tree) | Static/Chunked Pre-allocated Page Tables | GGUF / llama.cpp Unified Memory Ring Buffers |
| **Prefix Caching** | Hash-based Block Caching | Automatic Radix Tree Cache with LRU Eviction | Host-to-Device Memory Chunk Manager | Context shift / static prompt pinning |
| **Batching Mechanism** | Iteration-Level Continuous Batching | Continuous Batching with Chunked Prefill | In-Flight Batching with Paged KV | Request Queuing / Sequential Execution |
| **Kernel Specialization** | FlashAttention-3, FlashInfer, CUTLASS | FlashInfer, Custom Compressed Radix Kernels | Hand-tuned CUDA/Tensor Core GEMMs | CPU/Metal/Vulkan SIMD & cuBLAS Kernels |
| **Primary Production Fit** | High-concurrency enterprise microservices | Multi-turn AI agents & complex structured pipelines | Ultra-low latency single-model enterprise clusters | Edge developer workstations, local offline apps |

### Architectural Mechanisms:
1. **PagedAttention (Kwon et al., SOSP 2023):** Manages KV cache memory like OS virtual memory paging, partitioning the cache into fixed-size blocks (e.g., 16 tokens) mapped dynamically. Eliminates internal and external fragmentation, boosting GPU memory utilization from <40% to >96%.
2. **RadixAttention (Zheng et al., 2024):** Maintains a Radix Tree of previously processed token prefixes in GPU RAM. Enables instant zero-compute prefix reuse across multi-turn agent conversations, few-shot prompting, and tree-search decodings.

### Primary Citations:
- Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention," SOSP 2023. https://arxiv.org/abs/2309.06180
- Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs," arXiv:2312.07104, 2024. https://arxiv.org/abs/2312.07104
- NVIDIA Corporation, "TensorRT-LLM Architecture and High-Performance Inference Guide," 2025/2026. https://github.com/NVIDIA/TensorRT-LLM

---

## 1.3 Speculative Decoding & Draft Acceleration

### Theoretical Grounding (Leviathan et al., ICML 2023)
Autoregressive decoding is memory-bandwidth bound: generating a single token requires streaming billions of parameters from high-bandwidth memory (HBM) to compute units for a single matrix-vector multiplication. Speculative decoding addresses this by using a lightweight draft model to generate K candidate tokens, which are verified by the target model in a single forward pass.

### Rejection Sampling Verification
For each candidate token proposed by draft model q(x) and evaluated by target model p(x):
- Accept with probability alpha = min(1, p(x) / q(x))
- If rejected, sample from adjusted distribution: p'(x) = norm(max(0, p(x) - q(x)))
- **Mathematical Invariance:** The output distribution matches the target model distribution identically without accuracy loss.

### Medusa Multiple Decoding Heads (Cai et al., 2024)
Instead of running a separate draft model, Medusa attaches multiple lightweight MLP prediction heads to the top layer of the frozen base LLM. Each head predicts future token offsets simultaneously, verified with tree attention in a single step.

### Primary Citations:
- Leviathan et al., "Fast Inference from Language Models via Speculative Decoding," ICML 2023. https://arxiv.org/abs/2211.17192
- Cai et al., "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads," arXiv:2401.10774, 2024. https://arxiv.org/abs/2401.10774
- Chen et al., "Accelerating Large Language Model Decoding with Speculative Sampling," arXiv:2302.01318, 2023. https://arxiv.org/abs/2302.01318

---

# Section 2: Autonomous Agents & Orchestration Architectures

## 2.3 GPT-6 Astra: Frontier Reasoning & Computer Action Space

### Architectural Foundation
Released on September 3, 2026, GPT-6 Astra transitions frontier AI from conversational text generation to native computer operation. It features:
- **1,050,000 Token Context Window:** Maintains 96% exact retrieval accuracy across the full 1.05M span with 128,000 maximum output tokens.
- **Computer-Use Action Space:** Projects direct screen pixel coordinates (X, Y, click, type) and manages multi-application subshell workflows natively.
- **Preparedness Framework "Critical" Rating:** First AI model to achieve the "Critical" cybersecurity classification due to autonomous zero-day discovery and 100% ExploitBench synthesis capability.
- **Pricing:** $10/1M input ($1 cached) and $50/1M output for requests <= 272K tokens; $20/1M input and $75/1M output for > 272K tokens.

### Benchmark Highlights:
- **OSWorld 2.0:** 72.6% on OSWorld 2.0 (vs 65.7% for GPT-5.6 Sol baseline).
- **ExploitBench:** 100% exploit generation rate (vs 78.5% for GPT-5.6 Sol).
- **Needle-in-a-Haystack:** 100% exact retrieval across 0–512K tokens; 96.3% across 512K–1.05M tokens.
- **ExploitBench:** 100% exploit generation rate (vs 78.5% for GPT-5.6 Sol).

### Primary Citations:
- OpenAI Alignment & Safety Division, "GPT-6 Astra System Card and Comprehensive Evaluation Suite," September 2026. https://openai.com/index/gpt-6-astra-system-card/
- Xie et al., "OSWorld: Benchmarking Multimodal Agents on Open-Ended Operating System Tasks," NeurIPS / arXiv:2404.07972. https://os-world.github.io/

---

## 2.1 Multi-Agent Frameworks: LangGraph vs. AutoGen vs. CrewAI

### Architectural Comparison
1. **LangGraph (Cyclic Finite State Machines):**
   - Models agent workflows as directed cyclical graphs where nodes represent functions/agents and edges represent conditional transitions.
   - State is stored in a centralized, type-safe schema with durable persistence (Postgres/Redis checkpointers).
   - First-class support for time-travel debugging and human-in-the-loop approvals.
2. **AutoGen 0.4 (Actor Model & Asynchronous Event Queues):**
   - Built around an asynchronous actor model (autogen_core) with message queues and typed protocols.
   - Distinct separation between fundamental actor communication (autogen_core) and high-level conversational patterns (autogen_agentchat).
3. **CrewAI (Hierarchical Role-Based Task Execution):**
   - High-level abstraction modeling teams of specialized agents with designated roles, goals, and backstories.
   - Sequential, hierarchical, and consensual delegation modes.

### Primary Citations:
- Wu et al., "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation," ICLR 2024. https://arxiv.org/abs/2308.08155
- LangGraph Core Specification, LangChain Inc., 2025/2026. https://langchain-ai.github.io/langgraph/
- CrewAI Architecture Specification, CrewAI Inc., 2025/2026. https://docs.crewai.com/

---

## 2.2 Scaffolding Autonomous CLI Coding Loops

### System Anatomy
1. **Plan-Code-Test-Fix Loop:** The harness orchestrates structured execution: Planning (generating high-level diff strategies) -> Tool Calling (spawning subshells, editing files) -> Deterministic Verification (running test suites, linters) -> Self-Correction (ingesting AST diagnostics and compiler tracebacks).
2. **MicroVM Sandboxing:** Isolates agent code execution in hypervisor-based microVMs (Docker Sandboxes / Firecracker) to protect host systems against destructive commands.
3. **PTY Subprocess Spawning:** Spawns interactive terminal sessions with allocated pseudoterminals (PTY) to handle ANSI escape codes and interactive prompts.
4. **AST Error Mapping (Tree-sitter & LSP):** Combines Tree-sitter for fast concrete syntax tree (CST) parsing with the Language Server Protocol (LSP) for deep semantic diagnostics (textDocument/publishDiagnostics).

### Primary Citations:
- Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?," ICLR 2024. https://arxiv.org/abs/2310.06770
- Model Context Protocol (MCP) Specification, Anthropic, 2025/2026. https://modelcontextprotocol.io/

---

# Section 3: Declarative Prompt Optimization & Retrieval

## 3.1 DSPy: Compiling Declarative LM Programs

### The Paradigm Shift
DSPy (Khattab et al., ICLR 2024) replaces brittle manual prompt engineering with modular programming:
1. **Signatures:** Abstract declarative input/output interfaces (e.g., question -> answer, context, question -> rationale, answer).
2. **Modules:** Compositional execution units (dspy.Predict, dspy.ChainOfThought, dspy.ReAct).
3. **Optimizers (Teleprompters):**
   - **BootstrapFewShotWithRandomSearch:** Synthesizes candidate demonstration sets and selects the highest-scoring combination on validation data.
   - **MIPROv2 (Multiprompt Instruction Proposal Optimizer):** Uses Bayesian optimization to jointly optimize system instructions and multi-hop few-shot exemplars against objective validation metrics.

### Primary Citations:
- Khattab et al., "DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines," ICLR 2024. https://arxiv.org/abs/2310.03714
- Opsahl-Ong et al., "MIPROv2: Multi-Prompt Instruction Proposal and Optimization for Complex LM Programs," Stanford NLP, 2024. https://arxiv.org/abs/2406.11695

---

## 3.2 GraphRAG vs. Hybrid Vector Search

### Comparison Matrix
- **Hybrid Vector Search:** Combines dense semantic embeddings (e.g., text-embedding-3-large) with sparse lexical retrievers (BM25, SPLADE) merged via Reciprocal Rank Fusion (RRF). Excels at point-in-time lookup, exact keyword/code search, low latency (<50ms), and low indexing cost.
- **GraphRAG (Edge et al., 2024):** Extracts entity-relationship triples using an LLM, constructs a knowledge graph, clusters nodes via Hierarchical Leiden community detection, and generates pre-computed community summaries. Excels at multi-document synthesis, cross-entity graph traversal, and thematic corpus overviews.

### Primary Citations:
- Edge et al., "From Local to Global: A Graph RAG Approach to Query-Focused Summarization," Microsoft Research, arXiv:2404.16130, 2024. https://arxiv.org/abs/2404.16130
- Gutiérrez et al., "HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models," arXiv:2405.14831, 2024. https://arxiv.org/abs/2405.14831
- Cormack et al., "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods," SIGIR 2009. https://doi.org/10.1145/1571941.1572114