Sep 4, 2026

Notes: Attention, Silent Failure, and the Reference Signal

Working notes from a conversation tracing LLM mechanics down to a control-theory claim about knowledge work. Hard points first, then the connections between them, then the thesis they produce.

I. Hard Points — Mechanism

1. One rule, run at scale. An LLM is a single mathematical function: tokens in, a probability distribution over the next token out. Sample, append, repeat. Training is one operation — predict the next token, measure error, adjust weights — applied trillions of times. No rules for grammar, causality, or tone were written. They precipitate out because representing them is the cheapest path to lower prediction error.

2. The model is stateless. No memory persists between calls. Every turn, the entire conversation, system prompt, retrieved documents, and tool definitions are re-sent as one flat block. Continuity is an illusion maintained entirely by the application layer.

3. Attention is a function of three arguments. Attention(Q, K, V) = softmax(QKᵀ / √d) · V

Query — what this token is looking for
Key — how this token advertises itself
Value — what it hands over when attended to

All three are projections of the same hidden state through different learned matrices. The same token is simultaneously seeker, advertiser, and payload. The /√d scaling keeps softmax from collapsing to a spike; without it, gradients vanish and training fails to converge.

4. The KV cache is memory, not learning. Causal masking means a token’s Key and Value depend only on tokens at or before it — so once computed, they are frozen and can be cached. Queries cannot be cached; each is consumed the moment its token is generated.

Cost: roughly 2 × layers × kv_heads × head_dim × bytes per token. A large model at full precision approaches ~2.6 MB per token; a 100K context lands in the tens of GB of VRAM. This is why GQA, MQA, MLA, and cache quantization exist, and why serving engines borrow OS virtual memory (PagedAttention: fixed blocks, block tables, copy-on-write prefix sharing, LRU eviction).

Recompute beats retention because the plain text is ~400 KB and the cache is 40 GB. Cheap state, expensive state, and the economics are not close.

5. Attention is quadratic. Every token attends to every prior token. Double the context, roughly quadruple the compute. State-space architectures carry a fixed-size recurrent state instead — linear cost, but lossy by construction. They must decide what to forget, and they are measurably worse at retrieving an exact detail from deep context.

II. Hard Points — Failure

6. The arithmetic is not the failure surface. GPUs are reliable. Floating-point non-associativity introduces nondeterminism (batch-dependent kernel and reduction order), so identical prompts can yield differing output — but that is noise, not error. Nobody fumbles a dot product.

7. There is no error signal anywhere in the system. No stack trace, no failed assertion, no checksum. The model emits a distribution and samples. A correct answer and a confident fabrication are produced by the identical mechanism, with identical internal character. Every output is, from the system’s perspective, a successful completion.

This is the purest instance of the doctrine’s core case: an explicit system reporting full health while failing, because it possesses no channel through which failure could be expressed.

8. The specific silent degradations.

Lost-in-the-middle — retrieval accuracy sags for content in the middle of long contexts; nothing announces it, the answer just gets quietly worse
Quantized KV cache — saves memory, costs quality, invisible at the output
Calibration decay — RLHF tunes for human approval, and humans rate confidence highly; certainty and correctness decouple, invisibly from both sides
Tokenization artifacts — the model cannot count letters in a word it never sees as letters

9. The objective is plausibility, not truth. Training maximizes next-token likelihood, then human preference. Neither is truth. A fabricated citation is therefore not a malfunction — it is the mechanism working correctly against a target that is not the one anyone wanted.

10. There are no seams inside the loop. Instrumentation requires boundaries. Selection, weighting, and synthesis all occur inside a single function call. The model performs something resembling Selection, Interpretation, Synthesis, and Origination without the judgment that constitutes them — and that resemblance is what makes the failure silent. It does not look like automation overreaching. It looks like an answer.

III. Hard Points — Distributed Systems Parallel

11. Reconciliation loops have the error signal inference lacks. The Kubernetes controller is level-triggered: it re-derives from observed reality on every pass and drives the delta against declared state toward zero. That comparison step is precisely what inference has no equivalent of. An LLM is fire-and-forget; it never observes its own output against a spec, because nothing holds a spec.

12. HA systems exist because a node cannot certify its own health. Quorum writes and checksums encode the same premise the doctrine does: verification must originate outside the thing being verified.

13. Raft achieves agreement, not truth. Five nodes replicating a corrupt value produce a perfectly consistent corrupt cluster. Raft assumes crash-fault tolerance — nodes fail by stopping. It explicitly does not handle Byzantine failure, where a node stays up and returns plausible garbage.

That is exactly the LLM failure mode.

14. Ensembling does not fix it, because the nodes are correlated. Byzantine protocols assume independent failure. Models sharing training data, architecture, and RLHF pressure fail in correlated ways — they will confidently agree on the same fabrication. The quorum is compromised at the level of node independence.

→ The human is the only uncorrelated node available. This is a topology argument, not a sentiment.

15. “AI proposes, human disposes” is leader-proposes / quorum-commits. The proposal remains uncommitted until an independent participant acknowledges it. Same shape, arrived at independently.

16. Where the parallel breaks — and why it matters most. Kubernetes works because desired state is declarable. Three replicas, this image, that limit. The manifest is explicit, so the loop can be automated end to end.

Knowledge work has no declarable desired state. There is no manifest for is this correct, relevant, and worth someone’s time. The spec itself is tacit. The controller cannot be automated — not because the loop is hard to build, but because you cannot write down what it should reconcile toward.

IV. The Capstone

17. The machine cannot redefine the ideal condition. Even k8s has this limitation and papers over it. When conditions change, the controller does not notice. A human observed the drift, formed a judgment about what “good” now means, and rewrote the manifest. Every autonomous-looking loop in the stack is bounded by a target a person must keep re-authoring.

18. Relevance is a relation, not a property. The article did not change; the world did. The product shipped a new version. The workaround became the fix. The question stopped being asked. Nothing in the content differs — so a system that sees only content sees nothing at all. It is not misjudging relevance. It has no surface on which relevance could register.

19. Every available proxy is lagging and corruptible. Usage counts measure what people found, not what they needed. Age measures nothing — some articles are correct for a decade. Each metric measures the artifact rather than the fit, so optimizing against them drifts toward what is most retrievable rather than what is most true. Goodhart’s law at the target-definition layer, which is worse than Goodhart at the execution layer, because it is invisible.

20. A system cannot set its own reference signal. Control theory states it plainly. Redefining “ideal” requires standing inside the changed conditions and holding a stake in them — which is the Ba argument arriving at the same place from the other direction.

V. The Zhuangzi Turn

Chapter 26, External Things — Daoist, not Zen, though Chan absorbed it wholesale: the snare exists for the rabbit, the bait for the fish, words for meaning; once you have the meaning you can forget the words. Where can I find a man who has forgotten meaning, so I might have a word with him.

The mechanism reads as commentary on the parable:

Attention is a snare that catches meaning. Values are consumed into the next hidden state and discarded, every layer.
The KV cache is the refusal to forget — every snare and scrap of bait retained, quadratically expensive, on the chance a later token fishes the same water.
State-space models are the ones practicing the discipline: fixed budget, compress or drop, forget the trap and keep the fish. And they are worse at exact recall from deep context — which is either a design tradeoff or a small parable about what forgetting costs.

VI. Thesis

The limit of automation in knowledge work is not capability. It is reference.

A language model has no error signal — correct output and confident fabrication are produced by the same mechanism and are internally indistinguishable. That makes it a Byzantine node in a cluster whose other nodes are correlated with it, so no quantity of models can supply the missing verification. But the deeper constraint sits one layer above: the model cannot author its own target. Relevance is a relation between an artifact and a moment, not a property of the artifact, and when conditions move, nothing in the content changes to signal it. Every automatable proxy measures the artifact instead of the fit, so optimizing against them produces Goodhart drift at the target-definition layer, where it is invisible.

A control loop cannot set its own reference signal. Something standing in the changed conditions, with a stake in them, must keep re-authoring what “good” means.

Therefore AI proposes, human disposes is not a governance hedge or a comfort measure. It is the topological requirement for the loop to close at all — the only point at which the reference signal can enter the system.

Corollaries

Design for the entry point, not the guardrail. The human is not a safety net downstream of the machine; the human is where the target originates. Systems that treat review as a final gate have mislocated the function.
Do not let automation own relevance. Retrieval, ranking, and summarization can own retrievability. The moment they own relevance, the target quietly becomes whatever the proxy measures.
Correlated verification is not verification. Adding models to check models raises consistency, not truth.
Instrument the boundaries you have. You cannot instrument inside a forward pass. You can instrument the seams: what was proposed, what was accepted, what was changed, and by whom.
The tacit spec is the asset. If the desired state cannot be declared, then the people who hold it are not overhead in the pipeline. They are the specification.