← All research projects

Reasoning uncertainty · 2026

Entropy Is Not Frequency

A large-scale audit of the minority high-entropy pattern in reasoning tokens—and of the statistical shortcuts that can turn a useful rank rule into an overstated universal law.

Read the paper ↗ View code ↗
Distributional analysis of entropy among reasoning tokens
The audit separates token positions from token-type averages, aligns five uncertainty metrics by direction, and tests whether frequency can substitute for uncertainty.
1.116Bcached token positions
539,466reasoning trajectories
43,529decoded token types
51uncertainty caches

What exactly follows an 80/20 rule?

The original Beyond the 80/20 Rule study connects a minority of high-entropy “forking tokens” to effective reinforcement learning. This project independently audits the observational distribution and asks which conclusions survive changes in model, benchmark, metric, and unit of aggregation.

The minority high-entropy shape is robust. Its numerical threshold is not universal, and entropy cannot be replaced by token frequency.

Positions and types answer different questions

A position-level analysis asks which generated token occurrences fall in the uncertain tail. A token-type analysis first averages each metric over every occurrence of an ID, then asks which vocabulary items are usually uncertain. Common tokens can dominate position counts while retaining only moderate average entropy; rare types can show unstable extreme means.

That distinction explains why the original position-level 80th-percentile threshold of 0.672 and this audit’s type-average threshold of 1.219 are not competing estimates of the same quantity.

Agreement without interchangeability

After orienting Entropy, Confidence, Gini, LogProb, and SelfCert so that larger ranks consistently mean greater uncertainty, their top-20% token-type sets have mean pairwise Jaccard overlap 0.6932, with individual pairs ranging from 0.5362 to 0.8636.

Frequency remains only a weak proxy. Across 43,529 types, entropy has Pearson r = −0.0165 with raw frequency, Spearman ρ = −0.2959 with frequency rank, and Pearson r = −0.3464 after log-transforming counts. Very frequent connectives—including So, Let, Since, and Therefore—can still occupy high-entropy roles.

What this audit does not reproduce

  • No reinforcement-learning experiment is run, so the project does not reproduce the original gradient-masking performance claim.
  • A percentile specifies a selection budget, not a fixed natural threshold shared across all distributions.
  • Decoded strings can merge distinct IDs, and token-type averages hide multimodal context dependence.
  • The 51 caches span heterogeneous models and tasks rather than a balanced factorial design.
  • The results are descriptive and do not establish that entropy causally makes a token important.