Get in Touch

The "Hard Token" Rewrite Protocol: Converting Fluff to Data

Dec 21, 2025

LLMs do not "read" your content the way a human does. They allocate attention. In a Transformer, self-attention determines which tokens influence the model’s internal representation at each step. If your sentence is mostly adjectives, adverbs, and generic filler ("Soft Tokens"), you give the model fewer anchors to lock onto—and your paragraph becomes easier to paraphrase, compress, or skip when an answer system is selecting sources.

To win the citation, you must stop writing prose and start engineering Entity Density. This is the

"Hard Token" Rewrite Protocol.

To win the citation, you must stop writing prose and start engineering Entity Density. This is the

"Hard Token" Rewrite Protocol.

The Warning: Soft Tokens Are Invisible

If you write with adjectives, you are making your claims harder to verify.

In self-attention, tokens matter based on how they interact with other tokens in context. Concrete identifiers (dates, names, versions, measurable values) reduce ambiguity. Vague descriptors (“significant”, “fast”, “robust”) increase ambiguity—and ambiguity is where confident errors live. OpenAI’s research on hallucinations frames this as a broader incentive problem: when systems are rewarded for guessing instead of acknowledging uncertainty, they produce plausible but incorrect statements more often.

[Hard Tokens] are specific, verifiable data points (e.g., "$4.2M," "Python 3.12," "Q3 2025") that give the model—and the reader—clear reference points.
[Soft Tokens] are subjective descriptors (e.g., "significant," "leading," "fast") that add little to verifiability.

If your paragraph has 100 words but only 2 concrete identifiers, you are under-specifying the claim. In practice, the more your paragraph contains dates, numbers, named tools, and defined terms, the easier it becomes for an answer engine to extract and attribute it without guessing.

If you write with adjectives, you are making your claims harder to verify.

[Hard Tokens] are specific, verifiable data points (e.g., "$4.2M," "Python 3.12," "Q3 2025") that give the model—and the reader—clear reference points.
[Soft Tokens] are subjective descriptors (e.g., "significant," "leading," "fast") that add little to verifiability.

The System: The Adjective Purge (SOP)

You do not need a creative writer; you need a ruthless editor. This is the exact workflow to convert "Marketing Speak" into "Training Data."

Step 1: The "Red Pen" Scan

Scan your text for qualitative descriptors. Look for words like "many," "fast," "significant," "industry-leading," or "robust." Highlight them in red. These are Soft Tokens. They tell the AI nothing concrete about the physical reality of your system.

Step 2: The Data Injection

You must replace every red highlight with a specific number or proper noun.

Bad (Soft): "We grew significantly last year."
Good (Hard): "Revenue increased by 214% in Q3 2025."

Step 3: The Entity Anchor

Ensure the subject of your sentence is a Named Entity (Brand/Product/Tool) rather than a generic pronoun.

Weak: "It integrates with many tools."
Strong: "BU Clarity integrates with HubSpot and Salesforce."

This matters because Named Entity Recognition (NER) is a standard NLP capability: it labels sequences of words as entities like organizations, people, locations, etc. Named entities are easier to extract, track, and attribute than anonymous pronouns.

Step 4: The Syntax Check

Reformat lists into Key-Value Pairs (e.g., "Growth: 24%"). This structure reduces ambiguity and improves parse reliability. It is closer to the way systems consume data when they must produce or extract structured facts (schemas, keys, enums, required fields).

You do not need a creative writer; you need a ruthless editor. This is the exact workflow to convert "Marketing Speak" into "Training Data."

Step 1: The "Red Pen" Scan

Step 2: The Data Injection

You must replace every red highlight with a specific number or proper noun.

Bad (Soft): "We grew significantly last year."
Good (Hard): "Revenue increased by 214% in Q3 2025."

Step 3: The Entity Anchor

Ensure the subject of your sentence is a Named Entity (Brand/Product/Tool) rather than a generic pronoun.

Weak: "It integrates with many tools."
Strong: "BU Clarity integrates with HubSpot and Salesforce."

Step 4: The Syntax Check

Manual Editing vs. Python Audits

Don't guess your density. Measure it. Human editors are bad at estimating density. We use Python to calculate a rough "Hard Token Ratio" before hitting publish.

The Density Script

This script uses spaCy (an NLP library) to count Named Entities vs. total words. If your ratio is below 10%, rewrite.

import spacy

# Load the English language model
nlp = spacy.load("en_core_web_sm")

# INPUT: Your draft text
text = "Buclarity increased SEO traffic by 35% using Entity Schema in 2025."
doc = nlp(text)

# PROCESS: Count Named Entities (Hard Tokens)
hard_tokens = len(doc.ents)

# PROCESS: Count Total Words
total_words = len(doc)

# OUTPUT: Calculate Density
density = (hard_tokens / total_words) * 100
print(f"Hard Token Density: {density:.2f}%")

# TARGET: >10% for "Hub" content (heuristic, not a Google metric).
# NOTE: NER is imperfect—use this as a directional signal, not a pass/fail truth

Developer Note: You can use the "Hemingway Editor" to spot adverbs first, then use the "Hard Token" rule to replace them with data.

Python script output showing a Hard Token Density calculation of 12.5%, passing the quality threshold.

Don't guess your density. Measure it. Human editors are bad at estimating density. We use Python to calculate a rough "Hard Token Ratio" before hitting publish.

The Density Script

This script uses spaCy (an NLP library) to count Named Entities vs. total words. If your ratio is below 10%, rewrite.

import spacy

# Load the English language model
nlp = spacy.load("en_core_web_sm")

# INPUT: Your draft text
text = "Buclarity increased SEO traffic by 35% using Entity Schema in 2025."
doc = nlp(text)

# PROCESS: Count Named Entities (Hard Tokens)
hard_tokens = len(doc.ents)

# PROCESS: Count Total Words
total_words = len(doc)

# OUTPUT: Calculate Density
density = (hard_tokens / total_words) * 100
print(f"Hard Token Density: {density:.2f}%")

# TARGET: >10% for "Hub" content (heuristic, not a Google metric).
# NOTE: NER is imperfect—use this as a directional signal, not a pass/fail truth

Developer Note: You can use the "Hemingway Editor" to spot adverbs first, then use the "Hard Token" rule to replace them with data.

Failure Modes: When "Data" Becomes Spam

The system breaks when you confuse "specific" with "true."

The "False Precision" Trap

Do not add decimals to estimates to make them look like hard tokens. If you don’t have measurement-grade data, don’t cosplay it. Use rounded numbers, ranges, or confidence language—then cite the source. Precision without provenance is a trust leak.

Entity Stuffing

Jamming too many unrelated entities into one sentence confuses extraction and relationship mapping.

Broken: "Apple, Microsoft, and Tesla saw gains in 2025 while Python updated to 3.12." (Too many disconnected nodes).
Fixed: Keep it to Subject (Entity) -> Action -> Object (Entity).

The "Orphaned Stat"

Providing a number without a date is weak attribution. "Traffic is up 50%" is not fully citable because it lacks temporal grounding. "Traffic up 50% (YoY 2025)" is a valid hard token.

The system breaks when you confuse "specific" with "true."

The "False Precision" Trap

Entity Stuffing

Jamming too many unrelated entities into one sentence confuses extraction and relationship mapping.

Broken: "Apple, Microsoft, and Tesla saw gains in 2025 while Python updated to 3.12." (Too many disconnected nodes).
Fixed: Keep it to Subject (Entity) -> Action -> Object (Entity).

The "Orphaned Stat"

Providing a number without a date is weak attribution. "Traffic is up 50%" is not fully citable because it lacks temporal grounding. "Traffic up 50% (YoY 2025)" is a valid hard token.

References & Verified Data

[1] Transformer Attention: Self-attention is the core mechanism that computes token interactions in the Transformer architecture. Source: "Attention Is All You Need" (Vaswani et al., 2017).
[2] Hallucinations & Uncertainty: OpenAI research argues hallucinations persist partly because training/evaluation reward guessing over admitting uncertainty. Source: OpenAI Research.
[3] Named Entity Recognition (NER): Stanford NER (CRFClassifier) is a widely cited implementation of NER for labeling entities. Source: Stanford NLP Group.
[4] Schema-Constrained Structure: Structured Outputs enforce JSON Schema adherence, improving reliability for extracting/using facts as data. Source: OpenAI API Docs.

The "Hard Token" Rewrite Protocol: Converting Fluff to Data >

Last Post

See All Post

Abstract digital art with colorful squares and lines

Entity Schema: The Knowledge Graph Handshake

October 1, 2025