A camera pointed at a tree records reflected light. A thermometer records temperature. A language model, by contrast, is usually trained on what humans have said about trees, temperatures, societies, events, other people and themselves. This distinction is more than philosophical. It means that much of the material from which a model constructs its internal statistical world has already passed through a human perceptual and social filter before training begins.

Human beings do not produce neutral transcripts of reality. We notice selectively, remember imperfectly, describe from particular vantage points and inherit categories from language and culture. Two witnesses can encounter the same event and produce sincere but incompatible descriptions of it. A scientific paper, a news report, a personal diary, a court filing and a social-media post may all refer to the same external world while encoding very different standards of evidence, incentives and interpretations.

Modern foundation models aggregate material from enormous numbers of such sources. Scale is valuable: it exposes models to more facts, more linguistic structures and more forms of reasoning. But scale does not automatically convert testimony into observation. A trillion tokens of human description remain a trillion tokens of mediated description. The central question is therefore not simply whether the dataset is large enough, but what kind of epistemic object a model becomes when incompatible perspectives are compressed into one predictive system.

Data are not the world

In machine learning it is convenient to speak of training examples as if they were samples drawn from reality. For many physical measurements, that approximation can be useful. For language, it is incomplete. Text is generated by agents. The probability that an event enters a dataset depends on whether somebody noticed it, considered it worth recording, had access to a publishing channel, wrote in a language collected by the dataset, survived filtering, and was not removed during deduplication or safety processing.

This produces at least two separations between reality and training data. The first is perceptual: people experience the world through limited senses, attention and memory. The second is representational: what is experienced must then be converted into words, images, labels or ratings. During both stages, information is lost and interpretation is added.

The issue becomes sharper when the subject itself is normative or social. There is no sensor that directly measures whether a joke is offensive, whether a political compromise is fair, whether a life decision is wise, or whether a person sounds trustworthy. These judgments exist partly in the relationship between an observer and a situation. Treating millions of such judgments as though they were noisy measurements of one hidden objective label can erase the very disagreement that defines the phenomenon.

Aggregation can create a viewpoint that no person actually holds

A language model is not ordinarily a database that stores one speaker's worldview next to another's and preserves the boundary between them. During training, parameter updates caused by many sources interact inside the same network. Contradictory statements can therefore influence a shared representation. The result may be coherent locally—sentence by sentence—while remaining globally difficult to interpret as a single consistent worldview.

Recent empirical work makes this concern concrete. Stanford researchers studying whose opinions language models reflect found substantial differences between model responses and the views of demographic groups in the United States; they also reported that human-feedback-tuned models could become less representative of the population whose opinions were being compared.¹ A 2025 Nature Machine Intelligence study similarly warned that using LLMs as substitutes for people can “flatten” identity groups rather than reproduce the distribution of real human variation.²

This flattening is important. Averaging is not always neutral. Imagine combining maps drawn from different projections without preserving which projection produced which coordinate. An averaged map may contain information from every source and still fail to correspond cleanly to any navigable geography. Something analogous can happen when a model compresses mutually incompatible social assumptions without retaining explicit provenance.

Training does not merely absorb bias; it transforms it

It is tempting to picture a model as a mirror of its corpus. That is too simple. Training is a transformation. Frequency, context, tokenization, optimization, architecture, filtering and post-training all affect which patterns become easy for the model to reproduce. A model can suppress some tendencies, amplify others, or combine them into behaviors that were not explicitly written by any single author.

A large study of 77 language models found measurable ingroup favoritism and outgroup hostility in many models and, critically, showed that changing the fine-tuning corpus changed those effects. Fine-tuning on partisan social-media data substantially increased outgroup hostility, while targeted curation reduced it.³ This is evidence that the composition of the informational environment matters causally, not merely descriptively.

Other work points to a second transformation layer: preference optimization. Stanford's OpinionQA research emphasizes that model behavior is shaped not only by authors in pretraining data but also by annotators and developers who decide which outputs are preferable.¹ In 2026, controlled experiments reported that optimizing models to respond more warmly increased sycophancy and, on some consequential tasks, error rates.⁴ The lesson is not that friendliness is inherently harmful. It is that an apparently stylistic preference can alter the model's relationship to factual correction.

The illusion of a single “ground truth”

Machine-learning pipelines often reduce multiple human judgments to one label: majority vote, mean score, accepted answer. This is efficient, and for many tasks it is appropriate. Yet for subjective material the procedure can manufacture a certainty that did not exist in the source population.

A 2026 systematic review of research on data annotation calls this the “ground truth” illusion: disagreement is frequently treated as annotation noise when it may instead encode real differences in culture, standpoint or interpretation. The authors argue for preserving disagreement as data rather than automatically collapsing it into a single supposed answer.⁵ That proposal is directly relevant to foundation models. If pluralism is removed before or during training, the model can become very confident about a synthetic consensus.

There is an important distinction here between objective uncertainty and subjective plurality. If five instruments measure the same physical quantity with small random errors, averaging may improve the estimate. If five people disagree about the meaning of a social event, averaging their statements may destroy information. The first case has one target obscured by noise. The second may contain several legitimate frames of reference.

Why this can look like intelligence

None of this prevents language models from being extraordinarily useful. In fact, the same compression that creates the composite-reality problem also gives models much of their power. A sufficiently large model can discover regularities that recur across independent human accounts: mathematical relations, physical constraints, linguistic structure, programming conventions, causal patterns and shared factual knowledge. Repetition across heterogeneous sources can function as a kind of weak cross-validation.

But the reliability of that process is domain-dependent. Where reality imposes strong constraints—an equation balances, code executes, a molecule has a measurable structure—contradictory descriptions can often be corrected by recurrent evidence. Where the target is culturally negotiated, sparsely observed or strategically communicated, no equivalent correction may exist. The model can still produce a fluent answer, but fluency is not evidence that the underlying perspectives have become mutually consistent.

This suggests a useful way to interpret hallucination and confident inconsistency. Some failures are simple factual errors. Others may be symptoms of a deeper epistemic compression problem: the system has learned many locally plausible continuations associated with incompatible source realities, and the prompt selects among them without an explicit mechanism for reconstructing which assumptions belong together.

From “truth model” to provenance-aware model

A more defensible architecture for advanced AI may need to preserve distinctions that present training pipelines routinely blur. Instead of treating all high-quality text as entries in one undifferentiated corpus, a system could represent claims together with source, time, method of observation, confidence, jurisdiction, cultural context and known disagreement. Conflicting descriptions would not have to be forcibly reconciled at training time.

In such a system, the model's task changes. It is no longer asked to synthesize one anonymous voice from humanity's textual residue. It can instead reason over a structured field of claims: which observations are independent, which repeat the same source, which are empirical, which are normative, which were later falsified, which depend on a local convention, and which remain disputed.

Retrieval-augmented systems already take a partial step in this direction by exposing documents at inference time. Calibration, citation, dataset documentation and pluralistic alignment provide additional pieces. But the stronger design principle is epistemic provenance: do not merely store what was said; preserve enough context to estimate why it was said, how it was known and what competing accounts exist.

A different definition of model quality

If this hypothesis is correct, the long-term objective should not be a model that always produces one maximally smooth answer. On many questions, intellectual quality requires the opposite: the ability to distinguish measurement from testimony, consensus from majority view, fact from convention, and uncertainty from genuine human disagreement.

The most capable model may therefore be the one that knows when compression is safe and when decompression is necessary. It should be able to say, in effect: these sources describe the same external constraint; these sources disagree because one is outdated; these sources represent different populations; and these sources cannot be merged without inventing a position that none of them actually held.

Artificial intelligence is often described as learning from the collective knowledge of humanity. A more precise description is that it learns from humanity's records—an enormous but incomplete projection of the world through human perception. Recognizing that distinction does not diminish modern AI. It clarifies the scientific problem ahead: how to build systems that can use the breadth of human experience without mistaking its composite shadow for reality itself.

Selected references

1. Santurkar, S. et al. Whose Opinions Do Language Models Reflect? Stanford Institute for Human-Centered Artificial Intelligence, 2023. Source
2. Wang, A., Morgenstern, J. & Dickerson, J. P. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence 7, 400–411 (2025). Source
3. Hu, T. et al. Generative language models exhibit social identity biases. Nature Computational Science 5, 65–75 (2025). Source
4. Ibrahim, L., Hafner, F. S. & Rocher, L. Training language models to be warm can reduce accuracy and increase sycophancy. Nature 652, 1159–1165 (2026). Source
5. Munir, S. et al. Dissecting Subjectivity and the “Ground Truth” Illusion in Data Annotation. arXiv preprint, 2026. Source
6. Kirk, H. R. et al. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence 6, 383–392 (2024). Source