Training-data recall vs live retrieval
Every LLM starts with parametric knowledge — information learned during training. A model like GPT-5.6-terra or Claude Sonnet 5 has a knowledge cutoff date. Anything after that cutoff is invisible unless the engine provides live retrieval. Engines fall into two modes:- Ungrounded. The model answers from parametric memory only. Facts about recent products, pricing, or brand positioning may be outdated or invented. This is how base-model API calls behave without retrieval.
- Grounded. The engine searches an index or the live web, retrieves passages, and conditions the answer on those passages. Grounding reduces hallucination but does not eliminate it. See LLM engines for which ai-cmo.dev engines are grounded.
Retrieval-augmented generation (RAG)
RAG is the architecture most grounded engines use:- A retrieval system (vector database, search API, or crawler index) finds passages relevant to the prompt.
- The retrieved passages are inserted into the model’s context window alongside the prompt.
- The model generates an answer conditioned on both the prompt and the retrieved text.
Why grounded engines cite
Citation is not decorative. Grounded engines cite because the retrieval step produces source documents, and the model is instructed to reference them. Perplexity, for example, retrieves from its own index and attaches inline citations. ChatGPT with browsing retrieves from Bing and links sources. Gemini with grounding retrieves from Google Search. An engine that does not retrieve cannot cite. If an answer contains no citations, assume the engine answered from parametric memory.Why ungrounded engines hallucinate brand facts
Without retrieval, the model has only its training data. For a brand that launched after the training cutoff, the model has no information. It may invent plausible-sounding facts — a process called hallucination. This applies to brand names, product features, pricing, and reputation. Hallucination is not random. Models hallucinate along patterns they learned: a brand in a known category will be described using the category’s typical claims. This can create incorrect but confident-sounding statements.Why answers vary across engines
Each engine uses:- A different base model (GPT, Gemini, Claude, Grok)
- A different retrieval index (Bing, Google, proprietary)
- Different retrieval ranking and chunking strategies
- Different system prompts for grounding and citation
Why answers vary across days
Engines change. Models are updated, retrieval indexes are refreshed, system prompts are adjusted. A measurement from Monday may differ from Wednesday’s because:- The retrieval index crawled new pages
- The model provider deployed a new version
- The engine changed its grounding strategy
- The brand’s pages changed rank in the retrieval source
Temperature and sampling variance
LLMs do not produce deterministic output. Even with temperature set to 0, floating-point nondeterminism in GPU kernels causes variation. At nonzero temperatures, the model samples from a probability distribution over tokens. The same prompt can produce different answers on consecutive calls. This variance is the single most important reason why one-off checks are worthless. A single call to ChatGPT that does not mention a brand proves nothing. The tenth call might mention it. An engine that mentions the brand 40% of the time looks either invisible or dominant depending on which single call you check.Why ai-cmo.dev uses repeated sampling
The platform sends each prompt to each engine multiple times — the repeat count depends on the preset. Thesmoke preset repeats once. The deep preset repeats three times. This is not an oversight in the smoke preset; even single-repeat sampling across a large prompt set produces aggregate visibility data that is more stable than a single prompt check.
Without repeated sampling, the measurement captures sampling noise, not signal. The confidence interval around a score reflects the number of observations (n). See Visibility metrics for how to interpret score reliability.