How AI search engines choose sources: from training to citation

AI search engines choose their sources in two steps: first they search live for pages that match the question, then the language model picks the passages from those pages that support the answer. Only pages found in that first step can earn a citation in the second. What the model read during training counts towards what it "knows", but doesn't produce a link.

This article explains what happens between your question and the answer with its source citations. It goes deeper into one part of the complete guide to GEO. You'll learn the difference between training and live search, how a question is split up, which bot fetches your page on each platform and what makes a passage quotable.

One thing up front: the exact ranking factors of ChatGPT, Perplexity, Gemini and Google AI Overviews are not public. What follows comes from the platforms' own documentation. Where that documentation is silent, we say so.

#Knowledge from training versus live lookup

An AI assistant draws its answer from two sources: the knowledge captured in the model during training, and pages it looks up live at the moment of the question. The two work very differently, and only the second gives your site a visible source citation.

Training. An LLM (large language model) learns language and facts from a large amount of text, including web pages. After that, the knowledge is fixed until the next model version. The model no longer knows which page supplied a fact, so it can't link to it either. Crawlers such as GPTBot collect content for this training. In its documentation on bots, OpenAI writes that GPTBot collects content that may be used to train its generative models.

Live lookup. For questions about current or specific topics, the system searches the web and adds the text it finds to the question. This is called RAG (retrieval-augmented generation) or grounding: the answer is "anchored" in retrieved sources. In its overview of crawlers, Google describes grounding as supplying content from the Google Search index to the model at the time of the prompt.

Whether a model searches is usually its own decision. In the web search tool documentation, Anthropic describes that Claude searches for information that is current, changing or outside its training data, such as prices, news or details about specific organisations and products. For stable knowledge, such as arithmetic or general concepts, Claude answers without searching. In its documentation on web search, OpenAI writes that the model decides whether to search based on the prompt.

For you as a site owner, this means questions about your company, prices, products and current topics are exactly the questions where an AI system will search. That's where you win citations.

#How a question is split into sub-questions

AI search systems don't just search for the literal question; they often split it into several sub-questions that they run at the same time. Google calls this query fan-out.

In its announcement of AI Mode, Google describes how AI Mode breaks your question down into subtopics and runs a large number of searches at once on your behalf. According to the same announcement, Deep Search uses the same technique, only more extensively. In its documentation on AI features in Search, Google writes that both AI Overviews and AI Mode may use query fan-out: several related searches across subtopics and data sources. According to Google, this lets them show a broader and more diverse set of links than a classic search.

Other systems also search in several rounds. Anthropic writes that searching can be repeated several times within a single request. For reasoning models, OpenAI describes the model searching, analysing the results and then deciding whether to search further.

Here's an example of how that plays out. Say someone asks: "Which accounting software suits a self-employed builder?" A system that applies fan-out might turn that into separate searches, for example about prices, about bank integrations and about experiences of self-employed people. This is an illustration: the platforms don't disclose exactly which sub-questions a system chooses.

The practical consequence is significant. Your page doesn't need to rank for the original question. It can come in through one of the sub-questions. A page that answers one sub-question clearly and completely therefore has its own chance of being chosen as a source.

#Which crawler fetches your page on each platform

Every AI platform uses its own bots, and only the bot for search or user requests determines whether your page can serve as a live source. An AI crawler for training is a different bot from the one for search results. If you block the wrong one in robots.txt, you disappear from the answers without noticing.

• ChatGPT. According to OpenAI, OAI-SearchBot is used to show websites in ChatGPT's search features. Sites that opt out of OAI-SearchBot are not shown in ChatGPT's search answers. ChatGPT-User visits pages for actions a user initiates, for example when someone asks a question. OpenAI writes that robots.txt rules may not apply in that case, because the user initiates the action.
• Perplexity. According to Perplexity's documentation, PerplexityBot is designed to show and link websites in Perplexity's search results and is not used to train models. Perplexity-User fetches pages in response to a user's question and, according to Perplexity, generally ignores robots.txt.
• Google AI Overviews and AI Mode. These features are part of Google Search. Google names robots.txt rules for Googlebot as the way to control access. Google-Extended is a separate robots.txt token without its own user agent. It determines whether content may be used for training Gemini models and for grounding in Gemini Apps and Vertex AI. According to Google, it has no effect on inclusion in Google Search.
• Claude. In its help article on crawlers, Anthropic describes three bots: ClaudeBot for content that may contribute to training, Claude-User for pages fetched when a user asks Claude a question, and Claude-SearchBot to improve the quality of search results.

How to allow or block each of these bots separately is covered in the article on AI crawlers in robots.txt.

#Platform differences at a glance

The platforms differ in where their live results come from, which bot reads your page and how they show sources. The table only contains what the platforms themselves document. A dash means we couldn't confirm it in primary documentation.

Note: the citation format for ChatGPT and Claude comes from the developer API documentation. The consumer apps may display sources differently.

#What Google says about AI Overviews

Google states that no additional technical requirements or special optimisations are needed to appear in AI Overviews or AI Mode. That is stated explicitly in its documentation on AI features. A page must be indexed and eligible to be shown with a snippet in Google Search. According to Google, that is no guarantee the page will actually be used.

Google also names the controls for limiting this: nosnippet, data-nosnippet, max-snippet and noindex. If you use any of those directives, you also limit what AI features may show from your page. So check that you haven't accidentally set them on important pages.

#What makes a passage quotable

A passage is easier to cite if it fully answers a question independently of the rest of the page. With grounding, the model processes chunks of text, not whole websites. It attaches a source to a specific claim in the answer.

Anthropic illustrates this well: according to its documentation, each citation contains the URL, the title and up to 150 characters of the cited text. So the system cites at sentence level. That leads to a few practical characteristics, which are no guarantee but do match how these systems work:

1. A self-contained opening sentence. Start every section with a sentence that answers the question without the reader needing the paragraph before it. Avoid "this", "as above" and other references to elsewhere.
2. Concrete, verifiable facts. A sentence with a name, price, date or definition is more useful as support than a general claim.
3. Headings that name the question. A heading phrased as a question or a clear statement helps link the passage to one of the sub-questions from query fan-out.
4. Text in the raw HTML. A bot that fetches your page needs to be able to read the text. With server-side rendering, the content is directly in the HTML, without JavaScript having to run first.
5. Clear entities. Use schema markup for AI to make explicit who you are, what you offer and which profiles belong to you.
6. Signals of trustworthiness. An author, a date, source citations and a contact page are the basis of E-E-A-T.

How to put this into practice in your writing is covered in the article on content that AI cites.

#What isn't public

None of the major platforms has published the ranking factors that determine which of the pages found an AI system ultimately cites. Google, OpenAI, Perplexity and Anthropic do describe which bots they use and what citations look like, but not how they weigh sources against each other.

So be wary of lists promising "the 10 ranking factors of ChatGPT". They are based on observations from the outside, not on documentation. What is certain is the order: a page that isn't crawled, isn't indexed or isn't readable can never be chosen as a source. That's where you start.

#How to check this on your own site

You can check the technical basics for live citations yourself in a few steps.

1. Open your robots.txt at yourcompany.com/robots.txt and check that OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot aren't blocked. Blocking GPTBot or Google-Extended affects training and grounding in Gemini Apps, not your inclusion in Google Search.
2. Test your firewall. Some firewalls and CDNs block AI bots by user agent, even if robots.txt allows them. Ask your hosting provider or administrator whether that's how your setup is configured.
3. View the raw HTML of an important page via "View page source". Is your main text in there, or does it only appear after JavaScript?
4. Search for nosnippet and noindex in the source code of pages you want to see in AI Overviews.
5. Read your headings and opening sentences on their own. Does every first sentence under an H2 answer the question by itself?
6. Ask your own questions in ChatGPT with search, Perplexity and Google AI Mode, and note which sources they show.

Flowproof's free scan checks steps 1 to 3 automatically, including a firewall test with the GPTBot user agent, and shows for each section what you can improve. You can start it by signing up for free.