How AI Engines Choose Sources to Cite (What Is Known)

By Yahav Fuchs •

Short answer: the engines tell you who crawls, not how they choose. What is documented is access: which bots exist and how to allow them. What is not documented is the scoring that decides which retrieved pages end up cited. Everything beyond access is inference from observation, including ours, so hold it loosely and test it on your own prompts.

  • Documented: crawler names, purposes and robots.txt behavior.
  • Documented by Google: AI features need no special technical requirements beyond standard search eligibility.
  • Not documented by anyone we checked: the weighting that picks citations from retrieved pages.
  • Observed, across our own studies: matching the prompt’s wording and intent is the strongest consistent signal.
  • Engines differ, and they change. Check the date on anything you read, including this.

I wrote this post as of November 2026. Where I rely on a vendor’s documentation, I read it directly. Where I rely on third-party reports, I say so.

A model of what happens

Most answer engines follow a similar outline, even though the details differ:

  1. Interpret the question. The engine may rewrite it into one or more search queries.
  2. Retrieve candidates. It searches an index, which might be its own, a partner’s or a general web index, and gets a list of pages.
  3. Read and select. It reads some of those pages, fully or in part, and decides which support the answer.
  4. Generate and cite. It writes the answer and links some of the pages it used.

A page can fail at any step. It may not be retrievable at all (blocked or not indexed). It may be retrieved but not selected (a poorer match, or a source the engine trusts less). It may be used but not shown as a citation. When you ask “why was I not cited?” you are asking which step failed, and the answer is different for each.

What is documented, engine by engine

ChatGPT

OpenAI documents three crawlers. According to its bots documentation, OAI-SearchBot is for surfacing sites in ChatGPT’s search features, GPTBot collects content that may be used to train its models, and ChatGPT-User handles actions a user triggers. The documentation says that sites that opt out of OAI-SearchBot do not appear in ChatGPT search answers, though they can still show up as navigational links, and that the settings for the crawlers are independent.

What is not documented: how retrieved pages are ranked or which ones are cited. What we have observed in our own data is that the share of citations going to specific domains can shift quickly. In ReddGrow’s tracking of ChatGPT answers, Reddit’s share of citation rows fell from about 3% to under 1% over roughly a month in mid-2026, while other source types gained. That is an observation about outputs. Why it happened was not explained by OpenAI as far as we could find, so we do not claim a cause. See why ChatGPT cites Reddit and how to rank in ChatGPT search.

Perplexity

Perplexity’s bot documentation describes PerplexityBot, which surfaces and links sites in Perplexity’s search results and is not used to crawl content for foundation models, and Perplexity-User, which handles user-initiated fetches and generally ignores robots.txt because a person triggered it. To appear in its results, the documentation says to allow PerplexityBot and permit requests from its published IP ranges.

Beyond access, the selection logic is not documented. Perplexity shows its sources prominently by design, so citation behavior is easy to observe. In ReddGrow’s Reddit analysis, Perplexity cited almost only threads that were also cited many times by others, rather than one-off threads, which suggests a narrower set of favored sources. That is a pattern in our sample, not a rule from Perplexity. Details for the engine are in how to rank in Perplexity.

Google AI Overviews and AI Mode

Google is the most explicit. Its documentation on AI features says a page must be indexed and eligible to appear in Search with a snippet, that there are no additional technical requirements, and that you do not need new machine-readable files, AI text files or markup. Traffic from these features is included in the Web search type in Search Console’s Performance report.

In practical terms, classic search fundamentals apply: crawlability, helpful content, internal links. Google does not publish how it picks which of the eligible pages to link in a generated summary. In our YouTube citation study, Google’s AI summaries accounted for about 76% of the video citation rows in our data, so Google’s AI features are clearly a large consumer of video and forum content. See how to rank in Google AI Overviews.

Gemini

Gemini is Google’s assistant. Public guidance for publishers on Gemini specifically is thinner than for Search, and Google’s AI features documentation covers AI Overviews and AI Mode rather than the Gemini app. In our tracking, Gemini’s citation behavior differed from Google’s AI summaries: it made up a small share of rows in our YouTube data, and in our Reddit data Gemini cited almost only threads that were cited many times. Treat any claim about Gemini’s selection as observation. We cover it in how to rank in Gemini.

Claude

Anthropic offers web search in Claude. Third-party reports say the search backend is Brave Search and describe separate crawlers for training, search indexing and user-triggered fetches, but I could not find Anthropic’s own documentation in my searches, so I am treating those details as unverified. If Claude matters for your category, check Anthropic’s current published crawler guidance before changing robots.txt. Our guide is how to rank in Claude.

Copilot, Grok and Meta AI

Each has its own retrieval and its own level of documentation, and I have not verified their selection behavior from primary sources for this post. We have individual guides: Copilot, Grok and Meta AI. The safe approach is the same for all three: measure what they cite for your prompts and compare it to the others.

What we have observed about selection

None of the following is a platform statement. It is what our own citation data shows, and I will keep the caveats attached.

Prompt match is the strongest signal we measured. In ReddGrow’s studies of cited Reddit threads (994 threads, drawn from 42,407 unique threads in 90 days), LinkedIn content and YouTube videos (995 videos), pages that directly answered the prompt that surfaced them were cited more than loosely related pages. For Reddit threads, prompt words in the title and the opening separated heavily cited from once-cited threads better than upvotes. For YouTube, the share of prompt words in the title and description was the largest content signal, and videos that stated the question early did better.

Popularity is a weak signal. The median heavily cited Reddit thread had 11 upvotes. For LinkedIn, author follower counts were nearly identical between most-cited and once-cited content. YouTube subscriber count did not matter, though views per day did.

Format depends on intent. For Reddit, roundups and guides were more often heavily cited than opinion threads or troubleshooting questions. Best-of prompts were answered mostly by recommendation requests and roundups, and how-to prompts by how-to questions and guides.

Length is not a universal lever. Long Reddit posts did better, but YouTube video length did not matter, and LinkedIn articles that were cited were plain, around 1,200 words.

Engines differ in what they pick. In the LinkedIn study, Google AI and Perplexity cited mostly articles while ChatGPT cited mostly short posts, with the weakest topical match. Once-cited Reddit threads came largely from Google’s AI summaries.

Observation is not causation. Engines may cite these items because they already rank well or have comments that answer the prompt, not because of their layout. We say this in each study, and it applies here.

What this means in practice

Given how little is documented, the sensible approach is to do the things that work across steps of the pipeline:

  1. Be retrievable. Allow the relevant search crawlers (OAI-SearchBot, PerplexityBot, and Google’s standard crawling) and check firewall rules. Verify with server logs.
  2. Answer the exact question. Put the question’s wording in the title or heading and the answer in the first lines. This lines up with the strongest signal we saw.
  3. Be corroborated. For recommendation prompts, independent sources carry weight. Appear in roundups, review sites and community discussion that engines already cite for your category.
  4. Measure by engine. Build a fixed prompt list, run it repeatedly and record the sources. How to track brand mentions in AI search shows how.
  5. Re-check regularly. Citation behavior moves, and what worked in spring may not in autumn.

Claims I would be skeptical of

  • “Add this schema type and you will be cited.” Google’s documentation says no special markup is needed for its AI features. Other engines have not documented a schema effect that we could find.
  • “Publish a llms.txt file.” We went through the evidence in does llms.txt matter. It is unproven.
  • “Engine X always prefers source Y.” Source preferences shift. Ask for a date and a sample size.
  • “We reverse-engineered the algorithm.” Nobody outside these companies has, and this includes us.

How to check this yourself

You do not need to take anyone’s word, including mine. Pick 30 prompts from your category, run each several times in each engine, and record the sources. Count which domains and page types recur. That table of recurring sources is a more reliable guide to what each engine picks for your market than any general article, and it takes an afternoon.

If you want that tracked continuously, along with the Reddit threads behind your citations, ReddGrow does this for the engines listed on that page. Otherwise a spreadsheet and some discipline will tell you most of what you need.

Frequently Asked Questions

Do AI engines publish how they choose sources?
Only partly. Most publish crawler documentation and basic guidance, such as which bots exist and how to allow or block them. None publishes the ranking logic that decides which retrieved pages get cited, so most claims about why a page was cited are inference from observation.
Is getting cited the same as ranking in Google?
Not exactly. Google says its AI features use the same index and need no special technical requirements, so classic search fundamentals matter there. Other engines use different retrieval, and a page that ranks well in Google may not be cited in ChatGPT or Perplexity, or the reverse.
Do all AI engines cite the same sources?
No. In ReddGrow's tracking data, different engines cite very different mixes of domains and formats. For example, in our YouTube analysis Google's AI summaries accounted for most of the video citations, while in our Reddit analysis other engines leaned on a smaller set of repeatedly cited threads. Always measure each engine separately.
Can I block AI search crawlers and still be cited?
Generally no, for engines that rely on their own search crawler. OpenAI's documentation says sites that opt out of OAI-SearchBot do not appear in ChatGPT search answers, and Perplexity says to allow PerplexityBot to appear in its results. User-triggered fetchers may behave differently and are documented separately.
What is the single most reliable way to get cited?
Have a page or thread that directly answers the exact question, be readable by the engine's crawler, and be corroborated by independent sources. In our studies of Reddit, LinkedIn and YouTube citations, matching the prompt's wording and intent was the strongest consistent signal.