How Perplexity indexes content

Indexing, retrieval, and citation are different events

“Perplexity indexed my page” is often used to mean several different things. Discovery is when a system learns a URL exists. Crawling is a request to fetch it. Indexing is storing or organizing information for later search. Retrieval is selecting material in response to a particular question. Citation is the final choice to show a source alongside an answer. Perplexity’s public material does not expose a complete pipeline or promise that every discovered URL passes through all these stages.

For an agency report, name the observed outcome: a logged fetch, a Search API result, or a citation in a tested answer. These checks are not interchangeable.

What Perplexity says its crawlers do

Perplexity’s official crawler documentation identifies PerplexityBot as a crawler designed to surface and link websites in search results; it says this bot is not used to crawl content for foundation model training. The same page describes Perplexity User as supporting user actions and says it may visit a page when a person asks a question.

The documentation set is not perfectly aligned: the crawler page says the user fetcher generally ignores robots.txt, while the Help Center robots.txt article, last modified September 14, 2026, says a previous URL summary feature was disabled and that PerplexityBot follows robots.txt. Treat that behavior as subject to change; do not design around a presumed robots.txt bypass.

Robots.txt: control crawler access, not guaranteed disappearance

The Help Center says PerplexityBot will not index full or partial text that a site disallows through robots.txt, but a blocked page may still be represented by its domain, headline, and a brief factual summary. Blocking text access is not a reliable way to remove every trace of a URL. If a page must stay private, use appropriate access controls; robots.txt alone does not erase information already known elsewhere.

Perplexity’s crawler documentation says its settings work independently and can take up to 24 hours to be reflected. Verify the current official guidance before changing policy, especially where the content is sensitive. A basic rule that permits the named crawler is straightforward, but should only be deployed if public crawling is intended.

User-agent: PerplexityBot
Allow: /

What query time search documentation tells us

Perplexity’s Search API documentation describes real time web search, ranked structured results, and a distinction between results and generated answers. Perplexity’s Help Center separately describes a user experience that interprets a question, searches the web, summarizes information, and provides numbered citations. Together, these sources indicate that a question can initiate search and source selection. They do not disclose the data stores, crawl schedule, freshness rules, ranking weights, or citation threshold across Perplexity products.

The API is a developer facing search surface, not a public specification of the consumer product’s entire index. For broader retrieval concepts, see how LLMs decide what to cite and the RAG explainer.

A practical model for diagnosing a page

Use ordinary web access stages as a diagnostic framework, not as a claim about Perplexity’s private implementation. Confirm that the canonical URL resolves without a login, soft 404, or redirect loop. Check robots.txt and page directives, then inspect the returned HTML for the answer text rather than an empty interactive shell. Ensure internal links point to one canonical URL; review server or CDN logs and test relevant questions. Fix any concrete access problem before adding schema or rewriting copy.

The content chunking guide discusses how self contained sections help readers and retrieval systems use a page.

Make the answer easy to verify and reuse

A useful page needs no special “Perplexity format.” It should state the answer near the relevant heading, define unfamiliar terms, separate facts from recommendations, and show where important claims come from. For example, a software agency explaining its support policy should name the response window, the plan it applies to, the exceptions, and the date the policy was reviewed.

“We respond quickly” is hard to quote accurately; “Standard plan support is available on business days, with a target first response within one business day” is specific provided it is true and kept current. Link the primary policy, name who it applies to, and state exceptions. Structured data can clarify entities or page type when it accurately reflects visible content, but it cannot manufacture authority or guarantee retrieval. The answer first writing guide covers this editorial pattern.

A crawl and firewall check for site teams

If Perplexity requests seem absent, check infrastructure first. Review the live robots.txt response and page directives; inspect CDN, server, and WAF logs around the test period; and distinguish requests by user agent and source IP. Perplexity recommends checking published IP ranges rather than trusting user agent text alone. Use the current PerplexityBot IP endpoint and, where relevant, the Perplexity User IP endpoint, not a stale copied range. Match each agent only to its own ranges.

Confirm the response is the page, not a challenge, rate limit response, or empty template. Do not disable security controls broadly just to encourage a crawler.

Measure what you can observe

Separate access, visibility, and outcomes in an agency measurement plan. Access evidence includes verified bot requests and successful responses. Search visibility can be checked with repeatable queries and, where useful, the Search API; record the query, date, locale, result URL, and whether the page text appears. For citation monitoring, save the prompt, answer, cited URL, and context; responses can vary with wording and product behavior. Track qualified visits or assisted conversions where analytics can attribute them responsibly.

Do not equate a crawler hit with an impression, or a citation on one test with stable placement. The AEO KPIs and tracking guide offers a broader measurement framework, while the Perplexity SEO guide covers platform level planning. For the wider mechanics path, visit the How AEO works hub and compare the sibling guide on how ChatGPT cites web sources.