JW2D OÜ

How to get cited by Perplexity: crawlers, site settings and content

Yakiv Romanchuk, founder of JW2D · Published

Short answer

To appear in Perplexity answers, your site must be reachable for PerplexityBot — the crawler Perplexity says is designed to surface and link websites in its search results. Perplexity recommends allowing it in robots.txt and states that it is not used to crawl content for AI foundation models. After that, content decides: in the GEO research, which was also tested on Perplexity, adding quotations, statistics and sources increased a page’s visibility in answers.

Perplexity’s two crawlers

Agent What it does (according to Perplexity) robots.txt
PerplexityBot Surfaces and links websites in Perplexity search results; not used to crawl content for AI foundation models Respects robots.txt; Perplexity recommends allowing it
Perplexity-User Visits a page when a user asks a question, to help answer and link to the page; not used for crawling or training Generally ignores robots.txt, because the fetch is requested by a user

Perplexity publishes the IP ranges of both agents (perplexitybot.json and perplexity-user.json), so they can be allowed in a firewall or CDN, and notes that changes may take up to 24 hours to be reflected.

Technical checklist

  1. robots.txt. Make sure PerplexityBot is not disallowed — neither by name nor by a general “block all bots” rule.
  2. Firewall and CDN. Bot protection can block crawlers even when robots.txt allows them. If you use Cloudflare or a similar service, check the bot settings and Perplexity’s published IP ranges.
  3. Content in HTML text. In our practice, text that appears only after scripts run, or only in images, is read less reliably by AI crawlers.
  4. Search indexes. Keep the site indexed in Google and Bing as well: answer engines and their users rely on the open web, and good indexing helps everywhere.

What content gets cited

The research paper that introduced generative engine optimization tested its methods both on a benchmark and on the live Perplexity.ai. The authors report that adding quotations, statistics and citations of sources worked best, with improvements on Perplexity of up to 37% on one of their visibility metrics, while keyword stuffing gave little to no improvement.

In practice this means:

  • Answer the question in the first sentences of a page or section.
  • Use numbers: prices, timeframes, measurements, dates.
  • Cite sources: link to studies, official documentation and data.
  • Quote experts, including your own team, with names and roles.
  • Keep company facts consistent across your site, directories and profiles (our practice).

How to check whether Perplexity cites you

Perplexity shows its sources next to each answer. Ask it the questions your customers ask, in their languages, and record whether your pages are among the sources and who is cited instead. Repeat monthly with the same list to see the trend.

Sources

  1. Perplexity Docs. Perplexity Crawlers
  2. Aggarwal P. et al. GEO: Generative Engine Optimization. arXiv:2311.09735, KDD 2024

FAQ

How do I get my website into Perplexity?

Allow PerplexityBot in robots.txt and in your firewall or CDN, make sure the content is available as text, and answer customer questions clearly with facts and sources.

Does Perplexity train its AI on my website?

Perplexity states that PerplexityBot is not used to crawl content for AI foundation models, and that Perplexity-User is not used for crawling or training.

Should I block Perplexity-User?

Perplexity says this agent generally ignores robots.txt because the fetch is requested by a user. Blocking it also means users cannot get your page as a source when they ask about it.

How long until robots.txt changes take effect?

Perplexity says it may take up to 24 hours for its systems to reflect changes.

What content does Perplexity cite more often?

In the GEO research, which included tests on Perplexity, pages with quotations, statistics and cited sources gained the most visibility; keyword stuffing did not help.

Want AI to recommend your business?

We will check what ChatGPT, Gemini and Perplexity say about your niche today and show you where to start.

Discuss your project About the service WhatsApp

More articles