You may or may not not be bored of everyone talking about AI – but like it or not, it’s here and it’s not exactly going anywhere so at this point we have to either get on board or go and lay down in a dark room. I’ve chosen to get on board as I won’t have a business in a few years time unless I do. Like me you probably used to Google everything but now you’ve found yourself asking ChatGPT, Claude, Perplexity or Google’s AI Overviews to recommend a supplier instead of scrolling through ten blue links opening each one in a new tab. If your WordPress site isn’t part of those answers, you’re invisible to that whole group of buyers. Not many people make it to the traditional search engine results page anymore.
The good news is that most of what gets you cited is good, old-fashioned web practice done properly. There’s no magic plugin and no secret file. No need for dodgy link building sourced from a far away country. There are, however, a handful of WordPress settings that quietly lock AI tools out, and a few content habits that make your pages much easier for AI to quote.
In this article I’ll walk you through how AI assistants find and choose sources, what to check on your WordPress site, how to write pages that get quoted, and how to tell whether any of it is working. Following this… you’ll probably opt to lay down in a dark room – and I won’t blame you.
Why it matters now
AI answers are taking clicks that used to go to websites. A Pew Research study found that people clicked a result on 8% of Google searches when an AI summary appeared, compared with 15% when it didn’t. Only 1% clicked a link inside the summary itself.
At the same time, AI tools are sending more visitors than ever. Similarweb data reported by TechCrunch showed ChatGPT referrals to news sites growing from under 1 million to over 25 million in a year. That’s a pretty astonishing stat.
So there are fewer clicks to go round, and a bigger share of them comes from AI. Being the business an AI names in its answer is quickly becoming as important, if not more, as ranking on page one.
How AI assistants find your website
AI tools read your site in three different ways, and each one needs to be able to reach it:
- Training. Crawlers such as OpenAI’s GPTBot and Anthropic’s ClaudeBot collect pages to train future models. This shapes what the AI “knows” about you without searching. It’s slow, and it’s months out of date by the time a model ships.
- Search. When an assistant searches the web, it uses its own index or a search engine’s. OpenAI says ChatGPT search shares queries with Bing, and Google’s AI Overviews come from Google’s normal index. This is where most recommendations come from, so it’s the one to get right.
- Live visits. When someone pastes your link or asks about your business, tools like ChatGPT-User and Claude-User fetch the page there and then.
The practical upshot is simple. Your normal search engine optimisation (SEO) still does most of the heavy lifting, because AI search sits on top of ordinary search indexes. Google says outright that there are “no additional requirements to appear in AI Overviews or AI Mode”. If Google and Bing can’t see your page, no AI can recommend it.
The technical side: what to check on your WordPress site
1. Make sure the right bots are let in
Each AI company runs separate bots for training and for search, and you can treat them differently. Many businesses are happy to block training but definitely want to appear in AI search results. Here are the main ones:
| Bot | Company | What it does | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search | You won’t be shown in ChatGPT search answers |
| GPTBot | OpenAI | Collects pages for training | Your content stays out of future training |
| Claude-SearchBot | Anthropic | Indexes pages for Claude’s search | Your visibility in Claude may drop |
| ClaudeBot | Anthropic | Collects pages for training | Your content stays out of future training |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers (not training) | You won’t be surfaced in Perplexity |
| Google-Extended | Controls Gemini training and grounding | No effect on Google Search or AI Overviews | |
| Applebot-Extended | Apple | Controls Apple Intelligence training | No effect on Apple search results |
That last pair catches people out. Blocking Google-Extended doesn’t remove you from AI Overviews, because those are built from the normal Googlebot index.
You control all of this in robots.txt. In WordPress, Yoast SEO and Rank Math both let you edit it without touching the server. A sensible starting point for most businesses is to allow everything. If you’d rather keep out of training while staying visible in search, add:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
2. Check nothing is blocking them by accident
This is where we find most problems. Bots are often blocked without anyone deciding to block them:
- Cloudflare. Since July 2025, Cloudflare has asked new domains whether to block AI crawlers by default. If you said yes, or someone else did, check AI Crawl Control in your Cloudflare dashboard and set each crawler to Allow or Block.
- Security plugins and firewalls. Rate limiting and “bad bot” rules can turn away legitimate AI crawlers. Check your security plugin and your host’s firewall logs.
- The “Discourage search engines” box. Under Settings > Reading in WordPress, this box often gets left ticked after a site goes live. It hides you from everyone, AI included.
3. Get indexed by Bing, not just Google
ChatGPT search leans on Bing, so set up Bing Webmaster Tools and submit your sitemap. Yoast and Rank Math both generate one for you. If you’re a local business, it’s also worth claiming a Bing Places listing. In our own testing, Bing Places listings were regularly quoted in AI answers to “best X in London” type questions.
4. Add structured data
Schema markup tells machines exactly what a page is about. On WordPress, Yoast and Rank Math handle the basics: Organization or LocalBusiness, Article, Breadcrumbs. Make sure your business name, address, logo and social profiles are filled in correctly, and add FAQ markup where you genuinely answer questions.
5. Keep your important text in the HTML
If key information only appears inside a slider, a tab loaded by JavaScript, an image or a PDF, there’s a real risk a crawler won’t see it. A quick test: right-click your page, choose “View page source” and search for a key sentence. If it isn’t there, move it into the page itself. Fast, clean pages help too, as crawlers have limited time for each site.
6. llms.txt: nice to have, not essential
llms.txt is a proposed file that gives AI tools a tidy summary of your site. Yoast generates one for free and Rank Math has a module for it, so it costs you nothing. Just don’t expect miracles. Google says extra AI files “will neither harm nor help“, and none of the major AI companies has said it uses them.
Writing content that gets quoted
In September we ran 11 typical buying questions through an AI search tool, things like “WordPress developer London” and “how much does a bespoke WordPress website cost”. Then we looked at which pages it quoted. The patterns were very clear.
One page per service, named like the question
Almost every time, the page that got quoted was a dedicated page about exactly that thing, with the topic in the web address. For example /woocommerce-developer-london/ or /charity-web-design/. General “Our services” pages that list ten things in a grid barely got a look in.
We saw it on our own site too. Our page about AI and WordPress development was the first source quoted for that question. Nobody else had a page dedicated to it.
What to do: give each service you want to be recommended for its own page, with a clear title and web address.
Answer the question in the first paragraph
AI tools pull short, self-contained chunks of text. A page that opens with two paragraphs about your passion for excellence gives them nothing to use. Start each page with a plain one- or two-sentence answer: what you do, who for, where, and roughly what it costs. Then go into detail.
Use real facts and numbers
Vague claims don’t get quoted. Specifics do: prices, timescales, named clients, results, locations. In our test, the pricing questions were answered by quoting other agencies’ cost guides. Publishing your real price ranges is one of the easiest ways to get into answers your competitors are missing.
Structure pages like a good reference
- Use proper headings (H2, H3) that read like the questions people ask.
- Keep paragraphs short, with one idea each.
- Use lists and tables for steps, comparisons and specifications.
- Add a short FAQ section with honest, direct answers.
- Show who wrote the page and when it was last updated. AI tools, like people, prefer fresh information from a named expert.
What happens away from your website matters too
AI answers don’t only quote your own site. They also lean heavily on what other websites say about you, so a few things off your site make a big difference.
- “Best of” lists and comparison articles. In our test, one “top 10 agencies” article was the only source behind two of the most valuable questions. The agencies in it got named; everyone else didn’t. Find the lists in your sector and ask to be included, or write a fair comparison yourself.
- Review and directory profiles. Profiles on Google Business Profile, Bing Places, Clutch, Trustpilot and sector directories get read and quoted, ratings included. Claim yours, fill them in completely and ask happy clients for reviews.
- Consistent details everywhere. Use the same business name, address, phone number and description of what you do on every profile. If the AI sees five slightly different versions of you, it’s less sure who you are.
- Being mentioned by others. Guest articles, podcast appearances, case studies on a client’s or partner’s site, and local press all help AI tools connect your name with your specialism.
How to tell if it’s working
There’s no perfect dashboard for this yet, but you can get a good picture from three places:
- Ask the AI tools yourself. Write down 10 to 15 questions a customer would really ask, like “best [your service] in [your town]” or “how much does [your service] cost”. Run them in ChatGPT, Claude, Perplexity and Google every month or two, and note who gets named and which pages are quoted. Use a private window so your own history doesn’t skew the results.
- Check your analytics for AI visitors. In Google Analytics 4, look for visits coming from chatgpt.com, perplexity.ai, claude.ai, gemini.google.com and copilot.microsoft.com. Group them into their own channel so you can track them over time. Bear in mind that Google counts AI Overviews and AI Mode clicks as ordinary web search in Search Console.
- Check your server or Cloudflare logs for AI bots. If OAI-SearchBot, Claude-SearchBot and PerplexityBot never visit, something is stopping them. If they visit and get errors, something is blocking them.
Your AI optimisation checklist
- “Discourage search engines” is unticked under Settings > Reading
- robots.txt allows OAI-SearchBot, Claude-SearchBot and PerplexityBot
- Cloudflare AI Crawl Control is set the way you actually want it
- Your security plugin and firewall aren’t blocking AI crawlers
- The site is verified in Bing Webmaster Tools and Google Search Console, with sitemaps submitted
- Organization or LocalBusiness schema is complete and accurate
- Each key service has its own page, with the topic in the title and web address
- Every key page opens with a short, direct answer
- Prices, timescales and real examples are on the page, not hidden in a PDF
- Google Business Profile, Bing Places and directory profiles are complete and consistent
- You’re on the main “best of” lists for your sector, or working on it
- AI referral traffic has its own channel in GA4
- You test your key questions in AI tools every month or two
The short version, finally
AI optimisation isn’t a separate dark art. It’s good SEO, a technically sound WordPress site, and content that answers real questions with real facts. Most of the sites we look at aren’t missing a clever trick. They’ve got a bot blocked by accident, a services page trying to do ten jobs at once, or no prices anywhere.
If you’d like us to check how your WordPress site shows up in ChatGPT, Claude and Google’s AI answers, and fix what’s holding it back, get in touch. It’s what we do day in, day out, alongside WordPress and AI development and ongoing website support.
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web?
- Perplexity: Perplexity crawlers
- Google: Common crawlers, including Google-Extended
- Google: AI features and your website
- Google: AI optimisation guide
- Apple: About Applebot
- Cloudflare: AI crawlers blocked by default (July 2025)
- Cloudflare: Manage AI crawlers
- llmstxt.org
- Yoast: llms.txt
- Rank Math: llms.txt
- Pew Research: Google users are less likely to click on links when an AI summary appears (July 2025)
- TechCrunch: ChatGPT referrals to news sites are growing (July 2025)
- OpenAI: ChatGPT search help