Challenge
Beyond search engines, the practice of researching companies and services via generative-AI answers has been spreading. Yet conventional SEO measures alone cannot address how AI crawlers read a site or which pages they cite. Measures were needed to help AI correctly understand and cite the business.
Approach
- 01 Explicitly allowed major AI crawlers—ClaudeBot, GPTBot, PerplexityBot, and others—in robots.txt, and clearly stated the Sitemap
- 02 Served an llms.txt that summarizes the whole site structure for AI, in a form that keeps up as pages are added
- 03 Set up JSON-LD structured data—Organization, FAQ, articles, and more—so AI can interpret page content without misunderstanding
- 04 Confirmed that server-side WAF and access restrictions were not blocking AI crawlers, ensuring crawling actually goes through
- 05 Regularly measure how the business is described and cited by actually searching on major generative-AI tools
Key success factors
- Building the foundation for AI to read
- By aligning crawl permissions, Sitemap, and llms.txt, we first put the site in a state where AI can discover and retrieve it at all.
- Preventing misunderstanding with structured data
- By making the company, services, and FAQ explicit through structured data, we let AI cite the business without getting it wrong.
Solution
We set up AI-crawler acceptance (robots.txt / Sitemap), an AI-oriented site summary (llms.txt), structured data, and server-side crawlability as one continuous flow. We built a foundation on which generative AI can discover the corporate site and correctly cite the business.
Facing a similar challenge?
Whatever your industry or scale, let's start with a conversation.
Contact form
