Arnout Hellemans advises that you don’t have to provide AI crawlers with full access to all of your content.

Arnout says: “People really need to look at their technical setup.

With that, I mean the rendered version versus the raw HTML – especially with a lot of AI crawlers not rendering yet, and also because I've seen a lot of discrepancies in that area.”

Is the rendered version typically quite different for different search engines?

“As with a lot of things in SEO, it depends.

Sometimes, when websites are built using JavaScript frameworks, the content is actually different in the raw HTML versus the rendered HTML.

Titles might be different. There might not be schema markup, there might be different headings, etc., because those can be changed by the execution of JavaScript.

That can severely impact the discoverability of your page.”