robotcheck

A 1-10 readability score is useful, but the gap we kept worrying about was between readable and actually fetched. On our job board we added llms.txt, plain markdown mirrors of every page and a homepage that renders without JavaScript, so any parser would score it well. Then we still had to check separately that AI user agents get the same response a browser gets, because a CDN or WAF rule can hand them a challenge page and nothing in the HTML you analyse will show it. Does RobotCheck request the site with the real crawler user agents like GPTBot, ClaudeBot or PerplexityBot and compare the responses, or does it score the HTML fetched with a generic client?