Crawler

WebeonsBot

WebeonsBot reads a website when someone asks our free SEO audit to check it. It runs only on request, one site at a time. It is not a search engine and it does not build an index.

How it identifies itself

WebeonsBot/1.0 (+https://webeons.com/tools/seo-audit/bot)

What it does

• Reads robots.txt first and obeys it. Rules for the user agent WebeonsBot apply, otherwise the rules for *. A Crawl-delay is respected (up to 10 seconds).
• Reads public pages only, with GET and HEAD requests. It never submits forms, logs in, or runs scripts.
• Keeps to at most 4 requests at once and about 2 requests a second per site.
• Reads up to 25 pages for a quick scan and up to 1,000 for a full audit (up to 6,000 on sites we run ourselves), starting from the homepage and following links. The same site is crawled at most once a day; repeat requests get the saved result.
• For full audits it also checks a sample of up to 60 outbound links and 40 images with HEAD requests, to report broken links and oversized images.
• Refuses private and internal addresses, and stops at 5 MB or 10 seconds per page.

How to block it

Add this to your robots.txt. WebeonsBot then stops before reading any page and the audit says so.

User-agent: WebeonsBot
Disallow: /

Questions or problems

If WebeonsBot causes you any trouble, email hello@webeons.com with the time and your domain, and we will look into it.