For Webmasters

ARPAIBot Crawler

Information about our web crawler: what it does, how it works, and how to control its access to your website

What is ARPAIBot

ARPAIBot is the web crawler of ARPAI GEO. It visits public pages of websites at the request of our clients to analyze brand visibility in AI search systems.

ARPAI GEO is a SaaS platform for monitoring brand presence in AI-generated responses (ChatGPT, Perplexity, Google AI). The crawler collects publicly available company information to set up brand monitoring profiles and perform GEO site audits.

Technical Specifications

User-Agent
ARPAIBot/1.0 (+https://arpai-geo.com/webmaster.html)
Bot type
Web crawler
Protocol
HTTPS (GET requests only)
robots.txt compliance
Yes, fully compliant
Crawl rate
No more than 1 request per 3 seconds per domain
Crawls
Public HTML pages only
Does not crawl
Private sections, APIs, forms, files (PDFs, images)

What ARPAIBot Does

The crawler performs two types of tasks at the request of ARPAI GEO clients:

GEO Site Audit

Analyzes public pages of a client's website for AI indexing readiness: structured data presence, content quality, and technical parameters. Results are only available to the website owner (the client).

Company Profile Analysis

Reads publicly available information from the client's website homepage: company name, business description, and contacts. This data is used to configure brand visibility monitoring in AI search engines.

Data Protection & Privacy

ARPAIBot automatically filters personal data from collected information. The following are never stored:

We comply with GDPR and applicable data protection regulations. For more details, see our Privacy Policy.

How to Block ARPAIBot

If you do not want ARPAIBot to crawl your website, add the following lines to the robots.txt file in the root of your site:

# Block ARPAI GEO crawler User-agent: ARPAIBot Disallow: /

The crawler checks robots.txt before each visit and fully complies with all specified restrictions. Changes take effect on the next scheduled crawl.

Partial blocking: you can allow crawling of specific sections only. For example, to allow only the homepage:

User-agent: ARPAIBot Allow: /$ Disallow: /

Contact Us

If you have questions about ARPAIBot or want to report an issue:

Website
Operator
Sole Proprietor Dmitry Sergeevich Volchenko (TIN: 343571017093)

We respond to webmaster inquiries within 2 business days.

About ARPAI GEO

ARPAI GEO is a SaaS platform for monitoring, analyzing, and optimizing brand visibility in AI-generated search results. The service tracks how ChatGPT, Perplexity, and Google AI recommend your brand and helps improve your positions.

Key features: monitoring across 3 AI systems, geographic analysis by city and region, product monitoring, GEO site audit, source analysis, content generation, and promotion strategy.

Learn more about ARPAI GEO capabilities →