BrightCrawler


What is BrightCrawler?

BrightCrawler is BrightCloud's bot that collects content of webpages for the purpose of classifying websites.



Why is BrightCrawler crawling my site?

BrightCrawler attempts to find and classify every website on the world wide web in order to create the most comprehensive information repository of internet information available.



Does BrightCrawler crawl my site responsibly?

Yes. While we do crawl the home page of your site, we do not crawl beyond that if your robots.txt file prohibits it. We also go to great lengths to reduce our impact on your site's performance. For example, we use delays between page requests and will not overload a single IP address even if it's hosting multiple domain names. We also minimize the total number of times we access your site.



I feel my site is being abused by BrightCrawler, where can I contact you about it?

BrightCrawler is very lightweight and should not have an impact on your website's bandwidth, but if you still feel a need to address the issue, you can contact BrightCloud at brightcrawler@brightcloud.com