BrightCloud employs unique spider and crawling technology called BrightCrawler to visit web sites and pages and collect information for the purpose of classifying web sites.
BrightCrawler attempts to find and classify every website on the World Wide Web in order to create the most comprehensive information repository of Internet information available.
Yes. While we do crawl the home page of your site, we do not crawl beyond that if your robots.txt file prohibits it. We also go to great lengths to reduce our impact on your site's performance. For example, we use delays between page requests and will not overload a single IP address even if it hosts multiple domain names. We also minimize the total number of times we access your site.
BrightCrawler is very lightweight and should not have an impact on your website's bandwidth, but if you still feel a need to address the issue, you can contact BrightCloud at brightcrawler@brightcloud.com.