BrightCloud Master URL Database Overview

The BrightCloud Master Database is the largest and most accurate URL database in the world.

The BrightCloud Master Database has been built from the ground up to support the large numbers of sites that have to be captured and classified if a web filtering vendor intends on mitigating the threats across the Long Tail. Here are some key characteristics:

• More than 100 millions sites classified
• Over 70 categories
• A URL may have multiple categories
• Each category has a confidence score

Note that the BrightCloud Master Database is more than 5 times larger than the leading enterprise URL databases. The classification methodology utilizes a combination of state of the art automated URL classification technology, as well as human review.  This combination has enabled BrightCloud to build the largest and most accurate URL database in the world, extending web filtering's value propositions across the internet's Long Tail.  Additionally, because each of a given URL's categories have a confidence score, a rich policy model can be built for a given integration. 

View a full list of categories.

BrightCloud Feedback Loop

An important aspect of the BrightCloud Master Database is that it benefits from the browsing habits of all end users. Whenever a user visits an uncategorized page, that URL is anonymously logged and queued for categorization. The site is categorized via either automated or human classifiers and the database is updated. Consequently, subsequent end users benefit by having the most frequently visited web sites and their categories cached locally on the OEM's solution.

Latency

An often posed question about providing a URL database service via a hosted model is whether the latency associated with returning a category is a burden on the end user. There are three layers, all of which can contribute to reducing the latency:

• Local cache: Depending on the integration, the BrightCloud Agent may have the URL cached locally, which is an extremely quick lookup.
• BrightCloud Service cache: The BrightCloud Service also has a cache of classified URLs
• If a URL is not found in either cache, the service will query the BrightCloud Master Database.

Because the network packed is very small, and the cache and Master Database queries are very efficient, the BrightCloud Service will return an answer well before the elements of the web page itself have been received by the end user. In short, the BrightCloud Service is so efficient and lightweight that it can return a category faster than a typical web itself can be rendered.