The following specifications, articles and tutorials are reference materials to get started:
The REST interface for BrightCloud Web Services uses the OAuth protocol to authenticate incoming requests. More specifically, it uses the 2-legged variation of OAuth protocol, since there are only 2 parties involved in a BrightCloud Web Services transaction: the end user ("consumer," in OAuth terms) and the server hosting BrightCloud Web Services ("provider").
The end user role in a Web Services transaction requires registration with the BrightCloud website to generate a key/secret pair. This is a one-time activity. These are your BCWS Secret Access Key and your BCWS Access Key ID. Using the key/secret pair, the end user generates a digital signature in accordance with OAuth protocol for the BrightCloud Web Services request; this request will include the generated signature in the request header (as part of Authorization header). This "signed" request is then sent to the Web Services server.
The server's role in the transaction is to extract the digital signature from the Authentication header in the request, authenticate it, and return the appropriate response if the user is authenticated and the request is valid. Alternatively, the Web Services server will send an error if the authentication fails or there is another problem with the request.
Generating a key/secret pair
You can visit BrightCloud Web Services website to register and generate a unique key/secret pair to be used when making Web Services requests. For scenarios where a user needs Web Services integration with more than one application, a registered user can request more than one key/secret pair and name them appropriately for easy identification and use in their applications.
NOTE: For those who refer to the specification or 3rd party code later, the BCWS Access Key ID is called the Consumer Key and the BCWS Secret Access Key ID is called the Consumer Secret in OAuth parlance.
The rest of this document contains information on the mechanics of correctly creating the OAuth header to sign BrightCloud Web Services API calls. One of the advantages of using a standards based approach to authentication is that there are many libraries and example integrations available to use. There are plenty of OAuth libraries available in many languages; a good resource for these is the OAuth code page.
Signing a BrightCloud® Web Services request
For every request to BrightCloud Web Services, the user needs to generate a digital signature in accordance with the 2-legged OAuth protocol and send it as part of the request headers. The signature should be generated using the HMAC-SHA1 algorithm (BrightCloud Web Services does not support the PLAINTEXT or RSA-SHA1 algorithms).
The 2-legged OAuth approach is slightly different from the more common 3-legged OAuth approach. Since only two parties are involved in the 2-legged OAuth scenario, one doesn't need to request a Request Token or exchange that Request Token for an Access Token. Therefore, a blank string should be used in place of the Access Token and the Access Token Secret, which are never required. In summary:
Next, we'll walk through the mechanics of a sample signing.
Let's assume that the user wants to retrieve classification information for a URL "www.whatismyclassification.com" from BrightCloud Web Services. We'll also assume that the user has previously registered with BrightCloud Web Services and obtained the Access Key ID (aka Consumer Key) dpf43f3p2l4k3l03 and Secret Access Key ID (aka Consumer Secret Key) kd94hf93k423kf44. To sign the request, the user will use the HMAC-SHA1 signature method, and generate a Nonce string kllo9940pd9333jh and Timestamp 1191242096.
The OAuth information such as Consumer Key is included in the request using special OAuth Parameters starting with the 'oauth_' prefix, most of which are mandatory. OAuth does not allow any other parameter to use the 'oauth_' prefix. These OAuth parameters are not only sent as part of HTTP request, they are also used to calculate an OAuth signature for the request which, in turn is included with the other OAuth parameters in the request.
Calculating the OAuth Signature
Although OAuth specs allow for 3 different methods for calculating OAuth signature, BrightCloud Web Services currently supports only HMAC-SHA1 method. OAuth signature calculation for a request involves constructing a signature base string that serves as the HMAC-SHA1 text which will be "signed," constructing a key that serves as the HMAC-SHA1 algorithm key and then using the HMAC-SHA1 text and key in the HMAC-SHA1 algorithm to generate the signature.
There are three pieces to a signature base string:
These three strings are concatenated together, separated by an ampersand (&), to generate the signature base string.
URL Encoded HTTP method
For our sample request, the HTTP method to be used is GET. We start our signature base string with it.
URL Encoded Normalized URL
URLs are constructed from various elements, and take the general form of scheme://authority:port/path?query#fragment. The scheme and port are used to establish the desired connection, the authority is used in the Host header and to connect to the service provider, and the path and query parts are used in the request itself (the fragment is used locally by the browser and is not transmitted with the request).
The request URL is normalized as scheme://authority:port/path. Note that the query and the fragment are excluded from the normalized URL. The scheme and authority must be in lowercase, and the port must be present except for the default HTTP(S) ports which must be omitted (80 is omitted for http, and 443 is omitted when the scheme is https). The path must retain its case as some web servers are case-sensitive.
For our sample request, the REST resource URL that the user needs to access is:
After normalization, this becomes:
The URL encoding of the normalized URL yields the following string:
URL Encoded Normalized parameters
For our sample request, the following table contains our OAuth parameters.
These OAuth Parameters are collected together in their raw, pre-encoded form. All text parameters are UTF-8 encoded. After UTF-8 encoding, the parameters are URL-encoded. All the unreserved characters (letters, numbers, '-', '_', '.', '~') must not be encoded, while all other characters are encoded using the %XX format where XX is an uppercase representation of the character's hexadecimal value.
The parameters are then sorted first based on their encoded names, and if equal, based on their encoded values. Sort order is lexicographical byte value ordering, this is the default string sort method in most languages, and means comparing the byte value of each character and sorting in an ascending order resulting in a case sensitive sort.
The following table contains the OAuth parameters for our sample request after the required encoding and sorting.
For our sample request, the normalized parameter string looks like:
oauth_consumer_key=dpf43f3p2l4k3l03&oauth_nonce=kllo9940pd9333jh&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1191242096&oauth_token=&oauth_version=1.0 The URL
encoding of the normalized parameters yields the following string:
oauth_consumer_key%3Ddpf43f3p2l4k3l03%26oauth_nonce%3Dkllo9940pd9333jh%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1191242096%26oauth_token%3D%26oauth_version%3D1.0 To generate the signature base string, the three pieces generated above are concatenated together separated by an '&'.
For our sample request, the resulting signature base string is:
The HMAC-SHA1 signature method uses two secrets, the Consumer Secret and the Token Secret (which is blank in our case) as the HMAC-SHA1 algorithm key. To construct the algorithm key, each secret is UTF8-encoded, URL-encoded, and concatenated into a single string using an ampersand (&) character as a separator even if either secret is empty. Libraries should not assume the secrets are in plain ASCII text and ensure proper UTF-8-encoding and URL-encoding prior to concatenation.
For our sample request, the UTF8-encoded, URL-encoded Consumer Secret is:
The UTF8-encoded, URL-encoded Token Secret is:
The resulting HMAC-SHA1 algorithm key is:
Note the empty string after the ampersand. This is because our Token Secret is empty.
Using the signature base string generated in Step 1 as the HMAC-SHA1 text and the key generated in Step 2 as the HMAC-SHA1 algorithm key, the HMAC-SHA1 algorithm will generate an octet string which must be base64-encoded with equals ('=') padding.
Here's the signature we calculated for our sample request:
The calculated signature is added to the request using the oauth_signature parameter. When the signature is included in the HTTP request, it must be properly encoded as required by the method used to transmit the parameters.
BrightCloud Web Services requires OAuth parameters to be sent strictly as part of HTTP Authorization header as depicted in the sample above. Web Services doesn't yet support OAuth parameters delivered to it in the URL query element or in a single-part 'application/x-www-form-urlencoded' POST body.
Here's how our sample request would look after including the signature:
GET /rest/uris/www.whatismyclassification.com HTTP/1.1 Host: thor.brightcloud.com:80 Authorization: OAuth realm="http://thor.brightcloud.com/rest" oauth_consumer_key="dpf43f3p2l4k3l03" oauth_token="" oauth_nonce="kllo9940pd9333jh" oauth_timestamp="1191242096" oauth_signature_method="HMAC-SHA1" oauth_version="1.0" oauth_signature="NQAufeYLc5TDMjuHGupTB9L2zdw%3D"
OAuth is a great standards-based way of authenticating BrightCloud Web Services requests. As the community strengthens the libraries and implementations, every application developer benefits from those fixes and performance improvements. We strongly encourage you to download, use, and contribute where appropriate to the OAuth code base of your choosing.
% php cmdclient.php -k your_access_key_id_here -s your_secret_access_key_id_here -m GET -u http://thor.brightcloud.com/rest/uris/categories fHTTP/1.1 200 OK Etag: "9900072dbd616bdfe2867f2cfdeb33d1" Connection: close Content-Type: application/xml; charset=utf-8 Date: Mon, 20 Jul 2009 15:01:32 GMT Server: WEBrick/1.3.1 (Ruby/1.8.7/2008-08-11) X-Runtime: 183 Content-Length: 7973 Cache-Control: private, max-age=0, must-revalidate Set-Cookie: _bcws_session=BAh7BzoMdXNlcl9pZGkGOg9zZXNzaW9uX2lkIiUzYTEyZWMxMGQ4YTczZTRhYjRiNWUyZGViMWNhYjE3OQ%3D%3D--7b95c5741e696701acdbddf3919c703cac2d1254; path=/; HttpOnly Regular Response < ?BrightCloud version=bcap/1.1? > < bcap > < response > < status >200< /status > < statusmsg >OK< /statusmsg > < categories > < cat > < catid >68< /catid > < catname >Abortion< /catname > < catgroup >Legal Liability< /catgroup > < /cat > < cat > < catid >46< /catid > < catname >Abortion- ProChoice< /catname > < catgroup >Legal Liability< /catgroup > < /cat > < cat > < catid >48< /catid > < catname >Abortion- Pro Life< /catname > < catgroup >Legal Liability< /catgroup > < /cat > . . . < /categories > < /response > < /bcap >
This tutorial outlines the steps to create an account, install the Web Services PHP command line client, and walk through its use. This tool is helpful for making ad-hoc calls to the API as well as debugging the requests your programs make or even for low volume scripting.
Authenticating with BrightCloud® Web Services
BrightCloud Web Services uses the 2-legged OAuth standard to validate your identity whenever you issue a request. Authentication ensures that you don't get charged for operations you did not authorize.
Security always involves a secret. For BrightCloud Web Services, your secret is your Secret Access Key. Do not reveal it to anyone even if a request appears to come from a legitimate source, such as BrightCloud services. BrightCloud Web Services pairs your Secret Access Key with your Access Key ID. You will include your Access Key ID and a HMAC-SHA1 computed signature using your Secret Access Key in all BrightCloud requests. Web Services verifies that the sender of the request knows both the Access Key ID and the corresponding Secret Access Key and is not violating any other security measures set forth in the OAuth standard. Web Services does not process requests where the Access Key ID and Secret Key do not match or any other violations occur (e.g. reusing nonce values or timestamps which are out of line).
In the following samples, you simply add your Access Key ID and Secret Key to the sample files. For more information about authentication, please visit the OAuth.net website and our OAuth Integration for BrightCloud Web Services.
Setup the PHP command line client
To continue, you must complete the following steps:
To install the PHP client, simply decompress it into a subdirectory of your choosing and change into that directory. Be sure to check out our other sample programs and code in the developer section of our website as well.
Get the list of BrightCloud® categories
The first thing to do is to retrieve a list of BrightCloud categories and their associated Category IDs. These Category IDs are returned instead of the Category Name in order to conserve space. Therefore, the first thing your application should do is download the latest copy of our Category IDs and build a map of Category IDs to Category Name values.
To see what the REST call looks like to make this request you can use the "-d" flag of the command line client to "dump" the GET request instead of making the call to the BrightCloud Web Services server. You are going to make a GET request to the URL /rest/uris/categories.
% php cmdclient.php -k your_access_key_id_here -s your_secret_access_key_id_here -m GET -u http://thor.brightcloud.com/rest/uris/categories -d GET /rest/uris/categories HTTP/1.0 HOST: thor.brightcloud.com Connection: close Authorization: OAuth realm="",oauth_version="1.0",oauth_nonce="b1c80a7890ee3df421b3ac730d44fb12",oauth_timestamp="1248102047",oauth_consumer_key="gIQBlKQx5cxCeSGoLtbHsQ",oauth_token="",oauth_signature_method="HMAC-SHA1",oauth_signature="YbHGSU20CPVEWXcNt9lyCUmgTnk%3D"
The first thing to notice is that the Access Key ID (following the "-k" parameter) and Secret Access Key ID (following the "-s" parameter) were both required to make this call. They were used to generate the oauth_consumer_key (which is your Access Key ID) and the oauth_signature value (which is computed with your Secret Access Key ID) you see at the end of the OAuth Authorization header.
To see what the REST response looks like, simply remove the "-d" flag at the end of the command line and issue the response to the BrightCloud server.
You can see all of the various Category Name, Category ID, and even Category Group values we supply for your convenience. These Category Groups are only suggestions; you can organize and use the Categories however is most useful to you.
Retrieve all categories for a URL
To see what the REST call looks like for requesting the BrightCloud categories for a particular URL, you can again use the "-d" flag of the command line client to "dump" the GET request instead of making the call to the BrightCloud Web Services server. This time the GET request is made to /rest/uris/[url], e.g. /rest/uris/www.google.com. You can supply the protocol, i.e. http or https, or not. If you choose not to specify, we will assume you're asking about http.
% php cmdclient.php -k your_access_key_id_here -s your_secret_access_key_id_here -m GET -u http://thor.brightcloud.com/rest/uris/www.google.com -d GET /rest/uris/www.google.com HTTP/1.0 HOST: thor.brightcloud.com Connection: close Authorization: OAuth realm="",oauth_version="1.0",oauth_nonce="b1c80a7890ee3df421b3ac730d44fb12",oauth_timestamp="1248102047",oauth_consumer_key="gIQBlKQx5cxCeSGoLtbHsQ",oauth_token="",oauth_signature_method="HMAC-SHA1",oauth_signature="YbHGSU20CPVEWXcNt9lyCUmgTnk%3D"
To see what the REST response looks like, remove the "-d" flag at the end of the command line and issue the response to the BrightCloud Web Services server.
% php cmdclient.php -k your_access_key_id_here -s your_secret_access_key_id_here -m GET -u http://thor.brightcloud.com/rest/uris/www.google.com HTTP/1.1 200 OK Etag: "9900072dbd616bdfe2867f2cfdeb33d1" Connection: close Content-Type: application/xml; charset=utf-8 Date: Mon, 20 Jul 2009 15:01:32 GMT X-Runtime: 183 Content-Length: 7973 Cache-Control: private, max-age=0, must-revalidate Set-Cookie: _bcws_session=BAh7BzoMdXNlcl9pZGkGOg9zZXNzaW9uX2lkIiUzYTEyZWMxMGQ4YTczZTRhYjRiNWUyZGViMWNhYjE3OQ%3D%3D--7b95c5741e696701acdbddf3919c703cac2d1254; path=/; HttpOnly Regular Response < bcap > < seqnum >1< /seqnum > < response > < status >200< /status > < statusmsg >OK< /statusmsg > < uri >www.google.com< /uri > < categories > < cat > < catid >50< /catid > < conf >85< /conf > < /cat > < /categories > < bcri >88< /bcri > < a1cat >1< /a1cat > < /response > < /bcap >
You can see all of the various Category IDs and Confidence Scores (numbers between the conf tags) we supply for your convenience. The higher the confidence score, the more certain we are that the URL belongs to that particular category. Keep in mind that Confidence Scores will not add up to 100. For example, while a URL could belong to the Sports category, it could also independently belong in the Gambling category. Therefore, a sports betting site would have high Confidence Scores in both categories.
The PHP command line application is even more capable than this. You can make any of the calls documented in our BrightCloud Web Services REST API. Use the application's help command to learn about all of its degrees of flexibility.
To make a different BrightCloud Web Services REST call, specify the necessary request method ("-m") such as POST or PUT and the corresponding resource URL ("-u") to make the call. The PHP command line application will then convert these actions and your API credentials into a properly formed Web Services call and return the response. This output can then be filtered or written to a file so that other scripts can process it further.
BrightCloud Web Services is a RESTful API service for developers to access Webroot BrightCloud URL classification and reputation data. The two main pieces of information provided are:
Webroot embedded security partners use the Webroot BrightCloud SDK to gain authenticated access to BrightCloud Web Services. The SDK includes sample code for API access to Webroot's information service, and compiles on Microsoft® Windows® operating systems and many Linux distributions. Additionally, it includes production quality source code for local cache implementation, real-time updates, and subscription management. For more information, visit the BrightCloud® Threat Intelligence Services for Embedded Security Partners page.