Bright Data is well known for proxies, web unlocking and search APIs. However, their offerings span much further than this. In this article, we’ll explore Bright Data’s suite of products and how to use them.
When you’re finished reading, you’ll be able to answer the following questions.
- What are Bright Data’s flagship products?
- What is Bright Data’s Scraper Studio?
- What is Bright Data MCP?
- What is the Bright Data CLI?
- What is Bright Data’s new Discover API?
- How can your team use these products?
Overview of offerings from Bright Data
Bright Data’s offerings include all sorts of web data infrastructure. They offer web unlocking and SERP tools. Bright Data also offers cloud browsers and datasets. They’ve also launched a variety of new products centered around AI-native tooling and easier data access.
They also just launched a Free Tier plan supporting tools like the Unlocker and SERP APIs. All users get 5,000 requests per month for free.
Flagship offerings
- Unlocker API: Bright Data’s Unlocker API helps people and AI agents access the web reliably with CAPTCHA solving, automated proxy rotation and geotargeting.
- SERP API: Bright Data’s longstanding Search Engine Results Page (SERP) API. Get results from a variety of search engines in structured JSON format.
- Browser API: Spin up remote cloud browsers with automated proxy integration, geotargeting support and CAPTCHA solving.
- Scraper APIs: Run custom and prebuilt scrapers on demand. Bright Data supports a variety of scrapers for eCommerce, social media, news and other sites. A full list can be found here.
- Datasets: Access historical datasets with custom formatting and delivery integrations. Their Dataset Marketplace includes over 250 domains and over 350 historical datasets.
New and updated products
- Scraper Studio: Instantly generate Scraper APIs using natural language prompts. Broken scrapers are handled with a single self healing click. Users can easily deploy and schedule their scrapers.
- Bright Data MCP: Plug AI agents directly into Bright Data’s tools and services. Completely automate your data pipeline using AI agents.
- Bright Data CLI: Control scrapers, data APIs. Add AI agent skills and manage your Bright Data account directly from your terminal.
- Discover API: An AI-focused search engine with up to 1,000 results per request. Searches can use intents to improve search context.
A closer look at Bright Data’s flagship products
Unlocker API
The Unlocker API (also known as Web Unlocker) has been Bright Data’s flagship product for years. It offers automated proxy management, geotargeting and CAPTCHA solving. People and AI agents can choose from output formats such as HTML, Markdown and screenshots.
To use the Unlocker API in Python, you can send a simple POST request with params for your target URL and desired response format.
import requests
headers = {
"Authorization": "Bearer <your-bright-data-api-key>",
"Content-Type": "application/json"
}
data = {
"zone": "<your-zone>",
"url": "https://geo.brdtest.com/welcome.txt?product=unlocker&method=api",
"format": "raw"
}
response = requests.post(
"https://api.brightdata.com/request",
json=data,
headers=headers
)
print(response.text)We wanted a raw, plaintext response. As you can see below, that’s exactly what we received.
SERP API
SERP API is used for instant search results from a variety of engines. Responses are configurable between HTML, JSON, Markdown and screenshots. Google, Bing, Yandex and DuckDuckGo are supported. Simply run a search and receive structured search results.
The code below runs a search for pizza. Results should be received as JSON and written to a file.
import requests
import json
headers = {
"Authorization": "Bearer <your-bright-data-api-key>",
"Content-Type": "application/json"
}
data = {
"zone": "<your-zone>",
"url": "https://www.google.com/search?q=pizza",
"format": "json"
}
response = requests.post(
"https://api.brightdata.com/request",
json=data,
headers=headers
)
with open("serp-response.json", "w") as file:
json.dump(response.json(), file, indent=4)This next screenshot is an image of the JSON file containing our search results.
Browser API
From the Browser API dashboard, you can immediately configure a proxy powered headless browser with CAPTCHA solving. Select the “Playground” tab to view a prebuilt Puppeteer script for your custom browser.
Here is the code from the playground. It opens a browser using Puppeteer and performs a search using Booking.com.
import puppeteer from "puppeteer-core";
const URL = "https://www.booking.com/";
const BROWSER_WS = "wss://brd-customer-<your-username>-zone-<your-zone-name>:<your-password>@brd.superproxy.io:9222";
function addDays(date, days) {
var result = new Date(date);
result.setDate(result.getDate() + days);
return result;
}
function toBookingTimestamp(date) {
return date.toISOString().split('T')[0];
}
const search_text = "New York";
const now = new Date();
const check_in = toBookingTimestamp(addDays(now, 1));
const check_out = toBookingTimestamp(addDays(now, 2));
// This sample code searches Booking for acommodation in selected location
// and dates, then returns names, prices and rating for available options.
run(URL);
async function run(url) {
console.log("Connecting to browser...");
const browser = await puppeteer.connect({
browserWSEndpoint: BROWSER_WS,
});
console.log("Connected! Navigate to site...");
const page = await browser.newPage();
await page.goto(url, { waitUntil: "domcontentloaded", timeout: 60000 });
console.log("Navigated! Waiting for popup...");
await close_popup(page);
await interact(page);
console.log("Parsing data...");
const data = await parse(page);
console.log(`Data parsed: ${JSON.stringify(data, null, 2)}`);
await browser.close();
}
async function close_popup(page) {
try {
const close_btn = await page.waitForSelector('[aria-label="Dismiss sign-in info."]', { timeout: 25000, visible: true });
console.log("Popup appeared! Closing...");
await close_btn.click();
console.log("Popup closed!");
} catch (e) {
console.log("Popup didn't appear.");
}
}
async function interact(page) {
console.log("Waiting for search form...");
const search_input = await page.waitForSelector('[data-testid="destination-container"] input', { timeout: 60000 });
console.log("Search form appeared! Filling it...");
await search_input.type(search_text);
await page.click('[data-testid="searchbox-dates-container"]');
await page.waitForSelector('[data-testid="searchbox-datepicker-calendar"]');
await page.click(`[data-date="${check_in}"]`);
await page.click(`[data-date="${check_out}"]`);
console.log("Form filled! Submitting and waiting for result...");
await Promise.all([
page.click('button[type="submit"]'),
page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
]);
};
async function parse(page) {
return await page.$$eval('[data-testid="property-card"]', els => els.map(el => {
const name = el.querySelector('[data-testid="title"]')?.innerText;
const price = el.querySelector('[data-testid="price-and-discounted-price"]')?.innerText;
const review_score = el.querySelector('[data-testid="review-score"]')?.innerText ?? '';
const [score_str, , , reviews_str = ''] = review_score.split('n');
const score = parseFloat(score_str) || score_str;
const reviews = parseInt(reviews_str.replace(/D/g, '')) || reviews_str;
return { name, price, score, reviews };
}));
}When you’re ready to test the browser, click “Run request.”
Bright Data then launches a browser using their cloud infrastructure. As you can see in the response, the browser successfully navigated to the site.
Scraper API
Bright Data’s Scraper API lets teams run prebuilt scrapers on demand for fresh data whenever it’s needed. Users can also deploy custom scrapers. Once you’ve selected a scraper to use, you can configure and run it from the dashboard.
Here, we enter a URL into the LinkedIn profile scraper and run the scraper by clicking the “Run manually” button on the dashboard.
From the snapshots page, you can view the status of your scraper. As you can see in the image below, our scraper is running.
Once it’s finished, you can view the data you’ve extracted. In the image below, we’ve got our LinkedIn profile data laid out in a JSON file.
Datasets
Teams using the Dataset Marketplace can choose from hundreds of massive ready-made and curated datasets. Popular datasets include LinkedIn profiles, Amazon products and LinkedIn job data.
Once you’ve selected a dataset, you can view and download sample data from the set. You can speak to a data expert if you’re on the fence or you can purchase a dataset instantly by clicking a button.
Bright Data’s new and updated offerings in detail
Now, let’s take a look at Bright Data’s new and updated products. Scraper Studio has been reoriented into primarily a no-code platform to create and deploy scrapers. Bright Data’s MCP server gives AI agents access to commonly used tools. Bright Data CLI gives both humans and AI agents comprehensive access to Bright Data’s suite of tools.
Scraper Studio
To get started with Scraper Studio, select “Web Scrapers” on your sidebar. Then, click the “New” button. In the dropdown, choose “Develop a web scraper.”
Next, you’ll be prompted to input a URL and optionally add instructions. When you’re ready, click the “Start scraping with AI” button.
Next, Bright Data’s AI will fetch and analyze the URL. The AI model might ask additional questions. For instance, in the next image, the AI asked me if I wanted homepage data, articles or something else. I prompted it to scrape individual articles.
Scraper Studio then lets you review the schema generated for the platform. When the schema looks good, click “Approve.”
Scraper Studio then begins building and deploying your custom scraper. When it’s ready, you’ll receive an email alert.
You’ll now have a new dashboard you can use for your custom scraper. From here, you can initiate the scraper manually, schedule runs and setup automated API calls. You can also tweak the code.
MCP Server
Now, we’ll take a look at Bright Data’s MCP server with Claude Code. Open up your Claude config file and paste the following snippet to run Bright Data’s MCP server locally. Make sure to replace the API key with your own. Make sure you have NodeJS installed so npx can execute.
{
"mcpServers": {
"Bright Data": {
"command": "npx",
"args": [
"@brightdata/mcp"
],
"env": {
"API_TOKEN": "<your-bright-data-api-key>",
"GROUPS": "social,advanced_scraping"
}
}
}Next, we prompted Claude to find some basic YouTube data for random videos. According to Claude, the metadata scraper failed to return initially.
After another prompt, Claude fetched the full metadata. The next three images are all the metadata for a single video. There are approximately 40 fields. This is ideal for model training.
You can get even more video data for AI training using Bright Data’s video and media data APIs, which allow you to download full video, audio and caption content that you can use together with metadata like the data above.
Bright Data CLI
The Bright Data CLI gives people and AI agents access to nearly the entire Bright Data suite — right from the command line. Users can choose run the CLI interactively or in direct mode as well to pipe inputs and outputs through the shell.
To install the CLI, simply run the following npm command.
npm install -g @brightdata/cliYou can make CLI commands using the structure brightdata <command> -- <arg>. As you can see in the image below, we run a command to search for AI news and using the -o flag, we save the output to a Markdown file.
You can see the saved output from the file below.
The Bright Data CLI integrates seamlessly with AI agents. We use the skill command to handle AI agent skills. skill list shows the skills that are available for integration.
brightdata skill add allows you to add skills interactively. We chose to add the brightdata-cli skill to give our AI agents full access to the CLI tooling.
After choosing your skills, you can select additional agents to receive the skills.
The CLI quickly adds the skills to all of your selected AI agents.
Discover API
Bright Data’s new Discover API lets people and AI agents perform search queries with intent matching. This allows for better search context. Each result comes with a confidence score.
In the image below, we perorm a search for ai news. We get a variety of results such as science, news, insights and educational links. Some are news sites. Some are not.
In this next image, we add an intent to our search: find the latest ai news relevant to investment markets. Now, rather than a boatload of sites, all 20 of our results are relevant to AI and investment.
Conclusion
Bright Data offers one of the most comprehensive web data suites available. They support the old school staples like proxies and unblocking. Search capabilities can be integrated using their traditional SERP API as well as through their Discover API. Cloud browsers handle the need for live web interactions. Teams can also now completely skip coding entirely using Scraper Studio.
With their new Free Tier, the barrier to projects and enterprise grade web data has never been lower.
