Your AI agent works perfectly on localhost. It answers questions, follows multi-step reasoning, and pulls in live information from the web. Then you deploy it to production, and everything falls apart. Requests start returning 403s, pages time out mid-flow, and answers come back incomplete because a failed fetch broke the entire reasoning chain. Your function logs on Vercel fill up with retries and aborted runs, even though the code hasn’t changed.
This isn’t a bug, it’s just how the web works. Modern sites aggressively block automation using fingerprinting, behavioral analysis, rate limits, and CAPTCHA. The scraping tools you’ve been using weren’t built for AI agents that need to browse the web step-by-step, maintaining state as they go. Old libraries assume each request stands alone. But when an AI agent gets blocked, it’s not just losing one piece of data; the whole reasoning process falls apart.
This guide teaches you how to adapt agents for the issues mentioned above. You’ll learn how to deploy production-ready AI agents on Vercel using the Vercel AI SDK for orchestration and Bright Data for reliable web access via the Model Context Protocol (MCP).
Why production AI agents need reliable web access
Agentic systems turn web access into part of the control flow. A single task may involve a search, three follow-up page loads, a scrape, and a verification step before the model can answer. Each request depends on the previous one having succeeded, and failures compound. Setups built for batch scraping collapse when used for interactive agents. When a fetch is blocked or times out, the model doesn’t just lose a page; it loses the state that tells it what to do next.
There’s also a second-order effect that shows up in production. Tool calls from LLMs are highly regular. The same query patterns, the same headers, the same access paths. Anti-bot systems are tuned to detect exactly that kind of behavior, which means agent traffic gets flagged faster than a one-off script and web access becomes infrastructure for agents. Once requests are predictable and unblockable, reasoning, retries, observability, and cost control all become tractable problems instead of guesswork.
Understanding the Vercel and Bright Data stack
Using Vercel and Bright Data for building AI agents works because it cleanly separates agent reasoning from web execution. The agent decides what information it needs, and a dedicated web access layer figures out how to retrieve it without getting blocked. The two connect through a strict protocol boundary, so failures stay contained instead of leaking into prompts or destabilizing the agent’s reasoning.
On the agent side, the Vercel AI SDK is responsible for orchestration. It manages prompts, tool calling, streaming responses, and model selection, while remaining model-agnostic. From the agent’s perspective, tools are simply typed functions that return structured data. This keeps reasoning deterministic and testable, even as agents grow more complex.
Tool execution is handled through the MCP. MCP defines a JSON-RPC interface between the LLM runtime and external services, with server-side execution by default. This keeps credentials secure, avoids client-side fingerprinting, and enforces explicit schemas for inputs and outputs via Zod. Agents never guess what a tool might return. They either receive valid, structured data or a clear failure.
Bright Data supplies the tools and handles the complexity of modern anti-bot systems. Instead of issuing raw HTTP requests, the agent sends high-level MCP tool calls, like search, scrape, and extract, to Bright Data. Bright Data handles things like CAPTCHA handling, IP rotation, session continuity, and geographic targeting, then returns clean results the agent can reason over.
Setting up your development environment
Before you can try this setup yourself, you’ll need the following prerequisites:
- A recent Node.js runtime on your local machine
- A Vercel account on the free tier
- A Bright Data account
- An API key for the language model provider you plan to use (eg OpenAI)
In this tutorial, you’ll build a deep research AI agent using Vercel AI SDK and Bright Data APIs, and deploy it on Vercel’s Fluid Compute platform.
To start, create a new folder named vercel-bd-ai or something similar, and run npm init -y inside it.
Then, run the following command to install the required dependencies:
npm i @ai-sdk/openai @brightdata/sdk ai dotenv zodThe Vercel AI SDK (ai and @ai-sdk) provides the agent runtime and tool-calling abstractions. Alongside those, Zod will help with schema validation, and dotenv helps with the local environment configuration. Keeping schemas explicit from day one makes debugging far easier once tool calls enter the picture.
Configure environment variables early and treat them as production-critical. Create a .env file for local development and include your model provider key (OPENAI_API_KEY) and your Bright Data API key (BRIGHTDATA_API_KEY). Make sure this file is excluded from version control (by adding it to your .gitignore file) as well as the Vercel CLI (by adding it to your .vercelignore file).
When you deploy to Vercel, you will need to add these variables through the dashboard, so your local and production environments behave consistently.
It’s time to start building the Vercel function with your agent.
Building your first web-enabled Agent
Begin by creating a file named api/agent.js in your project root directory. This file will house your agent function. In this file, save the following code:
import 'dotenv/config'
import { generateText, stepCountIs } from 'ai'
import { openai } from '@ai-sdk/openai'
import { brightDataTools } from '../brightdata-tools.js'
let _tools
const getTools = () => {
if (_tools) return _tools
if (!process.env.BRIGHTDATA_API_KEY) throw new Error('Missing BRIGHTDATA_API_KEY')
_tools = brightDataTools({ apiKey: process.env.BRIGHTDATA_API_KEY })
return _tools
}
/**
* Vercel Serverless Function wrapper for the existing agent.
*
* - Accepts GET ?q=your+query or POST { "query": "..." }
* - Uses BRIGHTDATA_API_KEY from env for Bright Data tools
*/
export default async function handler(req, res) {
try {
const method = (req.method || 'GET').toUpperCase()
// POST: expect JSON body with { query }
if (method === 'POST') {
const body = req.body || {}
const query = String(body.query ?? '').trim()
if (!query) {
return res.status(400).json({ error: 'Missing query in request body' })
}
// Get or create tools (may throw if env missing)
const tools = getTools()
const model = openai('gpt-4.1-mini')
const result = await generateText({
model,
prompt: query,
tools,
stopWhen: stepCountIs(5),
})
return res.status(200).json({ text: result.text?.trim() ?? '', details: result })
}
return res.status(405).json({ error: `Method ${method} not allowed` })
} catch (err) {
console.error('agent function error:', err)
return res.status(500).json({ error: String(err) })
}
}This file exposes your agent as a Vercel Serverless Function. It accepts a natural-language query over HTTP, initializes an OpenAI model, and runs it through the AI SDK with Bright Data tools enabled so the agent can fetch live web content when needed. The Bright Data client is created lazily and reused, ensuring the API key is present and avoiding unnecessary setup on every request.
Each request is bounded and production-safe. The agent is limited to a small number of reasoning steps, errors are handled explicitly, and the final response is returned as JSON.
The next step is to define the Bright Data tools.
Bright Data offers a robust MCP with a host of pre-built tools that you can directly drop into your Vercel AI client using an MCP client instance. However, in this tutorial, you’ll learn how to create the necessary tools using Zod, Bright Data Search and Scrape APIs, Vercel AI SDK, and some helper functions.
Create a file named brightdata-tools.js in your project root directory. In this file, add the following import statements:
import { tool } from 'ai'
import { z } from 'zod'
import { bdclient } from '@brightdata/sdk'Then, define the brightDataTools object which will contain all the tools:
export const brightDataTools = (config) => {
if (!config?.apiKey) throw new Error('brightDataTools requires config.apiKey')
const client = new bdclient({
apiKey: config.apiKey,
autoCreateZones: true,
})
// You'll add the tools and helpers here
}This checks for the Bright Data API key and creates a Bright Data client using an available key.
Next, add the following helper function to brightDataTools:
const normalizeSearch = (raw, maxResults) => {
const text = typeof raw === 'string' ? raw : JSON.stringify(raw)
// If Bright Data returned JSON-ish data, try to parse it.
try {
const parsed = typeof raw === 'string' ? JSON.parse(raw) : raw
const items =
Array.isArray(parsed) ? parsed :
Array.isArray(parsed?.results) ? parsed.results :
Array.isArray(parsed?.items) ? parsed.items :
Array.isArray(parsed?.organic) ? parsed.organic :
null
if (items) {
const mapped = items
.map((it) => ({
title: it.title ?? it.name ?? '',
url: it.url ?? it.link ?? '',
snippet: it.snippet ?? it.description ?? it.summary ?? '',
source: it.source ?? it.domain ?? '',
}))
.filter((r) => typeof r.url === 'string' && r.url.startsWith('http'))
.slice(0, maxResults)
return { type: 'json', results: mapped }
}
} catch {
// fall through to markdown/html parsing
}
// Parse markdown links: [title](url)
const mdLinks = []
const mdRe = /\[([^\]]+)\]\((https?:\/\/[^)]+)\)/g
for (const m of text.matchAll(mdRe)) {
mdLinks.push({ title: m[1], url: m[2] })
}
// Parse html links: <a href="...">title</a>
const htmlLinks = []
const aRe = /<a\s+[^>]*href=["'](https?:\/\/[^"']+)["'][^>]*>(.*?)<\/a>/gi
for (const m of text.matchAll(aRe)) {
const title = String(m[2]).replace(/<[^>]+>/g, '').trim()
htmlLinks.push({ title, url: m[1] })
}
const merged = [...mdLinks, ...htmlLinks]
.filter((r) => r.url.startsWith('http'))
.slice(0, maxResults)
return {
type: 'text',
results: merged,
raw_preview: text.slice(0, 4000),
}
}This function normalizes search output into a small list of results with URLs to crawl. Bright Data may return markdown or HTML in its responses, but this helper will ensure that the agent will be able to handle both with light parsing.
Next, add the following helper function to brightDataTools:
const extractLinks = (content, baseUrl) => {
const text = typeof content === 'string' ? content : JSON.stringify(content)
const links = new Set()
// [text](https://...)
const mdRe = /\[[^\]]*]\((https?:\/\/[^)]+)\)/g
for (const m of text.matchAll(mdRe)) links.add(m[1])
// href="https://..."
const hrefRe = /href=["'](https?:\/\/[^"']+)["']/g
for (const m of text.matchAll(hrefRe)) links.add(m[1])
// relative: [text](/path) or href="/path"
const relMdRe = /\[[^\]]*]\((\/[^)]+)\)/g
for (const m of text.matchAll(relMdRe)) {
try {
links.add(new URL(m[1], baseUrl).toString())
} catch {}
}
const relHrefRe = /href=["'](\/[^"']+)["']/g
for (const m of text.matchAll(relHrefRe)) {
try {
links.add(new URL(m[1], baseUrl).toString())
} catch {}
}
return Array.from(links)
}This helps extract absolute URLs from scraped Markdown/HTML to support crawling.
Next, add the following schema-related helpers to brightDataTools:
const extractSchemaSpec = z.object({
fields: z.array(
z.object({
name: z.string().min(1),
type: z.enum(['string', 'number', 'boolean', 'array', 'object']),
description: z.string().optional(),
required: z.boolean().default(true),
}),
).min(1),
})
const buildZodFromSpec = (spec) => {
const shape = {}
for (const f of spec.fields) {
let base
switch (f.type) {
case 'string': base = z.string(); break
case 'number': base = z.number(); break
case 'boolean': base = z.boolean(); break
case 'array': base = z.array(z.any()); break
case 'object': base = z.record(z.any()); break
default: base = z.any()
}
shape[f.name] = f.required ? base : base.optional()
}
return z.object(shape)
}This helps create Zod schemas to validate extracted output from scraping calls for retrieving organized data.
Next, create the tools by adding this code to brightDataTools:
const tools = {
webSearch: tool({
description:
'Search the web for relevant articles and pages. Use this to discover sources before crawling or extracting.',
inputSchema: z.object({
query: z.string().min(1).describe('Search query'),
max_results: z.number().int().min(1).max(10).default(5).describe('Max number of results to return'),
}),
execute: async ({ query, max_results }) => {
try {
const raw = await client.search(query, {
searchEngine: 'google',
dataFormat: 'markdown',
format: 'raw',
})
return normalizeSearch(raw, max_results)
} catch (error) {
return { error: `Error searching for "${query}": ${String(error)}` }
}
},
}),
webCrawl: tool({
description:
'Crawl a URL up to a small depth by scraping pages and following discovered links. Returns markdown snapshots for visited pages.',
inputSchema: z.object({
url: z.string().url().describe('Start URL'),
depth: z.number().int().min(0).max(2).default(1).describe('Crawl depth (0 = just this page)'),
}),
execute: async ({ url, depth }) => {
const maxPages = 8
const perPageLinks = 6
const visited = new Set()
const queue = [{ url, d: 0 }]
const pages = []
try {
while (queue.length && pages.length < maxPages) {
const next = queue.shift()
if (!next) break
const { url: current, d } = next
if (visited.has(current)) continue
visited.add(current)
const scraped = await client.scrape(current, {
dataFormat: 'markdown',
format: 'raw',
})
const markdown = typeof scraped === 'string' ? scraped : JSON.stringify(scraped)
pages.push({ url: current, depth: d, markdown })
if (d >= depth) continue
const links = extractLinks(markdown, current)
.filter((u) => u.startsWith('http'))
.slice(0, perPageLinks)
for (const link of links) {
if (!visited.has(link)) queue.push({ url: link, d: d + 1 })
}
}
return { start_url: url, max_depth: depth, pages }
} catch (error) {
return {
start_url: url,
max_depth: depth,
pages,
error: `Error crawling ${url}: ${String(error)}`,
}
}
},
}),
webExtract: tool({
description:
'Extract structured fields from a single web page. Scrapes the page and returns data matching the provided schema.',
inputSchema: z.object({
url: z.string().url().describe('Page URL to extract from'),
schema: extractSchemaSpec.describe('Extraction schema spec'),
}),
execute: async ({ url, schema }) => {
try {
const scraped = await client.scrape(url, {
dataFormat: 'markdown',
format: 'raw',
})
const markdown = typeof scraped === 'string' ? scraped : JSON.stringify(scraped)
const outputSchema = buildZodFromSpec(schema)
const data = {}
for (const f of schema.fields) {
if (f.type === 'string') data[f.name] = ''
else if (f.type === 'number') data[f.name] = 0
else if (f.type === 'boolean') data[f.name] = false
else if (f.type === 'array') data[f.name] = []
else if (f.type === 'object') data[f.name] = {}
}
const validated = outputSchema.parse(data)
return {
url,
markdown, // evidence for the agent to fill fields accurately
data: validated,
note:
'Populate/overwrite the returned `data` fields using the `markdown` evidence in your agent reasoning.',
}
} catch (error) {
return { error: `Error extracting from ${url}: ${String(error)}` }
}
},
}),
}
return toolsTo make the agent accessible, add the following if block in the handler function in api/agent.js right below the const method = (req.method || 'GET').toUpperCase() line:
// Serve a simple form UI on GET
if (method === 'GET') {
const q = String(req.query.q ?? '').replace(/</g, '<')
return res
.status(200)
.setHeader('Content-Type', 'text/html; charset=utf-8')
.send(`<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<title>Agent</title>
<style>body{font-family:system-ui,Segoe UI,Roboto,Helvetica,Arial,sans-serif;margin:24px}textarea{width:100%;height:160px;margin-bottom:8px}button{padding:8px 12px}#result{white-space:pre-wrap;background:#f7f7f8;padding:12px;border-radius:6px;margin-top:12px}</style>
</head>
<body>
<h1>Agent</h1>
<form id="form">
<label for="q">Query</label>
<textarea id="q" name="q" placeholder="Enter query">${q}</textarea>
<br />
<button id="submit" type="submit">Run</button>
</form>
<div id="status" aria-live="polite"></div>
<div id="result"></div>
<script>
const form = document.getElementById('form')
const qEl = document.getElementById('q')
const submit = document.getElementById('submit')
const status = document.getElementById('status')
const result = document.getElementById('result')
form.addEventListener('submit', async (ev) => {
ev.preventDefault()
const query = qEl.value.trim()
if (!query) return
submit.disabled = true
status.textContent = 'Running…'
result.textContent = ''
try {
const resp = await fetch('/api/agent', {
method: 'POST',
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
body: JSON.stringify({ query })
})
if (!resp.ok) {
const err = await resp.json().catch(() => ({ error: resp.statusText }))
status.textContent = 'Error'
result.textContent = err.error || JSON.stringify(err)
return
}
const data = await resp.json()
status.textContent = 'Done'
result.textContent = data.text || JSON.stringify(data)
} catch (err) {
status.textContent = 'Request failed'
result.textContent = String(err)
} finally {
submit.disabled = false
}
})
</script>
</body>
</html>`)
}This renders a simple form when you go to /api/agent and allows you to make requests to the agent easily.
At this point, you’re ready to test the agent locally. Run npx vercel dev to build and run the function locally. Once the project is built, go to http://localhost:3000/api/agent and you’ll see a text box with a “run” button to make requests to your agent:
Try pasting a fairly detailed request like the following and click “Run”:
Find the official documentation for Model Context Protocol (MCP). Crawl the documentation up to depth 2.
Extract:
- core concepts
- architecture overview
- examples of real-world usage
Summarize findings with citations. Also, give me a list of tools you called to fulfill my request.And then you’ll see the agent in action:
The agent is now built and ready for deployment.
Deployment with Vercel fluid compute
Once the agent works locally, deployment comes down to choosing how much time and compute each request needs.
A common choice for running serverless functions is in Vercel Functions. This works well for most API-style interactions and short agent runs, including simple tool usage and single-step responses. However, as agents start to perform longer reasoning chains or make multiple sequential tool calls, they can run into execution time or resource limits depending on the plan and configuration.
Fluid Compute is designed for those heavier cases. It’s a serverless execution mode that allows functions to run longer and use compute more efficiently, making it better suited for agents that need to search, crawl, extract, and synthesize information in a single request.
Fluid Compute is enabled by default for new projects after April 23, 2025. For older projects, you need to go to your Function Settings and toggle the switch to enable Fluid Compute in your project:
Once enabled, run npx vercel deploy in your project and Vercel will build and deploy your functions to Fluid Compute. Also, add the two environment variables in Project Settings → Environment Variables:
And that’s it. You can access your deployed agent via the Vercel URL.
Understand Scaling and Operational Limits
Even if your agent code is perfect, production behavior will still be shaped by platform limits.
- Concurrency affects how many agent runs can happen at once.
- Cold starts add a few seconds occasionally, which can matter if your agent makes many sequential web calls.
- Cost is driven by execution time and request volume. Agents that browse and extract aggressively will cost more than agents that mostly rely on model reasoning.
Make sure you design your agent for predictable work. Prefer fewer, higher-quality tool calls. Validate outputs early. Cache repeated lookups when you can. Those choices matter more in production than almost any prompt tweak.
Monitoring and debugging agent performance
Once your agent is deployed, the question shifts from “does it run” to “does it run reliably.” Web-enabled agents are prone to failing quietly. Maybe a tool call returns malformed data, or a crawl step stalls, or a downstream synthesis step degrades because the retrieved content was thin. Without instrumentation, you end up debugging final answers instead of the system that produced them.
Start with logs and observability on Vercel. For every invocation, you should be able to answer three things quickly:
- Did the function time out or crash?
- Which step did the agent reach before failing?
- Which external call caused the slowdown or block?
Treat MCP tool calls like application metrics. Log each call with lightweight, structured metadata rather than raw content. At minimum, capture invocation ID, tool name, target (truncated), duration, success or failure, and error category. Over time, this gives you a clear success-rate picture and makes it obvious whether you’re dealing with model behavior or web access issues.
Advanced patterns and use cases
Once an agent is stable in production, the biggest gains come from shaping how it uses tools.
Browse with intent
Browsing with intent means every web request has a purpose tied to the agent’s current goal. The agent starts with a clear question, makes a targeted search, selects a likely source, and extracts only the information it needs instead of drifting through pages.
That focus reduces wasted calls and makes failures easier to recover from. Fewer, more deliberate requests lower cost, avoid bot detection, and keep the agent’s reasoning on track as workloads grow.
Follow explicit multi-step workflows
Most effective agents follow explicit multi-step workflows. A typical research flow searches first to identify high-signal sources, crawls only the top results, extracts specific fields instead of full pages, and synthesizes once enough evidence is gathered. Making this sequence explicit and enforcing a tight budget on tool calls prevents runaway browsing and keeps failures localized.
Pay attention to session management
Session management becomes important as soon as an agent touches the same domain more than once. Many sites behave unpredictably with stateless requests. Reusing a session across related tool calls preserves cookies, stabilizes identity, and reduces aggressive rate limiting. A simple approach is to generate a session ID per user request or conversation and pass it consistently through related web operations.
Control costs
Cost control follows naturally from these patterns. Cache repeated results, prefer targeted extraction over full crawling, batch decisions before browsing, and always use search to narrow the problem space.
Agents that follow these rules feel smarter not because they do more work, but because they do less unnecessary work.
Troubleshooting common integration issues
Even with a solid architecture, production agents will hit edge cases. The key is identifying the failure pattern quickly and fixing the root cause instead of layering retries and prompt workarounds. Most issues fall into a small number of categories and each has a mechanical fix once you know what to look for.
Rate limits and quota exhaustion
Rate limits and quota exhaustion are the most common issues. Agents may run normally for days and then suddenly start returning generic network errors or empty responses. The fix is almost always visibility and budgeting. Track how many web operations each request performs, cache reused results, and enforce per-request limits so a single prompt can’t consume an outsized share of your allowance.
Timeouts and blocking Errors
Timeouts and blocking errors usually point to excessive scope. Long crawls, deep link following, or slow sites can push execution past limits, even on extended runtimes. Don’t start by increasing timeouts. Reduce crawl depth, split large jobs into smaller steps, or move genuinely long operations to asynchronous workflows.
Repeated CAPTCHA or block failures on a specific domain are a signal to adjust access patterns, reuse sessions, slow request rates, or change geographic origin, not to retry harder.
Configuration failures
Treat configuration failures as a first-class concern. Missing environment variables, expired keys, or mismatched local and production settings tend to break everything at once. If all requests fail simultaneously, verify secrets first in your Vercel project settings and add startup checks that confirm critical configuration is loaded.
Once you approach troubleshooting as an application problem rather than a prompt problem, failures become predictable, fixable, and far less disruptive.
Conclusion
At this point, you have an AI agent that can reason in steps, access live web data without collapsing under anti-bot defenses, and run predictably in a serverless production environment. That shift separates experiments that work once from systems you can safely put in front of users.
The focus of this guide is separation of concerns. Agent logic stays focused on reasoning and decision-making. Web access is handled by a hardened layer that absorbs CAPTCHA, rate limits, and blocking behavior. Deployment and scaling are handled by Vercel, while web reliability is handled by Bright Data, with MCP keeping the boundary explicit and stable.
From here, progress should be incremental. Start small, validate real usage on free tiers, and add caching, session reuse, or async workflows only where they pay off. Reliable web access isn’t the end goal — it’s the baseline that finally lets AI agents do useful work in production.
