AI is changing what search needs to be
For decades, Google has been the dominant search engine on the internet. Things are changing. As AI gets more heavily intertwined with everyday life, humans aren’t the only consumers of search engine results. As things continue to evolve, there’s a new trend emerging — search engines for AI.
The shape of search engine demand is changing. People still need to search. Engines still need to output results for human eyes. However, they also need to give AI-friendly results. Search Engine Results Page (SERP) APIs are evolving with features like automated parsing and multi-engine support. Semantic search APIs are using Large Language Models (LLMs) to abstract away SERP outputs.
Today’s search engines are becoming more AI-friendly and machine-readable.
SERP APIs are a firehose of search engine data. They are indespensable for things like, competitive intelligence, Search Engine Optimization (SEO) and data discovery powered by web crawling. SERP datasets are incredible tools for data mining and general tasks across the web such as retail intelligence. They give structure to web search data.
Semantic search APIs solve a different problem. AI models come with limited context windows. They don’t always need raw search data. They need compact, trustworthy data — similar to the prompts we use when talking to chatbots. With semantic search, an AI model can perform a query and get an answer. It doesn’t need to sift through an ocean of data.
The friction between human-oriented search and machine-oriented retrieval
Traditional search engines weren’t built for machine-readable results. Your average search engine spits out a list of hyperlinks with descriptions. However, these descriptions are added by individual site developers as metadata. They are not unbiased summaries. Sometimes descriptions are spot on. Sometimes they’re embellished. Other times, they outright lie. Humans have the gift of discernment — we can look at bad results and simply ignore them. Unless they’re equipped with complex reasoning (most simple AI agents aren’t), AI models have a difficult time doing this.
Let’s take a look at a simple Google search results page. As you can see, it looks simple and easy on the eyes. Each result has a header with a link to the content. Below the header is a general description of the site.
For humans looking through a browser, this is very easy to read. When you look at the document itself however, it’s anything but clean or simple. The snippet below contains only a portion of our first result.
<div class="b8lM7"><span class="V9tjod" jsaction="trigger.mLt3mc"><a jsname="UWckNb" class="zReHs" href="https://spaceplace.nasa.gov/blue-sky/en/" data-ved="2ahUKEwjDz6GlupKRAxWsEDQIHb54AGYQFnoECCoQAQ" ping="/url?sa=t&source=web&rct=j&opi=89978449&url=https://spaceplace.nasa.gov/blue-sky/en/&ved=2ahUKEwjDz6GlupKRAxWsEDQIHb54AGYQFnoECCoQAQ"><h3 class="LC20lb MBeuO DKV0Md" id="_lVQoaYOxJ6yh0PEPvvGBsAY_71">Why Is the Sky Blue?</h3><br><div class="notranslate ESMNde HGLrXd ojE3Fb"><div class="q0vns"><span class="H9lube"><div class="eqA2re NjwKYd Vwoesf" aria-hidden="true"><img class="XNo5Ab" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAACqklEQVR4AaxSQ2AjABDM2XZt82zbtm2rtm2bSWrbtpPatt0521h8d0cEAKP+Zf/vAT4TyrSdftWPeFwqYtlIlbW0+sT62bJiiTQmfmI0dllzfnqAiVjBuYmcmu5XNYBtAfWgJ1aC060RNNZpmKIgj+Wi1yhRe49rFK9Y5U8VFE6n8Amm5p487/D2wHTR6wtmSj4qnCHzAjOM3cHqTAGfUzFoHEqwzb8OPlU9KDY3Ry4jG+JZBRDNIohoOh7ECqyNfntg9BVeo0mPD2O66E3s38kF4oq18BPcAAdjVzT1DWJoaAB5Ns4Ins2GMHoBhNHxI3Q+BwrJnrsIcxTk5oy/s7l19M1lWHpIEGqb9oPEsQnUx6IAhtHa1oa6hgYUZ2UjWngjQuexI3wmE7LuPHV6qwG7W/WGSQ/2Ycz1pSBcFQJBi4yrYa3oKi8H1cgSxR6+KKtvAbWmARRnVySyCSNsz5loSknljHcHDDMPTXh8AGOvLYHYOUF4mR5Ggp818pUNkG9shbLiEhzSCYQGOQaNra1Q90jCUgfqhY8ucD2N2DXVzBHXzy0BnvEADSzoqjiPpsZONHb3oKasAiHuPhC318VW51AwuDeCiVx16OMBuste3NymOY0Zp9cDUkwAhR1UkjCWK16H+uUzKBLcBvlNa0G4zIEZaiZvrO1nJ5UzfzwgrhReLyPuCuerl5F0ax/C753GWXVbTHt0CU6LaOHExAnaM0KY+PwEWOwzwEGuIn4RpN2HLYl8J5wu0JBq8zlJZWAiV4POowUnX5jj7N7NmHlrCwjXeDFPWRVcHq21H79/nUR2j2peVnJlJhupAuzEcrC5VIHGtQQ05kEYf3s9GC0iC19Nbc0zc7x5AWg6PxCXAvFxIH4OdO5L1dWPT8vPO1yruuqBKE1yIwC+sJn+pZPFvAAAAABJRU5ErkJggg==" style="height:18px;width:18px" alt="" data-csiid="lVQoaYOxJ6yh0PEPvvGBsAY_11" data-atf="4"></div></span><div class="CA5RN"><div><span class="VuuXrf">NASA Space Place – NASA Science for Kids (.gov)</span></div><div class="byrV5b"><cite class="qLRx3b tjvcx GvPZzd cHaqb" role="text">https://spaceplace.nasa.gov<span class="ylgVCe ob9lvb" role="text"> › blue-sky</span></cite></div></div></div></div></a></span><div class="B6fmyf byrV5b Mg1HEd"><div class="HGLrXd ojE3Fb"><div class="q0vns"><span class="H9lube"><div class="eqA2re NjwKYd" style="height:18px;width:18px"></div></span><div class="CA5RN"><div><span class="VuuXrf">NASA Space Place – NASA Science for Kids (.gov)</span></div><div class="byrV5b"><cite class="qLRx3b tjvcx GvPZzd cHaqb" role="text">https://spaceplace.nasa.gov<span class="ylgVCe ob9lvb" role="text"> › blue-sky</span></cite></div></div></div></div><div class="csDOgf BCF2pd ezY6nb L48a4c"><div id="_lVQoaYOxJ6yh0PEPvvGBsAY_82" jsdata="l7Bhpb;_;lVQoaYOxJ6yh0PEPvvGBsAY25" jscontroller="i8S0p" data-id="atritem-https://spaceplace.nasa.gov/blue-sky/en/" jsslot="" jsaction="rcuQ6b:npT2md;h5M12e" data-ved="2ahUKEwjDz6GlupKRAxWsEDQIHb54AGYQ2esEegQIKhAJ"><div class="MJ8UF iTPLzd rNSxBe eY4mx lUn2nc" aria-describedby="_lVQoaYOxJ6yh0PEPvvGBsAY_71" style="position:absolute" aria-label="About this result" role="button" tabindex="0"><span class="D6lY4c"><span class="xTFaxe z1asCe" style="height:18px;line-height:18px;width:18px"><svg focusable="false" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8c1.1 0 2-.9 2-2s-.9-2-2-2-2 .9-2 2 .9 2 2 2zm0 2c-1.1 0-2 .9-2 2s.9 2 2 2 2-.9 2-2-.9-2-2-2zm0 6c-1.1 0-2 .9-2 2s.9 2 2 2 2-.9 2-2-.9-2-2-2z"></path></svg></span></span></div></div></div></div></div>
The snippet above contains a small string of text and the vast majority of it is just HTML with embedded JavaScript. When we distill it down to just the text, this is what’s left.
NASA Space Place – NASA Science for Kids (.gov)
https://spaceplace.nasa.gov › blue-sky
Why Is the Sky Blue?
The actual HTML document is well over 90% browser code. Imagine feeding ten or even a hundred of these raw results to a human and expecting them to read the page. Your model’s context window gets used up almost immediately and the token cost adds up quickly. AI models can extract data from this but it’s often inaccurate and terribly inefficient.
Raw HTML data is very loosely structured and machines need structure. SERP APIs were originally built to address this problem for traditional, non-intelligent software. Instead of a swamp of HTML, these APIs converted the page content into structured data. In this next snippet, we render that same title data as a JSON object.
{
"website": "NASA Space Place – NASA Science for Kids (.gov)",
"url": "https://spaceplace.nasa.gov"
}
Real search APIs are more detailed than the snippet above but the concept is the same. Convert the raw, unreadable HTML into something lightweight and machine-readable.
What AI actually needs from web search
Outside computer vision, AI models don’t see the web the way we do. They don’t see layout, color or spacing. They see a document filled with irrelevant nonsense. Without structured data, we just expect them to magically find the relevant data from within the mess.
AI models need structure. Data needs to be separated from its presentation medium. Relevance needs to be separated from inline JavaScript. Context needs to be separated from raw HTML.
For years, JSON has been the standard format for machine-readable data. We can pack a list of JSON objects into a list called, sometimes called an array. That’s not the point though. JSON is portable and it’s agnostic. It doesn’t care if you’re working with JavaScript or Python. It doesn’t care if you’re human or AI. It just fits into standard programming environments. It was used with hardcoded software long before it was used with generative AI.
There’s another data format that works remarkably well with AI models — plaintext. Generative-Pre-trained Transformers (GPTs) are especially good with plaintext. They were originally designed for general purpose language. In plaintext format, models handle our data the same way they handle a prompt. This isn’t as structured as JSON but text models are built to understand text semantically.
Retrieval mode 1: SERP APIs as machine-native access to general web search
For general web search and result analysis, SERP APIs are almost unbeatable. We’re not chained to a specific search engine and our results come in a standardized format that can be utilized by all software — AI-powered and traditional.
Now, we’ll use Bright Data’s Python Software Development Kit (SDK) to perform a search and view the results. You can install the SDK using pip.
pip install brightdata-sdk
Here is our basic code example. We simply import our client and Python’s native JSON library. Next, we initialize the client and call the search() method. The search method takes a variety of keyword arguments but the two you should really pay attention to are query and search_engine. query takes in a list of search queries and search_engine allows us to choose from any number of engines offered by the provider. In this case, we use Bing. parse=True tells the API to parse the results for us — this is where our structured output comes from.
from brightdata import bdclient
import json
client = bdclient(
api_token="your-bright-data-api-key",
)
search_results = client.search(
query=["why is the sky blue?"],
search_engine="bing",
response_format="json",
parse=True
)
with open("search-results.json", "w") as file:
json.dump(json.loads(search_results[0]["body"]), file, indent=4)
Now, let’s take a look at one of the objects from the JSON output. We get all sorts of detailed output within a single structure. The object contains multiple links, a site title, the site description and our ranking number.
{
"link": "https://spaceplace.nasa.gov/blue-sky/en/",
"tracking_link": "https://www.bing.com/ck/a?!&&p=e1f4ae75e20d42fdfcad98e031105492e1988c6ae59187144df15d628c2c13e0JmltdHM9MTc2NDIwMTYwMA&ptn=3&ver=2&hsh=4&fclid=2593b546-5f53-69cf-3f5a-a3f75e6c6847&u=a1aHR0cHM6Ly9zcGFjZXBsYWNlLm5hc2EuZ292L2JsdWUtc2t5L2VuLw&ntb=1",
"display_link": "https://spaceplace.nasa.gov \u203a blue-sky \u203a en",
"title": "Why Is the Sky Blue? | NASA Space Place \u2013 NASA Science for Kids",
"site_name": "NASA Space Place",
"rank": 1,
"global_rank": 1
},
{
"link": "https://www.britannica.com/story/why-is-the-sky-blue",
"tracking_link": "https://www.bing.com/ck/a?!&&p=89c0ef8fbae04f29b377f2b21d9ff65e6e38b7bb074c5b5318a2fd9da8a2d59dJmltdHM9MTc2NDIwMTYwMA&ptn=3&ver=2&hsh=4&fclid=2593b546-5f53-69cf-3f5a-a3f75e6c6847&u=a1aHR0cHM6Ly93d3cuYnJpdGFubmljYS5jb20vc3Rvcnkvd2h5LWlzLXRoZS1za3ktYmx1ZQ&ntb=1",
"display_link": "https://www.britannica.com \u203a story \u203a why-is-the-sky-b\u2026",
"title": "Why Is the Sky Blue? | Britannica",
"site_name": "Britannica",
"extensions": [
{
"type": "fact",
"key": "Author",
"value": [
{
"text": "John P. Rafferty"
}
],
"rank": 1
}
],
"rank": 2,
"global_rank": 2
},
Since all of this is contained within a single schema, it’s built for machine-readability. In this format, AI models can analyze thousands of results and make inferences quickly.
SERP APIs let AI models use data from all types of search engines while still maintaining a structured, unified output format.
Retrieval mode 2: Semantic-first search engines built for natural language
Next, we’ll look at a newer search method called semantic search. Here, we’ll use the Tavily API to perform the same query and analyze the output.
We’ll start by installing the Tavily SDK.
pip install tavily-python
Here is our code. It’s very minimal — and that’s the point. This SDK is built for natural language summary, not structured analysis. We’re approaching the same query through a much different lens. Once we’ve initialized the client, we call qna_search() to perform a simple question and answer search. This outputs a string which can be piped directly into an AI model’s chat interface.
from tavily import TavilyClient
tavily_client = TavilyClient(api_key="tvly-dev-your-tavily-api-key")
response = tavily_client.qna_search("Why is the sky blue?")
print(response)
Here’s the output. Tavily performs the search for us. Then it gives a response with semantic understanding. The result is remarkably light. Please note, qna_search() is capable of hallucination. Results do need to be double-checked.
The sky appears blue because sunlight scatters more in shorter wavelengths, with blue light being scattered more than red. This scattering occurs due to particles in Earth's atmosphere.
If we want structured results, we can get those too. Instead of qna_search(), we use plain old search().
from tavily import TavilyClient
import json
tavily_client = TavilyClient(api_key="tvly-dev-your-tavily-api-key")
response = tavily_client.search("why is the sky blue?")
with open("search-results.json", "w") as file:
json.dump(response, file, indent=4)
Note that our results are contextually relevant but lack some of the finer details and nuance included from traditional SERP. Each result has a score attribute — Tavily is telling us (and AI agents) how well the content matches the query semantically.
{
"url": "https://www.nesdis.noaa.gov/about/k-12-education/atmosphere/why-the-sky-blue",
"title": "Why Is the Sky Blue? | NESDIS - NOAA",
"content": "Blue light is scattered more than other colors because it travels as shorter, smaller waves. This is why we see a blue sky most of the time.",
"score": 0.9983125,
"raw_content": null
},
{
"url": "https://www.reddit.com/r/askscience/comments/14566ig/why_is_the_sky_blue_do_i_understand_it_correctly/",
"title": "Why is the sky blue? Do I understand it correctly: : r/askscience",
"content": "Because the sun is intense, the scattering gives you lots of blueness in the sky. But because the sun doesn't travel very far through the",
"score": 0.99827254,
"raw_content": null
},
With semantic search, we can get an AI-powered answer to the query with contextual understanding. These APIs approach the problem differently. Rather than analyzing SEO, AI agents receive short, relevant answers so they can move on to their next task without reasoning or inference.
AI widens the retrieval landscape instead of replacing search
AI is quickly becoming a massive consumer of search data. Web search is not getting replaced. It’s branching into new modes. AI models can use SERP APIs for detailed analysis of search results across major search engines. Semantic search options like Tavily allow models to get precise results fast.
Neither of these tools is built to replace the other. They power separate use cases. As AI-powered search grows, we’re not consolidating toward a single search engine like we did in the early 2000s. We’re growing into a set of tools that can serve the broad use cases of web search.