Free Web Search for AI Agents: Scraping Bing with C# HttpClient (No API Key Needed)
I’ve been working on my Web Agent lately and wanted to set up a free Web Search fallback.
Google Search API was the first thing that came to mind. But after checking the official docs, I found out that Google has already stopped accepting new users for its Custom Search JSON API. Existing users also have to migrate before January 1, 2027.
It used to offer 100 free searches per day. Now you can’t even sign up for it anymore :’(
I’d previously played around with Exa Web Search API. It returns JSON directly and can even fetch webpage content for you. Pretty convenient, but once the free credits run out, you’re paying for it.
I figured DuckDuckGo might be a decent Google alternative. It has a plain HTML search page, so grabbing it with C# HttpClient and parsing the results should be easy enough, right?
Nope. Three consecutive tests on my local machine, all HTTP 403. Didn’t expect DuckDuckGo’s anti-bot protection to be that aggressive…
Then I decided to give Bing a shot. Turns out its regular search page was still accessible through a simple HTTP request. No Playwright, no Chromium, no headless browser.
So I added Bing to my Web Agent as a fallback for DuckDuckGo.
Scraping Bing Search Results with C# HttpClient
No extra NuGet packages. Just HttpClient + Regex, including the HTML parsing.
using System.Net;
using System.Text.Encodings.Web;
using System.Text.Json;
using System.Text.RegularExpressions;
internal class Program
{
static async Task Main(string[] args)
{
var queryText = "台北 甜點店";
var limit = 10;
using var handler = new HttpClientHandler
{
AutomaticDecompression =
DecompressionMethods.GZip | DecompressionMethods.Deflate
};
using var client = new HttpClient(handler);
client.DefaultRequestHeaders.UserAgent.ParseAdd(
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0 Safari/537.36");
client.DefaultRequestHeaders.AcceptLanguage.ParseAdd(
"zh-TW,zh;q=0.9");
var query = Uri.EscapeDataString(queryText);
// Prefer Taiwan and Traditional Chinese results.
var url = $"https://www.bing.com/search?q={query}&cc=TW&setlang=zh-Hant";
using var response = await client.GetAsync(url);
response.EnsureSuccessStatusCode();
var html = await response.Content.ReadAsStringAsync();
var results = new List<object>();
var matches = Regex.Matches(
html,
@"<li class=""b_algo""[^>]*>(.*?)</li>",
RegexOptions.Singleline | RegexOptions.IgnoreCase);
foreach (Match m in matches)
{
if (results.Count >= limit) break;
var block = m.Groups[1].Value;
var aMatch = Regex.Match(
block,
@"<h2[^>]*>\s*<a[^>]*href=""([^""]+)""[^>]*>(.*?)</a>\s*</h2>",
RegexOptions.Singleline | RegexOptions.IgnoreCase);
if (!aMatch.Success) continue;
var rawUrl = WebUtility.HtmlDecode(aMatch.Groups[1].Value);
var title = StripHtmlTags(aMatch.Groups[2].Value);
var finalUrl = ResolveBingUrl(rawUrl);
var snippetMatch = Regex.Match(
block,
@"<div class=""b_caption""[^>]*>.*?<p[^>]*>(.*?)</p>",
RegexOptions.Singleline | RegexOptions.IgnoreCase);
var snippet = snippetMatch.Success
? StripHtmlTags(snippetMatch.Groups[1].Value)
: string.Empty;
results.Add(new
{
title,
url = finalUrl,
snippet
});
}
var output = new
{
success = true,
engine = "bing",
results
};
var options = new JsonSerializerOptions
{
WriteIndented = true,
Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping
};
Console.WriteLine(JsonSerializer.Serialize(output, options));
}
static string ResolveBingUrl(string url)
{
var match = Regex.Match(url, @"[?&]u=a1([^&]+)");
if (match.Success)
{
try
{
var base64 = match.Groups[1].Value
.Replace('-', '+')
.Replace('_', '/');
var padding = (4 - (base64.Length % 4)) % 4;
base64 = base64.PadRight(base64.Length + padding, '=');
var bytes = Convert.FromBase64String(base64);
return System.Text.Encoding.UTF8.GetString(bytes);
}
catch
{
return url;
}
}
return url;
}
static string StripHtmlTags(string input)
{
var decoded = WebUtility.HtmlDecode(input);
var text = Regex.Replace(decoded, @"<[^>]+>", string.Empty);
return Regex.Replace(text, @"\s+", " ").Trim();
}
}Actual Search Results
Here’s what I got when searching for 台北 甜點店. Ten results, with the actual URLs already decoded.
{
"success": true,
"engine": "bing",
"results": [
{
"title": "2026 台北甜點推薦：12 間下午茶與甜點店，滿足你的甜食控靈 …",
"url": "https://blog.ocard.co/life/dessert-time-taipei/",
"snippet": "2026年3月13日 · 台北甜點、蛋糕推薦名單就在這！ 想跟朋友相約吃下午茶、甜點，或是購買生日蛋糕、母親節蛋糕、父親節蛋糕等， …"
},
{
"title": "2026 【台北甜點】推薦11間台北巷弄甜點店，高評價手工現 …",
"url": "https://gofunit.com/%E5%8F%B0%E5%8C%97%E7%94%9C%E9%BB%9E/",
"snippet": "2025年12月31日 · 喜歡吃甜點的女孩來到這堪稱點心王國的城市，一定不能錯過接下來小編要介紹的10間超夯台北甜點店！ 有專職千 …"
},
{
"title": "2026台北甜點店｜15間Dcard推薦台北蛋糕下午茶名單，精緻平 …",
"url": "https://mymarcy.tw/taipei-patisserie/",
"snippet": "2026年5月29日 · 本台北甜點推薦文收集dcard、threads眾網友分享的台北下午茶、台北蛋糕推薦，親自走訪分享過濾，還有不少令我 …"
},
{
"title": "【2026台北甜點推薦】精選11間高CP值平價甜點：學生小資族 …",
"url": "https://dalang.tw/台北甜點懶人包/",
"snippet": "2026年9月2日 · 獨家分享告訴你11間評分最高的台北甜點推薦，Dcard與PTT人氣推薦必吃的台北甜點，想找台北甜點伴手禮、台北平 …"
},
{
"title": "台北必吃甜點店｜精選40家下午茶懶人包、台北好吃甜點推薦、i…",
"url": "https://angelababy0822.com/post-466959821/",
"snippet": "2026年7月14日 · 嚴選台北超過50家好吃甜點、蛋糕下午茶懶人包 真心推薦的好吃、好拍、氣氛佳的網美必訪咖啡甜點午茶店 網 …"
},
{
"title": "甜點控必存！11家編輯私藏台北法式甜點名店，走進巴黎般高級的甜點 …",
"url": "https://rosy-arts.com/life/taipei-french-pastries/",
"snippet": "2026年7月10日 · 台北的法式甜點如此豐富，該如何尋覓那間命定店鋪？ Rosy編輯精選了11家超人氣的法式甜點名店，令人驚豔 …"
},
{
"title": "甜點推薦清單！19間名店統整，蛋糕、肉桂捲、泡芙、可頌通通有",
"url": "https://www.heritage.com.tw/post/best-desserts-in-taipei",
"snippet": "甜點推薦吃哪家？ 台北甜點推薦外帶買什麼？ 本文依烘焙、泡芙、可頌、法式與日式分類，整理19間好吃的台北甜點店名單與地 …"
},
{
"title": "台北排隊甜點清單｜10家人氣甜點，限量甜甜圈、草莓盒一次收…",
"url": "https://supertaste.tvbs.com.tw/hot/357894",
"snippet": "2026年1月3日 · 《食尚玩家》幫你把10家台北排隊甜點名店一次整理好，從甜甜圈、布丁、法式巧克力到台味粉條甜品通通入榜，還 …"
},
{
"title": "2026台北甜點店下午茶推薦 精選10家特色甜點 抹茶/巧克力專賣 …",
"url": "https://lizzzstyle.tw/%e5%8f%b0%e5%8c%97%e7%94%9C%e9%bb%9E%e5%BA%97%e4%b8%8b%e5%8d%88%e8%8c%b6%e6%8E%A8%e8%96%A6%2010%E5%AE%B6/",
"snippet": "2024年10月2日 · 且本篇甜點類型不只有一般蛋糕，更有台北巧克力專賣店、抹茶專賣店、人氣布朗尼、舒芙蕾......等，有些是有內用 …"
},
{
"title": "台北平價甜點店地圖：內行推薦5間高CP值口袋名單",
"url": "https://benjamintrips.tw/read-1473424",
"snippet": "2026年4月9日 · 想在台北找到高CP值的平價甜點嗎？ 本文親自走訪5間內行人才知道的平價甜點，從老字號豆花到創意千層蛋糕，每間 …"
}
]
}Bing’s URL Redirects Are a Bit of a Pain
Most Bing search results are wrapped in the b_algo class. Pulling out titles, URLs, and snippets with Regex isn't too bad. The annoying part is Bing's redirect URLs.
Get Donma Hsu 當麻許’s stories in your inbox
Join Medium for free to get updates from this writer.
Some links come wrapped in this u=a1... format. You have to strip the a1 prefix, deal with Base64 characters, and manually fix the padding just to get the actual URL.
All that work just to grab a search result. At least we’ve got AI to help with this stuff now :P
And since the whole thing relies on Bing’s HTML structure, Microsoft could change a CSS class tomorrow and break the Regex. But again, with AI around, fixing the parser isn’t exactly a big deal anymore, as long as they don’t start blocking requests too aggressively.
Plugging It into My AI Agent’s web_search Tool
My Web Agent already has a web_search Tool with just two parameters: query and limit. It returns 5 results by default, with a maximum of 10.
The logic is pretty straightforward. Try DuckDuckGo first. If it throws an HTTP error, hits a CAPTCHA, or returns nothing after parsing, automatically fall back to Bing. Only report a search failure to the LLM if both engines fail.
The LLM doesn’t need to know which search engine is running behind the scenes. It just calls the same Tool. If I decide to add Exa or another Search API later, I won’t need to touch the Tool Schema.
My system is designed for multiple users running at the same time. There’s no way I’m spinning up a Chromium instance for every single search. The memory and CPU usage would be ridiculous…
The actual project also uses a shared HttpClient, a 20-second timeout, and CancellationToken. I left those details out of the Console example to keep it short.
The Cost of Free
Scraping Bing and DuckDuckGo doesn’t require an API key, and there are no per-search API charges.
But let’s face it, this is still scraping. If a search engine blocks your IP, throws a CAPTCHA at you, or changes its HTML, you’re the one who has to deal with it.
Compared to something like Exa, you’re basically trading development and maintenance time for API costs.
For now, I’m keeping this as a backup search option for my Web Agent. No idea how long Bing will keep working this way. If it starts throwing HTTP 403 too, I’ll deal with it when that happens.
Enjoy it while it lasts…
Google’s official documentation: Custom Search JSON API
