The AI revolution has a massive, undocumented energy leak: the friction of legacy web crawling. Forcing LLMs to download visual code (HTML/CSS) just to extract semantic data is destroying corporate ESG goals. We are testing a solution in real-time.
The Premise: Business-to-AI (B2AI) Routing
Within the Project EOS (EndOfSearching) framework, we are conducting a live A/B test across multiple edge nodes. The hypothesis is simple: if we can detect an AI crawler at the network edge and serve it a pure JSON-LD vector instead of the visual website, we can drastically cut the digital carbon footprint (Scope 3 emissions) associated with RAG and LLM training.
Phase 1 Results: The 34% Baseline
Status: Completed
Our initial deployment utilized standard 'User-Agent' detection (identifying known bots like ClaudeBot or Googlebot). The data confirmed our theory, yielding a 34% reduction in bandwidth consumption per request compared to the unoptimized legacy node.
The Challenge: Dark Bots and Spoofers
A 34% reduction is a significant optimization, but it falls short of our architectural goal. Telemetry analysis revealed why: the internet is full of "Dark Bots". These scrapers actively spoof their identities, claiming to be human browsers (e.g., Chrome on Windows) to bypass basic filters, forcing our edge servers to deliver the heavy, energy-intensive visual payload.
Phase 2: Open Sourcing the Level 3 Router
To achieve true ESG compliance, we cannot rely on what a bot claims to be; we must look at where it lives. We have upgraded the B2AI Router to Level 3 Detection by analyzing Autonomous System Numbers (ASN) via Cloudflare's edge telemetry.
Because climate responsibility cannot wait for corporate paywalls, I am open-sourcing the core logic of our Level 3 Filter. Any developer can implement this in minutes to stop datacenter spoofers from wasting their server's energy:
// =====================================================================
// B2AI DETECTOR (LEVEL 3: DATACENTER / ASN VALIDATION)
// =====================================================================
const userAgent = request.headers.get('User-Agent') || '';
const asnOrg = (request.cf && request.cf.asOrganization) ? request.cf.asOrganization : '';
// 1. Basic ID: Catch official bots and lazy scrapers
const botRegex = /bot|crawler|spider|claude|perplexity|python|curl|wget|http-client|headless|puppeteer|playwright|postman|slurp|yandex/i;
// 2. The Datacenter Trap: Humans use residential ISPs. If it comes from a cloud provider, it's a machine.
const datacenterRegex = /amazon|aws|google cloud|google inc|microsoft|azure|digitalocean|hetzner|ovh|linode|contabo|alibaba|tencent|hosting|cloud|datacenter/i;
// 3. Final Verdict: It's a bot if the ID says so, if it's empty, OR if the network cable comes from a server farm.
const isBot = botRegex.test(userAgent) || userAgent.trim() === '' || datacenterRegex.test(asnOrg);
if (isBot) {
// Abort visual HTML. Serve pure JSON-LD vector (4KB).
return new Response(JSON.stringify(jsonLdData, null, 2), {
headers: { "Content-Type": "application/json;charset=UTF-8" }
});
}The new protocol is live. We are letting the global network stress-test this logic. The telemetry from the coming days will determine if we can push that 34% efficiency rate closer to total elimination of crawling waste.