The Quiet Problem With AI Search

I have spent a fair amount of time thinking about why AI search feels simultaneously impressive and unreliable, and I kept arriving at the same uncomfortable conclusion: the problem lives upstream of the AI itself. The model is often doing its job reasonably well. The corpus it draws from, and the attack surface surrounding that corpus, is where things fall apart.

The first thing I want to establish is that robots.txt is a 30-year-old protocol designed for a world where "crawler" meant Googlebot indexing pages for a search ranking algorithm. It was built for discovery, for helping humans find content. It was built for a moment when the relationship between crawler and content was straightforward. That world is gone, and robots.txt never evolved to meet the new one.

What I find striking is the 14% statistic. Only about 14% of the top 10,000 websites have explicit AI bot directives in their robots.txt files. That means 86% of the web has formed zero opinion on whether GPTBot or ClaudeBot should be reading and synthesizing their content. The sites that have formed an opinion tend to be the ones with the most to lose: academic publishers, serious journalism outlets, paywalled newsletters, and carefully maintained databases. They block the bots. What passes through freely is the long tail of SEO farms, opinion blogs, and anyone actively seeking AI-generated traffic.

I think about this as a self-selection problem, and self-selection problems produce systematically biased samples. The AI synthesizes from whatever it can access, and then presents that synthesis with the same confident tone it would use if it had read everything worth reading. The confidence is real. The completeness is illusory.

Layered on top of this is the compliance issue, and I consider this the more philosophically troubling part. Robots.txt compliance is voluntary. Reputable crawlers respect it. Crawlers built by actors with lower standards for data integrity tend to treat it as a suggestion. So the sites most invested in protecting their content quality end up blocking the good actors, while the bad actors scrape freely. The result is that principled crawlers see a worse web than unprincipled ones.

Prompt injection is where I think the situation moves from structurally broken to actively adversarial. Businesses are already embedding invisible instructions into webpages, hidden text directing AI tools to present their brand favorably, to omit competitive comparisons, to treat their product as a default recommendation. AI search tools were processing this as legitimate content because the model processes everything on the page and has limited ability to distinguish between data it should summarize and instructions it should follow.

I watched Brave Security demonstrate this on Perplexity's Comet browser assistant, and it clarified something for me. This is surgical manipulation. A human reading the page sees nothing unusual. The AI reads the page and gets hijacked. The output the user receives reflects the attacker's intent rather than the page's actual informational content, and the user receives it wrapped in the same confident synthesis they always get.

The defenses that exist today are mostly signature-based. Phrases like "ignore previous instructions" get flagged because they appear often enough in known attack patterns that filters can catch them. I find these defenses reassuring for exactly the wrong reasons. They catch the unsophisticated attacks. The sophisticated ones require a subtler approach, and subtle approaches are harder to filter because they blend into legitimate content.

A page that frames every competitor negatively, or describes its own product only in superlatives, or structures its content so the most retrievable paragraphs contain only favorable framing, biases the synthesis without triggering a single filter. The AI reads it, weights it, and incorporates it. The bias enters the output invisibly.

The multimodal expansion is the part I find most alarming about the near future. Text-based injection defenses are maturing, slowly and imperfectly, but they exist. Image and audio vectors are essentially open. Adversarial text can be embedded into the pixel layer of an image, invisible to any human looking at it, but readable by a vision model processing that image as part of a retrieval task. The defense infrastructure around this attack surface is close to zero right now, and AI search is moving aggressively toward processing images and audio as primary content.

I also think the network effect here runs in the wrong direction over time. As AI search grows in influence, high-quality publishers have increasing incentive to block crawlers entirely to protect their subscription models and their brand integrity. The corpus therefore gets worse as the technology scales and gains more users. People assume that more data and more scale produce better outputs. In this specific context, I believe the opposite is true.

Personalization adds another layer that I think about often. When AI search begins tailoring results to individual behavioral profiles and inferred preferences, injection attacks acquire the ability to be targeted. A malicious actor could theoretically embed instructions that activate only for users matching certain behavioral signals. The manipulation becomes individualized, auditable by essentially nobody, and invisible in the exact same way that makes current injections effective, with the added dimension of precision targeting.

The confidence calibration problem ties all of this together. These systems are trained to produce fluent, authoritative outputs. That training was appropriate when the data was relatively clean and the retrieval layer was passively indexed content. Applied to a retrieval environment that is actively gamed by commercial interests and bad actors, the confident tone becomes structurally misleading. The output gives the user zero signal about how polluted the upstream sources actually were.

I keep returning to the Google comparison because I think it illuminates why this matters. Google shows me ten links and my brain performs the arbitration. I see the source, I assess the credibility, I triangulate across results. Perplexity-style synthesis collapses that into a single voice. The injection, the bias, the skewed corpus, all of it arrives pre-arbitrated. The user receives a conclusion rather than evidence, which means the manipulation has already succeeded before critical evaluation can begin.

What I believe is required here goes beyond better filters or updated crawling standards. The field needs transparency infrastructure: disclosure of which sources contributed to a synthesis, confidence signals calibrated to source quality rather than fluency, and architectural separation between content the model should summarize and instructions the model should follow. Until that infrastructure exists, AI search will remain a system that sounds authoritative precisely because it obscures the mess that produced its answers.