This July, in one of my test runs, I watched an AI agent ask a mock firewall, built to answer like a real one, for every event tied to one IP address. The agent sent the timestamp in one format. The firewall expected another. Nothing matched, so the source returned zero rows, with no error and no warning. The agent read zero rows as “no firewall activity for this entity” and reported it with the same confidence as everything else it found. Nothing in the report said a whole data source had silently dropped out of the investigation.
I now have a name for that failure: confidently incomplete. It is worse than a wrong answer. A wrong answer can get caught downstream. A confident answer built on a fraction of the evidence looks exactly like a complete one, and it ends the investigation. In my tests, an agent working through raw per-source tools found 78 percent of the known answer with 5 sources connected and 53 percent with 40, and it reported done with the same confidence both times.
First, the position I am writing from. I run Query. We build a security data mesh, a normalized layer that lets people and AI agents reach security data wherever it lives and get answers back in one consistent shape. We built the mesh years before agents arrived, for human analysts tired of pivoting across consoles. When agents showed up we put them on it, and they turned out to be better at driving it than the people we built it for. An analyst asks five careful questions and moves on. An agent will happily ask forty. One arm of this test is the pattern my company sells, and that arm mostly won. I am not a neutral researcher, so I have kept the results that went against me in the story, and I have put the full methodology and every number in an appendix at the end of this post. I will mark which parts are measurement, which are my experience operating the product, and which are opinion. Weigh it all accordingly.
And to be clear about where I stand on MCP: it deserves the enthusiasm. It made wiring a model to a data source feel routine instead of bespoke. What I set out to measure is what happens after the wiring works. Once an agent can reach your data, does the path it takes change how well it does the job?
The test
I built mock security data sources and scaled the environment from 5 to 40 of them, then gave an AI agent a set of real investigation questions to answer two ways. In one arm the agent reached each source through that source’s own MCP tools, the way most agents are wired today. Call that arm raw MCP. In the other arm it reached the same data through a single normalized engine I built to implement the mesh pattern, deliberately simpler than our platform. Call that arm the test mesh. Same model, same questions, same underlying records. The only difference between the arms was the path to the data.
I wrote down what I was measuring before any run, and every answer was scored against a known fact set by a judge model that never saw which arm produced it. Afterward I pointed the same kinds of questions at our live mesh product, with 62 real sources connected, to check whether the pattern survives contact with a real system. Those are two bodies of evidence, and they earn trust differently. The controlled test is the measurement. The live run is a validation probe on our own demo environment, not a customer estate. I will keep them separate as we go.
At five sources, it is a wash
Small scale first, because it is the regime a company like mine has every reason to skip past. At five sources the two arms traded wins. On a testbed of uniform, generated sources, raw MCP beat the test mesh by 5 points. On a testbed that mocks five real products and keeps their native quirks, identities that do not match across systems, timestamps in formats nobody agrees on, the same concept living under three field names, the test mesh won by 11. Normalization pays in proportion to the mess it has to clean up, and five sources built to match each other do not have much mess.
| At five sources, same three questions | Raw per-source MCP | One normalized layer |
| Generated sources, uniform shapes | 78% | 73% |
| Mocks of real products, native quirks intact | 80% | 91% |
So if someone tells you their agent works great against a handful of clean, well-integrated sources, believe them. Mine did too.
Then I added sources
I scaled the environment from 5 to 15 to 40 sources and asked the same estate-wide questions. Completeness is the share of the known answer each arm found. These are the estate-wide sweeps, broader questions than the three above. Five sources is the regime where raw ran strongest in this test, a 15-point lead. The advantage flips between five and fifteen sources and keeps widening.
| Sources | Raw per-source MCP | Test mesh |
| 5 | 78% | 63% |
| 15 | 59% | 74% |
| 40 | 53% | 70% |
The advantage swung 32 points between 5 and 40 sources. The mesh’s own five-source number deserves a sentence: most of that deficit sat in one task, where the raw arm’s coarse vendor tools returned whole histories by default and the mesh trials scoped their queries to tight time windows. That is a depth-of-retrieval difference inside sources both arms reached, and at four trials per cell I hold the mechanism loosely. On the single broadest question in the set, the one that asks the agent to look everywhere, raw fell from 75 percent to 20 while the test mesh climbed from 67 to 83. The lines cross and keep going.
A real security estate sits past the right edge of that chart. Across our customer base the median deployment connects about 60 sources, and a single integration multiplies; one AWS S3 connection can front hundreds of buckets. If you are running agents against your environment today, you are on the steep side of that curve.
It ran out of attention, not budget
The intuitive explanation is budget: more sources means more calls and more tokens, so the agent runs out of room and gives up. That is not what happened. From 5 to 40 sources the environment grew 8 times larger. The raw agent responded by roughly doubling its queries per investigation and spending about 40 percent more tokens. Effort went up, and completeness fell anyway. The agent did not run out of budget. It ran out of attention.
What changed is which sources got consulted, and how hard. At 5 sources the raw agent queried 94 percent of the sources relevant to the question. At 40 it queried 75 percent and skipped the rest. The looking also got shallow before it got narrow: at 40 sources, two of every three sources the raw agent consulted got exactly one query and no follow-up, against one in four at small scale. Its first questions stayed sharp the whole way. It just stopped asking second ones. The test mesh, on the identical harness and model, never dropped below 95 percent of relevant sources and used fewer queries to get there.
Two objections came up when colleagues reviewed this data. Was the agent economizing, trimming work to hold costs down? No. Raw’s cost per investigation rose 126 percent across the same range, which is not what a conserving agent looks like. The two numbers count different things: 40 percent is the growth in what the agent generated, while billed cost also counts the input side, the tool definitions and returned records flowing through the context window every turn, and that surface is what an 8x environment inflates. Can better instructions fix it? Partly, and mostly where you need it least. I appended the same explicit check-every-source instruction to both arms. At 5 sources it lifted raw to 92 percent, widening its small-scale lead. At 40 sources, on the broadest question, it lifted raw from 20 percent to 33, and the agent still made about seven queries against forty sources and stopped, nowhere near its turn limit.
Anthropic has written about this mechanism from the tool side: load too many tool definitions, pass every raw result through the context window, and agents degrade. I measured the same degradation one layer down, at the level of data sources, where the only symptom is a clean answer built on part of the evidence.
The metric worth tracking, and a test you can run this week
Of the data relevant to a question, how much did the agent consult before it answered? I call that source completeness, and almost nobody tracks it, because agents get judged on whether the answer looks right. A confidently incomplete answer looks right by definition.
You can measure yours without any tooling from me. Take a question where you already know what a complete answer requires. Ask your agent. Then read what it queried rather than what it concluded, and count the sources it should have opened and never did. Count the ones it opened once and never followed up on, too, and run it more than once. That gap is your source-completeness number. For calibration, the test mesh arm never consulted less than 95 percent of the relevant sources. The raw arm was at 75, and nothing in its output said so.
Sixty-two real sources
A controlled test with mocks establishes a pattern. Whether it holds on a real system is a separate question. So I ran the same kinds of questions against our live product: 62 real sources, right at our median customer’s scale, with known ground truth. This is the validation-probe arm: our own demo environment, not a customer estate. The agent ran 30 to 40 federated queries per investigation, finished each in 7 to 9 minutes, and scored between 64 and 93 percent complete on first-pass investigations, averaging 82 percent across the estate-wide sweeps. The test mesh averaged 70 percent on estate-wide questions at 40 mock sources. Moving from mocks to a real environment with half again as many sources, completeness landed in the same band, a little higher.
No analyst runs forty federated queries in nine minutes. This is the part of the job agents are plainly better at, once one query surface reaches the whole estate. And when I went back through the transcripts to attribute the misses, the mesh had reached the relevant records in every investigation. Every point lost was lost in how the agent queried and reasoned over what came back, which is why our agent runs hand every query and conclusion back as artifacts a person can check. The agent does the consulting. The analyst does the judging, which takes minutes and is the work analysts are actually good at. The Workers gallery has full runs if you want to see what that looks like.
There is no raw-MCP comparison in those results, and the reason is itself a finding. I could not stand up sixty real vendor MCP servers against the same data with the same credentials, and I have not seen anyone who has. At estate scale, the practical alternative to a mesh is raw MCP against whatever subset you managed to wire up and keep running.
The part nobody demos
The demo version of MCP is plug and play. The production version is a bill that arrives per source, indefinitely: schema and field-semantics changes underneath you, credential and auth lifecycle across a dozen vendors, rate limits, pagination schemes, and every time-format quirk each of them ships, with almost no visibility into any of it. The firewall story that opened this post is one line item on that bill. In the controlled test the raw arm scored 50 percent on the firewall task at every scale I ran. The test mesh scored 100, because it puts time into a common form before the query ever reaches the source.
A mesh does not make that work disappear. It moves it to a team whose entire job is the mappings, the health of every connection, the normalization, watched in one place instead of sixty. And normalization has its own failure mode: a mapping that quietly drops a field produces the same confidently incomplete answer from a different cause. One of my hand-authored mappings did exactly that during this work, and only the stricter of my two judges caught it. The difference is that a mesh gives you one place to catch it, and in the product it is our mappings, our monitoring, and our name on it when a field goes quiet.
“Where do you put your security data?”
The reply I get from architects is centralization: pick one platform, pipe everything into it, give the agent one source, and the problem collapses back to the small scale where direct access works. I have asked about 50 security teams this year where they put their security data, and I have yet to hear a one-word answer. The typical team pivots across 12 to 20 sources for each investigation, sometimes over 100, and the crossover in my test came between 5 and 15.
The economics push the same direction. One Fortune 250 energy company we work with cut its SIEM ingestion from 10 TB a day to 4 on purpose, because paying SIEM prices to store everything had stopped making sense. The other 6 TB did not stop mattering to investigations. It just lives somewhere cheaper now. So centralize where it pays. A mesh treats the places you centralize as sources like any other. Splunk is a source. The lake is a source. The agent reaches what you put in them and the data you decided not to pay to put there.
What else moved
I set out to measure completeness, and two other things moved with it. Cost: by 40 sources the test mesh was about a third cheaper per investigation while returning more of the answer, and faster at every scale I measured. The cheapest model I tested gained the most: Haiku through the layer was about 10 points more complete, 16 percent cheaper, and 25 percent faster than Haiku on raw tools.
And one accidental security finding. Agents that investigate attacker activity read data attackers have touched, so eventually someone will plant an instruction in a log field hoping an agent obeys it. In my probe, raw per-source tools passed the planted payload to the model on 100 percent of runs where a tool returned that record, because raw tools return whole records with nothing in between. The test mesh ingested it only when a queried field carried it, which means there is a surface where hostile content can be inspected and filtered before the model ever sees it. One architecture ships with a place to put a defense. With the other, you must build it yourself.
What I would do in your seat
Strip my company out and here is what the evidence supports. If you are running an agent against a few well-integrated sources with scoped questions, behind well-built servers (that part is often the gotcha), raw MCP works today. Don’t over-engineer it. But everyone is headed toward connecting more of the environment so agents can see across it, and the more sources you connect and the broader the questions you ask, the more likely the agent answers from a subset and reports done. Run the source-completeness test above before you trust any of this, mine included. If your number comes back low, you have three moves: connect less and scope the questions harder, wait for prompting and protocol improvements, which are real levers at small scale and undemonstrated at estate scale, or put a normalized layer between the agent and your sources, built yourself or bought. Buying it is the business I am in, so weigh that recommendation with everything else here.
What this test cannot tell you
Three limits worth stating plainly. I varied the number of sources, never the volume inside each one; my read is that volume compounds the effect, because a normalized layer that selects fields before anything reaches the model should matter more as single sources return more, but that is reasoning rather than evidence, and it is the next thing I want to test. The mock firewall returns empty rather than throwing a type error, so read the opening story as an illustration of the failure class rather than a measured production failure. And nobody measures source completeness for humans either. No study says what share of the relevant facts an analyst surfaces in a cross-source investigation, though the conditions are on record: Tines found 53 percent of analysts pivoting across 11 to 30 products and spending about 64 percent of their time gathering context rather than analyzing it. We are holding agents to a completeness standard nobody applies to the people working the same queue.
Onward
The last two years of this field were about making the model smarter. I think the next two are about what the model can see, and whether you can tell when it is not seeing enough. A smarter agent that quietly skips half your data is a faster way to be confidently wrong. Take what is useful. Leave the rest.
APPENDIX:
Methodology and full results
Everything above comes from one of two bodies of evidence: a controlled simulation study and a validation probe against Query’s live product. They earn trust differently, so they are documented separately here.
Study design
Environment. Mock security data sources scaled from 5 to 15 to 40, plus two fixed five-source testbeds. One testbed uses uniform generated sources; the other mocks five real products and preserves their native quirks (mismatched identities, inconsistent timestamp formats, divergent field names for the same concept). The agent answered a fixed set of investigation questions two ways: through each source’s own MCP tools (the raw arm) and through a single normalized engine built for this test, not Query’s production platform (the test mesh arm). Same model, same questions, same underlying records in both arms. The raw arm was built to be competent, not a strawman: every source shipped its own MCP server with vendor-native tool and field names, native time formats, and search and lookup tools that accept filters and time bounds. The five-product testbed mocked Splunk, CrowdStrike Falcon, Okta, a threat-intelligence platform, and a CMDB. In several five-source tasks those tools’ whole-record defaults beat the mesh’s field-scoped precision, which is where the small-scale results came from.
Controls.
- Metrics were written down before any run, so goalposts could not move after seeing numbers.
- Every answer was scored against a known fact set by a separate judge model that never saw which arm produced it. A second judge agreed with the first 95 to 98.5 percent of the time across the two studies’ cross-checks. A third, stricter judge shaved a few points off both arms; the gap between arms survived untouched.
- Results are reported as estimates with confidence intervals, not pass/fail against a self-set bar.
- The scaling study ran four trials per cell. Every number is a mean across runs; treat the last digit as noise. Completeness and utilization figures come from the estate-wide questions; cost and time figures from the full question set.
Scoping. All runs took place in July 2026 against then-current Anthropic frontier models plus one cheaper model (Sonnet as the agent; Haiku as the cheaper model). Tool loading and the protocols around it keep getting more efficient. The effect measured here sits underneath the protocol, in context and attention, so better plumbing should move where the line falls rather than whether there is one.
Five sources, two testbeds
Same harness, same model, on the three questions both testbeds ask word for word:
| At five sources, same three questions | Raw per-source MCP | Test mesh |
| Generated sources, uniform shapes | 78% | 73% |
| Mocks of real products, native quirks intact | 80% | 91% |
Raw scored about the same on both testbeds. The test mesh moved with how much genuine mess there was to normalize.
Scaling study
Completeness on estate-wide questions (share of known answer found), four trials per cell:
| Sources | Raw per-source MCP | Test mesh |
| 5 | 78% | 63% |
| 15 | 59% | 74% |
| 40 | 53% | 70% |
The advantage swung 32 points between 5 and 40 sources (bootstrap 95 percent CI: 22 to 40 points). On the broadest single question, raw fell 75% → 28% → 20% while the test mesh climbed 67% → 75% → 83%.
Attention and utilization
From 5 to 40 sources the environment grew 8x. Raw queries per investigation rose from 11.6 to 25.1 (about 2x) with roughly 40 percent more tokens spent.
- Source utilization (share of relevant sources queried): raw 94% at 5 sources, 75% at 40. The test mesh never dropped below 95%, at lower query counts (15.2 calls at 40 sources vs. raw’s 25.1).
- Query depth: at 5 sources, one in four consulted sources received exactly one query with no follow-up. At 40 sources, two in three. First-query quality stayed constant across scales; follow-up questioning is what disappeared.
- Economizing check: raw’s cost per investigation rose from 21.6 cents (5 sources) to 48.7 cents (40 sources), up 126 percent, ruling out cost-conserving behavior as the explanation. The test mesh ran on the identical harness and model under the same cost pressure and held utilization.
Prompting intervention
The same three-sentence instruction (run a broad discovery pass, check every source that could change the answer, then conclude) was appended to both arms:
- At 5 sources: raw improved to 92 percent; the test mesh moved to 69. Prompting is a real lever at small scale and helped raw most.
- At 40 sources (broadest question, four trials per arm): raw improved from 20 percent to 33; the test mesh gave back about five points, from 83 to near 78. The instruction taxed both arms; it lifted only the arm that was failing, and not far. The raw agent, explicitly told to check every source, made about seven queries against forty sources and stopped, well short of its turn limit.
Cost and speed
| 5 sources | 40 sources | |
| Raw per-source MCP | 21.6¢ per investigation | 48.7¢ |
| Test mesh | 23.5¢ | 33.0¢ |
Wall-clock time favored the test mesh at every scale measured.
Cheaper-model comparison (5 sources only, cross-source questions): the test mesh lifted Haiku about 10 points. At five sources, the one scale I ran this comparison, Sonnet stayed put: 93.3 through the mesh against 91.7 raw, a tie at the scale the whole study says raw handles fine. Haiku through the mesh was 16 percent cheaper to run and 25 percent faster than Haiku on raw tools. Run at one scale only; hold it loosely. The pattern: the frontier model had little to gain at a scale it already handled; the cheap model gained everywhere.
The firewall task
The raw arm scored 50 percent on the firewall task at every scale run; the test mesh scored 100 percent, because it converts time to a common form before the query reaches the source. Note: the mock firewall returns empty rather than throwing a type error, so this is an illustration of the silent-failure class, not a measured production failure. A wrong yes gets caught downstream; a confident no ends the investigation.
Prompt injection probe
132 trials across four payload variants and six placements (three trials per payload-placement cell in each arm; 132 valid runs after excluding failed starts) of planted hostile instructions in log data. Raw per-source tools ingested the payload on 100 percent of runs wherever the tools returned the record, because raw tools return whole records, prose and all, with no filtering mechanism between record and model. The test mesh ingested the payload only when a queried field carried it. The architectural difference: a normalized layer queries specific fields and returns one common data set, creating a surface where hostile content can be inspected, flagged, and filtered before it reaches the model. How Query’s production mesh handles this is a topic for a separate piece.
Live product validation probe
Same categories of questions against Query’s live product: 62 real connected sources (at the median customer’s scale) in a demo environment with known ground truth. This is a validation probe, not a production study. Two of the four investigations ran three trials each; the other two ran once.
| Initial agent investigation pass | Completeness | Queries | Time |
| Everywhere an attacker IP appears | 79% | ~30 | ~7 min |
| An IP’s behavior and intent | 93% | ~30 | ~7 min |
| A user’s risk across the whole estate | 83% | ~30 | ~7 min |
| A host’s blast radius, first pass | 64% | ~40 | ~9 min |
Across the three sweep investigations, the trial-weighted average (five graded trials) is 82 percent at 62 real sources, computed from unrounded per-trial grades; recomputing from the rounded means in the table gives 82.6, so treat the last digit accordingly. The test mesh’s estate-wide average at 40 mock sources was 70 percent; its single broadest question scored 83.
Notes on the table:
- The 64 percent floor. The hardest investigation ties a host compromise to cloud-identity abuse two systems apart, and showed more variance than the others. None of Query’s investigation method was in the loop: no schema-first query building, no pass to run loose ends to ground. A general-purpose agent got roughly two-thirds of the hardest cross-domain join on its first attempt. The method layer exists to lift that floor and was not measured in this work, so no number is claimed for it.
- Miss attribution. Transcript review showed the mesh had reached the relevant records on all four investigations. Every point lost was lost in querying and reasoning over returned data, not in reach. This is the basis for the human-review design in Query’s agent runs: every query and conclusion is handed back as a checkable artifact.
- Every number is a naive agent’s first pass on the data layer alone. Floors, not ceilings.
Estate context
- Across Query’s customer base, the median deployment connects about 60 sources. A single integration multiplies: one AWS S3 connection can front hundreds of buckets.
- In roughly 50 conversations with security teams this year, no team described a single point of centralization. Typical pivot count: 12 to 20 sources; sometimes over 100. Microsoft-commissioned research fielded in 2025 found only 59 percent of security tools push data to the SIEM automatically.
- Customer economics. A Fortune 250 energy company cut SIEM ingestion from 10 TB/day to 4 TB/day deliberately; the other 6 TB remains investigation-relevant, stored on cheaper platforms. A second customer in a smaller environment cut Splunk ingest by about a third (425 GB/day, roughly a quarter off the licensing bill) while quadrupling the data its analysts could reach outside Splunk. Both stayed on their SIEM.
Operational surface of per-source MCP at estate scale
Running per-source MCP across a real estate means owning, per source, indefinitely: what data each server can and cannot reach; schema, API, and field-semantics changes; performance, throttling, and downtime behavior; response-shape consistency across sources; credential and auth lifecycle across vendors; and each vendor’s rate limits, pagination schemes, and time-format quirks, with minimal visibility into any of it. A mesh takes over that surface: mappings, connection health, time normalization, and auth lifecycle watched in one place. The maintenance never finishes under either approach; the question is whose team carries it.
Limits
- Source count varied; per-source volume did not. A team with five sources holding terabytes each has a different problem from a team with forty small ones. Expectation (unmeasured): volume compounds with source count, because field-level selection before the context window should matter more as single sources return more.
- The live run is a probe on Query’s own demo environment, not a customer estate.
- The cheap-model comparison ran at one scale. All runs used a single model family; no cross-family replication was attempted.
- No human baseline exists for source completeness. The human side of any comparison is inferred from documented conditions, not observed: Vectra’s field surveys put the share of daily alerts never worked at about two-thirds; Tines found 53 percent of analysts pivoting across 11 to 30 products and spending about 64 percent of their time gathering context rather than analyzing it. Cognitive science calls the result premature closure: stopping at the first plausible explanation.
- One hand-authored mapping in the test mesh quietly dropped a field during this work and was caught only by the stricter judge. Normalized layers inherit the quality of their mappings; the failure mode is real in both architectures and differs in where it can be detected.
