Connectivity is solved. Understanding and precision are not. Why retrieval systems need a semantic foundation.

I met up with a colleague at a conference recently. She was newer to the world of semantics and mostly excited by what she was hearing, with one note: semantic experts, she said, seemed to treat GraphRAG like a dirty word. They waved it off, as though the world of Retrieval-Augmented Generation (RAG) had come and gone with relatively disappointing results.

Elaborating further, she explained how some sort of retrieval mechanism is a necessity if you want to interact with LLMs at scale. Which raises the obvious question: why dismiss the most common retrieval mechanism outright? In reality, what the experts were dismissing wasn’t really the idea of RAG itself. It was retrieval systems that lack enough defined semantic grounding to consistently achieve the desired results.

 

The Modern Problem: MCPs

This argument applies anywhere a retrieval mechanism has to choose what to surface, including RAG implementations, but it is most pointed right now in the world of the Model Context Protocol (MCP). MCP has quickly become the de facto standard by which AI agents connect to enterprise tools and data sources, and retrieval is one of the most common things that happens through that connection. It has solved how AI systems make those connections, so the hype is deserved. However, the harder question, the one that determines whether any of this delivers what enterprises are paying for, has been largely avoided.

It is worth noticing what the acronym promises. MCP is the Model Context Protocol, but what it standardizes is the connection to the context, not the context itself. That is not a flaw in the specification: a protocol can move data, but it cannot tell you what the data means.

MCP is the latest case study that confirms an increasingly evident reality in the world of data connectivity: precision in retrieval comes from semantic grounding, and the absence of it is often what produces common failure modes. Treating any retrieval system as only a connectivity issue and ignoring the semantics problem is like welding a perfect pipeline only to run untreated water through it.

 

Connectivity Does Not Guarantee Precision

An AI system connected to your enterprise data can reach almost anything, but that is not the same as retrieving the right thing. The difference is what determines whether a retrieval-backed agent is producing actually useful answers or generating confident noise.

Surface similarity captures statistical patterns of meaning, but it does not formally distinguish related-but-distinct concepts. A retrieval layer that matches queries to data based on textual or vector similarity is doing real work, and modern embeddings encode valuable semantic relationships. However, they don’t disambiguate concepts deterministically. That kind of work still requires a human touch. If your domain uses “adverse event” and “safety signal” to refer to related but distinct things, embedding-based retrieval will often conflate them. If “account” means different things depending on whether it originates from your CRM, your ERP, or your contracts system, a model without a shared vocabulary has no way of telling them apart.

 

The Requirements for Precision

What is most often seen with industry-standard MCP architectures tends to be one tool per system: a Salesforce tool, a NetSuite tool, a contracts database tool, and so on. This is a common and reasonable pattern because it mirrors how enterprise systems are actually structured, and it keeps the toolset manageable. The question of how to expose retrieval is solved. The harder question is what happens before a tool gets called: how the agent decides which tool to invoke and what to ask it for. The solution: semantic resolution via a semantic layer.

The semantic layer sits between the agent and the MCP-connected tools. When a user asks a question, the agent doesn’t pass that question directly to a tool. It first passes through the semantic layer, where the question is resolved against a defined model of the business: what concepts are being asked about, which systems are sources of record for those concepts, and what typed parameters should be used to retrieve them. Only then does the agent call a tool, and it calls the right one with inputs defined by the business’s domain experts.

 

The visual above shows where the semantic layer sits and what it does between a question and the MCP tools that answer it.

 

A Worked Example (Adapted From a Real-World Case Study)

Consider a question an account manager might ask an agent before a meeting: “What are our open commitments to Acme Corp for Project Alpha?”

In a typical enterprise, this query touches three plausible systems. The contracts database is the system of record for legal commitments: SOWs, MSAs, amendments. The project management system tracks active deliverables and milestones. The CRM contains opportunities, calls, and the relationship history. An MCP implementation that wraps each of these systems exposes one tool per system.

Without a semantic layer, the agent has to make tool-selection decisions on the basis of tool descriptions written in each system’s native vocabulary. “Commitments” might match the contracts tool, the projects tool, the CRM tool, or all three at once. The agent invokes multiple tools to be safe, each returns a wide swath of loosely-typed records, and the model is left to interpret what came back. The response to the user is correspondingly imprecise.

With a semantic layer, “open commitments” resolves to a known concept that lives in the contracts system, “Acme Corp” resolves to a known counterparty, and “Project Alpha” resolves to a specific project. The agent calls only the contracts tool, with those resolved concepts handed in as typed parameters. The data that comes back is precise, traceable, and grounded.

 

The visual above illustrates the same question taking the two different paths, and where the costly work happens in each.

 

The Cost of Allowing Imprecision

The Acme query in the previous section is illustrative, but the cost is real. Without a semantic layer, an agent fans out to multiple tools, processes large payloads of loosely-typed records, and often performs follow-up reasoning passes to filter the results. With a semantic layer, it makes fewer, more confident calls and returns precise answers in fewer passes, possibly only one. With a single query, the cost of imprecision is invisible. For an enterprise running thousands of agentic queries per day, it’s a line item.

There is, of course, an offsetting investment: building and maintaining a formal semantic layer takes real work. SME time, governance, ongoing curation. But the comparison is not “free vs. paid.” Engineering-time interventions are a continuous spend that scales with usage. Semantic investment is front-loaded and compounds in the other direction: every new system added to the retrieval layer is cheaper and easier to integrate than the last.

 

Build the Foundation Early

Protocols define how things connect. Semantic models define what things mean. These are not substitutes for each other, and they cannot be sequenced in reverse. Enterprises that treat MCP as a complete strategy, rather than an interoperability layer that still requires a semantic foundation, will pay for the imprecision in tokens, latency, and engineering time, every day.

If I were to return to my colleague back at the conference, I’d tell her that her worry was misplaced. The experts are not dismissing RAG, GraphRAG, or retrieval mechanisms themselves, even if they think they are. What they truly dislike is retrieval that produces confident noise without any data to back it up. So the question was never which retrieval mechanism to bet on, it was whether the mechanism you use sits on a system of record it can cite, with real business data to validate every confident reply.

Your AI can already reach everything. Whether it can resolve anything depends on what sits under the connection. If meaning isn’t modeled, it’s inferred, or, worse, guessed. Give your agents tools that already speak your business language: tools that retrieve the right thing on the first pass, return answers you can trace, and get cheaper to scale with every system you add. Reach is the easy half, and it’s the half already solved. Resolution is the half you build, and it’s the part that stands between a confident answer and a correct one.

Looking to build a semantic foundation and refine your enterprise MCP capabilities? Contact us to learn more.