Why AI Agents Need Context Beyond RAG and Semantic Search

An agent can retrieve every relevant record, generate valid SQL and still produce a misleading forecast. In an example from Pinecone’s webinar, a one-day partner promotion on September 8 pushed projected Q4 mobile growth to +16%. Excluding the promotion from the forecast while keeping it in reported totals changed the result to +6%. The figures are illustrative; the underlying data and query were not the problem. The missing piece was context about how to interpret the data.

That distinction framed a November 4 webinar by Jörg Schad, Pinecone’s VP of Engineering. His central point: retrieval tells an agent what information is available, but a separate layer may need to decide which sources are current, trusted and appropriate for the task. That matters more when software acts on answers without a person checking each result.

This recap is adapted from From Data to Vectors to Context. The full webinar replay is available on YouTube.

Each retrieval method leaves a different gap

Keyword search uses an inverted index and relevance ranking to find exact matches at scale. The underlying ideas are decades old: inverse document frequency dates to 1972, and BM25 to 1994. But exact matching can miss different wording for the same idea. A search for “cancel,” for example, may not find a record that says “terminated subscription.”

Semantic search addresses that vocabulary gap. Techniques including word2vec, HNSW approximate nearest-neighbor search and BERT helped systems retrieve passages based on related meaning. Yet similarity alone can’t establish authority. A question about how a feature works might return both a design document from three years ago and a scan of today’s code. Both resemble the query; only one describes what runs now.

Retrieval-augmented generation, introduced by Lewis and colleagues in 2020, gives a model access to a knowledge base when it answers. That approach helps ground responses in external material, but retrieval by itself doesn’t verify that the returned source is the right one—or that the model used it correctly.

Recall measures coverage, not whether the answer is sound

Recall at K measures how many relevant documents appear among the top K results. It’s useful for checking whether retrieval surfaced the information. It doesn’t capture every condition that determines whether an answer is correct.

Suppose a current policy has replaced an older version. A search can return both, and a recall score may count them as relevant. An answer based on the outdated text can still be wrong. Access restrictions create a similar problem: a document might match a query but be out of bounds for the person or agent asking it.

Some gaps come from rules that similarity scores can’t encode. After a merger, one company might use a January-to-December fiscal year and another a February-to-January year. Search can return the records needed to answer a question spanning both businesses, but it can’t determine how the fiscal periods should be aligned. Adjusting embeddings or reranking won’t supply that missing rule.

The webinar’s examples point to the same division of work. The index can return the promotion spike, old and new policy versions, or documents from both companies. Context tells the agent how to treat those results: exclude the promotion from a forecast, select the current policy, or map the two calendars.

More input can make an agent less reliable

A person may notice when a forecast looks implausible and check the source data. An autonomous agent may act on its answer without that pause. Agents can also make retrieval calls at a volume that makes manual review impractical, while lacking the context to spot stale or unsuitable results on their own.

Loading every potentially relevant document into a model isn’t a simple fix. Chroma’s context rot report evaluated 18 LLMs and found that performance became less reliable as input length increased, including on deliberately simple tasks. On LongMemEval, focused prompts averaging about 300 tokens outperformed full prompts averaging around 113,000 tokens. The practical implication is that agents need targeted context, not simply a larger pile of material.

A context layer adds rules around search

Search still has a central role, and different tasks may call for different retrieval methods. Pinecone describes using dense vectors, sparse vectors and full-text search with BM25 ranking in one index. Exact keyword matching remains useful where wording matters; hybrid search can combine methods.

Above retrieval, a task context can hold summaries, entities, rules, policies and other knowledge artifacts prepared before a query arrives. It identifies which information applies, then makes lexical, vector or hybrid searches as needed. In the fiscal-year example, the records come from search, while the calendar mapping comes from context. The same pattern can determine which policy version takes precedence or how to handle a promotion.

Pinecone built Pinecone Nexus as a knowledge engine for agents. Its curation process tracks processed sources with content hashes, so later runs can focus on changed material. Answers include citations pointing to the file, page and exact text span behind a claim. Context-level instructions can set how the model handles conflicting sources.

Where this approach fits

Nexus won’t suit every retrieval workload, according to Pinecone. Straightforward RAG may be enough when the top results reliably contain the answer, source selection involves few complex rules, and a team can manage grounding itself. The product is a better fit for material such as contracts and research, which changes on a slower schedule. Frequently changing operational records—such as individual purchases on a large online store—call for a live, dynamic view that the webinar says Nexus isn’t designed for today.

The useful diagnostic is simple: check whether retrieval already found the necessary documents. If it did, the remaining work may be deciding how the agent should use them. Before spending a sprint on embeddings or reranking, teams can look for missing rules about authority, freshness, permissions or task-specific interpretation. Pinecone’s Nexus documentation describes the product; its trial page offers a way to explore it.