Agentic retrieval for Amazon Bedrock Managed Knowledge Base

Your users ask multi-part, comparative, and exploratory questions that span PDFs, slides, tickets, transcripts, and web content. Classic single-shot retrieval breaks down on these questions. Answers miss context, support tickets escalate, and analysts waste hours re-running searches. Agentic retrieval for Amazon Bedrock Managed Knowledge Bases is designed for these questions.

Consider two questions an analyst might ask: “Compare our 2020 and 2023 strategy. What changed?” and “What are the three biggest risks across product lines?” Neither is a single-shot lookup. A multi-intent question has no single point in embedding space that represents it well, so top-k chunks come back as an average of competing sub-intents.

Agentic retrieval plans and iterates over retrieval and can generate a response in the same call. This post focuses on why classic retrieval falls short on multi-part questions, how the AgenticRetrieveStream API works (including request construction and trace parsing), and when to choose it over the standard Retrieve API.

Why single-shot retrieval falls short

A worked example makes the gap concrete. Suppose you have ingested 25 years of Amazon shareholder letters into a Managed Knowledge Base. Your first question is direct:

“What is the most important message in the documents?”

The standard Retrieve API returns five chunks by hybrid score. In our test corpus, the top result is high-scoring but low-value: it covers the Amazon logo color scheme. The following chunks cover hiring, builders, and differentiation. The retriever did its job and ranked by similarity to the query embedding. But “most important message” is vague, and the scorer has no signal for what matters.

Now escalate to a realistic question:

“Compare how Amazon talked about hiring, long-term investment, and customer obsession in 2020 vs 2023. Where did emphasis shift?”

Single-shot retrieval must now serve three intents across two time periods with one query vector. You get one of two outcomes: a scatter of loosely related chunks, or a tight cluster dominated by the strongest signal. Neither is what a human analyst would do.

A human would decompose the question. They would search hiring in 2020, then hiring in 2023, then investment philosophy in each year. They would read the results, notice gaps, refine the search, and run it again. With enough evidence in hand, they would synthesize.

Agentic retrieval automates this workflow as an API.

Teams have addressed this gap with custom agent frameworks built on the Retrieve API. They prompt a model to plan and call Retrieve API in a loop, manage sub-queries, deduplicate, and decide when to stop. It works, but it becomes custom plumbing in every application, each instance carrying its own latency, cost, and reliability risks.

Agentic retrieval is a new retrieval mode in Amazon Bedrock Managed Knowledge Bases, available through the AgenticRetrieveStream API. Instead of one similarity search, it runs a planning loop driven by a foundation model: the model decomposes your question, retrieves against each part, judges whether it has enough evidence, and iterates as needed. It generates a grounded response by default in the same call. Set generateResponse=False to return retrieval results without generating a response.

You see every step of that loop as an ordered stream of trace events. Each event carries a step and a status, so you can see what the planner did and why:

A note on deduplication and the trace. Deduplication applies only to the final result event. When multiple sub-queries retrieve the same chunk, that chunk appears only once in the final results. Trace events remain available for reviewing the individual planning and retrieval steps.

Two parameters do most of the tuning. The first is maxAgentIteration, the ceiling on planning-plus-retrieval iterations. Use 3 for a single KB and 4–5 for multi-KB or comparative queries, though the planner may exit early after its evaluation step finds the evidence sufficient. See Quotas and limits for defaults and maximums.

The second is the foundation model configuration. The foundationModelType field determines whether agentic retrieval uses the service-managed model or a custom model. When you set it to CUSTOM, provide the model through foundationModelConfiguration.

Note on pricing: Agentic retrieval is priced per call, and the model you choose for the orchestration sets the rate. With the managed model, you pay $4 per 1,000 agentic retrieval calls plus $1 per 1,000 underlying Retrieve API calls. If you choose a model available in Amazon Bedrock, you pay standard pass-through Bedrock pricing for that model plus the same $1 per 1,000 underlying Retrieve calls. See more details on the pricing page.

Now that you have seen how the planning loop works, we examine the AgenticRetrieveStream API in practice.

The most direct case: one KB, one query, let the planner decompose. You send the request, then consume the stream, which delivers trace events, generated-response chunks when response generation is enabled, and a final result event.

That is the whole loop: one request, no orchestration code, and a stream you read top to bottom. The trace events tell you what the planner did. The result event gives you the deduplicated chunks. In production, log these trace events to Amazon CloudWatch for cost attribution, latency budgeting, and debugging. Each iteration is one planner invocation plus its sub-query retrievals, so the number of iterations is what drives both cost and wall-clock latency.

Multi-KB routing is what agentic retrieval does that the other APIs cannot. Register up to five Managed KB retrievers in one request and attach a natural-language description to each. The planner reads those descriptions to route each sub-query to the most relevant KB.

You consume the stream the same way. The routing decision shows up in the result: each chunk carries a sourceRetriever ID, so you can see which KB answered which sub-query. The descriptions are the contract the planner routes on, so write each as a one-sentence pitch for its KB. Vague descriptions lead to vague routing.

Choosing the right retrieval API

Managed Knowledge Bases give you two query surfaces: Retrieve and AgenticRetrieveStream. You pick based on the shape of the question.

Use Retrieve for short, well-scoped questions where one similarity search is enough. It has no planner, the lowest cost per call, and the tightest latency, and you keep full control over how you generate an answer.

Use AgenticRetrieveStream when questions are multi-part, comparative, exploratory, or span multiple knowledge bases. It decomposes the question, retrieves per intent, evaluates sufficiency, and can generate a grounded response in the same call, or return chunks for you to synthesize with a model you control. It costs more and takes longer because it makes multiple model calls, but it finds evidence a single search can miss.

We evaluated agentic retrieval on MuSiQue (Trivedi et al., 2022), a public benchmark of multi-hop questions. On these questions it showed 20 percent absolute recall improvement over single-shot retrieval, with the largest gains on the hardest questions. Single-hop questions show smaller gains (under five points).

Each MuSiQue question is designed to require two, three, or four retrieval steps, and a human annotator identified the exact document chain needed to answer it. That chain is an oracle decomposition: the ideal retrieval plan to measure against.

Two patterns stand out. First, the harder the question, the bigger the gain. The more steps a question requires, the more a single search misses, and the more agentic retrieval recovers. Questions that need several steps simply cannot be answered in one pass.

Second, the system takes about as many steps as the question actually needs. The benchmark tells us the ideal number of steps for each question (the chain a human expert traced). Agentic retrieval’s own number of steps stays within roughly one of that ideal, so it searches enough to answer the question without wandering.

We demonstrate how agentic retrieval solves a 4-hop query in the MuSiQue dataset.

Query: “When did the explorer reach the headquarters location of the group Study in Brown’s record label is part of?”

We illustrate the retrieval process using the following t-SNE plot over the corpus representation space. It shows how single-step Retrieve results (dotted-red boundary) compare to multi-step agentic retrieve. In each hop, agentic retrieve breaks down the query. It arrives at the document containing the answer.

These practices reduce cost, improve latency, and keep agentic retrieval predictable in production.

Security, governance, and observability

Agentic retrieval inherits the access model of Managed Knowledge Bases. The calling AWS Identity and Access Management (IAM) principal needs four actions: bedrock:AgenticRetrieveStream to invoke the stream; bedrock:Retrieve and bedrock:GetDocumentContent, scoped to specific knowledge base ARNs, to retrieve passages and full document content; and bedrock:InvokeModelWithResponseStream to invoke the planner model and stream the generated response. For the full policy and setup, see the documentation.

Two details are specific to agentic retrieval and worth calling out. Guardrails are configured through policyConfiguration.bedrockGuardrailConfiguration, and only the BLOCK action is supported; the MASK action is not supported with agentic retrieval. Because the planner generates intermediate sub-queries, those sub-queries are as important to audit as the final results. Log the full trace to Amazon CloudWatch so you can review what the planner searched for, not just what it returned.

For observability, Managed Knowledge Bases emit CloudWatch metrics for invocations, client errors, server errors, and throttles. AgenticRetrieveStream also emits the TotalIterationCount metric. Managed knowledge base telemetry integrates with Amazon Bedrock AgentCore Observability. See the documentation for the full metric list, namespaces, and log routing options.

This post explained why single-shot retrieval falls short on multi-part questions, introduced the AgenticRetrieveStream API, and showed when to choose it over standard Retrieve. Agentic retrieval moves planning, iteration, and sufficiency evaluation into the Managed Knowledge Bases service, so you get a single, observable API with consistent security, logging, and cost behavior instead of custom orchestration in every application.

Start with Retrieve for direct lookups, and reach for AgenticRetrieveStream when users ask multi-part, comparative, or exploratory questions, or when evidence spans multiple knowledge bases. To build your first agentic knowledge base, follow the companion notebook on GitHub. For API details, see the AgenticRetrieveStream API reference, or explore the feature in the Amazon Bedrock console.