Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions

Build a continuous integration and continuous delivery (CI/CD) quality gate that deploys an agent with role-based MCP tools, evaluates it, and blocks PRs when evaluation scores drop.

You shipped an AI agent on Amazon Bedrock AgentCore runtime. It calls tools through an MCP server protected by OAuth. Now you want CI to tell you when a code change makes its performance worse before it reaches production.

This post walks through a GitHub Actions pipeline that deploys an agent to AgentCore runtime and evaluates the agent with evaluation prompts using the AgentCore Evaluate API. If the agent regresses, the PR fails.

We’ll cover the full stack: a Strands agent that connects to an MCP server with role-based access control, a shared Cognito pool serving both machine-to-machine (M2M) and user-scoped auth flows, CDK infrastructure-as-code, and a unified evaluation script. The complete reference implementation is available in the accompanying repository.

Before diving in, here’s a quick primer on the building blocks. Skip ahead if you’re already familiar with them.

Why this matters: Without automated evaluation, agent quality is subjective. A developer changes a system prompt. The agent starts giving worse answers, and nobody notices until users complain. A quality gate catches this at PR time before it reaches production.

Here’s the scenario. You have an agent deployed on AgentCore runtime. It calls tools through an MCP server where some tools are public. Others are restricted by user role. Every time someone changes the system prompt, swaps a model, or updates tool configurations, you want to know: did the agent get better or worse?

Manual testing doesn’t scale. You need automated evaluation in CI. That means automatically deploying the agent in a dev environment, invoking it with representative prompts, scoring the responses, and blocking the merge if quality drops.

The complication: your MCP server uses OAuth with role-based access control. CI pipelines don’t have user context. How do you authenticate a headless pipeline against an OAuth-protected agent that forwards tokens to an MCP server expecting user roles?

AgentCore Evaluations: Where it fits

AgentCore Evaluations is the quality measurement layer in the Amazon Bedrock AgentCore platform. It sits alongside AgentCore runtime, which hosts your agent, and AgentCore Observability, a capability of Amazon Bedrock AgentCore that captures traces, completing the build → deploy → observe → evaluate lifecycle.

The service scores agent interactions using LLM-as-a-judge by default, with an option for code-based evaluation via AWS Lambda. It operates on OpenTelemetry traces, the same traces your agent already emits through AgentCore Observability. For on-demand evaluation, you provide span data directly in the API call; online and batch evaluation read from CloudWatch.

Three evaluation modes cover different stages:

Evaluators come in four categories:

The Evaluate API accepts sessionSpans (OpenTelemetry trace data from CloudWatch) and returns structured scores. Each evaluate() call must contain spans from a single session only. Mixing sessions causes a ValidationException.

The API also accepts optional ground truth through evaluationReferenceInputs. You can provide an expectedResponse (used by Correctness), assertions (used by GoalSuccessRate), or an expectedTrajectory (used by trajectory evaluators). Traces without ground truth fall back to ground-truth-free evaluation, so you only need to provide it for the turns you care about.

The pipeline deploys two AgentCore runtimes behind a shared Cognito user pool. One AgentCore runtime for the Strands agent and one for the MCP server:

A single Cognito user pool serves two auth flows:

The GitHub Actions pipeline deploys the agent stack to a dev environment, retrieves a JWT token from the provisioned Cognito instance to authenticate API calls, invokes the agent with an evaluation dataset, and analyzes the generated traces in Amazon CloudWatch Logs to assess performance against defined thresholds. It automatically approves or blocks the PR based on whether the overall score meets the acceptance criteria.

Handling OAuth-protected MCP servers in CI

When your agent calls an MCP server protected by OAuth, CI pipelines face a challenge: they don’t have user context. The MCP server expects a JWT with role claims, but a headless CI runner can’t complete an interactive OAuth consent flow.

There are three approaches for evaluating the agent, each with different trade-offs:

Decouple evaluation from live MCP calls entirely. A staging pipeline runs the agent with representative prompts, captures traces, and commits them as JSON fixtures. PR-time, CI evaluates those stored traces. No live invocation is needed.

The Evaluate API doesn’t need a live agent. It scores OpenTelemetry spans you provide. Your CI pipeline becomes deterministic (check existing traces) and you sidestep the OAuth problem completely.

Trade-off: You’re evaluating the staging deployment’s behavior, not the code in the current PR. The accompanying repo includes scripts/evaluate_stored_traces.py and sample fixtures in fixtures/ to get started with this approach.

Create a dedicated test user in your identity provider. Complete the OAuth consent flow once (interactively), cache the refresh token in AWS Secrets Manager. CI uses this token to invoke the agent as that test user.

Trade-off: Refresh tokens expire. You need a rotation mechanism or periodic manual re-consent.

Configure your MCP servers to support both M2M and user-scoped grant types. CI uses M2M tokens while interactive users go through the standard OAuth consent flow.

The MCP server middleware distinguishes between token types: M2M tokens contain scopes but no roles, so role checks are bypassed, and all tools are accessible. User tokens carry custom:roles claims, so tool-level access control is enforced. This bypass is secure because M2M tokens require a client secret that’s never exposed to end users. Only CI pipelines and the agent runtime can obtain these tokens, preventing untrusted callers from acquiring role-less tokens.

Trade-off: M2M tokens bypass role checks by design. If you need CI to test role enforcement specifically, use Approach B.

Use the decision tree below to find out which approach suits your use case:

Tip: Start with Approach A to get a quality gate running quickly. Graduate to Approach C (this post) for full end-to-end CI that tests the actual PR’s code changes.

Note: The Evaluation class from bedrock-agentcore-starter-toolkit handles trace collection from CloudWatch and scoring automatically, so you don’t need to manually query log groups or call the raw Evaluate API.

For Approach C, the MCP server uses three layers to support both M2M and user-scoped tokens. This is the key pattern that makes CI evaluation work alongside production role enforcement.

Layer 1 JWT validation (AgentCore): The platform validates signature, issuer, audience, and expiry before the request reaches your code. No implementation needed. AgentCore handles this through the Custom JWT Authorizer.

Layer 2 Header passthrough: request_header_allowlist=["Authorization"] on both runtimes makes sure the JWT reaches the agent and MCP containers. AgentCore forwards the caller’s Authorization header to your container unchanged.

Layer 3 Role-based tool access (AuthMiddleware): A FastMCP native middleware that reads the JWT through fastmcp.server.dependencies.get_http_headers(), decodes claims using PyJWT, and enforces custom:roles against the tool meta. M2M tokens (scopes but no roles) get full access. User tokens need the right role.

The middleware is added directly to the FastMCP server instance:

The CDK stack deploys everything in one command: Cognito pool, both runtimes, IAM roles, and pre-created test users. See infrastructure/stack.py for the full implementation.

Key resources created by the stack include:

The GitHub Actions workflow deploys the CDK stack in a dev environment and uses the created resources to run evaluation.

The unified evaluation script (scripts/agentcore_eval.py) handles the full pipeline: get token, wait for runtime, invoke agent, wait for traces, run evaluations, and gate on threshold.

The token acquisition uses the standard client_credentials grant.

Agent invocation uses HTTPS with a Bearer token (not boto3):

The script uses bedrock-agentcore-starter-toolkit’s Evaluation class to run evaluations, which handles trace collection from CloudWatch automatically:

Evaluation prompts cover the agent’s full tool surface including built-in tools, public MCP tools, and role-gated MCP tools:

The workflow runs on every PR to main that touches agent code, MCP server, infrastructure, or scripts. It deploys the CDK stack, invokes the agent, runs evaluations, posts results as a PR comment, and tears down the stack. See the full workflow for the complete implementation.

Warning: Runtimes stay in CREATING for a few minutes after CDK deploy returns, and invoking one before it’s READY fails with 424 Failed Dependency. The workflow polls get_agent_runtime (via the bedrock-agentcore-control client) until both are READY, then warms up the MCP server before evaluating.

Configure the following components to activate automated agent evaluation in your CI/CD pipeline.

Create a role with trust policy for your repo and permissions for CDK, Amazon Bedrock AgentCore, Amazon Cognito, Amazon Elastic Container Registry (Amazon ECR), and Amazon Bedrock. See the accompanying repo README for the full policy.

Everything else is read from CDK outputs at runtime.

AgentCore provides multiple built-in evaluators organized by what they assess:

This post uses four evaluators: GoalSuccessRate, Correctness, ToolSelectionAccuracy, and ToolParameterAccuracy. The tool-call evaluators are particularly relevant for agents with MCP tools. They verify the agent picks the right tool and passes correct parameters.

Tip: Start with four to five evaluators for CI. Add trajectory evaluators for tool-calling agents and safety evaluators (Harmfulness, Stereotyping, Refusal) for customer-facing agents. Use code-based evaluators for deterministic checks. Use the full set of periodic deep evaluations.

Testing the pipeline: Fail, fix, pass

A reliable way to build confidence in a quality gate is to watch it catch a real regression.

Deliberate failure: Change the agent’s system prompt to something unhelpful:

Push to a feature branch and open a PR. The pipeline runs and you’ll see:

Note: ToolSelectionAccuracy might still pass because the agent may select the right tool even with a bad system prompt. The evaluators measure different dimensions independently.

Fix and pass: Restore a proper system prompt, push again. The pipeline re-runs:

Warning: LLM-as-judge scores have inherent variance. As a result, the same prompt evaluated twice may produce slightly different scores. Set your threshold with some margin below your target reliability to account for this variance.

We ran this pipeline end-to-end. Here are the gotchas. Save yourself the debugging time.

Adapting for Microsoft Entra ID

This post uses Amazon Cognito, but the architecture is identity-provider-agnostic. If your organization uses Microsoft Entra ID (formerly Azure AD), the same pipeline works with targeted changes:

Everything else stays the same. The deploy script’s authorizerConfiguration.customJWTAuthorizer structure is identical, the evaluation script doesn’t touch authentication (it uses IAM through boto3), and the GitHub Actions workflow structure is unchanged.

Note: Despite these limitations, automated evaluation is strictly better than no evaluation. Even imperfect quality gates catch obvious regressions that manual review misses.

The accompanying repo provisions two AgentCore runtimes, a Cognito user pool, IAM roles, and pre-created users, so tear everything down when you’re finished to stop incurring cost. A single command removes it all:

If you deployed with npx rather than a global CDK CLI, run npx aws-cdk@2 destroy --force instead. In the dev stack, cdk destroy removes both runtimes, the Cognito pool and its app clients, the pre-created users, the IAM roles, and the M2M client secret in AWS Secrets Manager. The secret’s removal policy is set to destroy, so repeated deploys and teardowns stay clean. The CI workflow tears the same stack down automatically at the end of every run, because its CDK destroy step runs with if: always(). You only need this command for stacks you deploy yourself while following along.

The accompanying repository includes the complete implementation: CDK infrastructure for Cognito and both runtimes, an agent with MCP client and token forwarding, an MCP server with role-based access control, a unified evaluation script, a GitHub Actions workflow, and two walkthrough scripts. Use scripts/deploy_and_test_rbac.py for deployment and role-based access testing, and scripts/evaluation_pipeline.py for the evaluation pipeline.

To extend the project, you can start with Approach A by running python3 scripts/evaluate_stored_traces.py against the bundled fixtures without any deployment. From there, try adding a new role-gated tool to the MCP server (see mcp-server/README.md) or creating a custom evaluator with your own LLM-as-judge prompt. Maybe try adjusting EVAL_THRESHOLD per environment (for example, 0.7 for dev, 0.8 for staging, 0.9 for prod). You can also set up online evaluation for continuous production monitoring or compare on-demand to online evaluation for your use case.