Agentic systems are stochastic, context-dependent, and policy-bounded. Conventional QA—unit tests, static prompts, or scalar “LLM-as-a-judge” scores—fails to expose multi-turn vulnerabilities and provides weak audit trails. Developer teams need protocol-accurate conversations, explicit policy checks, and machine-readable evidence that can gate releases with confidence.
Qualifire AI has open-sourced Rogue, a Python framework that evaluates AI agents over the Agent-to-Agent (A2A)protocol. Rogue converts business policies into executable scenarios, drives multi-turn interactions against a target agent, and outputs deterministic reports suitable for CI/CD and compliance reviews.
Quick Start
Prerequisites
uvx – If not installed, follow uv installation guidePython 3.10+An API key for an LLM provider (e.g., OpenAI, Google, Anthropic).
Installation
Option 1: Quick Install (Recommended)
Use our automated install script to get up and running quickly:
(c) OPTIONALLY: Set up your environment variables: Create a .env file in the root directory and add your API keys. Rogue uses LiteLLM, so you can set keys for various providers.
Rogue operates on a client-server architecture where the core evaluation logic runs in a backend server, and various clients connect to it for different interfaces.
Default Behavior
When you run uvx rogue-ai without any mode specified, it:
Starts the Rogue server in the backgroundLaunches the TUI (Terminal User Interface) client
Default (Server + TUI): uvx rogue-ai – Starts server in background + TUI clientServer: uvx rogue-ai server – Runs only the backend serverTUI: uvx rogue-ai tui – Runs only the TUI client (requires server running)Web UI: uvx rogue-ai ui – Runs only the Gradio web interface client (requires server running)CLI: uvx rogue-ai cli – Runs non-interactive command-line evaluation (requires server running, ideal for CI/CD)
–host HOST – Host to run the server on (default: 127.0.0.1 or HOST env var)–port PORT – Port to run the server on (default: 8000 or PORT env var)–debug – Enable debug logging
–rogue-server-url URL – Rogue server URL (default: http://localhost:8000)–port PORT – Port to run the UI on–workdir WORKDIR – Working directory (default: ./.rogue)–debug – Enable debug logging
Example: Testing the T-Shirt Store Agent
This repository includes a simple example agent that sells T-shirts. You can use it to see Rogue in action.
(c) Run the evaluation and watch Rogue test the T-Shirt agent’s policies!
You can use either the TUI (uvx rogue-ai) or Web UI (uvx rogue-ai ui) mode.
Where Rogue Fits: Practical Use Cases
Safety & Compliance Hardening: Validate PII/PHI handling, refusal behavior, secret-leak prevention, and regulated-domain policies with transcript-anchored evidence.E-Commerce & Support Agents: Enforce OTP-gated discounts, refund rules, SLA-aware escalation, and tool-use correctness (order lookup, ticketing) under adversarial and failure conditions.Developer/DevOps Agents: Assess code-mod and CLI copilots for workspace confinement, rollback semantics, rate-limit/backoff behavior, and unsafe command prevention.Multi-Agent Systems: Verify plannerexecutor contracts, capability negotiation, and schema conformance over A2A; evaluate interoperability across heterogeneous frameworks.Regression & Drift Monitoring: Nightly suites against new model versions or prompt changes; detect behavioral drift and enforce policy-critical pass criteria before release.
What Exactly Is Rogue—and Why Should Agent Dev Teams Care?
Rogue is an end-to-end testing framework designed to evaluate the performance, compliance, and reliability of AI agents. Rogue synthesizes business context and risk into structured tests with clear objectives, tactics and success criteria. The EvaluatorAgent runs protocol correct conversations in fast single turn or deep multi turn adversarial modes. Bring your own model, or let Rogue use Qualifire’s bespoke SLM judges to drive the tests. Streaming observability and deterministic artifacts: live transcripts,pass/fail verdicts, rationales tied to transcript spans, timing and model/version lineage.
Under the Hood: How Rogue Is Built
Rogue operates on a client-server architecture:
Rogue Server: Contains the core evaluation logicClient Interfaces: Multiple interfaces that connect to the server:
TUI (Terminal UI): Modern terminal interface built with Go and Bubble TeaWeb UI: Gradio-based web interfaceCLI: Command-line interface for automated evaluation and CI/CD
This architecture allows for flexible deployment and usage patterns, where the server can run independently and multiple clients can connect to it simultaneously.
Summary
Rogue helps developer teams test agent behavior the way it actually runs in production. It turns written policies into concrete scenarios, exercises those scenarios over A2A, and records what happened with transcripts you can audit. The result is a clear, repeatable signal you can use in CI/CD to catch policy breaks and regressions before they ship.
