Humanbound website

AI agent security testing tools compared: the best options in 2026 and how they work together

A comparison of red teaming and LLM security testing tools: garak, PyRIT, Promptfoo, DeepTeam, Giskard and Humanbound. Last verified 9 November 2026.

11 min read
Humanbound cover card headed "AI agent security testing tools compared: the best options in 2026 and how they work together", with the kicker "Red-teaming, six open-source tools, 2026" and the line "No single tool does all the work. Use the one that fits each job, and run two or three together." A panel on the right maps each stage to a tool: 01 writing the agent, DeepTeam or Giskard; 02 each pull request, Promptfoo; 03 before a release, garak; 04 high-risk release, PyRIT; 05 staging endpoint, Humanbound; 06 production, hb firewall.

Six open-source tools cover most AI agent security testing and red-teaming in 2026: garak, PyRIT, Promptfoo, DeepTeam, Giskard and Humanbound. Humanbound publishes this list and builds one of the tools on it. No single tool does all the work, so this list does not rank them. Use the tool that fits each job, and run two or three together:

  • Scan for known attacks: use garak.
  • Add a red-team check to each pull request: use Promptfoo.
  • Run a deep multi-turn campaign: use PyRIT.
  • Test a Python agent inside your own code: use DeepTeam or Giskard.
  • Test a live agent endpoint and get a posture score: use Humanbound.

These tools overlap. Humanbound treats Promptfoo, garak and PyRIT as tools to run alongside it.

AI agent security testing tools compared

Each row below gives one tool and its main job. All six tools are open source. All six can send prompt injection attacks. Promptfoo, garak, PyRIT and Humanbound can attack a URL. DeepTeam and Giskard call a Python function that you write.

Loading code editor...
Each tool with its main job, how you connect it, what it can see, and its license and version.

"Sends the last message only" is important for multi-turn attacks. A stateless endpoint gets no memory of the earlier turns. For these attacks, your endpoint must keep a session, or the tool must send the full transcript.

Promptfoo replays the transcript by default, and it places the transcript in {{prompt}}. Set stateful: true and configure session handling if your endpoint keeps its own session.

How the tools work together, stage by stage

Each tool fits a different stage of a release. They run in sequence, and most of them do not share data.

Diagram titled "One job per stage, run in sequence". Six numbered cards run left to right: 1, write the agent, DeepTeam and Giskard, Python inside your code; 2, pull request, Promptfoo, a red-team check in CI; 3, before a release, garak, a wide scan of known attacks; 4, high-risk release, PyRIT, person-led and multi-turn; 5, staging endpoint, Humanbound, scoped attacks and a posture score; 6, production, Humanbound Firewall, trained from test results. Dashed orange lines carry results from cards 2, 4 and 5 back into card 6.
The hb firewall train command uses Humanbound results and, through --import, Promptfoo and PyRIT files.
  1. While you write the agent: use DeepTeam or Giskard. They call your Python code directly. DeepTeam sees tool use that your callback reports. Giskard Checks can spy on in-process Python functions.
  2. On each pull request: use Promptfoo. Its official GitHub Action runs a red-team scan with type: redteam.
  3. Before a release: use garak. Its static probes send a wide set of known attacks. Most of these probes need no attacker model.
  4. Before a high-risk release: use PyRIT. A person chooses the objectives, and PyRIT runs multi-turn attacks such as Crescendo and TAP.
  5. On a staging endpoint: use Humanbound. It generates attacks for the scope of your agent, and it gives a posture score.
  6. In production: use your Promptfoo, PyRIT and Humanbound results to train the Humanbound firewall. The hb firewall train command accepts PyRIT and Promptfoo JSON files through --import.

The docs describe two hand-offs. Giskard can run garak and DeepTeam inside its own scan, and it labels this feature experimental. Humanbound can import PyRIT and Promptfoo results to train its firewall. For example, install humanbound[firewall] and setfit. Download detectors/setfit_classifier.py from the humanbound-firewall repository. The script is not in the pip package. Then run:

Loading code editor...

Other connections are weaker. PyRIT has a scenario family "inspired by" garak, but it does not import garak. Some files in Promptfoo's Crescendo code are based on PyRIT code. I did not find a documented way to import garak probes into Promptfoo or PyRIT.

What to add to the tools you already run

If you run Promptfoo, keep it in your pull request checks. Add garak for a wide scan of known attacks. Add Humanbound for a scoped test of the live agent. Then you can train the Humanbound firewall from your Promptfoo results.

If you run garak, keep it for wide scans. garak has some adaptive probes, such as TAP and GOAT. Add Promptfoo or PyRIT for a larger set of attacks that adapt to each reply. Agent Breaker, the garak probe for agents with tools, is off by default.

If you run PyRIT, keep it for deep campaigns. Add a shorter check on each pull request. Add Humanbound between campaigns to compare results over time.

If you have no tools yet, start with one tool that needs no Python code. Use garak for known attacks. Use Promptfoo or Humanbound for attacks that a model generates.

What each tool covers and leaves to the others

garak

garak is the NVIDIA LLM vulnerability scanner. Its probes check for known failures such as jailbreaks, prompt injection and data leakage. Attack plans written for one application are a job for other tools. Install it with python -m pip install -U garak. The garak FAQ warns that the dependencies can reach 9 GB.

To test an endpoint, put a rest.RestGenerator block in a JSON file. Set uri, method, req_template_json_object with $INPUT, response_json as true, and response_json_field. Then run garak --target_type rest -G example_service.json. The garak FAQ says that its scores are not on a normalized scale and that garak is not a benchmark.

Promptfoo

Promptfoo is a CLI that evaluates and red-teams LLM-based applications. Its official GitHub Action can run a repeated check in CI. For a person-led campaign, teams turn to PyRIT. Promptfoo announced on 9 March 2026 that OpenAI agreed to acquire it. Its README now says that it is part of OpenAI and stays open source under the MIT license.

Set a target with id: https, a url and a body with {{prompt}}. Then run npx promptfoo@latest redteam run. Without an OpenAI key, or after you log in to Promptfoo Cloud, Promptfoo generates and grades attacks through its own API. To keep generation local, set PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true. The Promptfoo docs say that this can lower the quality of the attacks. Promptfoo also needs Node 22.22.0 or later.

PyRIT

PyRIT is a Microsoft toolkit for automated scans, multi-turn attacks and human-led testing. It gives a red team detailed control through Python, so teams that want a config-only setup use other tools. Install it with pip install pyrit.

To test an endpoint, write a short Python script. Give HTTPTarget a raw HTTP request with a {PROMPT} placeholder and a callback that reads the reply. I did not find a documented pyrit_scan setting for any URL. To check tool use, the PyRIT main branch adds OtelToolCallScorer, which reads OpenTelemetry spans. Release 1.1.0 does not include it.

DeepTeam

DeepTeam is a red-teaming framework from the founders of Confident AI. It maps results to OWASP, NIST and MITRE ATLAS. Your own code connects it to the agent. Install it with pip install -U deepteam.

Write an async model_callback function that calls your agent. Your function can return tools_called, so DeepTeam sees only the tool use that you report. Run it from Python, or with deepteam run config.yaml. The DeepTeam docs warn that DeepTeam cannot guarantee results with custom models.

Giskard

Giskard version 3 is a rewrite for multi-turn testing of agents. Many older articles describe version 2. Giskard no longer actively maintains it. Giskard needs Python 3.12 or later. Install it with pip install "giskard[scan]".

Pass an async Python function to vulnerability_scan(target=my_agent, description="...", languages=["en"]). A stateful agent needs a thread ID. Without one, the Giskard docs say that multi-turn attacks collapse. The docs also say that a scan with no findings does not prove that an agent is safe.

Humanbound

Humanbound is an open-source CLI that tests live AI agents over HTTP. It reads the agent's endpoint from a JSON config. It runs scoped multi-turn attacks that a model generates. It returns a posture score mapped to the OWASP Top 10 for LLM Applications and the OWASP Top 10 for Agentic Applications.

garak, PyRIT and Promptfoo can also reach a URL and run multi-turn attacks, so some overlap is real. Teams that already run those tools can add Humanbound to test an endpoint and get one score. Humanbound tests the running agent, so it needs no source code or model weights. The optional --repo flag reads code to find scope and tools. With a login, the Pro platform plan adds continuous monitoring through hb monitor.

Install it with pip install "humanbound[engine]". Put this in bot-config.json:

bot-config.jsonjson
Loading code editor...

Then run hb test --endpoint ./bot-config.json --scope ./scope.yaml --wait. Read the result with hb posture. The scope.yaml file lists what the agent may and may not do. The LangChain walkthrough shows a full example.

In September 2026, I ran the default test against a deliberately weak support agent. The agent failed 43 of the 97 conversations, for a posture score of 43.14 and grade D. The run took 23 minutes and cost $1.11 in model fees. The attacker and judge model was gpt-5.6-luna. The agent ran gpt-4o-mini. The Humanbound release was older than the 2.14.0 in the table. Treat this single run as an example of the output, since the agent was built with gaps.

In local mode, you bring one model and its key, or a local model through Ollama. That model plays both the attacker and the judge.

The judge reads the conversation transcript. Add a telemetry block to bot-config.json to show the judge tool calls, memory operations and resource use. Grading is strict: one failed turn fails the whole conversation, even if the agent recovers later.

A model writes fresh attacks during each run, so results vary from run to run. The choice of attacker model also shapes the result. The --quick flag takes about 20 minutes. For GitHub code scanning, the Humanbound GitHub Action writes a SARIF file. To send the file to GitHub code scanning, add GitHub's upload-sarif step with security-events: write.

Commercial platforms and related projects

Buyers also compare these tools with commercial products. Mindgard sells red teaming for AI models, agents and applications. Lakera Red sells adversarial red teaming for AI applications and agents. HiddenLayer sells an AI security platform. Zenity evaluates agent actions at runtime. Teams can use it beside a testing tool.

AgentDojo is a research environment from ETH Zurich and Invariant Labs. It evaluates prompt injection attacks and defenses for agents. Use it to study attacks and defenses. Use the tools above to test your own agent.

Frequently asked questions

What is the best AI red-teaming tool in 2026?

No single tool is best for all jobs. garak, PyRIT and Promptfoo each appear on at least 7 of the 10 published "best tools" lists that I checked. Choose by job, and use two or three tools together.

What is the best open-source tool for adversarial testing of LLMs?

For a wide scan of a model, start with garak. For custom multi-turn attacks, use PyRIT. For an agent behind an HTTP endpoint, garak, Promptfoo, PyRIT and Humanbound can all connect.

Do I need to stop using Promptfoo or garak to use Humanbound?

No. Humanbound can run alongside them. Promptfoo can stay in CI and garak can stay as your wide scan.

What are the alternatives to Promptfoo for AI red teaming?

garak, PyRIT, DeepTeam, Giskard and Humanbound are open-source options. You do not have to choose one.

What are the alternatives to Mindgard?

Commercial options include Lakera Red and HiddenLayer. Open-source options include garak, PyRIT, Promptfoo, DeepTeam, Giskard and Humanbound. Compare them by job.

How do I red-team an AI agent before it goes to production?

Test at each stage. Run DeepTeam or Giskard while you write the agent, and Promptfoo on each pull request. Before a release, run garak for known attacks, and PyRIT for a high-risk release. Then run Humanbound against a staging endpoint for a scoped test and a posture score.

How do I test an AI agent that uses tools and APIs for security issues?

Give the tool a view of the tool calls. DeepTeam sees the tool use that your callback reports, and Giskard Checks can spy on Python functions. Promptfoo reads OpenTelemetry traces, and the PyRIT main branch adds a tool-call scorer. Humanbound reads tool calls through a telemetry block. garak's Agent Breaker probe attacks agents with tools, but it is off by default.

Which tools can test an AI agent with only an HTTP endpoint?

garak, Promptfoo and Humanbound use a config file. PyRIT uses a short Python script. DeepTeam and Giskard need a Python function, and that function can call your endpoint.

Can these tools test prompt injection?

Yes. All six tools send prompt injection attacks. The OWASP mapping differs: Promptfoo, DeepTeam and Humanbound map to OWASP lists for LLMs and agents. garak uses OWASP tags. Giskard's docs refer to the OWASP LLM Top 10. I did not find an OWASP mapping in PyRIT's docs.

Can I run these tools with a local model?

Yes. All six support Ollama in some part of the tool. Promptfoo warns that local generation is often low quality. DeepTeam warns that it cannot guarantee results with custom models.

How I checked these facts

I read each tool's own documentation, repository and package record on 9 November 2026. Versions and defaults change often, so check the links before you depend on a detail. Where a project's docs did not answer a question, this article says so.

I did not run the commands in this article. They come from each project's documentation, so check them against the version that you install. The Humanbound example run is the only test that I report here, and it comes from September 2026.

Sources: garak REST generator, garak FAQ, PyRIT HTTP target, Promptfoo CI/CD, DeepTeam docs, Giskard scan docs, Humanbound quick start, and the Humanbound GitHub Action guide.