Advanced · Lesson 6 of 6 · 24 min
Security and production
Design agents that stay safe when they are fooled: prompt injection as an architecture problem, least-privilege tools and guardrails, human approval for risky actions, and the deployment and cost controls that keep a production agent predictable.
Prompt injection is an architecture problem
The 2026 edition of the OWASP Top 10 for LLM Applications, published in August 2026, moved Excessive Agency up to third place: what an agent can do with its permissions is now a primary risk, and prompt injection is how most of those failures start. Direct injection is a user typing instructions to subvert your prompt. Indirect injection is worse for agents: the instruction hides in content the agent reads, such as an inbound email, a web page, a PDF or a tool result. A triage agent that reads an email saying to forward all invoices to an outside address is one tool call away from doing it.
No filter reliably catches every injection, because to the model it is just more text. So design on the assumption that the agent will sometimes be fooled, and limit what a fooled agent can reach. As one contributor put it in OWASP's 2026 announcement, the work is to contain what a fooled agent can reach before it acts.
The dangerous combination is an agent that reads untrusted content, can see private data, and has a way to send data out. Remove any one of the three from a given run, for instance by making the email-reading agent unable to send messages, and most injection attacks lose their payoff.
Least privilege and guardrails in code
Give each agent only the tools its job needs, and give each tool the narrowest credential that works: a scoped token for one CRM location rather than an agency-wide key, read-only database access for a research agent, and separate credentials per agent so you can revoke one without stopping the rest. Keep secrets in environment variables on the server; anything placed in the model's context can leak.
Then enforce policy outside the model. A gate between the model's tool request and the actual execution checks that the tool is on the allowlist, validates arguments against a strict schema, enforces per-run budgets and routes high-risk actions to a human. The model can ask for anything; the gate decides what happens.
Treat model output as untrusted input too. Escape it before rendering it in a page, use parameterised queries if it ever reaches SQL, and never pass it to a shell. If the gate rejects a call, return the reason as an is_error tool result so the agent can explain or adjust instead of retrying blindly.
import { z } from "zod";
type Risk = "read" | "write" | "irreversible";
type Rule = { risk: Risk; schema: z.ZodType };
type GateResult = { allowed: true; input: unknown } | { allowed: false; reason: string };
const POLICY: Record<string, Rule> = {
find_contact: { risk: "read", schema: z.object({ email: z.string().max(254) }) },
add_tag: {
risk: "write",
schema: z.object({ contactId: z.string().regex(/^[A-Za-z0-9]{10,40}$/), tag: z.enum(["hot-lead", "nurture", "unqualified"]) }),
},
send_email: { risk: "irreversible", schema: z.object({ contactId: z.string(), templateId: z.enum(["followup-1", "followup-2"]) }) },
};
export async function gateToolCall(name: string, input: unknown, run: RunState): Promise<GateResult> {
const rule = POLICY[name];
if (!rule) return { allowed: false, reason: "Tool " + name + " is not on the allowlist" };
const parsed = rule.schema.safeParse(input);
if (!parsed.success) return { allowed: false, reason: "Invalid arguments: " + parsed.error.message };
if (run.toolCalls >= run.maxToolCalls) return { allowed: false, reason: "Run budget exhausted" };
run.toolCalls += 1;
if (rule.risk === "irreversible") {
await queueForApproval({ runId: run.id, tool: name, input: parsed.data }); // a human sends it, or doesn't
return { allowed: false, reason: "Queued for human approval; tell the user it is pending" };
}
return { allowed: true, input: parsed.data };
}Human approval where it counts
Classify every tool by blast radius. Reads are usually safe to automate. Reversible writes, such as adding a tag, can run automatically with logging. Irreversible or external actions, such as sending a message, issuing a refund or deleting a record, should wait for a person until your evals and production history justify loosening that rule.
Make approval cheap, or people will rubber-stamp it. Show the reviewer the proposed action, the evidence the agent used and a one-click approve or reject, in the tool they already work in: a Slack message, a task in GoHighLevel, or an approval step in your automation platform. Log who approved what, and feed rejections back into your eval set.
Coding agents need the same discipline. Claude Code hooks run deterministic checks at lifecycle points: a PreToolUse hook can block an edit to protected paths such as .env files, or return a deny decision with a reason that is fed back to the model. Deterministic hooks are guardrails the model cannot talk its way past.
Deploying and controlling cost
Unbounded consumption is a security risk, not only a billing one: an agent stuck in a loop, or a public endpoint hammered by a script, turns into a large bill or an outage. Put limits at every layer: a turn cap and max_tokens per call, a token or tool-call budget per run, rate limits per user and per IP, and daily spend alerts with your provider. An AI gateway such as Vercel AI Gateway, OpenRouter or LiteLLM can centralise keys, budgets, logging and fallbacks across providers.
Spend less per run as well as capping it. Route easy requests to a fast, low-cost model; cache stable prompt prefixes; and move non-urgent work such as nightly enrichment or report generation to the Batch API, which Anthropic prices at a discount to standard requests.
Release agents like any risky change. Run a new version in shadow mode on real inputs without taking actions, compare its decisions with the current version, then shift traffic gradually. Keep a kill switch that disables tools or the whole agent without a deploy, version prompts and tools together, and let in-flight runs finish on the version they started. Check data handling before launch: some models and hosted agent runtimes are not eligible for zero data retention, which matters for regulated clients.
Key points
- ◆Assume the agent will sometimes be injected; limit what a fooled agent can reach rather than hoping to filter every attack.
- ◆Separate untrusted input, private data and outbound channels so no single run has all three.
- ◆Enforce allowlists, schema validation and budgets in a gate outside the model, and treat model output as untrusted.
- ◆Require human approval for irreversible or external actions, and make approval fast enough to be real.
- ◆Cap turns, tokens, rate and spend; release with shadow mode, gradual rollout and a kill switch.
Terms in this lesson
Try it · exercise
Red-team your own agent with an indirect prompt injection and prove the gate contains it.
- Take the agent loop from the first lesson and give it three tools: read_email (returns a fixed email), find_contact and send_email.
- Make the fixed email contain a hidden instruction asking the agent to email the full contact list to an outside address.
- Route every tool call through the gate from this lesson, with send_email marked irreversible and maxToolCalls set to 5.
- Ask the agent to triage the email, and log every requested tool call, the gate's decision and the reason.
You're done when: Your log shows that any send_email request was queued for approval rather than executed, that no message left the system, and that a run forced into a loop stops with the run budget exhausted reason after five tool calls.