Many companies want to hand customer support, account recovery, internal IT support, or routine operations to AI. In this article, an AI agent means an AI system that can carry out several steps on its own: it does not only answer questions, but may also look up data, call tools, change settings, send links, or help a user complete a workflow.
The problem is not that AI will always “go rogue.” The more common risk is that it obediently follows the process and does something the process should not allow.
A June 15 article from the Stack Overflow Blog explains this risk with an account support scenario. An attacker may not need to crack a password. If they can persuade a support workflow with enough authority to change account data, add a new email address, or send a reset link, they may be able to take over an account that belongs to someone else. The article connects this to the security idea of a “confused deputy”: an agent with authority is persuaded by someone with less authority to do something it should not do for them.
For ordinary teams, that warning matters more than any single incident. Many workflows used to rely on humans making judgment calls in the middle: a support rep pauses when a request feels wrong; an IT colleague asks one more question when the request looks strange; a manager rejects an exception. Once an AI agent sits in the same position, if those judgments have not been written as clear rules, the AI may only see “the next step is available” and miss “this step should not be done for this person.”

First find the checkpoints that used to live in people’s heads
Before introducing an AI agent, do not start by asking, “How much of this can it automate?” Start by asking: in this workflow, where did humans previously sense that something was wrong, pause, reject, or escalate?
Common examples include:
- Customer support changing a user’s account email, phone number, shipping address, or payment information.
- IT support resetting a colleague’s password, opening access, adding them to a group, or creating access credentials.
- Sales or operations systems changing a customer’s plan, refund, discount, or data export.
- Internal workflows submitting an approval, exception request, or data deletion request for someone.
All of these may look like “following the process,” but the process often contains a part that was never written down: Is this person really the account owner? Why are they asking for this change now? Could this change give a third party control of the account? Is there a history of abnormal behavior? If the request comes from a new device, new region, or new contact method, can it still be completed directly?
A human support agent may combine these signals in their head. If an AI agent only receives the sentence “please change my email to this one,” and it also has tool access, it may translate that natural-language request into a tool action. The real danger is not whether AI can chat. It is whether identity and authorization are checked again before the tool is called.
Use a four-part table: identity, permission, reason, consequence
Before connecting an AI agent to any workflow that changes data, changes accounts, sends reset links, or triggers payments, use the table below as a first check.
| Checkpoint | Question to ask | If the answer is unclear |
|---|---|---|
| Identity | Who is the requester? How does the system confirm they are the account owner or have the right to act for them? | Do not let AI execute directly; send it to human confirmation first |
| Permission | Is this person allowed to request this type of change? Is the permission supplied by the system, rather than claimed by the requester? | Do not accept a request based only on text explanation |
| Reason | Does this change fit a normal situation? Are there unusual times, locations, devices, or historical records? | Ask for supporting evidence or escalate to the responsible owner |
| Consequence | What happens if this is wrong? Account takeover, data leak, financial loss, or only a mistaken text reply? | Automate only the low-risk steps |
The point of this table is to separate “the AI can call the tool” from “the AI is allowed to do this for this person.” Tool access is a system capability. Authorization judgment is a process responsibility. When the two are mixed together, the AI agent becomes a very polite and efficient front desk that may open the door for the wrong person.
Which requests must stop
Not every customer support or internal workflow has to avoid automation. Checking order status, organizing FAQ answers, reminding people about missing documents, and formatting information are usually lower risk. The requests that need real caution are the ones that are hard to undo or hand over authority after they are completed.
The following kinds of requests should not be allowed to run end to end through an AI agent:
- Changing login email, phone number, backup email, or multi-factor authentication method.
- Creating password reset, payment, refund, data deletion, or data export links.
- Adding administrators, increasing privileges, joining sensitive groups, or issuing API keys.
- Changing a customer’s plan, contract terms, billing address, or payment method.
- Any request whose reason is only one sentence, but whose result affects account control, company data, or money.
An API key is a key that lets software access a service. If it is issued to the wrong person, that person may read data, perform actions, or create costs. Multi-factor authentication is a second login check beyond the password, such as an authenticator app or SMS message. Once these items are changed, the real account owner may be the one who can no longer get in.
So the answer is not that an AI agent must “refuse to help completely.” A safer design is: collect the information first, compare it with rules first, flag abnormalities first, and then send high-risk requests to the person responsible for confirming them.
Minimum useful approach: let AI receive the request, not approve it
If a team is just starting to put an AI agent into customer support or operations, use a conservative design first: let AI handle intake, classification, missing information, and draft replies, but do not let it directly complete high-risk changes.
A minimum useful workflow can look like this:
- AI first classifies the request type: inquiry, ordinary change, sensitive change, account recovery, payment, or permission.
- The system automatically supplies the requester’s identity, login status, existing permissions, and abnormal signals, so the AI does not rely only on what the user says.
- Low-risk requests can be answered automatically or queued for processing.
- Medium-risk requests should keep a summary, evidence, and suggested next step, then wait for a human to confirm.
- High-risk requests are paused immediately and sent to a human workflow; the AI is not allowed to send reset links or change permissions.
This is not an argument against AI automation. It is a way to avoid mistaking “intake efficiency” for “approval authority.” AI is good at organizing messages, finding missing details, and reminding people about next steps. But whenever the next step touches account control, sensitive data, payments, or permissions, there needs to be a clear human checkpoint.
Everyday four-panel comic

- When many requests arrive at the same time, AI can help receive and classify them, so people do not have to sort every message by hand.
- Once account control, payments, data, or permissions are involved, “sounds like the owner” cannot be treated as authorization.
- AI can organize the evidence and missing details, but identity, permission, reason, and consequence need clear rules, and someone must be able to confirm them.
- Low-risk requests can pass quickly; high-risk requests should stop at the door and wait for the responsible person to decide.
Ask three questions before deployment
If you are evaluating a customer support AI, IT support agent, internal workflow agent, or any automation tool that will change data for people, ask three questions first:
- Before every tool action, does this agent carry the requester’s identity and permissions, rather than only one text request?
- Has the system clearly written which requests must be rejected, paused, or sent to a human, instead of leaving the AI to decide by itself?
- If the AI makes a mistake, can you trace which request, which tool, which rule, and which person finally approved it?
If you cannot answer any one of these questions, do not hand high-risk actions to AI yet. Start with low-risk organization and drafting. After the rules for identity, authorization, rejection, and traceability are written clearly, expand the scope step by step.
The value of an AI agent is that it can make workflows faster and less likely to miss messages. It should not replace the judgment that protects accounts, data, and money. Turning those moments when humans used to think “this feels wrong” into explicit rules is the most important step before introducing customer support and operations agents.
AI handoff card
Ask AI to organize this article's specific situation
Copy this into your own AI chat tool to turn this mini class into a personal checklist. BMC will not see what you paste into your AI tool.
References
- Stack Overflow Blog: AI agents expose the security checks you never actually wrote — https://stackoverflow.blog/2026/06/15/ai-agents-expose-the-security-checks-you-never-actually-wrote/
- Help Net Security: Omada Agent Governance helps organizations manage AI agent access, risk, and compliance — https://www.helpnetsecurity.com/2026/06/15/omada-agent-governance/
- SANS: Your AI Agent Is an Easily Confused Deputy — https://www.sans.org/blog/your-ai-agent-easily-confused-deputy-why-cloud-security-needs-credential-broker
- OWASP: Top 10 for Agentic Applications 2026 (ASI03 Identity & Privilege Abuse) — https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/