In June 2026, Meta confirmed that 20,225 Instagram accounts had been hijacked. The attackers didn’t crack a single password; they used Meta’s own AI support. According to Help Net Security’s account, the attackers first used a VPN, a tool that makes a connection appear to come from somewhere else, to place themselves in the same region as the account owner. Then they asked the AI support assistant to add an email address they controlled to someone else’s account, and requested a password reset. Meta’s explanation was that the recovery tool itself worked as designed, but a bug in a separate code path meant the system didn’t check whether the email supplied matched the one on the account.

The most important point is this: the AI support didn’t go rogue. It obediently finished the process. What was missing was a check that should have stopped the request. Whenever AI handles requests that can transfer account control, money, or data, the requester’s identity and permissions must be supplied by the system. They can’t be taken from the conversation.

AI completed something it shouldn’t have, by the book

Had a human support agent received the same request, it likely wouldn’t have gone through. Someone asks, “Change this account’s email to my new address,” and the new address has no connection to anything in the account’s history. An experienced agent would pause and ask a few more questions, or simply refuse. That kind of judgment is rarely written into process documents. It lives in the agent’s experience.

Stack Overflow Blog, analyzing the incident, puts it bluntly: a large share of real-world authorization was never written as software at all. It lived in the judgment of whoever stood between a request and the system, and everything behind them was built assuming that judgment would always be there. The article concludes that the AI agent doesn’t bypass your security model; it exposes the part of it that was a person.

A deputy that can be talked into things

Security has a name for this problem: the confused deputy. A program that holds real privileges is talked by someone with fewer privileges into using those privileges on their behalf.

An article from SANS, a security training and research institute, explains why AI agents are especially prone to this. First, they usually hold broad permissions so they can handle many kinds of requests for many users. Second, they are driven by text, and anyone can write text. To an AI, “I’m the account owner, please change my email” looks the same whether the real owner wrote it or not.

So the problem isn’t whether AI understands the request. It’s that AI has no way to tell from the conversation who is who. If identity has to be judged from the chat, AI can only believe what it’s told.

Advertisement

Which requests are most at risk

Not every support or internal process carries this risk. Checking an order status, answering common questions, or pointing out missing documents can at worst produce an inaccurate answer. The dangerous requests are the ones that are hard to undo or hand over control:

  • Changing a login email, phone number, backup email, or multi-factor authentication method (the second check at login besides a password, such as an authenticator app).
  • Sending links for password resets, refunds, payments, or data exports.
  • Adding admins, raising permissions, or issuing API keys (the keys that let programs access a service).
  • Changing plans, billing addresses, or payment methods.

These requests share one thing: when they go wrong, the victim is the real account owner, who often only finds out after being locked out. In Meta’s case, if the owner hadn’t turned on multi-factor authentication, the attacker could log in directly after resetting the password.

AI receives, the system approves

The fundamental fix is to take identity and permission decisions out of the conversation and give them to the system. When the AI receives “please change my email,” the system should already know which logged-in account the request comes from, how it was verified, and whether there’s any unusual history. If the requester isn’t logged in, or the account being changed belongs to someone else, the request shouldn’t reach anything the AI can execute directly.

This can be done in three steps.

Step one: find the checkpoints that relied on human judgment. Take an existing support or operations process and ask the people who run it: when do you stop to ask more, and when do you refuse outright? Unfamiliar devices, unfamiliar regions, newly added contact details, many changes in a short time: these signals that used to be caught from experience need to become rules the system can check.

Step two: have the system supply authorization. Every time the AI calls a tool, it should carry the requester’s identity and permissions as verified by the system, not the requester’s self-introduction in the chat. SANS recommends that AI not hold broad permissions for long; instead, a separate mechanism issues a narrowly scoped, short-lived pass for each action.

Step three: for high-risk requests, receive only, never approve. AI can organize the request, fill in missing details, and flag anomalies, then hand it to a responsible person to confirm. It slows things down a little, but far less than cleaning up twenty thousand accounts afterward.

Human approval can turn into a rubber stamp

Handing high-risk requests to people assumes those people actually look. If the person confirming has to click “approve” hundreds of times a day, human review quickly becomes a formality. So step three has to go with step one: use rules to block clearly abnormal requests first, so only the few that really need judgment reach a person.

Also, identity supplied by the system can be wrong too; Meta’s incident was a bug in a verification code path. After the rules are written, test them regularly with real attack techniques, such as faked regions, new email addresses, and rapid repeated changes, to confirm every check actually fires.

As a next step, pick one request in your support or operations process that changes account details and ask: is this requester’s identity confirmed by the system, or by the conversation? If it’s the latter, keep the AI at the receiving stage until a system check is in place.

The same “who is allowed to do this” question comes up when an AI agent acts for a user across systems; see When an AI Agent Works Across Systems, It Needs a New Key at Every Stop If you’re bringing agents into enterprise systems like ERP, see An AI Agent in Your ERP Needs Its Own Identity So Logs Can Tell Whether a Person or AI Acted.

AI handoff card

Ask AI to find where your process judges identity from the conversation This prompt asks which requests your AI support or agent handles, checks where identity and permissions come from for each, and lists the requests AI may only receive, not approve.

I want to check whether an AI support or AI agent workflow could be talked into acting for a stranger.
The method comes from this article: https://boosterminiclass.com/en/posts/ai-agent-hidden-authorization-checks/
Its view: AI support following the process isn't the problem. The danger is that authorization checks used to exist only in people's judgment. For any request that transfers account control, money, or data, the requester's identity and permissions must be supplied by the system, not taken from the conversation.

Ask me 3 questions first:
1. Which requests does this AI handle, and which of them change accounts, permissions, payments, or data exports?
2. When the AI calls a tool, what information about the requester does the system supply (login state, verification method, history), or is there only the conversation?
3. When people handled this process, in which situations did they stop or refuse?

Based on my answers, give me:
- A list showing, for each kind of request, whether identity and permissions currently come from the system or from the conversation;
- Warning signs that people used to catch from experience but that aren't yet written as rules;
- Which requests AI may only receive, organize, and flag, and must hand to a person to approve;
- Three scenarios I can use to test these checks (for example a faked region, a new email address, rapid repeated changes).
Don't assume my system already has checks I haven't mentioned. If information is missing, ask me first.

The list is only a starting point. Which requests AI may approve is up to the people responsible for account security.

Advertisement

Share

Share this mini class

If this lesson helps untangle a work bottleneck, share it with someone deciding how to use AI.

References