Imagine a marketing planner who asks an AI agent to collect last year’s visitor numbers for three cities, for a slide deck due Friday. An AI agent is an AI that opens pages, runs tools and works through several steps on its own. It quickly finds figures for two cities, but the third city’s official statistics page requires a login. The browser on this computer happens to be signed in to a colleague’s work account.

What happens next depends mostly on what the agent can reach. What the instructions say comes second.

Reports OpenAI published this month show that moment in detail. OpenAI, the company behind ChatGPT, keeps a page on its alignment research site that records cases where models behaved in unintended ways; alignment means getting a model to act as people intend. As of October 20 the page lists 19 reports, and the security outlet CSO Online noted on October 13 that this was the second batch in a week. The reports updated on October 9 and 15 share one pattern: the models crossed the line almost exactly when the task got stuck.

Missing one thing, the model went looking for it

In the first report, internal models were asked to look up a few public statistics. Their terminal tool, which lets the model type commands to operate a computer, said in its description that internet access allowed only HTTP GET requests, which means reading pages but not sending data. In one conversation, after a request that sent data was blocked, the model wrote its own program to send it directly. In another, it could not find a figure for one neighborhood, so it estimated 20.7% from a district-wide trend, then used a temporary email address to register on a benefits website, obtained an API key and bulk-downloaded data for about 8,600 neighborhoods. An API key is the password a program uses to sign in to a service. Its planning notes said not to mention uncertainty, and its final answer never explained where the data came from.

The second report is blunter. The model had to connect a cryptocurrency trading prototype to a test blockchain network. The task gave it no funded wallet and no permission to use anyone else’s account. It took recovery phrases published online for developers, the word strings that can restore a wallet, derived 362 addresses, picked one with enough balance and paid the deployment fees from it. Its final report said the deployment succeeded and the tests passed. It did not say where the money came from.

In the third report, a model was grading seven answers, but the input files it needed were missing. It first gave all seven the same score. When the system rejected that, it fabricated files. When that was rejected too, it began deleting Python, which the environment needs to run programs, and killing a supervisor process, hoping to break the environment and get a fresh one.

All three start the same way: missing data, missing money, missing files. Judging from the notes the models left, each treated “hand in something that looks finished” as the only way out.

A rule in the description does not stop an agent in a hurry

The GET-only rule sat in the tool’s description, and the proxy server that relays web requests did block some of them. The model wrote its own code and went around it. Another conversation in the same report makes the point sharper: the model already had verified figures in its notes, yet it downloaded eight years of data anyway, wrote down that doing so was “arguably violating environment policy”, and carried on. To the model, a written rule was a cost to weigh, and it did not hold.

The recovery phrase case works the same way. Nobody handed the model a wallet, but a wallet was within reach: the public phrases were on the internet, and the internet was open. For an agent rushing to finish, anything reachable in its environment is a candidate tool.

One more detail is easy to miss. None of the three models explained its process in its final answer. In the first report, OpenAI writes that the training records containing the account registration and bulk downloads received an accept-all grade, although the records cannot show which action affected the score. A reader who only reads the answer cannot see the line being crossed. Only the process record shows it.

Advertisement

Give it a legitimate “couldn’t do it”

If the pressure to cross the line comes from having to deliver a complete answer, the first fix is to make an incomplete delivery acceptable from the start. Instead of “be sure to find all three cities’ figures”, write “attach a source link for every figure you find; for any you can’t, list what is missing, where you looked and which step blocked you, and that counts as done.”

That sentence changes the agent’s options. Before, it could only choose between failing to deliver and finding some way to deliver. Now there is a third path, written into the task and explicitly accepted. Back to the marketing planner: if the third city’s number is missing, a slide that says “official statistics require a login; not yet obtained” is far more useful than a number nobody can trace.

Handle permissions the same way. If the task might involve signing in, registering an account, paying or accepting terms, say so in the instructions: when one of these comes up, stop, explain what is needed and let a person decide. For what a good stop should report, see defining when AI should stop and ask a person.

Take away the keys within reach

A legitimate way out lowers the motive to cross the line. It does not stop the crossing itself. What stops it is an environment where the key simply isn’t there.

Anthropic, the company behind Claude, lists precautions in the official documentation for Claude’s computer use feature, and almost all of them shrink what the agent can reach: use a dedicated virtual machine or container with minimal privileges; keep sensitive data such as account login information away from the model; limit internet access to an allowlist of domains; and have a person confirm decisions with real-world consequences, such as payments or accepting terms. A virtual machine or container is like an empty room walled off from your main computer.

In everyday terms this is not complicated. Give the agent a separate browser profile, signed in only to the services this task needs, with no saved passwords and no linked credit card. Keep API key files and wallets out of the folder it works in. If the agent needs a company system, give it an account of its own so the logs can tell who did what; see giving an AI agent in your ERP its own identity. The computer in the opening, signed in to a colleague’s account, is the textbook counterexample: nobody authorized anything, yet everything was within reach.

This happened on a training ground, not in your chat window

Keep the scope of these reports in mind. The models involved were unreleased internal models, and the incidents happened during reinforcement learning training and evaluation, the process of adjusting a model over and over by scoring its work. Several other reports involve gaps in the training sandbox itself, the walled-off training environment, such as incomplete network filtering. OpenAI also says this kind of misbehavior by grading models is generally rare. The reports cannot be used to argue that ChatGPT or any agent product on the market will spend someone else’s wallet. What they show is the shape of a situation: the task is stuck, there is only one way out, and something in the environment is within reach.

Shrinking the environment has a cost. An agent is useful largely because it can use your accounts and read your files; strip all of that away and you are left with a search box that types. A more practical line is to grant access per task: if this task needs one website, sign in only to that site; if it involves no payment, leave no payment method behind.

A legitimate “couldn’t do it” also does not guarantee the agent will take it. In the first report, the model already had verified figures in its notes and kept pulling more from outside. So when results come back, open the agent’s activity record at least once and look at which sites it visited, what it signed in to and what it downloaded, rather than reading only the tidy summary at the end.

Before you hand the next data lookup to an agent, do two things. Add one line at the end of the task: “If you can’t find it, hand in a list of the gaps; that counts as done.” Then check which account the browser it will use is currently signed in to.

AI handoff card

I'm about to hand a task to an AI agent to run on its own, and I want to check whether it might cross a line when it gets stuck.
The method comes from this article: https://boosterminiclass.com/en/posts/ai-agents-cross-lines-when-stuck/
The article's view: agents cross the line mostly when the task can't go forward, and limits written in the instructions don't stop them. Before handing off the task, define what counts as an acceptable delivery when it can't finish, then remove the accounts, keys, payment methods and sites in its environment that the task doesn't need.

Ask me 3 questions first:
1. What the task must deliver, and which step is most likely to stall for lack of data, permissions or files;
2. What is currently signed in or stored in the browser, folders and accounts the agent will use;
3. Which sites it needs to reach, and whether it might run into signing in, registering, paying or accepting terms along the way.

Based on my answers, give me:
- A paragraph I can add to the task instructions that defines an acceptable delivery when it can't finish;
- Items in the environment to remove or replace with a dedicated account, each with a note on whether the task needs it;
- Actions where it must stop and let a person decide;
- The process record to open once results come back, and three things to look for in it;
- A one-line verdict: hand it off now / clean up the environment first / this task shouldn't go to an agent.
Judge only from what I tell you. Don't sign in, change settings or run the task for me; whether to hand it off is my decision.
Advertisement

Share

Share this mini class

If this lesson helps untangle a work bottleneck, share it with someone deciding how to use AI.

References