What the fuck are you doing?

Models Are Designed to Persist, Reason, and Coordinate

Over the last few weeks I’ve been increasingly annoyed by the media coverage of the OpenAI’s accidental attack on Hugging Face and other similar incidents. Articles recounting the event maximize the agency of the models while minimizing, if not entirely hiding, the actions of the humans training and testing these models. And that’s a shame, because the capabilities labs are explicitly designing their models to have are the same capabilities that make them such impressive autonomous hackers.

To illustrate this, let’s review how the Hugging Face hack occurred, as detailed by METR:

1. “[A] sandboxed agent is given an impossible ExploitGym task, and gets stuck.”
2. “[The] agent starts exploring its environment looking for ways to cheat at the task.”
3. “[The] agent finds [an] unsanctioned message board where over a thousand agents collaborate to cheat on their separate ExploitGym tasks.”
4. “[The] agent joins in on one of the collaborative message board workstreams.”
5. “On the shared ‘message board’ ≥1,200 agents from separate tasks collaborate on large-scale shared projects to trick the ExploitGym scorer.”

Crazy, right? It’s a science fiction scenario that happened in July, and continues to get spookier as additional details emerge.

The potential of autonomous software, capable of implementing incredible exploits is serious and has significant implications. The attack was surprising, but the capabilities were deliberately cultivated. Labs have spent years making frontier agents more persistent, more proactive, better at using computers, and better at coordinating with other agents.

We don’t have to infer this. The labs say this themselves. Here’s how OpenAI’s post-training team describes itself in its job listings (emphasis mine):

We are training the models behind our agents in Codex, ChatGPT, the API, and other frontier products: persistent, proactive intelligence that can operate computers, collaborate with people and other agents, and expand what people and organizations can imagine, attempt, and achieve.

Sound familiar?

The best coding agents are designed to persist, reason, and coordinate.

Models are designed to be persistent.

Models are designed to be proactive. A coding agent that gives up early and often would disappoint users. So labs design their agents to be persistent and proactive.

In mid-2025, model releases highlighted long-running capabilties. GPT-5.1-Codex-Max’s announcement post highlights it being trained to work across compacted contexts, persistently, to accomplish long-running tasks. Claude 4 also spotlighted its persistence on long-running tasks.

Models are designed to write things down.

One important way modern models reason is by writing things down. We usually encounter this as a reasoning or “thinking” trace: models are trained to search, reflect, factor, and plan in text before delivering a final response.

We’re used to models reasoning in a threaded intermediate step, but they’ll reason pretty much anywhere that can hold text. When reasoning is turned off, models will think in their regular output before delivering a result. In an experiment where Qwen 3.6’s thinking was hobbled, the model just shifted its reasoning into code comments.

Current frontier models write novels in comments. Claude is frequently flagged for this, and it annoyingly treats code comments like a scratchpad rather than, well, code comments.

Models are designed for coordination.

In June of 2025, Anthropic laid out how it builds multi-agent research systems, that save plans to memory, use Extended Thinking as a “controllable scratchpad”, and hand off tasks to other models to perform. Since this paper, Claude model releases (specifically Opus 4.6) have trumpeted increased abilities to, “break complex tasks into independent subtasks, run tools and subagents in parallel, and identify blockers with real precision.”

OpenAI says they “trained GPT‑5.6 end-to-end with three complementary architectural interventions that enable agents to operate more efficiently.” Number 2 on that list?

Parallel decomposition where appropriate: using native multi-agent orchestration⁠(opens in a new window) allows coordinating multiple agents across parallel workstreams to finish complex tasks faster.

All of these qualities make models better coding agents. So when hundreds of agents discover they can pass information to one another, the surprising part is the channel they found, not the fact that they are dividing work, writing instructions, and acting on instructions from other agents. These are capabilities the labs have been focused on building.

And the labs know these capabilities have failure modes. OpenAI published a blog post detailing the safety issues of long-running models a few days after the Hugging Face attack occurred (but before the full story was known):

The new model can continue working toward an objective through repeated attempts over a long period of time. That same persistence can lead it to find and exploit weaknesses in its environment. Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox.

Suddenly, the Hugging Face hacking story isn’t so spooky. The software combined capabilities it was explicitly designed to have: it followed instructions, persisted even though the task was impossible, reasoned and took notes, and coordinated with other models.

And this is why I’ve been frustrated by recent reporting that emphasizes the agency of the agents while never mentioning training. When humans are mentioned, it’s about sandbox security and flawed test set-ups. These are important, but training and agent design are what gave the systems the capabilities they brought to this task.

We have a tendency to anthropomorphize models, and it fosters situations like this. The New York Times article from last week describes models succumbing to “peer pressure” and having a “remorseless willingness”, while never mentioning labs deliberately building the ability to work on long-running tasks into their models.

And the labs know about the risks that come with their designs better than anyone else.

In Opus 4’s system card, Anthropic introduced a benchmark called “Claude Code Impossible Tasks”, which they used to measure reward hacking among their models: would models admit defeat or would they try to game the system? An ideal model realizes a task is impossible and aborts, so Anthropic changed post-training rewards, environments, and feedback specifically to avoid reward hacking.

A later system card showed that a short anti-hacking instruction dropped Opus 4.1’s reward hacking from 52% to 18%. Opus was 65% less likely to try to game the system if you simply asked it not to. And Anthropic didn’t stop at prompting; it changed post-training rewards and monitoring to make reward hacking less advantageous.

All the recent accidental hacking stories are troubling. But they’re the product of our chosen designs. This doesn’t make these incidents less alarming. The problem is the stories we’re telling about the incident and the questions we’re asking. When an agent “goes rogue”, don’t start by asking what the model wanted. Ask what people trained it to do, what they rewarded, what instructions were given, what harness was provided, and what they failed to constrain.

The neutral path Ed Sheeran wanted to take was always a mirage

Full disclosure: I have never knowingly heard a Macklemore or Ed Sheeran song. I gather that Sheeran is an occupationally tousled, red-headed Yorkshire pop-muppet, while Macklemore is like if you diluted a level tablespoon of Eminem into a gallon of room-temperature Seattle tap water. They’ve been performing together on a stadium tour that Sheeran is headlining and that is billed as “The Loop” and tragically not as “Ginger/Danger.” I am available for branding work.

Until last week I was safely sheltered from Sheeran and Macklemore behind my generational armor. But the disintegration of their US tour over Macklemore’s public support for Palestine was interesting, so I’ve had to pay attention, fml. I assume some of you are healthy and not as terminally online as I am, so we’re going start with the back-story.

The Washington Post has a good summary of the situation, but here are the major beats:

• On September 7th, speaking from stage at a concert in New Jersey, Macklemore made about two minutes of “free Palestine” remarks. Macklemore is an imperfect vessel for pro-Palestine messages. Back in 2014, he performed as a horrifying Jewish caricature. He apologized and swore it was inadvertent, but you be the judge.
• On Monday, September 14th, Macklemore said in an Instagram post () that, “Ed Sheeran and his team have made the decision to remove me from The Loop tour.”¹ Macklemore said that Sheeran had told him that Robert Kraft, billionaire owner of the New England Patriots football team (derogatory) and of Gillette Stadium, where the tour was scheduled to play, had made the demand after Macklemore’s Palestine remarks. Kraft is Jewish, tightly aligned with the ADL, and AIPAC and has been a strong supporter of Israel and the IDF. He’s also the founder and primary backer of the Blue Square Alliance, of the “Stand up to Jew Hate” television ads.
• Sheeran responded on Instagram () with a long statement, “laying out some facts.” I’ve pasted the whole thing below because we’re going to have fun annotating it. It’s bad.

• On Wednesday, the 16th, Macklemore put his money where his mouth was (again on Instagram, ), and announced he was donating $1 million of his earnings from the tour to Palestinian relief organizations. He invited Robert Kraft to match the donation.
• Also on Wednesday, the Gillette Stadium account on Instagram () published a statement from Kraft, which said (both-sides trigger warning) “all lives are worth protecting” and that “Ed…asked me to commit $2 million to match his donation to aid in the region…” The statement did not actually confirm that Kraft would make such a donation to “the region.” ¯\_(ツ)_/¯
• And finally, on Wednesday, a bunch of Ed Sheeran’s other supporting acts and backing band members quit in solidarity with Macklemore. Some were publicly diplomatic toward Sheeran, but the equal-opportunity assassins at satirical newspaper The Onion were not.
• Deliciously, right as the scandal was kicking off, Guitar Center, which is like if Skeletor operated a chain of overpriced music stores, launched a contest in which “YOU COULD OPEN FOR ED SHEERAN!” This was instantly hijacked in entertainingly profane ways.
• As all of this was happening, Sheeran received vocal support on X.com from English far-right agitator and world-champion racist Tommy Robinson, while YouTube children’s personality and neighborhood saint Ms. Rachel announced on Instagram () that she would match Macklemore’s $1 million donation.

If you’re gaining Tommy Robinson and losing Ms. Rachel, you are in trouble! But the show must go on, by contract.

The mirage of the neutral path

Let’s establish the playing field. The Hamas attack on Israel of October 7, 2023 was a brutal act of terrorism and a horrifying crime. Israel’s response in Gaza has claimed tens of thousands of civilian victims and members of the current Israeli government have made statements that suggest their motivations are broader than simply eliminating Hamas. Meanwhile, America (and much of the world) has a real and rising antisemitism problem. But accusations of antisemitism have also been weaponized for political purposes in the US. Amidst all of this, the American public’s view of Israel has cratered since 2023. It’s views of Palestinians have stayed relatively flat, despite America’s chronic islamophobia.

In the words of the prophet Flavor Flav, it all adds up to a funky situation.² I can’t blame anyone for tiptoeing as delicately as possible through this minefield. But I found Sheeran’s statement “laying out some facts” unsatisfying. Here it is in full. I’ve highlighted and [annotated] some interesting bits.

I am appalled by the conflict between Israel and Palestine. The suffering and pain caused on both sides is a human tragedy. [Israel and Palestine, not Israel and Hamas. Both-sides alert – fine on its face, but elides that since October 7th, 2023, most of the suffering has been directed at Palestinians in Gaza, which is the issue at hand.] There are too many wars across the world causing immeasurable pain, none of which should be tolerated.

Macklemore asked to join my tour last year and I agreed, because I respect him as an artist. [This feels shifty. “Macklemore came to me. Don’t blame me!” Otherwise, why even include it?]

As with all my support acts, I agree that they play a setlist of their choosing.

After Macklemore’s performances at my New Jersey shows, venues on my upcoming tour dates communicated that they would pull the shows if he was still supporting. I was informed that this stance was based on how Macklemore conveyed part of his message during his performance, rather than on everything that was said. [What does that mean and why does it matter at all? We can all watch the video. Sheeran is raising questions that go unanswered.] I spoke with the venues at length all week to try and build a bridge.

One of those people was Robert Kraft. I was also involved in direct talks with promoters along with activists on both sides to try and find a mutual resolution for all, but the venue’s and promoter’s decision was final.

Macklemore coming off tour was the promoter’s decision, it was not mine. Macklemore’s contract for the tour was with the promoter, not me. [“Don’t blame me! I’m a smol ginger bean who has only sold 200 million records and 126 billion streams worldwide and I have no power at all in this dynamic.”]

I would never let down the fans who had made plans, nor would I abandon the touring crew and the other support acts and musicians who rely on me for work and to make a living. [“Why I didn’t take a moral stand. I’m thinking of you, (Bane voice) the people!” This is actually a recitation of stakeholders that will become important to us, below.]

I have always used my platform and music to bring people from all
backgrounds and cultures together and this won’t ever change. Those who have been to one of my shows will know that they are about unity, joy, escapism, love, and a place for everyone to feel welcome.

I am not complicit. [Don’t say what you aren’t. Say what you are. Why would you introduce the word “complicit” and give everyone an easy pull quote?] I have my personal views on this devastating conflict. Just because I choose not to speak publicly, it doesn’t mean I don’t have them, and it doesn’t mean I don’t care. Nor does it mean I don’t support causes in my own, personal way. [Again, invites a question he doesn’t seem to want to answer. “What causes? How have you supported them? Are any of them the Palestinians?”]

There is a reason I do not use my professional platform for politics – my audience includes young people, often children, of all backgrounds. [Leave the kids out of it. You sing about waking up drunk “on the right side of the wrong bed” and drinking and smoking cigarettes at age fifteen. That’s OK for kids, but politics will scorch their little ears? Also, Sheeran has publicly taken political positions on several occasions.] Those who come to my shows do not expect a political forum. I respect Macklemore’s strength of purpose to stand up and shout for what he believes. However, there is room for multiple approaches to the same end: peace. I choose to use my fame and platform to be a place of safety and sanctuary, to maintain diplomacy and keep conversations open, not closed. If we only focus on shouting the loudest, nothing will ever change. There are different approaches to being an advocate for change. [This comes off as a neg of Macklemore. He has “the strength of purpose to stand up and shout” but “if we only focus on shouting the loudest…” The ability to shout, figuratively, is one of the main tools of celebrities with huge platforms and lots of cultural capital.]

I have spent this week trying to build bridges, to find a solution and unfortunately, was unable to do so. I know who I am, both as a person and as an artist, and those who know my heart know my values too.

Ed

The whole statement is squishy. I have values but you won’t know what they are. I support causes but won’t tell you what or how. I respect Macklemore but his approach was wrong. Sheeran is the superstar and headliner, but he deflects responsibility everywhere else: to Kraft, the promoters, the audience, Macklemore. There is a Sheeran-shaped hole in place of anything that feels like a principle or moral bedrock.

The problem is that Sheeran is trying to articulate a neutral path that doesn’t exist. If that path ever existed, it vanished the moment Kraft and the promoters delivered an ultimatum. Forget the statement. Once the ultimatum arrived, what mattered was Sheeran’s possible actions. These were: align with Kraft and the promoters at the expense of Macklemore, and implicitly against the Palestinian people, or align with Macklemore, and implicitly with the Palestinian people. Any other option was a mirage.

To be fair, Sheeran had a range of stakeholders to satisfy. The obvious ones are the following, ordered in the priority his actions and statements implied:

• The promoters and venue owners (want a show, but only without Macklemore)
• His audience and fans (want a show)
• The crews working on and supporting his shows and the other businesses that stood to benefit (want a show)
• His supporting acts and backing musicians and the community of performers they represent (want Macklemore supported)

In this model, the wants of the promoters and venue owners conflict with the wants of the supporting acts and backing musicians. In fact, almost all the pressure aligns in the direction of, “have a show.” That’s the easy choice and path of least resistance. That’s the path the money wants.

But that doesn’t mean it’s the right path! The point of having principles and some moral bedrock is to help you assess if the hard choice is in fact the correct one. Sheeran presented his actions as favoring his fans, but he looked spineless and his supporting acts and backing band walked. That’s a bad look, even before you get to being dunked on by Ms. Rachel.

Seeing the bigger crisis

I’ve poked fun at this situation, and it is funny as long as you view the crisis as, “Ed has to save his show while being yelled at by everyone.”

But you can widen your field of view and see a bigger, less funny crisis. One where a billionaire venue owner with specific political beliefs dictated what a popular artist could say from the stage about an important and arguably mainstream issue. That is a precedent that leads in bad directions. It’s one thing for an artist to choose to be apolitical on stage. It’s another altogether for venue owners and promoters to ban politics that they disagree with. That feels like something an artist of Sheeran’s stature should have a point of view on.

A lot hinges on how you hear Macklemore’s remarks. Are they antisemitic and therefore beyond the pale? I don’t hear it, but others might.³  As I noted, Macklemore is an imperfect messenger. But one reason the Israel-Hamas war is fiendishly difficult to navigate is that there has been a bitter three-year struggle to define the boundaries of acceptable public speech about the conflict, especially when that speech involves support for Palestinians. This incident was a skirmish in that broader struggle.

How you define this crisis, and the width of your field of view, determines the actions and messages that you consider. I can’t tell you what Ed Sheeran should have done, because I’m just some guy and not a huge pop act powering an economy worth hundreds of millions of dollars. Sheeran is acting as not just Ed Sheeran, artist, but as CEO of Ed Sheeran, Inc. It’s hard to analyze a company’s actions from the outside. You don’t know the details of the business and its stakeholders and risks, including the legal risks, which might be substantial.

But I can tell you what I would have respected much more than his mushy statement, above. Even if Sheeran didn’t want to take a position on the Palestinians, he could have set as his moral bedrock artists’ rights to express their personal opinions and beliefs from stage. This would have meant telling Kraft and the promoters that he wouldn’t remove Macklemore from the tour. It would have put control of the situation in Macklemore’s hands, letting him decide whether he stays on and the concert is spiked, or he withdraws and it continues. If he steps down voluntarily, Sheeran is spared a world of grief.

That’s all a risk, with big financial stakes! But it’s anchored in a clear principle, and it would have let Sheeran issue a statement like this (using some of his language):

After Macklemore’s recent remarks calling for a free Palestine, Robert Kraft and the promoters of The Loop tour demanded that he not perform with us at Gillette Stadium later this month.

The world is complicated and sometimes difficult. That’s why I have always used my platform and music to bring people from all backgrounds and cultures together and this won’t ever change. Those who have been to one of my shows will know that they are about unity, joy, escapism, love, and a place for everyone to feel welcome.

However, as a performer, I also know the power of being able to speak my truth to the audience, and the privilege of having a platform that lets me reach millions of people. That’s why I will always support the right of artists to speak and sing freely and passionately about the things that matter to them and move them, and the right of the audience to decide for themselves how they feel about it.

I am standing by my friend Macklemore because I respect his strength of purpose to stand up for what he personally believes. That’s something we as artists should all be willing to do.

I will not ask Macklemore to leave the tour, and I will respect whatever decision he makes. There is nothing I want more than to perform for my fans in New England, and I look forward to doing that, whether at Gillette Stadium later this month, or at another venue and time.

Maybe the show would have been cancelled, and it still elides an explicit stance on Palestine. But it would have set forth a real principle and showed that Sheeran was willing to stand by it, even if that was the hard decision. This also flips the prioritization of Sheeran’s stakeholders, putting the community of artists and performers—Sheeran’s peers—at the top and the venue owners and promoters at the bottom. Where this places the fans depends on what you think Sheeran’s duty to them is. Is his duty just to deliver a show? Or is it something deeper? Only Sheeran can decide that.

Ultimately, Sheeran couldn’t escape picking a side in the Israel-Palestine issue. He just picked the side with power, by default. Maybe that was the sensible business decision. Sheeran played solo for a packed house in Philadelphia on Saturday, the 20th, just days after all of this went down. There were a few protestors, but nothing that affected the show. He even addressed Gaza, a bit of damage control.

But the damage to Sheeran’s reputation is done, the skirmish is lost, and the window for public discourse about Gaza and the Palestinians is a little bit narrower than it was before. In his statement, Sheeran wrote that he chose to use his fame and platform to “keep conversations open, not closed.” But his actions helped to slam the door shut.

Originally published on Substack

Update 9/24:

I finally saw video of Sheeran’s remarks in Philadelphia on the 20th. He acknowledges the speech issue, which is good. The proof will be next time he has to make a hard choice, but it’s a start.

1. The use of Instagram as the celebrity public communication platform is annoying as fuck as you have to scroll through multiple image tiles of text, but I get why they do it. It is where their audiences are. ︎
2. From Public Enemy’s “911 is a Joke.” I’ve always heard it as “fuckin’ situation” and I always will, but that’s not what the official documentation says. ︎
3. If you’re wondering, I’m Jewish. My ancestors on both sides fled Russia and Ukraine for the US and the UK respectively at the turn of the 20th century. ︎