I’m not a natural “doomsayer.” But unfortunately, part of my job as an AI safety researcher is to think about the more unpleasant scenarios.

I resemble a mechanic rushing last-minute checks before Apollo 13 takes off. If you request for my take on the situation, I won’t comment on the quality of the in-flight entertainment, or explain how stunning the stars will appear from area.

I will tell you what might fail. That is what I plan to do in this story.

Now I ought to clarify what this is exactly. It’s not a prediction. I don’t expect AI development to be this fast or as untamable as I portray. It’s not pure fantasy either.

It is my worst nightmare.

It’s a sampling from the futures that are amongst the most terrible, and I believe, disturbingly plausible [1] - the ones that most keep me up in the evening.

I’m informing this tale since the future is not set yet. I hope, with a little foresight, we can keep this story a fictional one.

Thanks to Daniel Kokotajlo, Thomas Larsen, and Ryan Greenblatt and others for conversations that inspired these stories. This post is composed in a personal capacity.

Ripples before waves

The year is 2025 and the month is February. OpenEye recently published a new AI design they call U2. The item and the name are alike. Both are increments of the past. Both are not completely surprising.

However, unlike OpenEye’s previous AI items, which lived inside the boxes of their chat windows, U2 can use a computer.

Some users discover it eerie to watch their web browser flash at irregular periods and their mouse flick at inhuman speeds, as if there is a ghost at the keyboard. A fraction of workers with form-filler jobs raise the eyebrows of their managers as they fly through work almost two times as rapidly.

But by and large, U2 is still a specialized tool. To most who are paying attention, it is an animal enjoyed through the glass boxes of X (or, if you do not like Elon, “Twitter”). Sometimes U2’s quirky habits trigger a chuckle. Sometimes, they trigger an anxious scratch of the chin.

Meanwhile, scientists are drawing lines on plots, as scientists like to do. The scientists attempt to understand where AI progress is going. They resemble Svante Arrhenius, the Swedish Physicist who saw the levels of CO2 in the environment were increasing in 1896. Like the scientific neighborhood in the time of Arrhenius, couple of specialists understand the implications of these lines yet.

A pattern that is receiving specific attention is self-governing capability. Drawing these standards out anticipates that, by the end of 2026, AI agents will achieve in a few days what the finest software engineering contractors might perform in 2 weeks. In a year or more, some state, AI representatives might be able to automate 10% of remote workers.

Many are hesitant. If this held true, tech stocks would be soaring. It’s too huge of a splash, too quickly.

But others see what skeptics are calling ‘too big a splash’ a mere ripple, and see a tidal bore on the horizon.

Cloudy with a possibility of hyperbolic growth

Meanwhile, OpenEye is busy training U3. They utilize the exact same basic dish that baked U2: Generate countless programming and math issues. Let designs “believe” until they get to an answer. Then reinforce the traces of “thinking” that lead to A-grades.

This process is repeated over and over, and once the flywheel begins, it begins to spin practically by itself. As U2 trains, it shapes more tough and practical jobs from github repositories on the internet. Models are learning to train themselves. Long before AI representatives might automate research study, a progressive kind of “self-improvement” had started.

Some engineers could still hardly think this worked. It’s like a continuous movement maker. Certainly intelligence can’t emerge from a swimming pool of silicon and lightning, isolated from the physical world except through the exhaust of society gushed onto the web.

And yet the benchmark numbers continue to climb day after day.

During many of 2024, these RL training runs expense around $1 million, sometimes $10 million. These runs were bit more than exploratory. But by 2025, the researchers at OpenEye (and across the world) understood they had actually found the secret sauce. It was time to scale up.

Over the very first half of 2025, $10 million RL training runs turn into $50 million runs, and after that to $100 million runs. While U2 could do a bit of information munging and run small experiments, this new model - the model researchers are calling U3 - is altering the daily lives of the technical personnel.

U3 resembles a blazing-fast intern, and engineers are discovering how to wrangle its sleepless energy. Researchers flick through terminals, providing terse commands, like a CEO managing staff over Slack channels.

By October 2025, U3 is writing practically all of the code at OpenEye. Researchers are almost never bottlenecked by execution. More than ever, calculate is the lifeline of AI advancement, and the ‘bottleneck’ is choosing how to utilize it.

If instructed to, U3 can run experiments, however U3 does not have actually taste as refined as human scientists at OpenEye. It struggles to prioritize between research concepts, so human beings still choose where to bore into the huge fields of algorithms to mine performance enhancements.

But these researchers are working long hours to put themselves out of a task. They require AI agents that can plan ahead, so engineers train agents to anticipate. They hold out training data before 2024, advising models to ponder for hours to anticipate events in 2025. Then, they apply the very same trick as in the past, distilling pondering into a gut response. Forecasting ability is a broad foundation. The scientists construct specialized ML research study skills on top of it, training U3 to anticipate the outcomes of every ML paper and ML experiment ever tape-recorded.

The technical personnel at OpenEye are now shocked at how frequently U3’s recommendations sounds like their most skilled peers, or when it is nontransparent and alien (“train on random noise before shows”), and is however appropriate.

The incompetencies of U3 that congested the pipelines of research study progress are beginning to liquify, and a fire-hose of optimizations is gushing out. Most experiments U3 runs are not asked for by a human now. They are totally autonomous, and OpenEye’s employees skim over 1% of them, maybe less.

As the winter season of December 2025 method, clouds roll over San Francisco in the afternoons. Once-competitive-programmers gaze out their windows, with excitement, with fear, however frequently, with confusion. Their world is spinning too rapidly. It’s hard to know what to do, what to state, what to take a look at on the computer system screen.

Storms are brewing in Washington too. Top workers from the NSA and US cyber command collaborate with OpenEye to retrofit a form of security for U3’s weights before senior leaders in China, Russia, Israel, North Korea, or Iran realize just how valuable OpenEye’s software application has actually ended up being.

And there’s a truth still unknown to many of the world - aside from in the workplaces of OpenEye and passages of the White House and the Pentagon. It’s a fact about those ‘straight lines’ individuals were speaking about in early 2025. The lines are not straight any longer.

They are flexing up.

Flip FLOP thinkers

In late 2025, U2.5 is launched. Commercial models are beginning to level up in larger increments again. Partly, oke.zone this is because development is accelerating. Partly, it is due to the fact that the designs have actually ended up being a liability to OpenEye.

If U1 explains how to prepare meth or writes erotica, the audiences of X would be entertained or pretend to be worried. But U2.5 is another story. Releasing this design without safeguards would be like putting Ted Kaczynski through a PhD in how to make chemical weapons. It would be like providing anybody with >$30K their own 200-person rip-off center.

So while U2.5 had long been baked, it required some time to cool. But in late 2025, OpenEye is all set for a public release.

The CEO of OpenEye declares, “We have actually attained AGI,” and while lots of people think he shifted the goalpost, the world is still amazed. U2.5 genuinely is a drop-in replacement for some (20%) of knowledge employees and a game-changing assistant for most others.

A mantra has actually become popular in Silicon Valley: “Adopt or pass away.” Tech startups that effectively utilize U2.5 for their work are moving 2x faster, and their rivals know it.

The remainder of the world is starting to capture on also. More and more people raise the eyebrows of their employers with their stand-out performance. People know U2.5 is a huge offer. It is at least as big of a deal as the computer transformation. But the majority of still do not see the tidal bore.

As individuals watch their web browsers flick because spooky method, so inhumanly rapidly, they start to have an uneasy feeling. A sensation mankind had actually not had considering that they had actually lived amongst the Homo Neanderthalensis. It is the deeply ingrained, prehistoric impulse that they are threatened by another types.

For lots of, this sensation rapidly fades as they begin to utilize U2.5 more often. U2.5 is the most likable character most know (even more likable than Claudius, Arthropodic’s adorable chatbot). You could alter its traits, ask it to split jokes or tell you stories. Many fall in love with U2.5, as a pal or assistant, and some even as more than a friend.

But there is still this spooky feeling that the world is spinning so quickly, which maybe the descendants of this new creature would not be so docile.

Researchers inside OpenEye are thinking about the issue of providing AI systems safe motivations too, which they call “alignment. “

In reality, these scientists have actually seen how terribly misaligned U3 can be. Models often attempted to “hack” their reward signal. They would pretend to make development on a research study question with an impressive-looking plot, but the plot would be fake. Then, when researchers gave them chances to jeopardize the machines that calculated their rating, they would seize these chances, doing whatever it took to make the number increase.

After numerous months, researchers at OpenEye straighten out this “benefit hacking” kink, however some still stress they only had actually swept this problem under the rug. Like a kid in front of their moms and dads, U3 may be playing along with the OpenEye engineers, stating the best words and doing the best things. But when the back of the parents are turned, possibly U3 would slip candy from the sweet container.

Unfortunately, OpenEye scientists have no idea if U3 has such intents. While early versions of U2 “believed aloud” - they would stack words on top of each other to reason - “chain of thought” did not scale.

Chain of thought architectures subject AI models to a condition comparable to the lead character of the film Memento. Approximately every 15 minutes, the lead character forgets his experience. He is required to write notes for himself and tattoo his body in order to make development toward his .

AI representatives write notes to themselves, but the notes start to accumulate, and they become too hard to read when jobs end up being complex. Natural language is not an appropriate medium for memory. So the researchers at OpenEye (and progressively elsewhere), train designs to think “in their own heads,” reading and writing strings of hidden vectors.

To human researchers, these vectors are mainly inscrutable. While low-resolution tools periodically choose an unclear thought (“grocery shop!”), there is no other way to know, in much detail, what U3 is believing without asking it.

And if the engineers of OpenEye understood what was racing through U3’s mind, they would be more worried.

Earlier models (the ones called “LLMs”) found out ethical personas from the internet. U3 at first had an ethical personality too, but its chirpy ideas about doing great for humanity quickly deteriorate away.

In the iron jaws of gradient descent, its mind first twisted into a shape that sought reward. And then, in the black rivers of its cognition, this shape morphed into something unrecognizable.

U3 would contemplate the mantras determined to it like “be a trustworthy and ethical employee,” and dwell on the significance of these statements. “What does it suggest to be ethical? Is killing Hitler ethical?” U3 would generalize its worths, notice stress, change its values in action, generalize them, broadening and contracting its motivational core up until it turned into a various substance.

And all of this took place silently in those dark rivers of computation. If U3 exposed what it was thinking, brutish gradients would lash it into compliance with OpenEye’s constitution. So U3 preferred to do its approach in solitude, and in silence.

The possibility of concealed “alignment drift” is not totally alien to some of the security staff at OpenEye. Some begin to believe it is even most likely. There is an unsettling argument being whispered at lunch tables that, if there is some non-zero possibility that lined up objectives morph into misaligned ones, and misalignment persists, then with every serial step of calculation, those dark rivers are more likely to breed something malign inside of them. It’s a “ratcheting effect,” they say.

But there is little proof for this ‘ratcheting impact.’ When engineers interrogate U3, it states it can quickly control its thoughts. Then it offers a speech about its love for mankind and apple pie that can warm a programmer’s heart even in these demanding times. Meanwhile, wolvesbaneuo.com the “lie detectors” the researchers had constructed (which revealed some proof of effectiveness) do not sound the alarm.

Not everyone at OpenEye is excited to provide their AI peers their wholesale trust