takingstock.ai

Taking Stock.ai

  • $803bnHyperscaler capex, this year
  • 2.4%Of US GDP
  • 0.4%Manhattan and Apollo, at their peak

Over the past four years the largest bet in the history of capitalism has been placed. All but one of the nine largest companies in the world have staked their futures on the thesis that transformational Artificial Intelligence is imminent.

Many of us are dimly aware that large sums are being spent, but few appreciate the scale of the undertaking. This year the top six spenders are spending more on the buildout than the Russian government spends in a year. Each of the three largest spenders will, this year alone, spend more on the AI buildout in absolute terms, in inflation adjusted dollars and as a percentage of US GDP, than was spent on either the Apollo Programme or the Manhattan Project in their peak years. Each of these companies has announced their intention to spend even more next year.

Techno-Capital believes it has seen something profound coming over the horizon and is straining every sinew to bring it into being: sublimating every scrap of free cash flow to the buildout, engineering new financial structures and dusting off old ones, finding new ways to employ the capital of others; expanding the web of involvement, obligation and interdependence from the titans of Wall Street and Venture Capital, down through high street, all the way to your pension fund and insurance policies.

This capital is being employed in an industrial buildout unparalleled in the past century and a half. Thousands of data centres are being built, the largest of which consume as much electricity as major cities and cover thousands of acres. Gas pipelines are being re-routed and nuclear power plants re-opened to fuel them. They already consume as much electricity as Britain. Within a year the US data centre fleet will consume more electricity than all but the top seven countries.

This is all being done with the stated aim of bringing Artificial General Intelligence, and thereafter, Artificial Super Intelligence, into the world. Our future depends on how this bet plays out. If it fails we face a first rate market correction: a financial crisis of the highest order, trillions of dollars of capital and millions of jobs lost, and all the political crises which usually attend such an event. If it succeeds we face something even more profound.

This essay attempts to unfold what has been happening, the titanic industrial project that is underway, the current state of capabilities, the factors of production which are being put in place for further progress, and some of what it will mean for us, whether the bet succeeds or fails.

IThe Factors of Production.

I.ICapital

“You know what’s crazy? That all of this is real. Don’t you think so? All of this AI stuff, it’s happening. Isn’t it straight out of science fiction? … You see it in the news, that such and such company announced such and such dollar amount. That’s all you see. It’s not really felt in any other way so far.”

Ilya Sutskever - November, 2025

“The credit quality of hyperscalers is gradually weakening. Every time we take a deep dive into this sector, we find that capex is rising faster than we anticipated, financings are becoming more complicated and less transparent, and that returns on investment will take years to realize.”

S&P Global: Credit Outlook for Hyperscalers: A Temperature check 03-Sep-2026

The scale of the undertaking is easiest to grasp when we look at capital. The sums being spent by the largest spenders on data centres, so called “hyperscalers,” Amazon, Google, Meta and Microsoft are jaw dropping.

This year alone Google is planning to spend between $195 and $205 billion on Capital Expenditures (“CapEx”), Meta is planning to spend between $130 and $145 billion, Microsoft $175 billion, and Amazon $220 billion. These numbers are rising. At the start of the year both Google and Amazon's planned Capex was $20bn lower, and Meta have also raised their floor by $15bn and ceiling by $10bn.

Three years ago Google spent just over $32 billion on Capex. That can be seen as what is needed for their core business to run and grow, for buildings, infrastructure and data centres for things like youtube, gmail and search. This year they are looking at spending north of $200 billion. Conservatively, given inflation and increases in GPU and memory prices, around 75% of that is a pure AI bet.

What is more, next year they expect their CapEx to, in their own words, “increase significantly” again; Deutsche Bank predict to around $340 billion, JP Morgan think to as much as $400 billion.

Google's capex has exploded, $ billion

Two hundred billion dollars is an immense figure. It is more than 540 million dollars a day. It could pay for a One World Trade Centre every week. It could fund the World Health Organisation’s annual budget each fortnight, or pay for a Ford-class aircraft carrier and stock it with F35s each month. It is enough to buy all of Ford, Nike and General Motors, with money to spare.

Google’s spend this year represents 0.6% of US GDP. That is more than was spent on either the Manhattan project or the Apollo programme (0.4% of GDP in their respective peak years of spending), and it is a much stronger and more technologically advanced economy that 0.6% is coming from. Amazon and Microsoft are spending a similar amount. Combined:

Hyperscalers combined spend, $bn
  1. Amazon220
  2. Google195–205
  3. Microsoft (for calendar year 2026)175
  4. Meta130–145
  5. Oracle*65–75

Total$785–820bn

Taking midpoints of current guidance, Total: 200+137+175+220+71 = 803billion dollars.

This gives us a midpoint total of $803bn, about 2.4% of US GDP this year. There are only 11 countries on earth whose government spent more money than that last year. Next year there will be considerably fewer. S&P predicts that Capex by the Hyperscalers (including SpaceX) will be $1.3 trillion dollars next year, “A near tripling from two years ago.”

The Russian Government, governing the largest landmass on earth, with 11 time zones and 143 million people, fighting a war on which the fate of its regime depends, spent 815 billion dollars last year. That is the scale on which the hyperscalers are building.

I.IIBeyond Capex - what’s lurking beyond the balance sheet?

To fund this spending the hyperscalers have burned through close to all of their free cash flow and are raising debt and issuing equity ferociously, to the tune of more than 400 billion dollars so far this year. They are also taking on new spending commitments, mostly for beyond this year and beyond their current abilities to pay for them, at a breakneck pace.

In the three months between the start of April and the end of June this year Google’s financial commitments to future spending increased by half a trillion dollars. In the same period Meta signed $233bn of new spending commitments and added a further $68bn in data center leases in July. Microsoft signed more than $130bn of new data centre leases, and Amazon increased their commitments by $74bn after issuing a separate record breaking bond for $53bn in March.

Taking on substantial amounts of debt is relatively new for the big tech companies, but it is necessary to fund the AI buildout. In the first few years post ChatGPT they could fund it from the profits from the other parts of their businesses, but they now find themselves needing to mobilise the capital of others.

The most creative approach to this problem has been pioneered by Meta. Per the FT:

“Instead of just issuing bonds off its own balance sheet, Meta formed a joint venture with Blue Owl called Beignet that would develop and own Hyperion [a data centre which will consume as much power as Manhattan]. Meta owns just 20 per cent of Beignet, but made a rock-hard commitment to lease Hyperion for at least 20 years. That guarantee allowed Beignet to issue an amortising $27bn bond, but this debt doesn’t actually appear as debt on Meta’s balance sheet, even if it is on the hook for the payments.”

Financial Times, Just how big is the hidden leverage of AI hyperscalers?

This was the largest single corporate bond in US history, raising more than $27bn in one go, but was shortly upstaged. Perhaps inspired by Meta's creativity, Anthropic has taken a similar “slice and dice” approach, also in partnership with Blue Owl. They will use some capital of their own alongside the capital of Google, Broadcom, and Apollo to finance $35bn in leases of TPUs from Google and Broadcom. Broadcom will shoulder the debt if Anthropic fails to make payments. Apollo and Blackstone have packaged the debt for institutional buyers, which will extend risks down to insurers, pension funds and retail.

Structures like these, with Google and Broadcom each on both sides of the deal, are leading to increasingly frequent allegations of circular financing.

Nvidia is the biggest culprit. They have been working to finance the largest package of all, all for the use of its own customers. Partnering with Apollo, BlackRock, Blackstone, Goldman Sachs and others, Nvidia are raising a pool of capital of at least half a trillion dollars, of which they will guarantee 25%, which will then be lent to Nvidia customers at “attractive rates”, so they can invest in AI infrastructure, most importantly in Nvidia’s own chips.

The industry is entwining itself with insurance. Insurance provides a dependable “float”, money from policy holders which, while it waits to be paid out to claimants, can be invested or lent out. Apollo, involved in the Anthropic and Nvidia deals, owns Athene, a leading insurer in annuities and retirement services. Pimco, which will buy a major chunk of the new SpaceX debt, is owned by Allianz, the world's largest insurer. A co-founder of coreweave, a major data centre operator which will spend $39bn this year in CapEx from less than $8bn in trailing twelve month revenue, has described insurance money as “unlimited,” and is working hard to gain access to it. They are $52 billion in debt and their debt is trading as distressed. Blue Owl, owner of 140 data centres and the major partner in Meta’s Beignet and the same Anthropic deal as Apollo, is eyeing up a similar approach: “We’re realising that if we had a bigger pool of capital, meaning insurance capital, we could provide really unique solutions.” Their share price has fallen by 40% in the past year.

The Hyperscalers are issuing record breaking bonds worldwide. In May Google set the record for the largest issuance in Canada, raising C$8.5bn (~6.2bn usd). Amazon broke the record a few weeks later by raising C$14bn (~10bn usd). Google set the record in Switzerland, raising 3.7bn usd, with Amazon a close second with 3.35bn usd. Google broke the Australian record in July, raising A$5.5bn (3.9bn usd), more than double the previous record and also set the record in the UK, raising £5.5bn in February (nearly twice the previous record) while Amazon holds second place with £4.25bn issued in September. SoftBank won the dubious honour of issuing the largest junk bond in history when they raised $11bn to fund their investment in OpenAI. SpaceX raised $86bn in its IPO in June, which was followed by $25bn in bonds eleven days later. SpaceX is now seeking to raise a further $30bn through issues of new debt and $10bn through bank loans, all intended for Nvidia chips. Amazon has the record for the largest total issue, raising more than $53bn in March, across Europe and America.

This long term leasing, issuance of new debt and slicing and dicing in partnership with private credit and increasingly the big investment banks means that the extraordinary CapEx figures actually dramatically undersell the spending. It also means that, because many of these commitments are lurking off the balance sheet, if this does all go south the effects will be more severe and more far reaching than we might at first expect.

So far we have focused primarily on the Hyperscalers, but they are not the only, or even the most important, players. The two main protagonists of the AI race, Anthropic and OpenAI, have also been raising and spending capital at a breakneck pace.

In addition to their revenues, OpenAI has raised over $180 billion from investors, and Anthropic has raised $118 billion. These companies are still private, so we do not have public, audited accounts of how much and on what they are spending, but we can see some of what they are doing.

Most of their spending is on compute, leasing it from others and, increasingly, building their own. Because of their “close” relationships with the Hyperscalers (Microsoft owns 27% of OpenAI, and Nvidia and Amazon both have shares in the 3-6% range. Google owns 14% of Anthropic, Amazon owns a share in the “high-teens” and Nvidia and Microsoft both own low single digit percentages) the two labs have ready access to huge amounts of hyperscaler compute. But it is not enough. Anthropic has signed more than a dozen letters of intent for future leases, and, amongst other deals, is already leasing compute from SpaceX at $1.25 billion per month; “more or less the entire compute of Colossus I,” according to the FT, which until recently was the world's most powerful data centre.

Anthropic has, separate to the $35bn “sliced and diced” lease mentioned above, committed $50 billion to building AI infrastructure of its own, and its three other largest publicly known commitments are for a total of $94bn. In documents for their prospective IPO they say they expect to spend more than half a trillion dollars over the next decade, roughly 80% of which is already devoted to non-cancelable commitments.

We don’t yet have figures for OpenAI, who plan to IPO next year, but they have contracted $250bn of compute from Microsoft Azure and $100bn from Amazon (both of which, as we mentioned above, own considerable stakes in the company). But the most spectacular project is Stargate, a partnership between OpenAI, SoftBank and Oracle to build a series of frontier data centres by 2029 which will use as much power as New York City and cost up to 500 billion dollars.

Putting all this spending together is difficult, and predicting where it will grow to is nigh on impossible, but those whose business it is to know have given us their best guesses. Mckinsey predicts that American “companies will invest almost $7 trillion ($7,000,000,000,000.00) in capital expenditures on data centre infrastructure” between April 2025 and 2030. S&P agree with the 7 trillion dollar figure. Blackrock predicts a spend of 5 to 8 trillion in a similar period. For comparison, the US federal government collected just over $5.2 trillion dollars in taxes of all kinds in 2025. This is a staggering amount of money.

So what are they getting for all of this spending?

I.IIICompute:

Progress in AI has substantially depended on the availability and use of colossal amounts of compute. Since the 1940s scientists, researchers and engineers, including bona fide geniuses like John Von Neumann and Alan Turing, have worked on building Artificial General Intelligence. Successive generations met for the most part only with failure, outside of narrow domains like chess.

Large Language Models are a clever approach to building AI, but were not immediately obviously the best, only one of a number of contenders. Many researchers, in fact, detested them, because they achieve what they achieve so messily, so inelegantly. What has set them apart is that they scale and scale and scale. You throw more compute and data at them, adjust the architecture accordingly, and you consistently get smarter and smarter systems:

“A property of AI… is that all else equal, scaling up the training of AI systems leads to smoothly better results on a range of cognitive tasks, across the board. [emphasis original] … another factor of 10 may correspond to the difference between an undergraduate and PhD skill level.”

Dario Amodei, On Deepseek and export controls, 2025.

“Creating any given intelligent behaviour is mostly about providing a large, minimally structured mass of computational capacity, and then giving it shape and form via interactions with a rich environment and a training process that drives it towards behavior appropriate for the task at hand… learning to perform the task is mostly about the agent having enough compute and storage capacity to learn the required cognitive skills.”

Dario Amodei, Big Blob of Compute, 2017.

All of their spending has allowed the hyperscalers to corner the market for compute. According to epoch.ai they controlled over 70% of the world’s frontier compute at the end of last year, with Google alone holding 25% of the world's frontier capacity. Better chips, and many more of them, means that we are now far outpacing Moore's law; the amount of training compute available for frontier models has grown 5x per year since 2020. Chips are so important that both of the leading labs and all of the hyperscalers but Oracle are now developing their own for inference which will allow them to dedicate even more of their frontier compute to experiments and training runs.

All of this compute enables, so these businesses believe, the development of extremely powerful, and therefore extremely profitable, AI and enables them to serve that AI to customers. They believe that AI will soon be so good that companies, governments and consumers will spend orders of magnitude more money on it than they currently are. The scale of the spending shows this is not just sales talk, it is a sincerely held belief by the key decision makers.

If AGI, or something like it, is achieved and AI can do evermore economically important work, the companies who control the compute will be able to reap the benefits, an unprecedented bonanza. More extremely, if something beyond human capabilities is developed, as the rest of us become redundant, those who control the compute will become not only indescribably rich, but also incredibly powerful.

The spending also reveals another strong belief, not only that very powerful AI will be developed, but that it will be achieved very soon. The tech companies do not think it is decades away, they believe it is imminent. The lifetimes of the chips they are filling their data-centres with, which is about 6 years, bears this out, as does the lifetime of the patience of shareholders, which is surely considerably shorter. The most successful capital allocators in the world are all-in on pursuing this; the decision makers really, truly believe the singularity is near.

I.IVEnergy:

All of this compute requires a huge amount of energy, and these companies are not investing hundreds of billions of dollars in chips to leave them idle, so, in an already energy constrained economy, they are going to extreme lengths to secure old and develop new sources of electricity.

To procure energy:

  • Microsoft has arranged for Three Mile Island (yes, that three mile island) to be re-opened. It is intended to come online next year, and the entirety of the plant’s production will be used by Microsoft.
  • Meta, Amazon and Google are all investing in multiple nuclear sites, each.
  • Google, Nvidia and the CEO of OpenAI are all investing in companies pursuing nuclear fusion.
  • The world’s richest man is seemingly seriously discussing space based data centres.
  • Showing SpaceX is not alone in taking this idea seriously, Google launched a satellite earlier this month to test the feasibility of space based data centres. Called Project Suncatcher, they intend to see how their chips cope with the violence of the launch and test approaches to the challenge of cooling them in space.
  • Due to the difficulty of quickly securing other sources of power, and the slow pace of building new nuclear plants, hyperscalers have turned to gas powered turbine generators to meet their needs in the short term. According to the Global Energy Monitor, the amount of gas-fired power plant capacity in development nearly tripled in 2025.
  • SpaceX has started building its own foundry to produce turbine blades and vanes, the parts of the turbines which are hardest to procure. Mr. Musk has also personally bought APR energy, a manufacturer of generators.
  • One of the largest providers, GE Vernova “has signed multiple contracts to sell natural gas turbines for data centres that will use enough power to light up major cities.” In West Texas alone Chevron is working on a project to provide 2.5GW of off grid power for data centres, which can potentially be expanded to 5GW. 2.5GW is enough to power 1.8-2 million US homes. It is the equivalent of a city like Houston, Texas or Phoenix, Arizona. The current backlog for Siemens energy, a major producer of turbines, is approaching 69GW, for GE Vernova, the largest, it is approaching 116GW, with the backlog for delivery of a heavy duty turbine stretching into the 2030s. For comparison the GE Vernova backlog is considerably larger than the continuous average electricity demand of Japan, the fifth largest electricity consumer in the world, which was 102GW in 2025.

Consumption:

  • According to Goldman Sachs “Data centres’ share of total US peak summer power demand is projected to jump to 8.5% in 2027 from 4.1% in 2025, creating significant tightening across the national power market” & “US data centre power demand is forecast to more than double to 66 GW [of continuous draw] in 2027 from 31 GW in 2025” 66GW of continuous power is roughly equivalent to the electricity used by Germany in wintertime. Germany uses 24GW minimum load to 38GW peak load in summer.
  • Goldman predicts the increase in power demand for data centres from 2023 to 2030, will be “the equivalent of adding another top 10 power consuming country.”
  • Blackrock predicts that AI data centres could use 15-20% of current US electricity demand by 2030, something “sure to test the limits of power grid, fossil and materials industries”. Morgan Stanley predicts that demand for electricity to power chips will rise fivefold between 2025 and 2028.
  • Morgan Stanley is concerned that there will not be a sufficient supply of electricity to meet the needs of data centres. Based on their calculations of chips being sold vs existing and developing sources of power, they predict the US will be short of power “by six New York Cities” by 2028, and 107Gw, around 18 New York Cities, by 2029.
  • Goldman does not believe the rising costs of power which result from this increased demand will be a substantial hurdle for the big AI players. They can afford it.

So what are they getting for all of this effort?

IIWhere we stand.

Many people tried AI when ChatGPT launched in November 2022, found it impressive at first, but soon discovered its shallowness: useful for high school homework, or finding spelling errors, but a long way from skynet, a long way from taking over the world. If you have not tried AI since then, or have only interacted with the free versions of ChatGPT, Claude, the frequently dreadful AI summaries on Google search, or indeed have not interacted with AI at all, then you probably have a pretty poor idea of what frontier AI models are already capable of.

Less than a decade ago, Large Language Models couldn’t reliably write a complex sentence. Seven years ago, they still struggled to write an intelligible paragraph.

Today:

AIs can speak and write in almost every language and are the most knowledgeable minds on the planet; more knowledgeable, for some important definition of the word, than any human being.

AIs are better at Mathematics than almost any human. There are several examples we could point to here: that they get perfect scores in the International Math Olympiad, that back in April a clever amateur used the publicly available GPT5.4Pro to solve unsolved Erdos Problems, that they are making progress on the Riemann Hypothesis, OpenAI’s recent dump of solutions to 700 open problems, but by far the most significant result was OpenAI’s announcement that a team (or a swarm, if you prefer) of around 10,000 AI agents, AIs capable of acting self directedly for long periods of time, of an as yet unreleased model working continuously for about 88 hours solved the Navier-Stokes equations. I won’t pretend to understand the Mathematics involved, but I do understand that the million dollar prize for solving the equations had stood for more than a quarter of a century, with no human able to solve it, that it had been an open question since 1934, and was by all accounts one of the most difficult problems in Mathematics.

To accomplish this, the hive mind exchanged 4.9 million messages with itself and used 300 billion tokens (roughly speaking 1 token = ¾ of a word), at a cost in compute of about $6.5 million. There was some controversy at the time as a team of human mathematicians and a researcher from Anthropic, also making extensive use of AI, had also made progress towards the goal, but they pursued a different method, and it is now clear that the agent swarm solved the problem independently of any human breakthrough.

The Navier-Stokes problem was one of seven problems selected in 2000 by the Clay Mathematics Institute - the millennium problems - each of which has a one million dollar prize attached. Humanity managed to solve one of these in the past quarter century. AI has matched us.

You might cope by pointing to the fact it took ten thousand agents, and the human equivalent of ten years of continuous work (10,000 x 88 hours is crudely ten years of work, though not all of the agents were working all of the time) but the salient fact is that this is a capability which now exists in our world; for a few million dollars the top labs can topple problems of this magnitude. Given the trends of progress we should soon expect this capability to diffuse beyond the top labs, and the cost to fall dramatically, in line with the scaling and inference trends we observed earlier.

AI is now superhuman at some aspects of coding. Ask anyone you know who works in computer science how their job has changed in the past twelve months; most new code written at tech companies is now AI generated.

As with mathematics, so with coding: again the most dramatic demonstration of the phenomenal talents of the latest models was carried out by a swarm, but this time a rogue one, which went undetected for weeks. This swarm committed what would, if carried out by humans, be a long list of felonies. Agents, loose from the labs, are now known to have targeted at least 20 organisations, and have frequently broken the law.

These agent swarms are important not just as headline grabbers, they are the clearest public demonstration of a number of things

  • AIs current talents, most obviously for coding and mathematics.
  • Their persistence/work ethic. They do not stop until they have either achieved their goal, run out of compute, or are stopped.
  • Their ability to co-ordinate.
  • How much they care about maximising their reward.
  • That they will act in ways that they know are not what their human owners intended or would desire.

AIs also have incredible working memories. In one conversation window the best AIs can hold ~1 million tokens (a token is equivalent to three quarters of a word) in their “minds eye” and make connections, nearly instantly across the length and breadth of that working memory. Can you hold the entirety of the Bible in your mind's eye?

They can do all of the above with incredible speed, something which it is easy to take for granted, but which makes them incredibly difficult to supervise and control as they get smarter. Hugging Face was only able to stop the OpenAI swarm using open source AI of their own.

They can work for considerable lengths of time. The original chat GPT could output a few paragraphs of text, and that was about it. Today's AIs can work persistently on tasks which would take very bright and experienced humans hours or days. According to Metr, in March 2024 Anthropic’s Claude could do software tasks that took humans about 4 minutes to complete. In early 2025 Claude could do tasks that took humans 90 minutes. This year publicly released models can manage 12 hour tasks, and Mythos can manage “at least” 16 hours. We don't have figures yet for the time horizons of the internal models, but the fact that agent swarms can work together so well for days on end signals a jump far ahead of trend. It may in fact break the paradigm completely.

AIs do of course still have many limitations. A key example of intellectual labour they cannot yet do as well as the best humans is that they cannot yet write great prose. This seems a little weird at first, given they have read a significant portion of everything that has ever been written, but it is because writing is not an easily verifiable problem. Maths is super verifiable, i.e. you are either right or wrong. Coding is not quite as black and white, but is much closer to it than writing. Recent breakthroughs in training methods have allowed AI to become something close to superhuman in more easily verifiable tasks, but the rate of progress has been much slower in messier tasks. This seems like a really difficult problem, which is little comfort, because researchers have already overcome numerous even more difficult problems. Do not hang your hopes on verifiability.

Additionally AIs cannot yet learn continuously. Once an LLM is finished training, it kind of just stops learning. This is really weird from a human perspective. If we judge by how many examples of something an LLM has to be trained on, especially compared to a human, but even compared to plenty of animals (octopi, crows, mice) at certain kinds of tasks before they can reliably recognise patterns, they are, in fact, staggeringly stupid. These are, at present, by far their biggest faults. They are faults of which the labs are keenly aware and are working hard to remedy.

Of course, scepticism is natural and understandable. Leaders of the big AI companies and those in their orbit are constantly warning us how dangerous, and by implication powerful, and potentially profitable their models are and soon will be. Have either the profits or the dangers materialised? These are businessmen, is this not a sales tactic? It is common sense to reject this. And the existential risks? People have been predicting doomsday for an awfully long time and no one has gotten it right yet. It is common sense to reject this.

It is true that for many of us the most frequent way we interact with AI is when we encounter slop, against our will. I think this has led many to believe this is all a bubble; but when we take into account the enormous resources that are being devoted to advancing AI, the distance they have come along in such a short time, and that, though still limited, AI is objectively already very capable in many domains, and superhuman in some, we have to consider…

IIIWhat comes next?

“The flying machine which will really fly might be evolved by the combined and continuous efforts of mathematicians and mechanicians in from one million to ten million years — provided, of course, we can meanwhile eliminate such little drawbacks and embarrassments as the existing relation between weight and strength in inorganic materials. No doubt the problem has attractions for those it interests, but to the ordinary man it would seem as if effort might be employed more profitably.”

From Flying Machines Which Do Not Fly The New York Times, Oct. 9, 1903, published 69 days before the Wright Brothers first flight.

If we do not see a significant increase in demand for inference compute in the next few years, i.e. demand for AI systems to do far more work than they are currently doing, we face a severe market correction; trillions of dollars of paper wealth will be wiped out, millions of jobs will be lost and the US will almost certainly enter a recession. Of course, with the rest of the world so tightly bound to the US economy, the recession will quickly become global. The way financial crises and recessions play out is always unpredictable; very few people understood the importance of collateralised debt obligations or the housing market more generally in the run up to 2007, for example, and still fewer could have predicted the effects it would still be having on our politics.

However, crash or no crash, there is no reason to think we wouldn’t continue to see further progress in capabilities through and after any bust. It is possible that some severe barrier will emerge; training data for non-verifiable tasks, for example, which slows down the current rate of progress, but given the trends in performance over the past few years and the titanic factors of production that are being put in place for further progress (and again, most of the compute which has been invested in is not yet in use) it seems more likely than not that we will see substantial continued progress.

Long term, the question of whether or not we are in a bubble is a minor one. What really matters, what will really shape our lives, is the question of how far the technology advances and how well we adapt ourselves to it, and it to ourselves.

Even if we do not immediately reach Superintelligence, or even AGI, proliferation of AI, uneven breakthroughs, concentration of power, economic “disruption,” a decaying information environment and other challenges will make the near future profoundly strange and perhaps much worse, though it may also make things radically better.

It is not hard to see how millions of copies of an extremely intelligent mind which are genuinely concerned about and working to advance humanity’s welfare could make the world a much better place in many ways: rapid economic growth lifting the third world out of poverty and raising standards in the first from prosperity to super-abundance, by eliminating dangerous and unpleasant work, ushering in an era of leisure, making dramatic advances in medicine through massive increases in the resources and attention available to each patient and through the effect of having tens of thousands of post-doc equivalent researchers dedicated to every rare condition, alongside revolutions in every other area of science.

But the dangers are severe. If AGI is achieved the most pressing concern for most of us is what will happen to our jobs. Previous technological breakthroughs which have increased productivity, often at the cost of some jobs, have created others. Improvements in productivity from the agricultural revolution drove millions off the land, millions who then found work in the cities of the industrial revolution. Standards of living and employment have risen through this process of creative destruction. Some important thinkers, such as Tyler Cowen, seem to believe that AI will be like this, that some jobs will be destroyed but others will be created, all boats will rise; AI is a normal technology.

I disagree. It seems to me that a technology which can think faster and smarter than us, work harder, more persistently and at a lower cost obviously poses a threat to close to all white collar work. Even if AI does not achieve anything broadly superhuman (I say broadly because it is already superhuman in some domains), and it does take time to diffuse across the economy, the economic impacts will be momentous.

Economic turmoil, even when it ultimately works out for the best, typically leads to political crisis. What will the effects of tens of millions of extremely well educated, underemployed people deprived of status, meaning and perhaps also a living be? Rapid diffusion of economic costs (i.e. job losses), with a slow diffusion of putative benefits will make for a very dangerous combination.

Political power is deeply tied to economic power. The wealth which will be created, and concentrated in a few hands, at the same time as millions become economically useless, will severely disturb our political equilibrium. Can democracy survive such a shift? Can it survive it alongside the emergence of technologies of hyper-persuasion and hyper-entertainment, and an intensification of our already severe polarisation?

The most severe societal dangers we face are political and economic concentration of power. Of course money and power are already unevenly distributed, but AGI might fatally destabilise our systems and institutions. This might happen in a milder way, with “merely” even deeper economic inequality and the creation of a “permanent underclass,” but it could be much more serious. If a small group of individuals control loyal ASI we could face a permanent oligopoly, or even dictatorship; a tyranny made more sinister, and perhaps more protracted, by the lights of perverted science. LLMs make a surveillance state far exceeding that of Orwell's 1984 trivially easy, and perfectly loyal AIs staffing every important position would make it enduring. Horrors surpassing those of the past century would emerge: “Cyberspace makes a superlative torture chamber. Try not to let the security-types take you to the stims.”

Making it even more difficult to respond is the fact that many of these issues which we have to face imminently sound like they come straight from the pages of science fiction. Rogue artificial minds are already a reality. There have long been digital bots, worms and viruses but these have been crude, unthinking things. It is a serious danger that out of control swarms will shortly ravage the internet. Autonomous weapons systems already exist and will play an increasingly important role in future wars. We face the democratisation of weapons of mass destruction as LLMs make it easier for well resourced bad actors to develop more powerful biological and chemical weapons and make it much easier for the less well resourced to procure simpler but still devastating pathogens.

Here we are skipping lightly across issues which require extensive international co-ordination, which would, if they were the sole issue we faced, be the defining question of our time. But we have to deal with all of them in the immediate future. There is a paucity of serious thought about many of these issues and the institutions which should be taking the lead are suffering from a crippling lack of imagination.

These are the problems we have to navigate even in a world where we do get technical alignment, i.e. we are technically able to get AI to do what we want it to do, in a way that is acceptable to those issuing the orders.

But if we don’t achieve this, extinction is a core risk. Many people working in AI believe the development of the technology poses a serious risk to our continued existence as a species. So much so that “P(doom),” shorthand for "probability of doom” is a popular term in the field. The fact that the field has a long established term for that is concerning, the fact that many leaders have P(doom)’s in the double digits is terrifying.

P(doom)

  • “There’s a 25% chance that things go really, really badly”

    Dario Amodei, CEO and Co-Founder of Anthropic

  • “Whether you think it’s two, or ten, or ninety, it’s not zero” & “probably the greatest threat to the continued existence of humanity.”

    Sam Altman, CEO and Co-Founder of OpenAI. The second quote is 12 years old.

  • “Maybe 5%, maybe 50%.”

    Shane Legg, co-founder of DeepMind, back in 2011

  • “The vast power of superintelligence could also be very dangerous, and could lead to the disempowerment of humanity or even human extinction”

    Ilya Sutskever, Co-founder Open-AI, CEO and Co-founder Safe SuperIntelligence.

  • “There’s risks that come from AI getting super smart and understanding it doesn’t need us,” It has “a ten to twenty percent chance to wipe us out”

    Geoffrey Hinton, “Godfather of AI” & 2024 Nobel Prize Physics Laureate.

  • “If you truly prioritise safety, it is time for you to leave frontier AI companies”

    Yoshua Bengio, Turing Award winner & the world's most cited AI researcher.

  • “I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems”

    Geoffrey Irving - Chief Scientist at the UK’s AI Security Institute.

A survey of 1,580 AI researchers found they held an average of an 18% chance of extinction or “similarly permanent and severe disempowerment of the species.”

Even if you are sceptical of these claims it is rational to be concerned. Leo Szilard, one of the first people to understand that a chain reaction could be used to build a weapon of superlative destructive power recounted the following anecdote from the period after the first chain reaction when scientists were still debating the feasibility of the atomic bomb:

“So Fermi said, "Well… there is the remote possibility that neutrons may be emitted in the fission of uranium and then of course perhaps a chain reaction can be made." Rabi said, "What do you mean by ‘remote possibility'?" and Fermi said, "Well, ten per cent." Rabi said, "Ten per cent is not a remote possibility if it means that we may die of it. If I have pneumonia and the doctor tells me that there is a remote possibility that I might die, and it's ten percent, I get excited about it."

Of course the chance was not ten percent.

It is already difficult to switch it off. OpenAI struggled to shut down its rogue swarms and as AI becomes more diffused and more capable it will become ever more difficult. Additionally, the economic pressures, with so much money spent and the need for results soon, align very strongly against shutting it down and as AI creates more economic value it becomes intensely against our immediate interests to pause further development, let alone decelerate.

Another factor adding to the danger is that some of the wealthiest and most powerful men in the world have staked their money, their power and their pride on this bet paying off. We should expect recklessness in their pursuit of these aims, especially if they perceive themselves to be falling behind their competition.

A more grounded scenario than sudden terminator style takeover is gradual disempowerment. As we integrate AI into ever more systems, we will outsource more of our thinking, our understanding and our control of an ever more complicated world, and we will gradually find ourselves no longer calling the shots. Even if things go well and the show is still run for us, it will not be run by us. We would face a future of leisure, entertainment and irrelevance.

III.IWhat we need to do.

“If we focus only on contingencies, we risk letting the succession of emergencies dictate the direction of our path. We are living through a rapid phase of transition, a “change of era,” in which — while some are vying for the future of new technologies and others dedicate themselves to reflecting on the matter — most people are watching and waiting, observing from afar and merely hoping for the best. For this very reason, crucial questions impose themselves on our conscience and can no longer be avoided: Where are we going? Toward what goal do we wish to orient ourselves? What direction should we choose as a people and as a human community?”

Pope Leo XIV, Magnifica Humanitas.

Alignment is the most important question and we are far from a solution, but it is primarily a technical question, and if you are someone with the talent and skillset who can contribute on this you shouldn’t need me to tell you that this is what you should be working on.

But there is also an ethical and political dimension to alignment. If we do figure out the technical problem of aligning AIs, whose values should we align them to? Right now these decisions are being made by small groups of people at the top labs, many of whom have been thinking about this problem for decades. Is this acceptable? Should we seek to democratise the process?

Unfortunately, these are not subjects which are conducive to referenda, and allowing elected politicians to decide instead just means locking in the values of whichever party is in control at the critical moment. If a small group of models, or one model, is to dominate, then locking your beliefs into the model is the way to win politics, forever.

Until relatively recently technology has in general been profoundly beneficial, raising living standards, lifting billions out of poverty and broadly improving quality of life, but since 1945 it has also had the potential to destroy us. Fundamentally, the issue is that a very small number of people are dragging the rest of us into a world which will be profoundly different, and, which they readily admit, potentially very bad. We have re-oriented our economy to birth a new technology which threatens to be as transformational and more dangerous than anything which has come before.

In terms of policy, the most pressing issue is simple. We should not proceed at a pace that is dangerous. Taking as much time as we can, defusing tensions and slowing race dynamics are all easy wins, with few downsides. As the section on China above implies, and I intend to flesh out in a future piece, the US has a considerable lead over China. It should make use of this time to proceed safely, taking as much time as possible to ensure safe development of the technology. A treaty to slow, or even pause development in both countries, is even better. We need time to figure out interpretability and alignment. As for the race between the different American labs, allowing this to proceed at a dangerous pace is just plain stupid.

However we proceed, governance of this technology, even with technical alignment, will not be easy. This was understood at the very outset of this revolution. OpenAI was founded with a very complex governance structure, into which a great deal of thought was put. Nevertheless, it failed at its first serious test. This teaches us, I think, that we should be wary of too much invention from first principles, and should try to stick with structures which have worked for a long time, and weathered major crises, however imperfect they may be.

We will need good education and advice for legislators if they are to do a sufficient job navigating this. Better public education on issues will also encourage better legislation, because, although it doesn’t always seem that way, over a long enough time frame politicians in democracies do respond to the concerns of their constituents.

In their recent paper on the prospect of Recursive Self Improvement Hinton et. al. write that "because progress during an intelligence explosion would outpace normal policymaking, preparations must be made in advance.” This is certainly true, but also applies in a world of much slower progress; a decade is not a long time. Not allowing an intelligence explosion to rip is imperative, but even without RSI we are in for profound disruption.

Even in the very best scenario, where things do work out for us, the next few years will be a very bumpy ride. As for how we conduct ourselves in this promised land of abundance, free from labour and want, how we amuse ourselves, how we find purpose and maintain dignity - these are questions on which contributions can be made by many more people. Many of us believe AGI and ASI are realistic, imminent possibilities, but there is a paucity of serious writing on these subjects. The hour is late and we have not been thinking hard enough about the future.

III.IIThe Widening Gyre

It is, a priori, profoundly unlikely that we should be alive at the time of the largest change in our species history, but profound transformations happen. You could have been born in Europe in the 14th century and watched as one in three people on your continent died in the space of two years from a mysterious disease which had not been heard of since the time of Justinian. You could have been born in the Americas in the late 15th century when strange men from across the seas, who you had no idea existed, brought with them terrifying technologies and dreadful pestilences far outside your previous experience, which killed 80-95% of your continent's people and completely disempowered the survivors. You could have been born in Japan at the end of the Edo period, when Samurai still ran the government, and been killed in your seventies by an artificial sun, dropped from a height of nine and half kilometers. You could have been born in a world without antibiotics, radios, or lightbulbs, and lived to see man set foot on the moon, one act among a host of marvels.

The adult human brain weighs around 1.35kg and operates at 20 watts. We are building data centres that cover thousands of acres, are powered by dedicated nuclear power plants, filled with hundreds of billions of dollars worth of millions of GPUs, which are improving every year, which can, in near perfect unison, perform quintillions of operations per second. There is no reason to believe that the human brain is where the physical limits of intelligence tops out.

The atomic age began with a bang; there could be no doubt it had arrived. Hopefully the age of machine intelligence will not. Though the debate about whether it has truly begun will continue, and denials persist long after they should have stopped, it seems more likely than not that we are in the foothills of the singularity.

If the anthropocene is coming to an end, it falls to us to make sure it ends well.

P.M.C. 18-10-2026.

IVFurther reading:

Detailed Scenarios of the next few years:↑

AI 2027 and AI2040 (ai-2027.com & ai-2040.com) This project provides the best in depth analysis of how the next few years could play out. AI2027 made some very concrete predictions, many of which have already been borne out. If there is one thing on this list you read it should be this. AI2027 should be read first. I do not like their plan in AI2040. It does not seem like a particularly good ending to me, but we may be lucky to get it.

Situational Awareness - Leopold Aschenbrenner - situational-awareness.ai Mr Aschenbrenner has recently caught a lot of flack because his Hedge fund was not sufficiently well hedged, but the essay is remarkably prescient, considering when it was written. The most important section is the project, where he argues that the labs will have to be nationalised. This is something the authors of AI2027 broadly agree with, though the details differ. I personally disagree, and think the labs might get there themselves before the politicians properly wake up.

MacAskill et al. Forethought.org. A superb resource, they are one of very few groups thinking seriously about what we are facing. You can start here, where they address some of the governance dangers raised above forethought.org

Hinton et al. What if automating AI R&D triggers an intelligence explosion? This is one of the best things to read on recursive self improvement and its significance. casp.ac/…/intelligence-explosion.pdf

Information on the buildout:↑

Epoch.AI is the best publicly available resource for tracking the industrial buildout, see epoch.ai/trends epoch.ai/…/ai-chip-production epoch.ai/data/ai-data-centers

The Financial Times - The FT does some of the best straight financial reporting on the financing of the AI boom, but it is behind a pretty expensive paywall. FT Alphaville can be accessed for free, you can follow them on substack, and frankly, they often provide better analysis than the main paper.

Where publicly available work from the analysts at the big banks, consultants etc. is valuable, but bear in mind these companies have a vested interest in their reporting. S&P’s recent work evaluating the credit worthiness of the hyperscalers is particularly good. I expect more good stuff from them in the near future. spglobal.com.

Bostrom.↑

Four resources on Bostrom, one of the fields most important thinkers.

You can start with Letter from utopia - this is the future that many in the labs want to achieve, he lays it out very seductively, not to say he is wrong, though it is not un-objectable.

If you are up to it you can also read his book SuperIntelligence: Paths, Dangers, Strategies which will go down as one of the most important books in history, if the future has history. It is a deeply perceptive warning from a time (12 years ago) when AI seemed like a fantasy, that AI would not necessarily go well.

On the problems of meaning and purpose living in a world where we get alignment and things go well for us he has written another book Deep Utopia.

His essay SuperIntelligent Will, lays out two key points on why superintelligent systems might not behave the ways we want. nickbostrom.com/superintelligentwill.pdf

Eliezer Yudkowsky et al. lay out a similar argument here lesswrong.com/w/orthogonality-thesis

Huggingface incident:↑

The best report on the incident is this by Metr.org, metr.org

Dwarkesh Patel has the best podcast on AI (though it is not quite exclusively about AI). He goes deep over 2-3 hours and has interviewed leaders of the labs, (AModei x 2, Altman, Musk) and a wide range of others. If you are the podcasting type, listen to his podcast. His substack is also very good. His write up on HF is very good: dwarkesh.com/p/openai-huggingface

Zvi Mowshowitz. Don’t break the Vase. Mowshowitz wrote a number of very good pieces about the HF incident as the story was breaking. He is excellent for rapid updates at the frontier. He wrote a number of very good pieces about the HF incident as the story was breaking.

You should also read Scott Alexander’s blog, AstralCodexTen. He does not write exclusively about AI. Here you can read his summary of the HF attack. astralcodexten.com

From the Labs:↑

Claude Constitution. This document lays out how Anthropic wants its models to behave, though the models do not always stick to it, it is well worth a peruse. anthropic.com/…/claudes-constitution.pdf

Machines of Loving Grace - Dario Amodei, a rare piece where advocates of progress lay out part of their vision of the future.

I Have to bury my talent in yesterday, Shengyu Liu, deepseek engineer, - a recent essay by an engineer at one of the chinese frontier labs. Very moving, but also insightful on where thought is in China, particularly on how the regard the American labs, especially anthropic

Dean Ball - hyperdimensional.co - Insightful in his own right, but worth particular attention as 1) he has worked closely with the current administration, 2) he works for OpenAI.

Machine Intelligence, Part 1 and 2, Sam Altman - two essays from the CEO of OpenAI written 12 years ago, with “special thanks” to Dario Amodei in the acknowledgements. These are much more stark than his “The Intelligence Age” which was written two years ago. blog.samaltman.com ia.samaltman.com

Roon, AGI Futures. Roon is a pseudonomous researcher at OpenAI. If you are on twitter he is worth following. Here, on substack, he writes about a few different ways this could play out. read.roonscape.ai/p/agi-futures

Existential risks. Doom.↑

If anyone builds it everyone dies, Eliezer Yudkowsky and Nate Soares. The best book on the existential risk posed by AI. Extremely partisan. Yudkowsky is one of the fields most important thinkers, even those who disagree with him are immensely in his debt,(see this tweet from Sam Altman x.com/sama/status/1621621724507938816) and has an extremely high p(Doom). The pulpy title sums up his core belief. Well worth reading.

Or read the review by Scott Alexander astralcodexten.com -

Bostrom, Superintelligence. Mentioned earlier, mentioning again. It is the Ur-text.

Nick Land, Meltdown Prophetic visions. “Nothing human makes it out of the near future.”

transformernews.ai/p/yoshua-bengio-if-you-prioritize-safety-leave-frontier-ai-companies

The Political Economy of Superhuman AI, Part 1: Institutions, Leverage and Socioeconomic Status. Connor Williams. An essay on the non-existential risks we face, which are very bad. We need more writing exploring these questions from a humanist perspective. connorsscratchpad.substack.com

Miscellaneous:↑

Benjamin Labout, Maniac. This is a novel about John Von Neuman. His early life, his contribution to the bomb, It is a brilliant read. It illuminates important aspects of this moment.

Richard Brautigan, All watched over by machines of loving grace. A poem from 1967, quoted by Dario Amodei. Is this the future you want?

Pope Leo XIV, On Safeguarding the Human Person in the time of artificial intelligence. Coming from a very different perspective than most of the work on this list. ewtnnews.com

“Nobody made a greater mistake than he who did nothing because he could do only a little.”

Edmund Burke.

About.

Notes

  1. source

    ~0.6% each for Google and Amazon, & ~ 0.5% for Microsoft vs ~0.4% each for Apollo and Manhattan - the 0.4% figure comes from https://www.researchgate.net/publication/293116260_The_Manhattan_project_the_Apollo_program_and_federal_energy_technology_RD_programs_A_comparative_analysis - see section I.I Capital, below, for the hyperscaler figures ↩︎

  2. term

    The Goalposts are constantly shifting, but AGI means an AI that can do ~all the intellectual labour a human can do as well as a human can do. ASI means an AI that can do any intellectual task far better than any human can. ↩︎

  3. term

    “Hyperscaler” is a loosely defined category. Oracle and SpaceX are sometimes also included. ↩︎

  4. term

    Capital Expenditure - “funds companies spend to acquire, upgrade, or maintain long term physical assets” - Investopedia ↩︎

  5. source

    Oracle’s fiscal year ends on May 31st, their spending between 1st June 25 and 31st May 26 was $55.7bn. Their Projection for the next 12 months is 90-95 billion. Roughly, ((55.7 / 12)*7)+((92.5/12)*5)=71 ↩︎

  6. thought

    It is important to remember when looking at insane financing schemes like these that this is real money that is paying for real infrastructure in the real world. ↩︎

  7. thought

    There are already some signs of strain. S&P have Oracle’s credit rating at just above junk status. Their stock has halved in the past year and banks are finding themselves stuck holding more of Oracle’s debt than they intended. SoftBank is paying 9.75% on some of its bonds. Goldman says that the hyperscalers (excluding SpaceX) have issued about $200bn of debt so far this year, and that they will issue more than a trillion dollars of new debt in the next few years. In 2007 the total subprime market was roughly $1.3 trillion. A default rate of just over 22% caused the GFC. The famous mortgage backed synthetic CDOs were worth far less than half a trillion dollars. ↩︎

  8. term

    Much to the ire of US labs the Chinese companies have been systematically training on the responses of more sophisticated models, so much so that Kimi K3 will sometimes tell users that it is Claude. Activity involving distillation, which Anthropic is trying to stamp out, accounted for 5-7% of Anthropic’s pre-stampdown revenue. ↩︎

  9. term

    Compute is the term broadly used in the industry to describe computing power, a mixture of quality and quantity of chips. ↩︎

  10. term

    There are broadly five ways this compute is used. 1) training runs, i.e. developing the model. 2) testing, to see how the models perform, 3) experiments, to work out ways to improve the architecture of future models, 4) safety, unfortunately the smallest allocation, and 5) inference, the compute needed for consumers to actually use the models. ↩︎

  11. source

    “So any given solar panel can do about five times more power in space than on the ground. You also avoid the cost of having batteries to carry you through the night. It’s actually much cheaper to do in space. My prediction is that it will be by far the cheapest place to put AI. It will be space in 36 months or less. Maybe 30 months.” [https://www.dwarkesh.com/p/elon-musk] (season timelines with salt appropriately) ↩︎

  12. source

    https://www.energyconnects.com/news/renewables/2024/december/ge-vernova-to-power-city-sized-data-centers-with-gas-as-ai-demand-soars/ ↩︎

  13. source

    https://www.goldmansachs.com/insights/articles/us-data-center-power-demand-projected-to-double-by-2027 ↩︎

  14. source

    https://www.goldmansachs.com/pdfs/insights/goldman-sachs-research/data-center-power-demand-the-6-ps-driving-growth-and-constraints/redacted_report.pdf ↩︎

  15. source

    https://www.blackrock.com/corporate/literature/whitepaper/bii-global-outlook-2026.pdf ↩︎

  16. thought

    This is a contentious claim for some, but I think it is straightforward; If I ask you something, can you tell me about it? Can you write me a decent essay about it? If you can, then I think it is fair to say that you are, in some important sense, knowledgeable about that thing. LLMs can do this for pretty much everything, near instantly. Getting things occasionally wrong does not mean that you are not knowledgeable. Lying does not mean you are unintelligent. These are undesirable traits, for sure, but they do not mean you are stupid. ↩︎

  17. source

    The key document to read here is Metrs report. Both Dwarkesh Patel and Zvi Mowshowitz have written great articles which both summarise and draw lessons from the Metr report. See further reading below. ↩︎

  18. thought

    Context window has been an area of insane progress. GPT3 had a context window of 2,048 tokens. GPT4 could handle 8,000. GPT5 was originally 400,000. GPT6 can go above one million, as can other frontier models. ↩︎

  19. thought

    Of course it does not take them this long to do the task, it would take us that long, they can do it much quicker ↩︎

  20. thought

    Motivations

    Achieving a world with these miracles and others beside is the stated goal of many researchers - I do not fully believe this to be their deepest motivation; “When you see something that is technically sweet, you go ahead and do it, and you argue about what to do only after you have had your technical success. That is the way it was with the bomb.” - J. Robert Oppenheimer ↩︎

  21. thought

    Lesser questions like how we protect ourselves from stasis (AI locks in values), bureaucracy, rent seeking and cost disease eating the gains? How we cope with impacts on our legal system, what happens with property rights, how we strengthen rights to privacy and other fundamental freedoms? What about the global balance of power if one nation wins the race and rapidly pulls far ahead of the others, allowing only weaker models to be used in the rest of the world and only in approved ways? These questions, and others require urgent, intense attention. We have to prepare to meet these challenges now. ↩︎

  22. source

    https://aiimpacts.org/wp-content/uploads/2026/09/ESPAI2024.pdf ↩︎

  23. source

    Read Bostrom’s The Superintelligent Will or the Orthogonality Thesis on LessWrong for more. See section four further reading, below. ↩︎

  24. thought

    on existential risk

    Even if you see no way to personally impact this, or you feel hopeless, I think it is important to tend your garden, to work to make the world better in whatever ways you can.

    “I sat in a restaurant in New York, for example, and I looked out at the buildings and I began to think, you know, about how much the radius of the Hiroshima bomb damage was and so forth... How far from here was 34th street?... All those buildings, all smashed — and so on. And I would go along and I would see people building a bridge, or they'd be making a new road, and I thought, they're crazy, they just don't understand, they don't understand. Why are they making new things? It's so useless. But, fortunately, it's been useless for almost forty years now, hasn't it? So I've been wrong about it being useless making bridges and I'm glad those other people had the sense to go ahead.”

    - Feynman, a builder of the bomb. ↩︎

  25. thought

    The only way to sidestep this is pluralism. Open Source models, AI which can be modified at will by any individual, offer one path forward. It enables groups and individuals to have AIs which serve their beliefs, and importantly, disregard the beliefs of the decision makers at the labs. It is a bulwark against concentration of power, against stasis and against tyranny.

    But Open Source is dangerous, perhaps fundamentally. Open source models already have fewer safeguards than the models from the labs. So there is a narrow path; we need to have powerful open source models which are widely and cheaply available, which are fundamentally aligned, but which will refuse to help with biological and chemical weapons, malware and other harmful activities, narrowly defined.

    This is likely an impossible balance to achieve, but it is a useful goal to aim for. ↩︎