#7: AI timelines, AI skepticism, and lock-in

Feb 3

That man is born merely for a few, who thinks only of the people of his own generation. Many thousands of years and many thousands of peoples will come after you; it is to these that you should have regard.
— Lucius Annaeus Seneca

Future Matters is a newsletter about longtermism and existential risk by Matthew van der Merwe and Pablo Stafforini. Each month we curate and summarize relevant research and news from the community. The version crossposted to the Effective Altruism Forum includes a bonus conversation with a prominent researcher. You can also listen on your favorite podcast platform and follow on Twitter. Future Matters is also available in Spanish.

Research

Ajeya Cotra’s biological anchors model to forecast AGI timelines consists of three parts — an estimate of the compute required to train AGI with 2020 algorithms, a projection of how these compute requirements decrease over time due to algorithmic progress, and a forecast of how the size of training runs will increase over time due to declining hardware costs and increased investment in AI training. Tom Davidson’s What a compute-centric framework says about AI takeoff speeds extends Cotra’s framework to incorporate a more sophisticated model of how R&D investment translates into algorithmic and hardware progress, and also to capture the “virtuous circle” whereby AI progress leads to more automation in AI R&D and in turn faster AI progress. This results in a model of AI takeoff speed, defined here as the time between AI being able to automate 20% of cognitive tasks to being able to automate 100% of cognitive tasks. Davidson’s median estimate for AI takeoff is approximately three years. This is an impressive and significant piece of research, which we cannot summarize adequately here; we hope to feature a conversation with the author in a future issue to explore it in more depth. The full report is available here. Readers are encouraged to play around with the neat interactive model.

AGI and the EMH, by Trevor Chow, Basil Halperin, and J. Zachary Mazlish, highlights the tension between the efficient market hypothesis and the hypothesis that transformative AI will arrive in the next few decades. Transformative AI will either raise economic growth rates if aligned or raise the risk of extinction if unaligned. But either of these disjuncts imply much higher real interest rates. (This implication follows from both intuition and mainstream economic theory.) Since we are not observing higher real interest rates, we should conclude either that timelines are longer than generally assumed by the EA and alignment communities, or that markets are radically underestimating how soon transformative AI will arrive.

Zac Hatfield-Dodds shares some Concrete reasons for hope about AI safety [🔉]. A researcher at Anthropic (writing in a personal capacity), he takes existential risks from AI seriously, but pushes back on recent pronouncements that AI catastrophe is pretty much inevitable. Hatfield-Dodds highlights some of the promising results from the nascent efforts at figuring out how to align and interpret large language models. The piece is intended to “rebalance the emotional scales” in the AI safety community, which he feels have recently tipped too far towards a despair that feels is both unwarranted and unconstructive.

Holden Karnofsky's Transformative AI issues (not just misalignment) [🔉] surveys some of the high-stakes issues raised by transformative AI, particularly those that we should be thinking about ahead of time in order to make a lasting difference to the long-term future. These include not just existential risk from misalignment, but also power imbalances, early AI applications, new life forms, and persistent policies and norms. Karnofsky is inclined to prioritize the first two issues, since he feels very uncertain about the sign of interventions focused on the remaining ones.

Lizka Vaintrob argues that we should Beware safety-washing [🔉] by AI companies, akin to greenwashing, where companies misrepresent themselves as being more environmentally conscious than they actually are, rather than taking costly actions to reduce their environmental impact. This could involve misleading not just consumers, but investors, employees, regulators, etc. on whether an AI project took safety concerns seriously. One promising way to address this would be developing common standards for safety, and trustworthy methods for auditing and evaluating companies against these standards.

In How we could stumble into AI catastrophe [🔉], Holden Karnofsky describes a concrete scenario of how unaligned AI might result in a global catastrophe. The scenario is described against two central assumptions (which Karnofsky discusses in previous writings): that we will soon develop very powerful AI systems and that the world will otherwise be very similar to today's when those systems are developed. Karnofsky's scenario draws heavily from Ajeya Cotra's post, Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover (see FM#4 for a summary of the article and FM#5 for our conversation with Cotra).

In Managing the transition to widespread metagenomic monitoring [🔉], Chelsea Liang and David Manheim outline a vision for next-generation biosurveillance using current technologies. An ambitious program of widespread metagenomic sequencing would be great for managing pandemic risk, by serving as an early warning for identifying novel outbreaks. But getting to this stage requires first addressing a number of important obstacles including high costs and privacy concerns.

In Technological stagnation: why I came around [🔉], Jason Crawford outlines some arguments for the ‘great stagnation’ hypothesis—the view that technological and scientific progress have slowed down substantially since the 1970s. Crawford’s main argument is qualitative: while we have seen significant innovation in IT since the 1970s, we’ve haven’t had many major breakthroughs in manufacturing, energy, and transportation, whereas previous industrial revolutions have been characterized by innovation across all major sectors. Crawford offers some quantitative arguments, pointing to US GDP and TFP growth rates. This was a readable post, but we remain to be convinced of the stagnation hypothesis: the qualitative arguments were hand-wavy, and the macro data looks pretty inconclusive for the most part (see also Alexey Guzey’s criticism of an influential paper on the topic).

Holden Karnofsky's Spreading messages to help with the most important century [🔉] considers various messaging strategies for raising awareness about the risks posed by transformative AI. Karnofsky favors approaches that help others develop a gears-level understanding of the dangers of AI, that communicate that AI alignment research is uniquely beneficial, and that focus on the threats AI poses for all humans. By contrast, he believes we should de-emphasize messages stressing the importance and potential imminence of powerful AI, and those that stress the dangers of AI without explaining why it is dangerous.

Literature review of transformative artificial intelligence timelines, by Keith Wynroe, David Atkinson and Jaime Sevilla, is a comprehensive overview of various attempts to forecast the arrival of transformative AI. The authors summarize five model-based forecasts and five judgement-based forecasts, and produce an aggregate of each of these two forecasts types based on Epoch members' subjective weightings. The Epoch website also lets readers input their weights and see the resulting aggregate forecasts. We found this literature review very useful and consider it the best existing summary of what is currently known about AI timelines.

Misha Yagudin, Jonathan Mann & Nuño Sempere share an Update to Samotsvety AGI timelines. In aggregate, the forecasting group places 10% on AGI by 2026, and 50% by 2041. This represents a shortening of timelines since Samotsvety last published similar numbers, which put ~32% on AGI by 2042.

Eli Dourado's Heretical thoughts on AI [🔉] argues that artificial intelligence may fail to have a transformative economic impact even if it transforms other aspects of human life. Dourado notes that, for many of the largest sectors in the economy—such as housing, energy, transportation and health—, growth has been slow primarily because of regulation, litigation and public opposition. Progress in capabilities, however impressive, may thus fail to precipitate an economic transformation.

In Longtermism and animals [🔉], Heather Browning and Walter Veit argue that the interests of non-human animals should be incorporated in longtermist priority-setting, and that this could meaningfully affect decision-making about the long-term future. As the authors mention, this is closely relevant to questions on the ethics of digital minds.

One-line summaries

Paul Christiano shares Thoughts on the impact of RLHF research [🔉] (reinforcement learning from human feedback), which was a focus of his alignment work at OpenAI in 2017–20.
Nuño Sempere shares his highly personal skepticism braindump on existential risk from AI.
Richard Chappell’s Text, Subtext, and Miscommunication is a particularly thoughtful discussion of the recent Nick Bostrom debacle.
Generative language models and automated influence operations [🔉], by Josh Goldstein and collaborators, investigates the impacts of large language models on efforts to influence public opinion and considers possible interventions to mitigate these risks.
In We need holistic AI macrostrategy [🔉], Nick Gabs argues that research on macrostrategic questions related to AI alignment should be a top priority.
MIRI released a conversation between Scott Alexander and Eliezer Yudkowsky [🔉] covering analogies to human moral development, “consequentialism”, acausal trade, and alignment research opportunities.
In Air safety to combat global catastrophic biorisks [🔉], Jam Kraprayoon, Gavriel Kleinwaks, Alastair Fraser-Urquhart, and Josh Morrison argue that extending indoor air quality standards to include airborne pathogen levels could significantly reduce global catastrophic biological risks.

News

Allison Duettmann interviewed [🔉] theoretical physicist Adam Brown on potential risks and opportunities for the future for the Existential Hope podcast.

Applications for the 2023 PIBBSS Summer Research Fellowship are open until Feb 5th.

Dwarkesh Patel interviewed [🔉] Holden Karnofsky on transformative AI, digital people and the most important century for the Lunar Society Podcast.

Michael Osborne and Michael Cohen, from the University of Oxford, gave evidence [🔉] on AI risk to the UK Parliament’s Science and Technology Committee.

Jack Clark gave an educational presentation on AI policy to the US Congress’s AI Caucus.

Kelsey Piper predicts what will likely happen with AI in 2023 [🔉]: better text generators, better image models, more widespread adoption of coding assistants, takeoff of AI personal assistants, and more.

In episode #29 of Manifold, Steve Hsu talked about ChatGPT, large language models, and AI.

Sigal Samuel interviews Holden Karnofsky [🔉] on reforming effective altruism after SBF.

Jack Clark published a new issue of Import AI [🔉] on smarter robots via foundation models, a new small but mighty medical language model, and a multilingual coding assistant made by Baidu.

The London Futurists Podcast interviewed [🔉] Anders Sandberg on the Fermi paradox, the aestivation hypothesis, and the simulation argument.

Everyone’s least favorite tool for communicating existential risk, the Doomsday Clock, has been set to 90 seconds to midnight this year.

Michaël Trazzi interviewed [🔉] DeepMind senior research scientist Victoria Krakovna about arguments for AGI ruin, paradigms of AI alignment, and her co-written article 'Refining the Sharp Left Turn threat model'.

David Krueger talked about existential safety, alignment, and specification problems for the Machine Learning Safety Scholars summer program.

Benjamin Hilton has updated [🔉] his estimate of the number of full-time equivalent (FTE) working directly on the problem of reducing existential risks from AI, from 300 FTE to 400 FTE.

80,000 Hours published [🔉] a medium-depth career profile of information security in high-impact areas, written by Jarrah Bloomfield.

Charlotte Stix published [🔉] a new issue of the EuropeanAI Newsletter.

Applications are open for the course “Economic Theory & Global Prioritization”, taught primarily by Phil Trammell and sponsored by the Forethought Foundation, to be held in Oxford in August 2023. Apply now.

Worryingly, Google will “recalibrate” [🔉] the level of risk it is willing to take when releasing AI products.

Meanwhile, OpenAI has received a new $10 billion investment from Microsoft.

The New York Times asks Are we living in a computer simulation, and can we hack it?

The RAND Corporation is accepting applications for the Stanton Nuclear Security Fellows Program, open to postdoctoral students and tenure track junior faculty, as well as to doctoral students working primarily in nuclear security. Apply now.

Aisafety.training is a useful new website collecting information on AI safety programs, conferences and events.

Epoch, a research group forecasting the development of transformative artificial intelligence, has released a report summarizing their main achievements in 2022.

EA Global: Bay Area will run from 24–26 February. Apply here by 8th February.

Conversation with Lukas Finnveden

To read our conversation with Lukas on AGI and lock-in, please go to the version of this issue crossposted on the Effective Altruism Forum.

We thank Leonardo Picón and Lyl Macalalad for editorial assistance.