I had the good fortune to speak with Daniel Kokotajlo, an ex-OpenAI safety researcher. He is known for taking a stance on the dangers of AGI, even going so far as to sacrifice OpenAI equity to be able to speak out.

Daniel believes that AGI will arrive within 3 years. There is only one primary cluster of skills that still has to be developed, which is long horizon tasks and agency skills. Daniel also assigns a 70% probability that AGI will go catastrophically wrong. We need to change our trajectory to have a good chance of survival.

To communicate the absurdity of the situation, Daniel gave two analogies that our alternative timelines. One imagines that human cloning had proceeded unhindered, and we now have 12-year-old super geniuses. Another imagines that octopuses had been bred for intelligence and companies now have octopus customer support instead of human. Unfortunately, strange as these scenarios are, our real situation is actually worse than either of them.

#agi #alignment #openai

Daniel Kokotajlo
https://www.lesswrong.com/users/danie...

Daniel Kokotajlo: “Unclear how to value the equity I gave up, but it probably would have been about 85%...”
https://www.greaterwrong.com/posts/ko...

OpenAI Insiders Warn of a ‘Reckless’ Race for Dominance
https://www.nytimes.com/2024/06/04/te...

A Right to Warn about Advanced Artificial Intelligence
https://righttowarn.ai/

OpenAI expands lobbying team to influence regulation
https://www.ft.com/content/2bee634c-b...

OpenAI Bolsters Lobbying Arm Amid Regulatory Pressure
https://www.pymnts.com/news/artificia...

Alex Blechman: “Sci-Fi Author: In my book I invented the Torment Nexus...”
https://x.com/AlexBlechman/status/145...

0:00 Intro
0:30 Contents
0:38 Part 1: When AGI will come
0:44 Daniel Kokotajlo is a safety researcher
1:05 Non-disparagement agreement
1:30 Daniel walked away from $1.7 million
1:56 OpenAI changes course after publicity
2:15 Right to warn about AI, New York times
2:51 I had a conversation with Daniel directly
3:14 Background and "What 2026 looks like"
4:03 Daniel is producing in-depth AI reports
4:30 Current projection for AGI: within 3 years
5:28 What is left for AGI?
6:16 Agents capable of operating autonomously
6:47 Part 2: Why AGI is dangerous
7:07 AGI could backfire of its own accord
7:50 We don't know how to control these systems
8:29 Why AI is more challenging than historical examples
8:54 Tech workers are too close to the tech
9:20 Torment Nexus and other bad examples
10:10 Two alternative world analogies
10:19 Analogy 1: Human cloning
11:02 12-year-old super geniuses
11:43 Society getting used to clones, too late to change
12:21 Analogy 2: Superintelligent octopuses
12:48 Octopus tanks for customer support
13:10 Fast breeding cycle
13:29 Science fiction book recommendation
13:48 How similar are these timelines to our reality?
14:26 Physical bodies vs observable compute
15:24 Tell your friends
15:34 Part 3: The moment of creation
15:47 Intelligence explosion likely to start in secret
16:23 High-stakes decisions in secret
16:51 Government involvement needs to increase
17:31 We've been getting boiled along with the frog
18:26 Desire for unnecessary level of secrecy
19:03 Proposal: keep the public informed at a high level
19:39 Impose reporting requirements on companies
20:37 Why the dead man's switch is helpful
21:24 Conclusion
21:55 AGI is likely to be kept secret
22:38 What we can do about it
22:58 Outro