Published On Nov 27, 2020
In this special edition, Dr. Tim Scarfe, Yannic Kilcher and Dr. Keith Duggar speak with Professor Gary Marcus, Dr. Walid Saba and Connor Leahy about GPT-3. We have all had a significant amount of time to experiment with GPT-3 and show you demos of it in use and the considerations. Do you think GPT-3 is a step towards AGI? Answer in the comments!
00:00:00 Connor's take on LinkedIn
00:00:47 Show teaser
00:20:02 Tim Introduction
00:26:55 First look at GPT-3, python sorting
00:31:05 Search strategy in LMs
00:38:28 Character analogies and Melanie Mitchell
00:44:27 Substitution cipher
00:47:21 Database prompt
00:53:00 Broader Impact Generation
01:02:47 Gary Marcus Interview (Robust.AI)
01:29:11 Connor Leahy Interview (Eleuther.AI)
01:32:29 Connor -- Tabular data
01:33:41 Connor -- other surprising examples?
01:34:54 Connor -- Is interpolated stuff new?
01:37:43 Connor -- structure of the brain / How GPT works
01:41:21 Connor -- Why cant GPT-3 reason?
01:46:30 Connor -- Missing information problem and ideas on our our brains work
01:54:28 Connor -- Topology of brain/models
01:58:49 Connor -- Hardware lottery / LSTM / Transformer
02:01:41 Connor -- NNs are just matrix program search
02:10:32 Connor -- Google -- information retrieval, the new paradigm, how to extract info from GPT-3, RL controller on top?
02:19:38 Connor -- Database example / "pattern matching is Turing complete"
02:23:55 Connor -- Did gpt3 understand?
02:26:30 Connor -- Are the GOFAI people right?
02:27:40 Walid Saba on GPT-3
02:30:41 Walid -- What is understanding and pattern recognition
02:35:56 Walid -- Chomsky would be happy
02:42:13 Walid -- Redefining success
02:46:05 Walid on Hinton
02:47:34 Walid on software 3.0
02:53:11 Keith -- We use machine learning because we cant write code to do the same thing
02:59:36 Keith -- What is pattern recognition and understanding
03:14:06 GPT-3 trials -- Turing Dialog
03:15:35 GPT-3 trials -- Mary Enjoyed a Sandwich
03:16:19 GPT-3 trials -- BBC has five offices in Germany.
03:16:55 GPT-3 trials -- Database prompt
03:20:23 GPT-3 trials -- Python
03:20:31 GPT-3 trials -- Patterns
03:21:01 GPT-3 trials -- Database again
03:25:11 GPT-3 trials -- GPT-3 experiment -- the trophy doesn’t fit in the suitcase
03:27:32 GPT-3 trials -- Scrambling words
03:30:41 GPT-3 trials -- PDF cleanup example (Gwern)
03:35:03 GPT-3 trials -- Word breaking and simple text patterns
03:37:16 GPT-3 trials -- Typing of entities
03:38:30 GPT-3 trials -- Basic Python append
03:39:07 GPT-3 trials -- Automatic programming?
03:42:31 GPT-3 trials -- Passive aggressive dialog input
03:44:39 GPT-3 trials -- symptoms of depression
03:45:43 GPT-3 trials -- Red shirts reasoning challenge
03:49:59 GPT-3 trials -- Binary encoding
03:50:36 Concluding statements from Walid, Tim and Yannic
Pod version: https://anchor.fm/machinelearningstre...
Connor Leahy:
/ connor-j-leahy
/ npcollapse
Eleuther.AI Discord -- / discord
Gary Marcus:
/ gary-marcus-b6384b4
/ garymarcus
https://www.robust.ai
Walid Saba:
/ walidsaba
/ ontologik
https://ontologik.ai