We dive into the Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution paper, a technique, competitive with GPT-2, that can use diffusion techniques to generate text.

--

Get Oxen AI 🐂 https://oxen.ai/

Oxen AI makes versioning your datasets as easy as versioning your code! Even is millions of unstructured images, the tool quickly handles any type of data so you can build cutting-edge AI.

--

Paper 📜 https://arxiv.org/abs/2310.16834

Links + Notes 📝 https://www.oxen.ai/blog/arxiv-dives-...

Join Arxiv Dives 🤿 https://oxen.ai/community

Discord 🗿   / discord  

--

Chapters
0:00 Intro
3:55 Modeling Probability Distributions for Generative AI
7:12 Problem #1: No Black Box
10:44 Solution #1: Train a Network to Approximate the Probability Mass Function
13:48 Problem #2: The Normalizing Constant, Z_theta, is Intractable
15:15 Solution #2: Autoregressive Modeling
17:15 Solution #3 (Real Solution): Model Score, Not Probability Mass
25:50 Learning the Concrete Score Through Diffusion
33:00 Evaluation
36:18 So What?
41:00 Takeaways