Published On Jul 21, 2023
❤️ Become The AI Epiphany Patreon ❤️
/ theaiepiphany
👨👩👧👦 Join our Discord community 👨👩👧👦
/ discord
Hey guys! I've been uploading less frequently lately since I started my startup - as I'm trying to focus and learn as much as possible.
Until I get back into the rhythm I'll be experimenting a bit with the channel: expect talks from my Discord server, live streaming, etc.
Here is a talk we had in my Discord server with Tri Dao from Stanford, talking about his recent work on Flash Attention 2.0!
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
✅ My Flash Attention blog: / eli5-flash-attention
✅ Horace's blog: https://horace.io/brrr_intro.html
✅ Flash Attention paper: https://arxiv.org/abs/2205.14135
✅ Flash Attention 2.0 paper: https://tridao.me/publications/flash2...
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
⌚️ Timetable:
00:00 Main talk starts - intro & motivation
10:20 Behind the scenes: how Tri got started with Flash Attention
16:40 Motivation: modelling long sequences
19:50 Brief recap of attention
23:15 Memory bottleneck, IO awareness
35:10 Flash Attention 2.0 improvements
37:20 Behind the scenes of Flash Attention 2.0 (refactor of CUTLASS 3)
43:45 Future directions
46:45 Q&A
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️
If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!
The AI Epiphany - / theaiepiphany
One-time donation - https://www.paypal.com/paypalme/theai...
Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💼 LinkedIn - / aleksagordic
🐦 Twitter - / gordic_aleksa
👨👩👧👦 Discord - / discord
📺 YouTube - / theaiepiphany
📚 Medium - / gordicaleksa
💻 GitHub - https://github.com/gordicaleksa
📢 AI Newsletter - https://aiepiphany.substack.com/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
#flashattention #tridao #mlsys