❤️ Become The AI Epiphany Patreon ❤️
  / theaiepiphany  

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦
  / discord  

Hey guys! I've been uploading less frequently lately since I started my startup - as I'm trying to focus and learn as much as possible.

Until I get back into the rhythm I'll be experimenting a bit with the channel: expect talks from my Discord server, live streaming, etc.

Here is a talk we had in my Discord server with Tri Dao from Stanford, talking about his recent work on Flash Attention 2.0!

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
✅ My Flash Attention blog:   / eli5-flash-attention  
✅ Horace's blog: https://horace.io/brrr_intro.html
✅ Flash Attention paper: https://arxiv.org/abs/2205.14135
✅ Flash Attention 2.0 paper: https://tridao.me/publications/flash2...
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

⌚️ Timetable:
00:00 Main talk starts - intro & motivation
10:20 Behind the scenes: how Tri got started with Flash Attention
16:40 Motivation: modelling long sequences
19:50 Brief recap of attention
23:15 Memory bottleneck, IO awareness
35:10 Flash Attention 2.0 improvements
37:20 Behind the scenes of Flash Attention 2.0 (refactor of CUTLASS 3)
43:45 Future directions
46:45 Q&A

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
💰 BECOME A PATREON OF THE AI EPIPHANY ❤️

If these videos, GitHub projects, and blogs help you,
consider helping me out by supporting me on Patreon!

The AI Epiphany -   / theaiepiphany  
One-time donation - https://www.paypal.com/paypalme/theai...

Huge thank you to these AI Epiphany patreons:
Eli Mahler
Petar Veličković

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

💼 LinkedIn -   / aleksagordic  
🐦 Twitter -   / gordic_aleksa  
👨‍👩‍👧‍👦 Discord -   / discord  

📺 YouTube -    / theaiepiphany  
📚 Medium -   / gordicaleksa  
💻 GitHub - https://github.com/gordicaleksa
📢 AI Newsletter - https://aiepiphany.substack.com/

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#flashattention #tridao #mlsys