AI Generated code is bad. But the demos do their best to hide that fact. Even Claude 3.5 Sonnet, which the company says "raises the industry bar for intelligence" But in their demo video, they gave Claude a REALLY easy problem to fix, it did a lousy job, and their video glossed over the bad parts.

But, it's not just Claude. In conjunction with ‪@Proof_news‬ and a new AI tool they've built for creators, I've broken down some different problems that human Software Developers can do, but - despite being trained on all the information and having near total recall - AI's just cannot get right.

Video Interview with Proof News about this investigation:
   • Proof Ingredients: Is AI going to rep...  


00:00 AI Code is still not good
00:45 Work with ‪@Proof_news‬
01:39 Intro to the Claude 3.5 Sonnet Demo video
02:26 Transferring code from video to an editor so I can demo it
02:50 What Claude thinks the bug is - and why it's wrong
03:45 The 3-character fix Claude missed
04:03 But Claude's test code is even worse
04:07 Hey, Claude - that's a square, not a circle
04:34 Don't test by starting with the answer
05:24 Bad demo practices contribute to the hype cycle
05:33 Collaborating with @Proof_news
06:06 Generalizing from "Claude got this wrong" to "AI's in general get this wrong"
06:58 Other kinds of Software problems AI does a poor job of
07:51 A classic problem, with a modern solution the AI's can't code
09:47 When AI's replace human programmers, this is what is lost
10:17 Planning and Estimation - or the lack thereof
11:16 AI coders fail at even small stuff, but the AI companies talk big
11:58 Companies are probably going to get better at not getting caught in poor demos
12:13 More videos about things AI's can't seem to get right
12:27 Wrap up

Links:
Video Interview with Proof News about this investigation:
   • Proof Ingredients: Is AI going to rep...  

Claude 3.5 Sonnet Coding Demo Video
   • Claude 3.5 Sonnet for agentic coding  

Source of "raises the industry bar for intelligence" quote
https://www.anthropic.com/news/claude...

Proof News Resources
https://proofnews.org/
   • Was your favorite YouTube channel use...  
   • Proof Ingredients: How AI companies s...  
   • Proof Ingredients: Teacher Gives AI M...