In this video I share my concept of autonomous AI crawler built with n8n that can navigate through the website and extract desired data.

In this tutorial I cover:
1. Concept of AI crawler built with n8n (how it works and what are its capabilities)
2. Pros and cons of the AI crawler
3. Quick walkthrough for main AI crawling agent, text tool and URL tool
4. Using Web Unlocker proxy for better result accuracy
5. Adding multiple agents to the workflow to retrieve more data

Download example n8n workflows from this tutorial:
1. Single agent with tools: https://n8n.io/workflows/2315-autonom...
2. Multiple agents with tools: https://gist.github.com/workfloows/48...

Resources:
1. Understanding agents (official n8n docs): https://docs.n8n.io/advanced-ai/examp...
2. Proxy used in the video (Web Unlocker): https://brightdata.com/products/web-u...

My other tutorials:
1. How to automate Gmail with n8n AI:    • How to automate Gmail with n8n AI  
2. Web scraping data with n8n and Puppeteer:    • Web scraping data with n8n and Puppeteer  
3. Build undetectable Amazon scraper with n8n, Puppeteer and Scraping Browser:    • Build undetectable Amazon scraper wit...  
4. How to automate Notion databases using n8n:    • How to automate Notion databases usin...  
5. Getting started with native 🦜️🔗 LangChain nodes in n8n:    • Getting started with native 🦜️🔗 LangC...  

Subscribe my newsletter: https://workfloows.com/
Visit my n8n creator profile: https://n8n.io/creators/workfloows/
Visit my Gumroad profile: https://workfloows.gumroad.com/
Follow me on Twitter/X:   / workfloows  
Follow Workfloows on LinkedIn:   / workfloows  

Disclaimer: I cannot be held responsible for any consequences resulting from the use of the information provided in this tutorial. Make sure to obtain proper authorization before engaging in web scraping activities, or consider using proxies to protect your online presence and ensure ethical scraping practices.

Create your Bright Data account and get $10 credit: https://brdta.com/workfloows

Create your n8n cloud account here (affiliate): https://n8n.partnerlinks.io/workfloows

Screen recording software that I use (affiliate): https://www.screen.studio/@jmMwX

0:00 AI crawler concept
1:15 Pros and cons of AI crawler
1:47 Part 1: Main AI crawling agent
3:35 Part 2: Text scraper tool
5:32 Part 3: Proxy and CAPTCHA solver
6:43 Part 4: URL scraper tool
8:10 Part 5: Crawling with multiple agents